Where to store DB and the files

[sinus]

Though Mario gave us in his cool helpfile some information, I am not sure:
In my computer, I have two different hard-drives.

- 1 Hitachi HD with 1 TB, two partitions, 500 MB each
- 1 Western Digital WD, 4 TB, two partitions

The program IMatch is (like normaly) on my hard-drive C.

So I have a drive C, and on the same drive, another partition, called D.

Then my second hard-disk, with 1.66 TB (in fact it has 4 TB, but I can use only 1.66 TB, that is a pity, but another story and not important here), has two partition, F with 1.3 TB, and G with 0.3 TB.

I planned now to store all my real files on partition F on the second harddrive, this, because it is a huge partition.

Now my question:
Is it wise to store the IMATCH-database on the same harddisk, like the files (the second harddisk) or is it better to store it on the first harddisk, the same, where the program is installed?

I guess, this is not that important really, but I wonder, if this could have some impact on the speed of IM5?

[jch2103]

In general, it's usually preferable to put your IMatch database and your image files on separate physical drives.

Also, if your computer supports it, putting your IMatch database on a USB 3 memory stick as Mario has recommended will generally enhance performance. If your computer doesn't currently support USB 3, you can get a USB 3 expansion card fairly cheaply (mine cost $20 US from Amazon).


The other performance enhancement, an SSD (for OS and database), is more expensive, although prices keep declining.

John

[lnh]

In general, it's usually preferable to put your IMatch database and your image files on separate physical drives.

Also, if your computer supports it, putting your IMatch database on a USB 3 memory stick as Mario has recommended will generally enhance performance. If your computer doesn't currently support USB 3, you can get a USB 3 expansion card fairly cheaply (mine cost $20 US from Amazon).


The other performance enhancement, an SSD (for OS and database), is more expensive, although prices keep declining.

John

At this point there isn't much price difference between a high performance USB3 memory stick and an equivalent sized consumer SSD, but pricing on both is subject to deals which can be found at any given time. There is a large performance difference between the better USB3 sticks and the cheap ones. Got a SanDisk Extreme 64GB USB 3.0 to boot Linux Mint from and it's extremely fast. Right now Amazon (US) is selling that one at a very low price. The price of that particular drive varies wildly on Amazon almost day to day and sometimes sells for more than double the price today.

Edit addition:
Not sure which version of Windows you're running, but if you go the SSD route, make sure TRIM is enabled in the OS (Google the topic and you'll find lots of help). Also my experience with SSD's on notebooks is to disable any power management to the SSD. The internal firmware on the SSD keeps file integrity in order, and if it isn't allowed to run most of the time things can go bad (it happened to me). Ditto for SSD's which might be used in a removable drive bay on desktops.

[Richard]

I agree with John but would have listed the choices with best first.

1. A quality SSD
2. A FAST USB 3 memory stick.
3.  IMatch database and your image files on separate physical drives. With the database on the fastest drive available.

[jch2103]

2. A FAST USB 3 memory stick.

Technology continues to march ahead: Corsair has introduced its Voyager GTX USB 3.0 flash drive (128Gb, 256GB) that incorporates an SSD controller for what should be greater speed. Still a bit pricey, but I assume others will soon be competing in this niche.

John

[Canay]

This is what my invoice says my computer has:
   3TB 7200 RPM SATA Hard Drive 6.0 Gb/s + 256GB SSD

This has really confused me because I've only ever had one C drive. Now I have a C drive and a D drive. Right now I have my pictures on the 3TB drive (D) in My Picture folder and my database is on the same hard drive in a different folder. My operating system is the only thing on the 256gb SSD (C) drive. Am I understanding that it would be better to move the data base to the 256gb SSD drive? Is the reason for speed? Or is it for security in case of a hard drive failure?

[Richard]

Am I understanding that it would be better to move the data base to the 256gb SSD drive?

I would recommend doing so. You will see faster database access.

[ColinIM]

(....) Is it wise to store the IMATCH-database on the same harddisk, like the files (the second harddisk) or is it better to store it on the first harddisk, the same, where the program is installed? (....)

On that last part of your question Markus (highlighted here by me), the good news is that once IMatch is running, there's no significant benefit or detriment in storing your images or your IMatch database on a different drive from where IMatch is installed.  The excellent advice in the replies above will bring huge benefits regardless of which drive IMatch is 'launched' from.

But please allow me to stretch this topic a little by offering two caveats about USB 3.0 memory sticks and the "interfaces" on our computers into which they're plugged. (My blurb below is aimed cheekily at our wider readership Markus, not necessarily offered to yourself!)

About running IMatch when its database is stored on a USB 3.0 based external drive, or (especially) if it's stored on a USB 3.0 rated memory stick:

Caveat 1. 
I'd recommend that when choosing a USB 3.0-graded memory stick to support our complex, "I/O busy" IMatch database, we should "play safe" and avoid any device which boasts "cheapness" or "economy" before it boasts "long-term warranty" and reliability.  There are millions of thumb-drives / memory sticks to choose from, but they're far from 'equal' in how they interact electronically with our computers.

Or to put it another way; we should use at least as much discretion in our choice of an 'IMatch memory stick' as we'd use when choosing the SD Cards or the CF Cards to plug into our expensive digital cameras.

Caveat 2. 
Yes indeed, a USB 3.0 based device can be fast, but I urge you to test, test and test again, over an extended period of time (30 minutes or more), to confirm how reliably your chosen USB3.0 device handles data transfers, especially random reads and writes, whenever that device is connected specifically to your working computer.

It doesn't matter how well or how reliably your USB3 device performs on other computers!  You should confirm that it works reliably in the exact PC+device relationship which matters the most in this database context.

I base this cynicism on my own experience with an otherwise stable and reliable Windows computer.  I suffered recurring and (what is even worse) rare data-transfer errors when I tried to move large amounts of data to and from my USB devices.  Errors occurred whether I transferred small nibbles of data (as with image files) or with larger blocks of data (as with huge ZIP files).

I'll skip the details of my diagnostic struggles over many months, working to find the cause alongside other technical people and with multiple permutations of USB3 external drives and with two different brands of USB 3.0 interface cards, but the conclusion is that - on my current OEM-built, Intel i7, Windows 64-bit system - I no longer trust USB3 for anything other than occasional docs or snapshot files etc..  I now rely only on the internal SATA-based drives which have proven to be as 100% reliable as we'd expect them to be.

Of course YMMV ... but I'd urge fellow IMatch users to test and verify whether your confidence is fully justified in using a USB3 device on your specific system for your IMatch database files.  After all, it's a database on which you'll invest many many hours of your time!

(Rather than me recommending any "disk test" tools to test your favourite USB3 memory sticks, you could give each memory stick a good work-out by running a "contrived backup" onto it, followed most importantly each time by the backup program's verify phase, and repeating this a number of times.

Choose a bunch of files which will almost fill the remaining space on your USB device, and then, if the verify phase reports "no problems" every time you do this, then you've got a good solid USB3 and PC relationship  )

/steps off USB3-cynical-soap-box

Colin P.

[Richard]

I find your experience interesting. I have been using a USB hard drive for backups, moving files between computers, and restoring files for about six years. No problems. I also use three thumb drives and never had a problem. Not even with the cheap ones. On the other hand I have lost files on hard drives due to sectors going bad. Both cases point to the need for frequent backups of all data.

For many forum members English is, at best, a second language. Except for a few very common ones, I hate to see abbreviations like: "YMMV". If they have the Oxford dictionaries they can find what it means but it does not appear in my Webster.

[ColinIM]

Hi Richard,

(....) Both cases point to the need for frequent backups of all data.

I agree 100%.

(....) I hate to see abbreviations like: "YMMV". If they have the Oxford dictionaries they can find what it means but it does not appear in my Webster.

Darn it, I agree here too, 100%!  OK, perhaps about 98%.

How ironically frustrating (for me) that it should be me who receives a gentle prod about using acronyms on a Forum!!!!!!!!!!! 

My reply to you on this second point about acronyms was (as usual for me) getting awfully long-winded, so I moved it onto a new post on the Off Topic board ...

... see On using acronyms like YMMV on a multilingual board such as this one

Join me there ... ?

Colin P.

[sinus]

Wow, thanks for all answers ... I will answer later, SORRY, because I have to take pictures these days a lot abroad (Zürich, Basel, Luzern, Engelberg ...) hence I am not often at my PC.

Have a good time ... Markus

[sinus]

Hi all (in German we can say "Hallo zusammen or Hallo alle zusammen"), but in English I do not know, maybe "Hi folks" or "Hello everyone" or ...

Thanks for your answers, I really appreciate them! So, like John wrote, I will put the db and the files on two different harddisks.

And, Inh, I have Win 7, but no USB3-support. And to buy such a card would not be a problem, but to open the pc, insert it, install it ... and at least MAYBE have troubles with drivers and so on, I do fear, so I will wait with this, and maybe better buy another computer in 2 or three years. My computer is about 3-4 years old now.

Richard, your liest is very helpful, I would like to have SSD, but they are still to expensive, I think, and the formfactor, what I must have, is 3.5" and SSD have mostly 2.5".

Colin, your remark about once IMatch is running, is also helpful. Did not know this. And of course, also your other part of your post is very interesting. And, thanks to your other post, triggered by Richard's (good) remark, I do now know, what YMMV means

IMatch is now very stable and I hope not a long time, then I will switch my daily working db from IM3 to IM5. I have worked for this and IM5 is ready, I think, with all relevants preferences.

Thanks!

[lnh]

...
And, Inh, I have Win 7, but no USB3-support. And to buy such a card would not be a problem, but to open the pc, insert it, install it ... and at least MAYBE have troubles with drivers and so on, I do fear, so I will wait with this, and maybe better buy another computer in 2 or three years. My computer is about 3-4 years old now.

Richard, your liest is very helpful, I would like to have SSD, but they are still to expensive, I think, and the formfactor, what I must have, is 3.5" and SSD have mostly 2.5".
...

If installing a USB3 card isn't something you'd like to do, adding a SSD isn't too hard depending on your system. Small, compact computer enclosures can be a challenge, but a generously sized tower enclosure shouldn't be too bad. You can also take that SSD to your next system in several years, so the investment isn't too bad. Mounting kits for placing a 2.5" device in a 3.5" bay are readily available. I know you're in Switzerland, but see this Newegg http://tinyurl.com/lkltyq4 for an example, and there are many companies which make this sort of thing. Price-wise, you probably can't compare Newegg or Amazon in the US with other regions, but 64GB consumer SSD's from name brands start at $50USD and a very large IMatch DB will fit on 64GB (assuming the actual files are located elsewhere). Not sure if TRIM is enabled by default on W7 (it is on W8.1), but it's easy to check and modify if needed.

[Erik]

Just to relate, my own system has its C: drive on a 480GB SSD and two other physical, standard SATA HardDrives.  My photos are on one of those drives, and a music collection is on the other.  Most of my software and the system is on the SSD.  Most other Documents are on the same drive as the photos.

Originally my IMatch database and cache were on one of the other hard drives.  IM was a bit slow, and then I moved the database to the SSD.  The speed increased immensely.  The cache is still on the standard hard drive.  SSDs are becoming quite reasonably priced. 

[ColinIM]

Regarding what I think is indeed the standard 2.5" form factor of SSD drives - the last two OCZ branded SSD's I purchased came with a neat metal plate which could optionally carry the SSD drive and allow it to fit neatly into a 3.5" drive-aperture.

BUT ... because of the following superb and increasingly well known properties of SSD drives, it's feasible to place them almost anywhere inside a computer case, in any orientation, and with fewer concerns about keeping an air-flow around them when compared to a conventional hard drive (although they do need to have some free air around them) -

1.  SSDs run consistently 'cool'. My SSDs rarely get warmer than the metal surface on the sides of my computer case.

2.  (as we know) SSD drives do not 'spin', which entirely eliminates any concerns about mounting them vertically or horizontally or at any other angle.  As long as the SSD drive and its connections are secure and stable, the SSD drive can be tucked into any reasonable 'void' inside your computer case.  And then their compact, 2.5" form factor becomes a big advantage 

(There are differing opinions about the impact on the life of the spindle-bearings of conventional, spinning hard drives if they aren't mounted almost completely horizontally or vertically.  And their fast-flying head-actuators can be influenced by skewed mounting angles too.  Vibration is also a big concern for the very latest, high capacity spinning hard drives; vibration caused for example by a grumbling cooling fan mounted nearby ... but SSDs are (within reason) also immune from vibration worries!)

I could ramble even longer about them ... !!  But yes, basically they're blooming marvellous devices 

Colin P.

[marco88]

Hi,

Having slowish responsiveness generally (even after disabling count on categories/collections/folder) and enabling batch processing of updates I am keen to test out moving the dbms on a USB 3.0 drive.

Sadly my USB ports are only 2.0. Is that not useable? i.e. too slow?


Kind regards
Marc.

[Richard]

 

Is that not useable? i.e. too slow?

I tend to think that USB 2 would be too slow but the proof would be to compare read/write speeds for your hard drive with those specified for USB 2. Of course the ultimate would be an internal Sold State Drive. Then you would not need to worry about USB.

[sinus]

Hi,

Having slowish responsiveness generally (even after disabling count on categories/collections/folder) and enabling batch processing of updates I am keen to test out moving the dbms on a USB 3.0 drive.

Sadly my USB ports are only 2.0. Is that not useable? i.e. too slow?
Kind regards
Marc.

Me too! Sadly only USB 2.0
But I think, as others point out, a USB 3.0-card should be cheap.

But to be honest, I do not want do it now, because I fear the installation and all this stuff. I have already installed therefore an internal new harddisk, this was easy, because I had only to put it in the computer like a CD-ROM (there is such a wagon (insertion), in what I could put my harddrive (SATA).

But even then I had massive problems to install it, I had bluescreens and so on, and was forced to change things in my boot-section (hit Del at the start of the computer, like 100 years ago

Without the help of Joerg (joel23; THANKS!) I am not sure, if I could have done it.
Phew, in ATARI's times computers was easier, I built then even 2-3 computers myself, but nowadays the big amount of software and hardware is simply overhelming - except it is the profession in the IT-business.

[Gerd]

Hi,

I have now changed from an older i5 to an newer i7 notebook (HP Envy17, i7-4702QM, 16 GB main mem.). It contains a SSHD (hybrid HD) with 1TB as C: and I upgraded this with a buil-in 2nd SSG with 256GB.
This gives me an important speed win, if I work with my Excel-sheets (10 -> 280GB with sometimes 900.000 records).

No probs to implement the SSD. I kept my OS (Win 8.1 Pro 64bit) on C:, I only put my data on the SSD. I also moved only my IM5-db to the SSD, programs and pics remain on C:.

My IM5-db is now 5,3 GB and contains 125.000 pics. The time to open IM5 and can do something is now 1 min., the time to get a selected pic the first time in the viewer takes another 1,5 min, later it only some seconds (0 to 2 sec., mostly 0 sec). To get all pics available in the file-window takes now 4 sec.

IM needs only more time, if new pics have to be added, indexed and is reading the Meta-data. For 44 pics it takes less 1 min.

And my experience with USB is, that an HD with USB 3.0 connected to USB 3.0 needs half the time than connected to USB 2.0.

Overall I'm very happy to have upgraded to a quicker system. IM5 reacts now nearly 'immediately' ... thanks here to Mario for his IM5-improvements! :-)

[joel23]

Hi,

Having slowish responsiveness generally (even after disabling count on categories/collections/folder) and enabling batch processing of updates I am keen to test out moving the dbms on a USB 3.0 drive.
Sadly my USB ports are only 2.0. Is that not useable? i.e. too slow?

Kind regards
Marc.

Why not connecting a HDD directly to a SATA or eSATA port?  In case your motherboard only has SATA-I ports but a free PCIe slot, get yourself a 30€ PCIe <-> SATA-II/III extension card and a 60€ SATA-II/III HDD and you will get the maximum out of the disk, 100-150MB/s.
If your PC already has SATA-II/III ports but all are taken, get a SATA multiplier (make sure it is FIS-based) - a multiplier is good enough to feed up to 3 SATA-III HDDs (assuming max. 150MB/s for HDDs) on ONE SATA controller/port at full speed.

Anyway. The first question should be: where is the real bottleneck?  Even a SATA-II HDD should be able to feed IMatch on normal use, assuming the HDD is not fully packed and not much fragmented.  If you on a notebook, tscha... sorry.