Database speed

[Carlo Didier]

As I just got my first USB3 disk (2.5" 2TB), my other ones still being USB2.
So I thought I'd do a little test on the speed by opening a copy of my iMatch database on my internal SSD, an internal HDD and the external USB3 drive and see what the difference is.

Here are the results (simply opening the same database by double-clicking on the imd5 file). The second time is for re-opening after closing it once. Apparently Windows can cache a lot in memory (16GB RAM):
- SSD:  23s (15s)
- HDD:  38s (15s)
- USB3: 54s (15s)

The same iMatch was used with the same settings ( cache, etc), only the location of the database changed.
For each test, I rebooted the PC and waited ~5 minutes before doing the test.
No virus checking on iMatch files.
I waited for the iMatch icon in the task bar to flash to stop the clock.
Not a scientific benchmark really, but a good indicator of the speed gains of SSDs ... 

[Ferdinand]

Which USB3?  They vary a lot and I'm surprised it was so slow.

[Mario]

The Windows file system cache is very good. It it has enough memory available, it can even hold the entire database in memory...

IMatch also uses a lot of in-memory caches itself, to keep database objects and metadata in local memory. It also uses predictive caching, guessing the data that will be needed soon and loads it into local memory when the database is idle. This is especially important when the database is ion a slow media like an external disk or even a NAS box.

Interestingly and expected, after switching to an SSD for database storage, a number of my algorithms in IMatch are now CPU-bound. Which means that IMatch could work faster if I had 8 or 16 CPUs instead of 4 (Xeon). Before the SSD, the algorithms were disk-bound, which means that the CPU's had to wait often for the disk to deliver the data.

All the little screws and gimmicks which make IMatch run as smoothly as possible...
I tune things all the time. For build 5.4.10 I have made parsing File variables faster, because a new feature also released with the 5.4.10 will benefit from that...

[Mario]

Which USB3?  They vary a lot and I'm surprised it was so slow.

There are indeed vast differences. The German computer magazine c`t tested a lot of the hardware, and some of it is only barely faster than USB 2.0. Especially with external drive bays. And of course USB sticks, which have a USB 3.0 plug but internally only use USB 2.0 parts...

[Carlo Didier]

Which USB3?  They vary a lot and I'm surprised it was so slow.

There are indeed vast differences. The German computer magazine c`t tested a lot of the hardware, and some of it is only barely faster than USB 2.0. Especially with external drive bays. And of course USB sticks, which have a USB 3.0 plug but internally only use USB 2.0 parts...

Mine is a WD Elements 2TB 2.5" USB3. Task manager showed write speeds of 80-120MB/s when I copied files to it. That's up to 960Mbit/s, more than twice the maximum speed of USB2. Of course, it depends on the size of the files (large ones copy faster) and how fast the source is (in this case the internal HDD). Also, the drive is BitLocker encrypted, which may cause a small performance hit.