Performance Tuning: Image Import

[abgestumpft]

Hi,

I did some test with importing RAW files and different number of threads.
Most of the runs I only did once, so there is some room for error. But in general it should give a good indication...

My Setup was:
AMD Ryzen 3700x - 8 core / 32 GB RAM / Nvidia GTX 1060 6GB / Files on classical HDD
Folder with 974 Olympus .ORF RAW files (20MP)

Settings found in Preferences -> Application:

1. Reading Metadata
Threads for metadata import = 0  - 2:30 - 3:00min
Threads for metadata import = 32- 0:40 - 1:00min


2. Adding and Uploading files
Prefer photools.com RAW processing = NO
Use embedded preview = NO

Threads for file import = 0 - 20:46min
Threads for file import = 8 - 07:05min (80-90% CPU, 25GB RAM usage, ca. 120MB/sec Disk)
Threads for file import =12- 06:08min (90-95% CPU, 100% RAM usage, ca. 120MB/sec Disk)
Threads for file import =16- 05:30min (100% CPU, 100% RAM usage, ca. 120MB/sec Disk)


Prefer photools.com RAW processing = YES
Use embedded preview = YES

Threads for file import = 0  - 02:52min (25% CPU, ca 120MB/sec Disk)
Threads for file import = 16- 01:39min (80-100% CPU, ca 230MB/sec Disk)


Summary:
On Multi-Core Systems it definetly makes sence to do some testing with "Threads for metadata import" and "Threads for file import" (especially if you're using the RAWs).
For now I'm using "Threads for metadata import" = 32 and "Threads for file import" = 16 (using embedded JPGs via iMatchs RAW processor)
This way the import of around 1000 images takes around 2:30min in total (vs. ca. 5:30min with default settings)

[Mario]

Thanks for sharing.

[abgestumpft]

Just a small update, that SSD makes a difference 
During the tests on the HDD the 230MB/sec very much look like disk maximum...

Now with around 4000 files on SSD instead of 1000 files on HDD

Setup now:
AMD Ryzen 3700x - 8 core / 32 GB RAM / Nvidia GTX 1060 6GB / Files on SSD (Samsung 860 EVO 1TB - SATA)
Folder with 3948 Olympus .ORF RAW files (20MP)

Prefer photools.com RAW processing = YES
Use embedded preview = YES

1. Reading Metadata
Threads for metadata import = 32 - 2:00min (100% CPU, 2,5GB RAM, 220-300MB/sec disk)

2. Adding and Uploading files
Threads for file import = 16 - 02:10min  (80-90% CPU, 1GB RAM, 600-800MB/sec disk)


So 4000 RAW files imported in 4:10min

[mastodon]

If I have an  i5-3570, that has 4 threads, Samsung 840 EVO 250 GB. What to automatic do? Is it worth to use manual settings for this hardware setup? Actually I see the writing is sometimes very slow, even with 10 pictures (takes more than 10 sec). But this is occasionally.

[Mario]

None of these settings can affect how fast ExifTool updates files.
This depends on the file format, the file size, if the file needs to be spliced first to make room for new metadata etc.
Usually this is totally I/O bound so throwing more processes at it might do no good if the disk is already at 100%.

[grish]

Hello,
I have an old second hand  workstation with more RAM and processors than I could ever use. (256 GB,  bi-xeon 32 proc).
But is there a way to say to Imatch "take all you need, don't be shy !"
No swap file of course.
Put all threads to 32 as show here
Could using a RAM disk be useful ?

I have perhaps 50k pics and I use RAW files when possible.

curently begining importing with 8 GO RAM used and 6% of processor and less than 1mo/s. I guess is preparing the database before the real job.


Thank's for your help !
grish

[Mario]

50K images is nothing.
I create databases with that size all the time, for testing purposes.
A few hours, tops. Images on local hard disk, database on SSD. 12 cores.

When I understand you correctly, you are experiencing performance problems?
Are these files stored on a local disk, SSD, network?
Database on SSD?

Throwing more processors at the job won't help if the disk does not deliver the data fast enough. If the CPUs are idle, the disk speed is the problem.

RAM is not an issue, IMatch is happy with less than 1 GB. The only feature that uses lots of RAM is the Viewer, to pre-cache images in memory.

Do you use WIC codecs? Which?

Switch IMatch to debug logging (Help menu > Support)
Import one folder with a few hundred files.
ZIP and attach the log file. This will show us some numbers.

Important: In 90% of all "IMatch is slow" cases, the problem is the virus checker. Scanning the database or images all the time while IMatch is working.