Imatch crashes during import

[Uwe]

Hello,
I am currently trying to import approx. 200000 files into Imatch. Unfortunately Imatch crashes every now and then. I cannot recognise the cause. I then have to restart Imatch and the import will continue.
system event:
Fehlerhafter Anwendungsname: IMatch2023x64.exe, Version: 23.14.0.2, Zeitstempel: 0x66ab760d
Fehlerhafter Modulname: ntdll.dll, Version: 10.0.26100.2454, Zeitstempel: 0x7cb6b6a8
Ausnahmecode: 0xc0000374
Fehleroffset: 0x00000000000881f5
Fehlerhafte Prozess-ID: 0xC24
Fehlerhafte Anwendungsstartzeit: 0x1DB67F22400AD65
Fehlerhafter Anwendungspfad: C:\Program Files\photools.com\imatch6\IMatch2023x64.exe
Fehlerhafter Modulpfad: C:\WINDOWS\SYSTEM32\ntdll.dll
Berichts-ID: bd3491f7-e064-4cf2-b6cb-6a4244028469
Vollständiger Name des fehlerhaften Pakets:
Fehlerhafte paketbezogene Anwendungs-ID:
Regards, Uwe

[Mario]

This is not uncommon. I explain all that in the Oh no - IMatch has crashed! help topic shown to new users after creating their database. Also take note of the information given in A Note About Performance

When IMatch crashes, you usually get a message and then IMatch produces a DUMP file. See Debug Dump File for more information.

If this does not happen, the crash happened somewhere outside of IMatch, e.g. in a WIC codec or other operating system component used by IMatch. ntdll.dll is a part of Windows, for example.

Adding 200,000 files produced over maybe 10 or 20 years and mangled with whoknowswhat software is prone to cause some issues like IMatch stalling with corrupted files or a WIC codec (when you work with RAW files) crashing, taking IMatch with it. A network stack collapsing under heavy load, NAS boxes becoming unresponsive under prolonged load, the system running IMatch becoming unstable under prolonged load ... 1,000 reasons.

If you use the default settings for performance, maybe dial them down a bit to reduce the stress on your computer. Some systems become unstable when IMatch is maxing out the system for a prolonged time. An unstable system can cause all kinds of effects. See Process Control (Advanced Setting) for more info.

Also, always include the ZIPped log file (see log file) created for the IMatch session where the problem happened. Backup the log file before restarting IMatch. This gives us a minimum of information to work with, e.g. if IMatch ran into a bad batch of files with many corruptions, if WIC produced errors etc.

[Uwe]

you got the debug-logfile py e-mail.
no dump was created by the IMatch crash
regards, Uwe

[Mario]

you got the debug-logfile py e-mail.
no dump was created by the IMatch crash
regards, Uwe

I'm usually several days behind with emails. I get a ton of emails every day.

If no DUMP file was produced, a system component, WIC codec, driver or something else crashed and took IMatch with it. Nothing I can diagnose without a DUMP file, sorry.

Reduce load as explained above, process files in batches of 10K to 20K files as outlined in the help (see like above). If you use WIC, enable photools.com RAW processing in the Application settings to rule out a crashing WIC codec.
If IMatch crashes, just restart and it will continue where it left of.

[Mario]

I have read your emails and replied.

As assumed, your system is totally overloaded and you must reduce the number of threads.
IMatch takes 400 seconds (!) to read a single CR2 file via WIC / LibRaw, and that's bringing everything down and reduces performance to a crawl.

And since the system is maxed out for a long time while IMatch is processing your 200,000 files, "stability issues" will creep up, caused random errors or even WIC codecs or drivers to crash. And when this happens, Windows terminates the "hosting" process. which is in this instance IMatch.

This is why IMatch either becomes too slow to react (unresponsive) or is terminated by Windows without a DUMP file being produced (because the crash happens "outside" of IMatch or Windows just terminates IMatch).

Dial in thread counts between 2 and 8 in "Process Control" in Application Settings, and things should work a lot more smoothly.

[sinus]

And I would, even when the system and IMatch are fine running, cut these 200'000 images in several pieces, makes not a lot more work and takes some stress away from the system. 

[Uwe]

Sorry, but I may have a wrong expectations or understanding. No matter how long the 'import queue' is, the application must organise the order of processing. The overall sequential process can only be as fast as the slowest individual step. What is the difference between importing 1000 files and 100000? Only in the overall runtime, because nothing changes in terms of content. Certain actions could also run in parallel, possibly a process that processes a small JPG file is completed faster than a 50MB RAW file and can therefore be written to the DB sooner, even if it was only read after the RAW file. From my experience in the past, the aim has always been to maximise the utilisation of all units involved in the process and only invest in new hardware when the hardware and software are already performing at their maximum. I can't explain the reason for the 400 second wait either (any advice is welcome), because the process involves a new internal 16TB SATA (for the files) and an internal 1TB SATA SSD (for the DB). CPU: 13th Gen Intel(R) Core(TM) i7-13700 2.10 GHz, RAM: 64.0 GB ,motherboard ASUS Prime Z690-P
PS. regardless of the settings of the process control, current each parameter = 2, the CPU load is always 100%
Regards, Uwe
Update: I switched from WIC to Photools RAW process, result: CPU load = 15% and not 100% with WIC

[Mario]

What is the difference between importing 1000 files and 100000?

It's much more likely that you run into way more problem/corrupted files when process 100,000 instead of 1,000 files.
"Issues" in WIC codecs and other components used to process your files will accumulate over time, and the more files you process in a row, the more likely is is that WIC codecs may crash, since IMatch cannot release them to clear out the junk.

Processing 100,000 files maxing out your system for hours and hours is much more likely to reveal instability issues than short comparatively short bursts of 5K or 10K files. I give these recommendations from years of experience, and the very first section of the help topic shown when database is created explains about that.

I can only tell you what I see in your log file. IMatch is processing many files at once, and your WIC (or LibRaw) requires 400 seconds to read a single image from your N: drive. 

The CPU load is at 100% because, again, from what I can see in the log file, many if not most of your files contain XMP face regions - and this means that the IMatch AI is taking the imported face regions, tries to detect a face fingerprint within the region and then stores everything in the database. Face recognition is a very CPU-bound process. But with only 2 threads on a 12 core machine, the CPU should never go that high.

I create test databases with 30K, 50K and 100K regularly on my notebook (fast .m2 SSD and images on external 2TB SSD) and I never saw this high CPU load with only 2 threads. And with the new adaptive system in IMatch 2025, things work very different anyway.

Update: I switched from WIC to Photools RAW process, result: CPU load = 15% and not 100% with WIC 

There you have it.

Windows WIC codecs have issues with your CR2 files. Since Canon does not deliver a WIC codec and has abandoned their developer support program, the only chance to process these proprietary files is the WIC codec maintained by Microsoft or the open source project LibRaw. If LibRaw works better for your CR2 variant, by all means, use it.
It may be the opposite with CR2 or CR3 files created with different camera bodies or firmware versions. It's all a bit of a mess.