How long does it take to add new files?

Started by ChicagoGhost, October 17, 2024, 09:25:45 AM

Previous topic - Next topic

ChicagoGhost

I have 150K photos to add, running on a 32-core (64-thread) CPU.
Sometimes, it says 12 hours to go, but sometimes 30 minutes.
It has already been an hour, and about 1/3 has been processed.
I see in the help that I need to run a Database Diagnosis every 50,000 files.
So I have to be in front of the PC to do this.
I suggest adding an option to let the user decide whether IMatch should do the Database Diagnosis automatically (during the process of adding files). Is there such an option?

---update
The IMatch window just closed by itself.
When I started it again, I chose to start normally.
But then it loaded the previous database I worked with, not the one that was running when the window closed.

Mario

#1
QuoteI see in the help that I need to run a Database Diagnosis every 50,000 files.
So I have to be in front of the PC to do this.
You don't need to run diagnosis. Running compact & optimize can streamline your database performance after adding 50K of files. But it's just a performance tip. If your database is on a SSD, you can let IMatch add all files at once - if everything is stable and now corrupted files are in the collection.

QuoteSometimes, it says 12 hours to go, but sometimes 30 minutes.
Unless you virus checker is interfering and playing games, this just means that the files being processed perform very differently. If you mix RAW with JPG with video files, the estimate will vary a lot. IMatch calculates the estimate based on a window of files seen recently, and if the recently seen 20-50 files took 0.5 second each to process (JPG) and now IMatch hits 50MB+ RAW or 1GB TIFF files which take 2 to 30 seconds to process, the estimate will go up of course.

QuoteThe IMatch window just closed by itself.

This is a sign that IMatch was terminated by Windows because of a component or external library it used crashed.
If IMatch crashes, it produces a DUMP file. See The Debug Dump File
If Windows terminates IMatch because a WIC codec crashed, a shell thumbnail handler crashed, or one of the external helpers IMatch uses to process PDF and video files crashed, Windows will terminate IMatch without IMatch having a say.

Always secure the log file in such cases, to support a postmortem analysis. See log file for details.
Make a copy of the log before restarting IMatch, ZIP and attach. If you let IMatch process 50K files in one go, the log file might be too large to upload, though. Upload to your cloud space and post a link in that case.

Processing 150,000 files in one go is prone to reveal some badly damaged or corrupted files. Without a log file it's impossible to tell which files caused the problem.

Processing this many files (I don't even know which kind of files you use) is also prone to accumulate "issues", like WIC codecs not releasing memory etc. This is why the help says to add files in batches. And maybe close IMatch every 30,000 files so it and Windows can release all allocated resources.

If your CPU has 32 cores and 64 HT (which makes it something corporations run multiple web sites on), IMatch will try to use them all. Which usually means that the disk containing the database and the disk(s) containing the files are the limiting factor. Reducing the number of threads used by IMatch might improve performance. Maybe try 8 to 12 for everything except face recognition. See Process Control (Advanced Setting) under Edit menu > Preferences > Application. Tick the "Show extended settings" box at the bottom to see all settings.

QuoteBut then it loaded the previous database I worked with, not the one that was running when the window closed.

If you have more than one database, IMatch remembers the last loaded database when it closes.
Since it was terminated by Windows, it had no chance to remember the database loaded in that session and thus loads the database you had open when the last session closed successfully. Normal behavior.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook