IMatch Freezing Endlessly

[Darius1968]

I'm attaching a log file about IMatch freezing for long periods of time.  Specifically, I had nine files in a selection, which were given to the viewer.  I simply used the keyboard right-arrow key to traverse through all of those images.  So far, so good.  Then, when I hit the "ESC" key to exit the viewer, I was at the file window, with those nine images still selected, as expected.  However, the  blue circular-wait icon was in effect for over a minute, before IMatch was done processing.  The same thing happened again.  I closed out the database, so that I could restart the program.  Here IMatch again froze, and I had to finally force-close the program.  So, this is the log file for your review.  Thanks. 

[Mario]

Your database is massive.
It has 400,000 files and almost 25,000 categories.

I see that IMatch reports that the database is locked.
IMatch is busy scanning folders for modifications, which is very slow on your system.
My guess would be that your virus checker is blocking IMatch or the database for periods of time.
Or that your system cannot keep up and that the disk and database system are struggling with the way you use IMatch and the massive number of files.

Make sure the database is on a SSD.
Remove unneeded data-driven and formula-based categories.
Make sure your virus checker is leaving the database and IMatch alone.

Close some expensive panels like the filter panel, category panel.
All these work great, but 400,000 files in the database and keeping everything up to date all the time may be too much for your system.
Not even IMatch can work miracles.

Keep in mind that many stock photo agencies manage way less images on dedicated systems than you try to manage in a single IMatch database.
Consider splitting your database into two databases of 200,000 files each to reduce the computing requirements.

[Darius1968]

My database is on as SSD.  How would I go about determining in which hierarchy my categories have the greatest concentration.  My guess would be one of the data-driven categories, but all those are set to manual refresh only.  My system is a year-old i9 processor with 16 GB RAM.  I'm not really doing anything that is changing these files.  Also as I've said before, I have the IMatch program file and the database both excluded from my antivirus.  What else can I exclude that might help IMatch? 

[Mario]

I see many entries for AddOrUpdateFolder so IMatch seems to connstanly scan something...

I have a 650,000 files test database on a 5 year old PC and I have never seen this kind of locks and database panic messages.
Something is interfering with IMatch. Did you double-check the virus checker? This is the source of problems in 90% of all "IMatch is slow" reports these days...

[Darius1968]

The following screenshot shows that I have both the actual IMatch program file and the folder in which it resides excluded, as well as the database itself. 


So, is there anything else I should be excluding? 

[graham1]

I have in my main photo database some 625,249 images, almost all RAW files, and 20,450 keywords and synonyms. The database file size is 17.99GB. The database and cache are on a SSD and the files are all on internal hard non-SSD drives.

I too was having apparent freezing issues, but I expected that with this number of files.  It is not perfect yet, but it is improving.  My main computer is also about a year old, an i9 with 32GB RAM.

I decided to rebuild my database slowly.  I catalogue my images into one top level folder per year.  I have systematically added one year at a time, letting it run overnight if necessary, and running compact and optimise after each run.  My cache is set to allow 400GB, and has so far reached 281GB.

As I have added files, it has slowed down, and now if I add any more, thumbnails are often not created.  In order to get the thumbnails, I right click the folder not showing them and force an update, which cures the problem.  For each folder I do this for, it speeds up.  It needs a methodical and patient approach, but I feel I am getting there.  If you are prepared to leave it running overnight to optimise, and rescan where necessary, I think it will reach an acceptable performance level.

The exact same images and keywords are in my Lightroom catalogue, which has been built up slowly over the years.  That, too, works, but when I comes to keywording, it can be a real dog - pause for a moment after 1 or 2 characters have been entered in the Keywords filter, and it locks up for (literally) a minute or two.  I am far preferring IMatch for keywording.  Presently I use a small on the fly catalogue for keywording the images after they have been developed in Lightroom, but I hope that once I have got the database fully tuned, I will be able to do everything from my main database, which amongst other things will mean I do not have to keep my keywords synchronised.

So, in summary, I think performance is improving with effort being put in to maintain the database, but it takes time.  I think it will be worth the investment.

Graham

[Mario]

I have in my main photo database some 625,249 images (...)

In real life, there is no free lunch. We all know that. You said that in your post.

If you go to one of the 'big' DAM vendors (Canto, FotoWare, Extensis, AssetBank, Widen, ...) and ask them for some specs and price quotes for managing 700,000 assets, you will learn some new things.
Words like dedicated server platforms, scaling, multiple servers to spread the load etc. A lot of on-premise hardware or rented cloud computing power.
In short: Managing this amount of file takes a lot of resources. And money.

While I try to make IMatch faster with every version (and I've made a big step in IMatch 2020), there is no free lunch. No magic.

Searching 500,000 files instead of 100,000 files will take 5 times as long. Actually, it will take even a bit more due to physics and database technology....
Sorting 100,000 files instead of 10,000 files will take 10 times as long.
Keeping a data-driven / formula-based category up-to-date for a 500,000 files database will take 5 times longer than for a 100,000 files database.
And IMatch, by default, has 20 or so data-driven categories...

The same is true for collections. For the timeline. For ...

IMatch is really fast and pulls a lot of tricks to cache things, calculate things in parallel in the background etc. But, there are always limits. And physics.

It may already make a massive difference if a user has the category panel open. Or the category or collection filter open in the Filter Panel.
Every action the user performs may invalidate all collections and categories in the database. And, if IMatch needs up-to-date categories / collections to update the user interface, a specific constellation may force IMatch to update these slow categories / collections immediately. And often.
Which then causes tens of thousands of disk operations, which then, inevitably, blocks other database access operations. And, in the end, may even stall the IMatch user interface for several seconds.

If you only have a simple file system browser who only cases for the 500 or 5,000 files in a folder, the total number of files on your disk is irrelevant.
But when you add half a million files to a DAM, which needs to keep track of everything, things look different.
While IMatch is capable to deal with large collections of files, it cannot overcome hardware limits or plain database performance characteristics.

[Darius1968]

If I had a lump sum of files that didn't have a valid thumbnail, and I performed on these a force update, could that cause interactions with an antivirus that would slow things down? 

[Mario]

No idea.

Rescanning files is something that IMatch does in the background. Of course this causes a lot of 'action', from invalidated categories and collections to face detection.
All these operations use the database, and if too much is going on, some parts on IMatch will have to wait for others to finish. This is normal. But if too many things in IMatch have to wait for too long because other things take to long to do their thing, IMatch will log performance warnings to the log file and the user interface may become sluggish at times.