[BBD] Imatch stalled on deleting files

[ubacher]

I deleted 6 files. Took too long. Logfile (in debug) shows Failed to get transaction or CS lock for 'This Week' etc.

Logfile attached.

[Mario]

You are uploading a log file that spans 3 days?
Can you at least tell me "when deleting 6 files took too long"?

Your database has about 400,000 files, 25 GB file size on drive D. Is D: your fastest SSD?
Importing a new tag set from a new ExifTool version takes about 2 minutes, which is twice as much and more than it should take. Many folders are enqueued for rescan. That's on Sunday.

There are a lot more then 6 files removed over the 3 days.
Typical execution time less than 0.1 seconds.

I see a number of CIMTaskFile::DeleteFiles, which take between 1.5 and 3 seconds. 90% of the time is spent waiting for the Windows DeleteFile function to finish. I see some much slower DeleteFile logs, where most of the time was spent updating a large batch of data-driven categories.

I see a ton of #SLOWCAT entries for categories named 'WORKING' or WhereUsed'. And they are calculated often. Calculating categories is expensive. Avoid having too many expensive categories visible, because this forces IMatch to recalculate them very often. Or set them to manual when you don't need them often.

I see entries like [13000ms #sl] PTFileWnd::LoadGroup where the File Window took 13 (!) seconds to load. Why?

Your system reports 8 processor cores, which would indicate that this is an older system ? or lower end system?
Managing 400,000 files is quite a lot of data to handle, many stock photo agencies get by with far less images.
There are limits of what IMatch can do, and if you delete a file and IMatch needs 3, 5, or 10 seconds to update all affected categories, because the drive is maxed out or not performing too well, there is little I can do. If IMatch is busy i the background keeping categories and collections updated, and you delete a file, which requires hundreds of objects to be removed from the database, plus invalidates all categories again, things might be slow at times.

Maybe consider upgrading your PC. For 400,000 managed files, a modern PC with 16 or 24 cores, m.2 SSD storage and 32 GB RAM would surely be a noticeable performance upgrade. My laptop has these specs, and it did cost me about 1000€ 18 months ago.

[ubacher]

Can you at least tell me "when deleting 6 files took too long"?

Sorry, I expected you would search for Failed to get transaction or CS lock.
Look for 02.26 09:57:22+

I had no problems with slow CATS in the past. Some of the rported I could at least understand, the one of WhereUsed is a plain category tree without any calculated etc. cats.

Yes my system is old, an upgrade is planned for this summer. The SSD is an M2 Samsung 970 EVO Plus.
The images are on a spinning disk. 
I just see this much delay since IM 2025.

I see entries like [13000ms #sl] PTFileWnd::LoadGroup where the File Window took 13 (!) seconds to load. Why?

I thought you could shed light on this.

[Mario]

The failed to get a transaction is just an indicator that the database is too busy. A collection tried to update, could not open a transaction in the allowed time slide and re-enqueued itself. A normal operation when the database is busy, e.g. while calculating many #SLOWCAT categories.

WhereUsed is a plain category tree without any calculated etc. cats.

Set still it takes more than 5 seconds to retrieve it. Other other categories calculated at the same time max out the database. IMatch Workflow categories, even the Unassigned Files categories take a long time. A database with 400,000 files will just take 10 times as long to calculate a category than a database with, say, 50,000 files. Physics.

IMatch was also often busy updating folders and indexing files. Maybe your system is overloaded?
Change the performance profile to "Balanced" to see if this makes a change.

I cannot consider this a real bug. Your computer is just not that fast, and you combine that with a corporate-size database managing 400,000 files. Something has to wait, sometimes.

Move the categories reported as #SLOWCAT under a common parent and close it to hide the categories. This reduces the number of times IMatch has to recalculate them.

Consider switching categories you don't need often to manual update.
If you don't use (all) of IMatch Standard / Workflow categories, delete them. Updating these categories can be expensive, and for a 400,000 files database on an older PC, it can be really expensive.