Reading metadata after write-back takes very long

[Ger]

Mario,

I'm very regularly experiencing extreme long wait time when reading metadata after a write back. It happens with different files.
I don't have an idea what this is causing, maybe the debug log file gives you an indication on where I must look.

Thanks

ger

[Mario]

In this log file, a whole bunch of files is written.
This complicates things a bit, since IMatch writes multiple files in multiple threads in parallel...

Write backs take between 1s and 3s for JPEG files, which seems a bit on the slower side.
Metadata is propagated too.
8 files written in about 8 seconds. OK, although slightly on the slower side.
11:02:13 => Write-back completed at

A number of data-driven or formula-based categories is updated, using metadata searches regexp like "\|Source$".

11:02:15 => Metadata read for 3 files in 1984ms.

11:02:16 => Metadata read for 5 files in 2109ms.

IMatch updates data-driven and other categories and collections.
Database suddenly becomes very slow, collection updates re-scheduled because database is not responding fast enough.

IMatch reports #BLOCK events, meaning database operations have been aborted because it could not open a transaction. Wait times go up from a few milliseconds to several seconds. Some background processes abort after waiting for a minute or longer. Exceptional!

Constant calls to "InnerSearchMetabase", which is used by data-driven / formula-based categories and some filters in the Filter Panel and/or the File Window search bar.

Database access times become supert-slow. Basically, the database grounds to a total halt.
140 seconds (!) to import metadata for a file into the database. Usually this takes maybe 200 milliseconds. Nasty.
The database is suddenly so slow that it takes IMatch 2 minutes to create a temporary table. Normal is 50-100ms.

The database is small, less than 90K files.
I don't even see such slow response times for databases with a million managed files.

Your system reports 12 processors with 24 cores HT and should deal with such a small database easily.

The only times I see such effects is usually when a virus checker kicks in, constantly scanning the database after each access, slowing it down 1000 times. Which virus checker do you use and did you make an exception for the folder containing the database?

Please check data-driven categories you have created, e.g. 'Meta' to see if they perform well.
Delete data-driven categories you don't use, e.g. under "IMatch Default Categories" and "IMatch Workflow Categories".

[Ger]

Thanks, Mario.

The I basically selected one file for write-back, but with versioning it becomes about 20 or so. 

Anti-virus program is the standard Windows Defender and image folder (d:\Image Library) as well as program folders as well as database is excluded.

indeed, the database has (complex) data driven folders. They work, but if these are the causing the problem, I might have to find an easier solution (maybe an app to run separately to create these folders).

Interesting you notice the |Source$, that's probably somewhere a remnant of an old setting. Will check this.

Attached the log file with the request for writing back one (tif) file, also showing ##BLOCK delays.

ger

[Mario]

IMatch reports that the category 'Meta' is unusually slow. Also 'Work', 'Theme'
And even the default category "Unassigned Files", which is strange for a database with less than 100K files.
Not sure what the Meta, Work and Theme categories do or how you have created them.

The write-back and ingest times are OK, but I see constant calls to "SearchMetabase", which is a routine used to search for tag values, e.g. by data-driven categories or formulas. And this seems to slow down everything real bad.

Try this:

Close IMatch.
Make a copy of the database and then restart IMatch and open the database copy.
Remove the aforementioned categories.
Run a diagnosis.

Do a write-back again and see if the problem still happens.


If not, the categories are the problem and I would like to have a copy of the database to see why this performs so badly. Maybe it performs badly only on your computer or this is a more general issue.

Contact me via support email address and I can send you instructions for removing the thumbnails from a copy of your database. This shrinks the database considerably and you can then upload it to your cloud space and send me a link. Removing the thumbnails will not affect performance so I should be able to reproduce the problem here.

[Ger]

Hi Mario,

First of all, I'm sorry for the delayed response, but last week I had to travel and had no time to test.

I removed the obsolete formula-based category referring to |Source$ and processing seems already much better. The formula in the category definition was erroneous as the |Source$ reference did not exist anymore.

Attached the log with the write-back of the same tiff file i sent earlier.

I will run several more tests; and I will try to remove some of the data-driven (formula) categories.
- 'meta' contains several formula-driven categories
- 'Work' seems strange. It contains a set of regular categories, without any special category
- 'Theme' is a category group under @Keywords; the |Source$ formula referred to here.

Thanks for your analysis and effort here. Much appreciated

ger

[Mario]

No #slow operations or #blocks reported anymore in this log file. Looks good.