Database optimization

[HGF]

Hello, I'm crawling ~ 170.000 pics with different types (ARW, NEF, JPG XCF, Tiff etc.), in 2.800 folders.
I have 1.674 categories configured, my database is ~5.5 GB. Sometimes, mostly when searching for categories it needs up to minutes to get a result. DB diag and optimization is done. DB on SSD, pics on fast HDD, CPU and memory should be okay. Are there any hints, tips how to optimize data structure or/and cat structure? Any experiences? Is there a log available to identify what task is taking so much time?
Thank you in advance, HGF

[attachment deleted by admin]

[Mario]

How do you search for categories?
In the Category View?
In the Category Panel?
Using a Filter?
...

Please provide us with a minimum info, e.g. the log file (in debug mode) from a session where you searched for categories. Without a log file I cannot even start to guess.

[HGF]

Sure Mario, I'll provide everything you are requesting.
1. Start of IMatch
2. In Media&Verzeichnisse I clicked on Gimp Test (3)
3. Took several minutes to provide 3 thumbnails

Attached, screenshot and dump file.

Remark: This is not typically for all folders. A lot of them are visible very fast. Maybe this has something to do with xcf files? Maybe search in cat is slow if xcf is in the search result?
Maybe the dump tells more.

Thanks HGF

Did some additional test: Take such long time just on the first click when Imatch has startet. Each click afterwards (2nd, 3rd try is fast). Maybe the thumbs will not be stored in the database?

[attachment deleted by admin]

[Mario]

IMatch did not have cached data for an internal index and needed to load this from the database. This took a whooping 25 (!) seconds. And it had to be done after you first clicked on a folder to show the thumbnails.

Usually IMatch caches that info in the database so this needs to be done only once. 
Once this data has been loaded and cached, things are fast. IMatch stores the cache when the database closes normally.

I see that IMatch reports that your last session did not properly shut down. Did you have a crash? Aborted IMatch in the task manager?
If you close the database normally once, the cache will be written to the database and the slow effect will be gone.

[HGF]

Mario, I had long response time and sometimes hangs when clicking through folders. If a hang occurred I closed by Task Manager. I'll do some more investigations to provide you more details. HGF

[Mario]

When you force IMatch to close, it cannot properly shut-down the database. And the cache will not be written, which means that it takes as long the next time you start IMatch. Start IMatch once, switch to a folder, wait until IMatch has re-created the missing cache and then close IMatch normally. It will be much faster the next time you start it.

Don't shut-down IMatch in the Task Manager without very good reasons. Check if IMatch is still using CPU and disk (and thus is buds). Check the log file repeatedly to see if IMatch is still working (writing stuff into the log file). Only when this is not the case for several minutes, IMatch may be actually stuck.

[Richard]

Check if IMatch is still using CPU and disk

Just a couple of days ago IMatch seemed to hang so I checked Task Manager and it reported zero CPU usage. However in a moment the usage went to 1 percent, then back to zero, then 1 percent again. Most of the time it reported zero usage and just occasionally 1 percent and once shot up to 9 percent. During this time I could see that my hard disk was being used occasionally and no other application would have been accessing the disk but IMatch.

I gave IMatch time to get everything straight and I than had the response that had been missing and which had made me think it had hung.It appears to me that far too many users are too quick to assume and simply close IMatch using Task Manager and instead of curing problems, they compound their problems.

[HGF]

I'll take care on this, thanks HGF