[UE] Data-Driven categories no longer update

[JohnZeman]

I'm having a strange problem I've never had before.  All of a sudden IMatch will no longer update data driven categories when I do a manual rescan of DD categories.  It appears that it is but afterwards nothing has changed.  If I exit IMatch, which for a while took much longer than it should, then restart IMatch again IMatch would tell me it didn't shut down properly.  So I did a database diagnostics (twice) but no problems were found.  Rebooting the computer didn't solve the problem either.

The one thing I did earlier today that I've never done before is I configured my file relations to add DxO PhotoLab .dop files to the DNGs in my database which seems to work just fine.  Then I told IMatch to rebuild all relations since I'd never used any file relations before but when it appeared that it was going to take 150 hours to rebuild all relations for my 128,000 photo database I closed IMatch to kill the process then restarted it.  That seems to be when the problem started.

Now IMatch seems to close properly but it still will not scan DD categories.

Log files attached.

Thanks Mario.

Note I may be a little slow responding, this is my busiest time of the year at work.

Edit: I may not have attached the right log file the first time.

[JohnZeman]

Well the problem appears to have cleared now after I also did a database compact and optimize operation.

Strange.  I'll keep an eye on this to see if it happens again.

[Mario]

IMatch may be still processing your relation updates. How long this takes depends on how you have configured them.
It is easy to shot yourself in the foot and make IMatch process billions of operations, bringing it down to a crawl.

1. Show us how you have set up your file relations (screen shots will be good).
2. Switch IMatch to debug logging via Help > Support. The log files show nothing unusual.
3. Make sure category updates are enabled under Edit > Preferences > Background Processing.

You may want to clear the processing queues via Database > Tools > Clear Processing...
Just shutting down IMatch while it is processing relation updates will not abandon the process. IMatch performs this in the background and queues everything. The queue will not be cleared when IMatch closes and IMatch continues automatically when you start it again.

[JohnZeman]

I should have thought to clearing the processing queues but it was over my noon hour and I was in a rush. 

I've now cleared the processing queues and have enabled debug logging.

I have update data driven categories enabled in preferences > background processing but I have most of my DD cats set to No for automatic updating.

Below is a screenshot of my file relations configuration.  I just recently started using PhotoLab 2 to process my original files and besides the DNG buddy .dop files I created similar relations for .dop sidecars for TIFs, PSDs, and JPGs.

I think we can close this bug report as user error.

Thanks again Mario.

[Mario]

You have set IMatch to search the D:\Mine\Photos\Original and 5 levels of sub-folders for buddy files. This means that IMatch has, for each file which is identified as a master, search all files in this folder hierarchy. Which may be a lot (?) and can cause IMatch to work real hard.

For example, if this hierarchy contains only 50,000 photos and IMatch scans a folder with 100 masters, it has to do 5 million operations (in the file system, which is real slow, plus in the database). This can take a long time. Buddy files need to be searched in the file system (different than versions) and this is 100 or 1000 times slower than searching the database.

Do you really spread your buddy around in that folder hierarchy, in arbitrary folders?
Usually buddy files are in the same folder as the master, or in a sub-folder of the folder containing the master.
And in that case you can reduce the number of files IMatch has to search greatly by searching only in the master folder and maybe one level deeper.

[JohnZeman]

I don't know how I managed to misread that page in help, it's certainly well written and plain as day to see but I sure did. 

My buddy files are in the same folders as the masters so I've now changed my file relation settings to Master and Specified folder only.

Thanks Mario.

[sinus]

I don't know how I managed to misread that page in help, it's certainly well written and plain as day to see but I sure did. 

My buddy files are in the same folders as the masters so I've now changed my file relation settings to Master and Specified folder only.

Thanks Mario.

Cool, John, now your DD will be even quicker, hopefully.