Entries stuck in Processing Queue and IMatch does not close properly

[Tveloso]

After working in IMatch for a while today, I found that there were 55 entries in the Processing Queue, which were not being removed:

   

When closing IMatch, it took quite a while longer than usual, with this dialog staying up that whole while:

   

...in some cases without having been completely painted:

   

Simply closing and re-opening IMatch gave the same behavior (with the slow shutdown, and then a report of an improper prior shutdown at the next start).

But IMatch did seem to eventually close properly at each shutdown, despite that longer wait time (putting up the backup reminder choices - where I selected "Skip This Time").  There was at least one time where IMatch did not give the message about an improper shutdown at the next start (I think when I had turned on Debug Logging) but usually it did give that message.

I ran a Database Diagnostic, which turned up no significant errors/warnings (just 5 instances of "Missing Keywords for Entity n" which were fixed), and the Processing Queue remained unchanged.

Should I clear the processing queue?  I'll send a Debug Log to the support email address...

[Mario]

The shut-down in your log file looks normal. The update queue in the IMatch engine terminates all threads during shut-down and all terminate very quickly (within one or two seconds for all).

Except, the Relation Manager thread, which takes one minute to shut-down. Which is unusual.
No additional data is available in the log file.

It appears the thread has been forced to wait by something. If you have buddy file rules, a good candidate would be a Windows function call utilized to search the file system for buddy files perhaps? But that would leave some traces in the log.
All I can tell is that terminating the Relation Manager thread pauses the shut-down for one minute (16:35:25 to 16:36:25) and the last operation logged before that was UpdateFolders, which means that the Relation Manager was scanning folders for buddy files and new versions.

I don't know about your relation settings, the file system layout, if there are maybe offline folders causing Windows file system functions to stall for extended amounts of time etc.

Did you change anything regarding relations recently?

You can empty the queue if this solves the problem. The removed entries will be added to a category for review.

[Tveloso]

Thank you Mario.  This does appear to be related to one of my Buddy Relations as you advised.

Here is some TL;DR on how I arrived at that conclusion (which can of course be skipped):

I had been working in a folder that's a part of my "old storage scheme", which contains lots of files that have yet to be "fully processed".  I was working in small batches there, doing some Face Annotation updates (thanks to help from my daughter) and deleting a few files...(I suspect it's those deletions that might be responsible for the "stuck" processing queue).

After Mario pointed to the possible involvement of a Buddy Rule, I tried a Refresh Relations (F4,R) in that folder.  This seemed to get stuck as well:

   

...so this seemed to confirm a File Relation involvement.

When I was developing my File Relations, I did lots of Refresh Relations (F4,R), and even in folders containing about the same number of files as the one in question here, the process always completed within a second or two, so this seemed unusual.

I cancelled that refresh process, and after looking over my Relation Rules, I found that a pair of them, that I normally keep disabled, were currently enabled, so I thought perhaps they might be the problem...(they were Version, and not Buddy Rules, but I thought they might possibly still be the issue), so I disabled them, and did another Refresh Relations (F4,R) for the folder.  The result was the same (the process still seemed "stuck", and I had to cancel).

So I disabled all my Relation Rules (did a Refresh Relations to ensure it ran to completion - which of course it did), then I enabled each rule, one-at-a-time, doing a Refresh Relations (F4,R) in between.  There were a couple of other (both Version and Buddy) Rules, that specified specific SubFolders Where to search for the Versions/Buddies...for example:

   

...and the Folder I was working in (which is a part of my older directory structure), actually did not contain such SubFolders.  I didn't think that it should cause a problem though, for a relation to specify a non-existent Where to search folder, but wondered if maybe that might actually be the issue (it was not).

Each time (as I enabled more rules), the refresh took just a second or so to complete, until I activated this Buddy Rule:

   

...where the refresh seemed "stuck" again.

(In case the above ScreenShot is not legible, that Rule has a Master Expression of ^(img_)*[0-9_-]+.jpg$, a Link Expression of {name}_back\.jpg$, and specifies the Master Folder only for Where to search)

None of the files in the Folder in question should be eligible to become Buddy Files per that Rule's config, but perhaps the very act of checking the contents of that folder, caused some other problem during the Windows function call...perhaps since the folder in question here is on a NAS...(although refreshing relations in other folders on that same NAS, while that Rule was Active, didn't cause the issue).

So it seemed to be this rule that manifested the problem (even though the rule itself was probably not really the cause).

*-*-(end of TL;DR section)-*-*


While I was investigating this, those 55 Processing Queue Entries were staying out there (even when I had the offending Buddy Relation disabled), so I was worried that I would need to clear the Processing Queue (which I thought likely contained some needed entries - apart from the one causing the "blockage" related to the Buddy Rule), but when I returned to looking into this (with that Buddy Rule still disabled), IMatch then successfully began processing those 55 entries that were previously "stuck" in the Queue, so this is no longer a concern.

And closing and re-opening IMatch no longer gives the issues I was seeing previously (of IMatch reporting an improper prior shutdown). So I think I'm back to normal (and have re-enabled that Buddy Rule).

But the issue with that Buddy Relation seems to still exist in that one Folder, and I'm thinking that it's not actually the Relation Rule itself causing the problem, but probably any Buddy Rule, configured to look for files in the same folder, might cause this issue in this particular folder. 

I tried a Refresh Relations (F4,R) in other folders, containing about the same number of files (also on the same NAS), and it completed in a second or less.  But the folder where the issue originally happened (and where a Refresh Relations still seems to get stuck) remains a problem.  There's probably a specific file, or set of files there, that are corrupted maybe?...

I suppose this entire post is TL;DR...(sorry about that)...but bottom line, everything is good now...(and once I get that "bad folder" fully processed into my new storage scheme, I expect I'll find the underlying issue (i.e. the file(s) causing this problem there).

[Mario]

I don't see an issue with a buddy relation that uses {p0}\FolderThatDoesNotExist.
I've tried this  doing a F4,R on 10,000 files (entire small test database, selecting the Database node) and it took only a few seconds. No buddy files were fond, since FolderThatDoesNotExist  does not exist.

Anything specific I should try?

[Tveloso]

I don't see an issue with a buddy relation that uses {p0}\FolderThatDoesNotExist.

Yes, I have no issues with the relations that are set up that way (not even in the Folder where I was having a problem).

It's actually a Buddy Rule that's configured to look in the Master Folder, that's causing the Refresh Relations to stall - but only in that one folder.

I just wondered (briefly) as I was re-enabling my rules one-at-a-time during testing, if perhaps the {p0}\FolderThatDoesNotExist arrangement could possibly be a cause, but discounted that possibility pretty much right away...and as expected, none of the Rules set up that way, caused any problem at all (in any Folder I tried).

So despite the continued issue with refresh relations in that one folder only, everything is working well...(and I suspect that once I fully process the files in that folder, into my new directory structure, I'll hit upon the File(s) actually causing that problem.

[Mario]

I need more details.
How, exactly, is the buddy relation rule set up?
How many files in the folder. File naming schema?

I don't recall a similar report, ever. So this could be just as well your virus checker interfering or it could be a really exotic issue with relations...if it's something in IMatch, I would like to reproduce it and fix it, when needed.

[Tveloso]

How, exactly, is the buddy relation rule set up?
How many files in the folder. File naming schema?

The Buddy Relation has:

    Master Expression . : ^(img_)*[0-9_-]+.jpg$
    Link Expression . . : {name}_back\.jpg$
    Where to Search . . : Master Folder
    Direction . . . . . : Specified Folder only

...and there are 5,467 Files in the folder in question (which resides on a NAS), and none of the files in that folder will be "picked up" by that Buddy Rule.

But I don't think this is going to be something in IMatch.  The folder in question contains photos from my daughter's old phone, where there was once an issue with the MicroSD Card, and I think that there are some corrupted files there.

I tried opening each of the four Problem Files groups that the Dashboard identifies for my Database:

   

...and then using the Folder Filter to see if any of them were in the "problem folder", but none of them was.

It will probably be a while before I get to working on those older files again.  But I'm thinking that once I do, I will probably file the File(s) causing the issue.

[Tveloso]

Mario, tonight I had a batch of about 200 Files that, when processed with a Renamer Preset (called "Store Photos" - which I use often), took a fair amount longer than usual for those files to "get stored"...(it took well over 40 minutes - when I would have expected no more than 10 or so minutes).

Most of the 200 Files in question had Versions/Buddies, so in all it was actually closer to 400 Files to process (but 40 minutes is still quite a bit longer than I usually see for about that number of files).

I'm not sure if this might be somehow related to the issue described in this topic (the "problem folder" was not involved here - but the files were being moved to that same NAS), but I thought I should mention it just in case (since File Relations were involved in that operation).

I noticed in the log, that there are some relatively long delays, all calling out the same Cache File  (<CachePath>\140539.jpg).  When I looked up that OID, it was for the first file that I had selected when I ran the Renamer Preset.

So, in the Media and Folders View, with that file selected, I had pressed Ctrl+A, then Ctrl+F2 to run the Renamer.

I'll send the Log to the support email...(unfortunately it's not a Debug Log).

[Mario]

1. Move the files to your local SSD and repeat.
If this works fine, the NAS is the issue.

2. Disable your buddy rules and repeat. Works OK? Buddy rules somehow are the problem.

NAS can cause all kinds of problems, especially if you connect via Wi-Fi and not a cable. Accessing files, writing metadata etc. can be 10 to 1000 times slower than doing the same on a local SSD or hard disk.

NAS are designed for video streaming, archiving, to run a database server etc.
For usage patterns caused by IMatch e.g. when writing metadata (pulling the file over the network, creating a temporary copy of the file on the NAS, writing metadata, when successful delete the original file) things may become slow, depending on the NAS hardware. Expensive pro-grade NAS boxes make far less problems than "home" NAS products. Sudden delays, drops in transfer rates, seconds-long delays/stalls when Windows is trying to access the file system on the NAS, which is simulated by the SAMBA software running on the NAS etc. can slow down IMatch severely. Use the monitoring tools in your NAS and Windows task manager to pinpoint the source of the problem.

If this happened suddenly, there might be a problem with the virus checker. Or a recent patch of the Linux system or the SAMBA software. See 1. for a way to test.