Re: Collections>Pins>Red (Improperly Updated)

[Mees Dekker]

This seems to happen to me too. When in the "media and fioders" I marked 8 files with a green PIN. IMatch does not do anything in the background; it is fully "at rest". 

However: in the Collections view the PIN count shows 1, but in the file window, there are clearly 8 files marked with a green PIN. See screenshot.

Switching to another view and/or another collection and/or waiting some 5 minutes makes no difference. The PIN count stays incorrectly at "1". 

After closing and reopening Imatch the count is now correct and is updated correctly when adding/removing a green PIN.

[Mario]

This thread is about two years old. It's usually better to start a fresh thread that to revive a thread zombie.

I've tried to reproduce this. Starting with 20 files in each of the pin collections, I've switched to the M&F View and added 10 files to the red pin collection by clicking the pin. Switching back to the Collections View, the red pin collection updates after about 0.5 seconds, changing from 20 to 30. Correct.

I did the same with blue and red pins, bookmarks and dots. Always worked.

Do you have a filter active in the Filter Panel? Quick Filter Ribbon? File Window search bar?
Collection Filter open in the Filter Panel?
Do the collections refresh when you click into the Collection tree and press <F5>?

The collection counts are re-calculated on demand in the background. If there is an issue that does not trigger the update or the background threads are somehow idle, this would explain what you see. But I currently don't see how this can happen.

[Mees Dekker]

This thread is about two years old. It's usually better to start a fresh thread that to revive a thread zombie.

OK I'll keep that in mind for the next time

Do you have a filter active in the Filter Panel? Quick Filter Ribbon? File Window search bar?
Collection Filter open in the Filter Panel?
Do the collections refresh when you click into the Collection tree and press <F5>?

There are no filters active, nor is the Quick Filter Ribbon or the File Window search in use. Collections do not refresh by clicking in the Collection tree and <F5> is pressed. I tried all that, but only closing and restarting IMatch solves the problem.

[ubacher]

Interesting - just yesterday I encountered the same problem. I had 10 files for which I set Pin Red.
When switching to collections->red pins it showed me all the files but beside the red pin in the index it showed "1".

Since there had been reports about these occasional mismatches I just ignored it.

[Mario]

I could so far not reproduce this. I've tried both small and large databases.
Even when I'm in the Collection View (e.g. on "pin green") and have many collections expanded (dot, pin, rating, label, ...) whenever I change a dot, pin, rating, label, the collection tree control updates after a very short time.

I've looked at the code but could not find something apparently wrong. This is quite mature and old code.

If the tree repaints a collection item, it asks IMatch for the text. IMatch then checks if the collection has the "must update" flag set. If not, IMatch gets the collection count and returns the formatted number.
If the collection needs an update, IMatch marks the node as "pending" and enqueues it for a background thread to update the collection in the background. The node then shows no number but something like "Red (...)" to indicate that the node is currently being updated.

The background thread processes the queue. The next collection is fetched from the queue and then the thread tries to update it. A collection may fail to update for several reasons, including locks or the background update queue being busy.
If the update fails, the collection remains in the queue and the background thread processes the next one.
If the update works, the background thread notifies the tree control that the count for a node has changed and then the tree redraws the item to show the new number.

Yes. There is a lot going on in IMatch, even with simple things like the numbers shown in the Collection tree

While trying to find a failure mode, I've added logging statements to see if the background update mechanism works, if and how often it fails, if failed collections were properly re-enqueued etc. And then I've tried for maybe 30 minutes on two computers to make the problem reported by Mess happen. But I could not. Even running indexing and face recognition in the background only delayed the update, but all nodes updated as soon as the background queue finished.

I've made some optimizations in the code while working with it (refactoring) and maybe, maybe, this fixes the issue on your side (it's more performant now). Without a way to reproduce this, I cannot know. Spending over an hour on this without an actual result is what I'm willing to do at the moment. There are many other things on my list.

If you find steps to reproduce this reliably, let me know.

[Mario]

How to ruin a perfect weekend. And a Monday. And... 

While trying to reproduce the effect reported by Mees, ubacher and at least another user at some point, I could not find a way. It always worked.

Unless I changed the green label of a file to blue and then back.
Suddenly, the "Blue" collection in the Collections tree showed 0 as the number of files, not 5, as it should have.
I've spent Saturday, Sunday and a couple of hours on Monday to find out why that happens.

It also happens only in the release build of IMatch (which is what I ship to customers) and not in the debug build with thousands of extra checks I use when developing. Which makes this a timing issue or, more likely, a race condition of sorts. Which are always tough to figure out and fix.

Collections are truly complex. Similar to categories. They are processed in multiple threads, multiple parts of IMatch access and sometimes modify them at the same time, the user interface wants to know how many files are in a collection and so on.

Today I finally figured out what is causing this effect. But I cannot explain it yet or how to prevent it. After thinking in parallel threads for two days, my brain's kinda mushy and I'll have to put another day or two into this.

Changing the label of a file from green to blue works fine. The metadata of the file is updated, the file is removed from the green collection and added to the blue collection. All work well.
But the reverse operation, changing the label of the file from blue to green (or yellow or red or ...) always causes the count in the collection tree to become 0. But the changes are stored correctly in the database, which means than closing and reopening the database corrects the display issue.

As usual, I will continue to analyze this until I find the true reason and then a way to fix this. How unfortunate that this takes so much time.

[Mario]

Now I know what happens. It's a multi-threading race condition. Darn.
Let's see how to fix this.

[Jingo]

Interesting that it happens only in the specific instance of those color changes... can't imagine there is "extra" code involved when those colors are chosen to swap... will be interesting (from a coding/developer perspective) to hear why this particular condition produced the race condition!

[Mario]

It's not necessarily linked only to this collection type. It's just that, on my computer, I can reproduce it that way. May be something else on another computer.

Labels is one of the "dynamic" collections. It can contain any number of labels, and the labels may change when the user adds new files to the database or sets a label manually via the Metadata Panel, a MD Template or similar.

When IMatch has to recalculate the "Label" root collection, it has to create children for each unique label "text" used in in the database. So it drops and re-creates child label collections dynamically when needed.

When collections have to be recalculated, e.g. you "moving" a file from "Green" to "Blue", IMatch first updates the database tags and then the collections in a way that minimizes database operations. Collections are also not immediately stored in the database, but enqueued and written asynchronously (non-dynamic collections only).

Furthermore., dynamic collections (Label, Annotations, ....) are calculated on demand. If something changes them (e.g a new face is added), IMatch marks the collection as "needs update". That's very fast and good for performance.
Only when something wants to know how many files are in a collection, the "needs update" collections are actually re-calculated. Such a request may happen at the same time when another part of IMatch is updating the same collection. Or the collection is currently written to the database.


Under some yet unknown conditions, two parallel threads appear to modify the same collection at the same time - although I'm sure I have everything protected by a mutex / critical section construct to prevent exactly that. Collections are modified and read by different threads all the time, that's their normal mode of operation.

Since all this happens only in "release" mode, my debugging capabilities are severely limited. And since it happens in parallel code, with varying timing, things are very hard to understand.

Luckily, some of the changes I've made allow me to "see" that the problem exists in the debug build in IMatch.
The counts shown in the Collection View in the debug version of IMatch are always correct (!) but there are some memory leaks reported when IMatch shuts down, related to collections. This only happens when a collection is modified by multiple threads at the same time, causing issues with the memory management. That's a clue I can use. If the memory leak is gone, the problem is gone (assuming the counts are also correct in the release build).

This is truly one of the most irritating problems I've had to work on for a very long time. I will prevail in the end, as always.

[Mario]

Aaand another day gone without results.
I've re-implemented all collection routines to be extra super-duper thread safe for multi-threading environments. Added a ton of new tests. And guess what - it still fails...<really bad swear word>...

DAMN. DAMN²

 It appears I still don't understand the problem. I've found a problem, but not the problem.



The code I've worked on today is from 2018, and worked apparently fine since then. No changes recorded in the version history for many years. Don't fix what ain't broken and all that.

[Darius1968]

I'm the original poster of this thread.  I've experienced this bug going all the way back to IMatch 5!  

[Mario]

Found it!



Geez, this was a truly tough nut to crack.

It turned out that, under very specific combinations of operations and file list layout in a collection, a partial comparison routine for bit vectors (which is the data structure IMatch uses for collections internally) reset a counter value in a specific branch / case, but did not mark the counter as invalid afterwards.
If the collection then was asked for how many files it contains, the answer was zero - even if the collection contained files! And this only happened in the release build (what you use) after the compiler has optimized the code and in specific parallel execution situations.

I was lucky that the number of files (aka bit distribution over ca. 19,000 bits) in my green and blue collections was just right to cause this problem in a reproducible manner. This allowed me to add more and more logging code to figure out what happened and where and when until i was finally able to monitor specific memory regions for unexpected changes. Which finally led to the cause of the problem and solution. Phew!

Now I can go back and what I've planned to do on last Saturday...

[sinus]

Cool!
The reward of really hard work.