Link faces to Categories

[nacho02]

Hello,

I'm a bit confused because I thought I had this running some time ago... here is the question. I have people whose faces have been detected and I have in the person editor assigned a category to the person. So far so good. The catagory is color-coded, so I would expect to run the face recognition and once the face has been assigned to a person, see a) the image assigned to the category and b) see the color in the viewer. 

It has been some time since I last played with faces, but I thought I had this working. Not anymore, it seems.

The images are not assigned to the category and (hence) no color-coding is visible.

Can someone help?
Thanks!  

[Mario]

Works here.

I have a person which assigns the color-coded categories A (red) and B (blue).

When I assign the person to a face in an image, the image is added to categories A and B and the color bar in File Windows shows the color and also the collection gadget in the Viewer.

When I link the person to a file, the file is also added to categories A and B and the color-code shows in File Windows and in the Viewer.

[nacho02]

OK... then I need to somehow test this, because it doesn't work for me.

Thanks Mario! 

[Mario]

Since no other user reported something similar, do the usual things: check that the categories are assigned, if the are color-coded, if there are warnings logged to the IMatch log file after you assigned a person to a face or linked a person to a file etc. This might provide some clues.

[Tveloso]

The catagory is color-coded, so I would expect to run the face recognition and once the face has been assigned to a person, see a) the image assigned to the category and b) see the color in the viewer.

Just to be sure...those assignments must also be confirmed ones.  If after running Face Detection, the resulting Face Annotations remain unconfirmed, then any Category/Keyword settings in Person Editor, will not actually be applied yet.

[Mario]

@Tveloso: good point.

[nacho02]

Mario. I think the issue has more to do with writing the metadata to the files. I've created a short video in 720px. Can I post it here?

Yes, the confirmation is of course something to take care of. Unfortunately, even with confirmed faces it did not work..

[Mario]

Are the categories assigned when you assign a person to a face?
Or link a person to a file?
What happens during write-back?
Are the files removed from the categories?
Are the person assignments removed?
Which metadata settings do you use for persons and faces?
Which file format do you process?
Do you use metadata templates that are triggered during file import?
Do you have any category-related options set under Edit > Preferences > Indexing?
If it worked before, what has changed?
More details are helpful. A video is usually not, because it does not answer all these questions.

I can write back the files with the A and B categories assigned and nothing changes.
The face annotations are written as XMP face regions into the JPG XMP record and the XMP sidecar file for the RAW file I tested.
Category assignments are not affected by write-back.

[nacho02]

Are the categories assigned when you assign a person to a face?


BTW. Using version 2023.14.2.

When unknown face has been detected and I manually assign it, yes. Category gets assigned. I can see it under Categories / current for the image.

If I select an image and run face detection, even though it works, it will not assign categories. See log file.


At 16:50 I change to debug and run face recognition on an image with three people. One is auto detected and confirmed. One is wrongly detected and one os detected but not confirmed. No category assignment. At 16:57 I manually assign persons to faces. I see two out of three categories. these have color coding. At 17:01 I write the metadata. Tag PersonInImage. No visual change. 2 / 3 catagories assigned.


Or link a person to a file?

Not done that yet.

What happens during write-back?

Nothing much, tbh. Category stays assigned but no color.

Are the files removed from the categories?

No.

Are the person assignments removed?

No

Which metadata settings do you use for persons and faces?

Good question. Under preferences Metadata2 I export face annotations to XMP, but I don't see much more that might answer this question, sry.

Which file format do you process? In this case JPG

Do you use metadata templates that are triggered during file import?

No

Do you have any category-related options set under Edit > Preferences > Indexing?

No

If it worked before, what has changed?

Can't say.. I'ts been a while since I last used faces.

More details are helpful. A video is usually not, because it does not answer all these questions.

I get it. No prob.

I can write back the files with the A and B categories assigned and nothing changes.

Same here. Nothing changes. 

The face annotations are written as XMP face regions into the JPG XMP record and the XMP sidecar file for the RAW file I tested.
Category assignments are not affected by write-back.

[Mario]

If I select an image and run face detection, even though it works, it will not assign categories. See log file.

Is the person assigned and confirmed automatically by IMatch? Because if the assignment is not confirmed, the categories are not assigned.

When you assign a person to a face, you will probably also confirm it. Then the categories are assigned and color-coding works. When the person is not confirmed, the categories are not added.

Nothing much, tbh. Category stays assigned but no color.

This is unlikely. When the category is still assigned, IMatch will show the color in color base in File Windows (unless you have set the option for the category to not show in File Windows).
Do you experience problems with other color-coded categories.

[nacho02]

Hi Mario,

I let IMATCH assign and confirm the person, yes. And still, there is no category assignment nor color-coding (because of the lack of category).

Writing the Metadata does not change this.

No, I have no other issues with color-coding.

There are instances where in cases where there are several persons on the image and IMATCH recognizes them, ONE of the persons will be properly categorized, but the other two. Feels like there is no consistency in the results, for some reason.

I see this is one of those things that is like opening a can of worms... I'll stick to assigning categories manually.

Thanks Mario!

[Mario]

I let IMATCH assign and confirm the person, yes. And still, there is no category assignment nor color-coding (because of the lack of category).

Show me the person editor for the person you have tried and how you have configured the categories you use for that person.
I have tested this with existing and even some new categories and it works fine. As soon as the assignment is confirmed, the person is assigned to the categories.

I've tried your case with images showing multiple persons, too.
After running FR, the correct persons were assigned and confirmed (because IMatch was sure). Two of the persons have color-coded categories A,B and C and the image was assigned to all three categories.

I don't see any issue with that, and no other user has reported the same problem so far.

[nacho02]

Hi Mario, I'll show you my settings later. 
I'm not thinking / saying this is any general issue of the software, rather with my database / my set-up.

At some point it might make sense to actually move away from the old DB (it's been essentially the same one for like 16 years) and start fresh, but that is prabably a different story alltogether.

BR,
Ignacio

[Mario]

rather with my database / my set-up.

That's why every detail matters.

At some point it might make sense to actually move away from the old DB (it's been essentially the same one for like 16 years) and start fresh, but that is prabably a different story alltogether.

I don't think this is necessary. My own database (one million files) and 3 databases I work with every day have been created with IMatch 3 back in 2015 or so. All work fine.

[nacho02]

Here are the settings you asked for and an example of the issue in 3 mins.

https://filedn.eu/lsxR0VpMJ7ekwBu6zjAJOh0/IMATCH%20Faces%20Ignacio.mp4
BR,
Ignacio

[Mario]

I've looked at the video.
At 1:30 you run a FR in the Viewer. Both persons are detected a person is assigned to each face.
The categories of both persons are assigned (blue and salmon).
This is the behavior we expect.

Then you select 2 files and run face recognition. Faces are found and Aulde is assigned as a person. But the category is apparently not added. Also not for the young man.
When you assign the person manually to the young man and correct Andra to Aulde in the other image, the correct categories are assigned again.

Now we know what we did before. It does not (always) work for your database on your computer. We still don't know what is causing this. There are no warnings logged to the log file in debug logging mode (Help > Support) I assume?

This was not reported before and despite trying to produce exactly the same situation here (by running FR on images with multiple persons, making IMatch not recognize one person but the other etc. categories assigned to persons were always assigned. I've tried to reproduce this with two databases on my PC and notebook.

Please upload this test database to your cloud space and send an email with a download link (include a link to this topic) to support email address. If this behavior depends on the database and not your system, I can hopefully reproduce it here and see what's causing this.

[axel.hennig]

I was able to reproduce this.

• Started with a new test-DB.
• Checked that in "Preferences > Application > Face Recognition" my settings are the same as in Ignacios settings (video at 01:09).
• Added 7 images (all very similar and showing three different people).
• "Trained" two out of this seven images and created three people (Adam, Bill, Chris) and added for each of this three people a category (within the Person Editor) to assign this person to. The categories were: "Persons|Adam", "Persons|Bill" and "Persons|Chris".
• Run "Detect faces and create Face Annotations" on the other five images.
• All persons have been found and automatically confirmed, but none of them has been assigend to a category.

[axel.hennig]

Debug-log attached.

[Mario]

Do the categories you have created exist in the Category View?

[axel.hennig]

Yes, they exist in the Category View, but just because I've created them there before I did run "Detect faces and create Face Annotations".

I think my steps were (for a second test-DB with other images, i.e. not the 7 images mentioned above):

• Import 12 images (4x3 images, where the "3" images are exactly the same, i.e. 3 times Arie, 3 times Bill,...)
• Create 4 categories (Arie, Bill, Chris and Denise)
• Run "Detect faces and create Face Annotations" on one Arie image, one Bill image, one Chris image and one Denise image.
• After this "Detect faces and create Face Annotations" run, all 4 images are opened in the Viewer and F2 is pressed when the rectangle with the found face was focused. A name was entered and an "assign to Category" category.
• All 4 images are afterwards within one of the above mentioned categories.
• Second run of "Detect faces and create Face Annotations" on the remaining 8 images.
• Each face is correctly found and automatically confirmed by IMatch (settings).
• All remaining 8 images are not assigend to any category.

[nacho02]

I'll upload the DB to my space and post here. Thanks! @Mario and @Axel!

And again, I'm pretty sure this is not some sw malfunction per se, just with my setting / DB.

BR, 

[axel.hennig]

If it is just within your DB my question would be: Why am I able to reproduce it?

[Mario]

I'll look into it. When Axel can repro it, I should be able to do it too.

[Mario]

This took quite some time, but I think I've found it

This was hidden deep in the multi-threaded guts of background face recognition.
To make things fast, IMatch injects custom functions into SQLite that SQLite can use when performing certain operations - like "reclustering" persons to assign persons to faces.

If a person is found and the "distance" in vector space is lower than the threahold for automatic assignment configured by the user (certainty), IMatch links the person to the face and then checks if the person has keywords and or categories to process and if there are keywords or categories missing that need to be added to the file (the file may already have some/all of the keywords/categories) and marking a file as pending when it was not really modified would be sub-optimal.

When doing all the checking and keyword processing, a configuration error caused the category assignment part to skip assigning the categories. And there you have the source of the problem!

Another, related issue, could cause category assignments to be skipped in interactive mode (e.g. working in the Viewer).
Unnecessary to say that this all quite complex and to keep the code maintainable, there are many small routines which do one job, and they are called as needed.

One routine did the keyword and category assignments. But only if the person assigned to the file has actually changed.
To determine this, it uses the IMatch "face index" to check if the entity id is different or there was no entity assigned before. And only in that case the keywords and categories were assigned

In some particular situations and order of operations, the face index was updated with the new person (entity) before the function which does the keyword/category assignments was called. And it then determined that the entity has not changed and so no keywords and categories were assigned in that case.

I've had to change quite a bit of code on several layers to communicate the fact that, although the entity has not changed, keywords and categories have to be processed anyway. Jet another "special case" to figure out and deal with.

I could have made all of this simpler. But I thought the ability to assign keywords and categories based on the persons assigned to faces would be useful. And it is! But all these features add complexity and a change made two years after can cause unwanted or hard to detect side-effects down the chain.

I'll do a bit more testing, but I think I have test-cases for all combinations of person features already.
 

[axel.hennig]

Always impressive how fast you are in fixing.

Thanks.

[Tveloso]

I would occasionally see a few instances of this warning:

Code Select Expand

Warning: Missing keywords for entity [N] for file [N]. Fixed....in the Database Diagnosis Log, and I imagine that it was related to this.

[Mario]

Always impressive how fast you are in fixing.

Thanks.

I process bugs always first. Working on "new stuff" while bugs pile up in the queue has ruined many good projects.

[Mario]

I would occasionally see a few instances of this warning:

Code Select Expand

Warning: Missing keywords for entity [N] for file [N]. Fixed....in the Database Diagnosis Log, and I imagine that it was related to this.

Could very well be. A bit of randomness was involved, even some computer-performance-dependent factors. Tricky one that one was,

[philburton]

And debugging can be real challenge sometimes, and sometimes bugs sneak past pre-release testing,  That's life in the world of non-trivial software.

[nacho02]

Welll.... what can I say? Wow for the explanation what was happening...

Happy to have helped debug a bit with this issue of mine.

Thanks to Mario once more for the extraordinary customer support! and looking forward to the next IMATCH Version!