Face Manager's Similarity Sort does not seem to work correctly

[Tveloso]

I have noticed that if I sort the Confirmed Faces in Face Manager by Similarity and not Date (in an effort to do a quality check and identify any incorrectly assigned persons), it seems as though the similarity sort might not be working exactly right.  There are indeed many contiguous thumbnails that appear very similar, but sometimes they are "broken up" by others that don't belong in that group. 

For example, a large swatch of green thumbnails in this ScreenShot is broken up by warmer colored thumbnails.  And two thumbnails that are very nearly duplicates of each other (the photos were taken just seconds apart) appear far apart from each other in the grid:



Also, the Sort by file date & time toggle appears to be backwards?  When it's in this position, I would expect the DateTime Sort:



...and in this position, the Similarity Sort:



...but it seems to be the opposite.

[Mario]

I've fixed the sort by issue already. Luckily now I also have a bug report id to link it to.

For the other problem, no idea. This just sorts by the similarity the AI has calculated for the face. Theoretically, less likely faces have a lower similarity and this is what the sort is based on. There can be "jumps" or "breaks", when the AI considers another face more similar. There are many factors involved and the decision the AI makes is not accessible algorithmically. That's just how AIs work. They make mistakes, they hallucinate. I've seen similar effects even with extremely expensive models like Google and others run in their clouds. Somethings the AI "calculates" the wrong person for a face or the similarity calculated is a lot different from what a human would suggest. A few different pixels in the face can make a huge difference. That's just how it is.

I've checked an the sort by distance works correctly - which means that the AI has calculated less distance for the "wrong" image following the comparison image in your screen shot. Nothing I can do about this. Shifting the face annotation by a few pixels in the Viewer can make a huge difference sometimes.

[Tveloso]

Would replacing the Similarity Value returned by the AI, with one that IMatch itself calculates (similar to what's done for newly indexed files), help this at all?...(or would adding such a post-processing step for the Face Thumbnail be too much effort for not a lot of benefit?)

Or, is there perhaps another parameter (the "confidence level"?) that can be sorted by instead?...(or is this the same as the similarity value, and what's already being used?)

I'm just thinking out loud, about a way to maybe aid in performing a quality check on Confirmed Faces...to have any incorrectly assigned ones "stick out" somehow...

[Mario]

You are looking at the faces the IMatch AI has assigned to a person. The distance/similarity/confidence value specifies how confident IMatch is that the face belongs to the person, in the interval 0..1. And this is what this sort uses.

[Tveloso]

Thank you Mario.

How are Manual Face Annotations treated, when the sort is by the distance/similarity/confidence value?  I would expect to see them all together either at the beginning or the end, but it seems like they might be mixed in with the "true faces"?...

[Mario]

The distance for manual face annotations is always 0.

[Tveloso]

This then suggests that when the Similarity Search is used in Face Manager, that all Manual Face Annotations should be listed first, ahead of all the "real faces".  But I'm pretty sure I saw that they were interspersed within the real faces.

I'll check again in the morning to make sure I'm correct about this.

Could it be that the sort by similarity, is within some other attribute?...(that might also explain why blocks of similar fiaces are "broken up" by blocks of others, as shown in the Screenshot in my original post)...

[Mario]

The sort is really simple.
IMatch retrieves all "face records" for the person and either sorts them by the distance value (0..1) or by the time stamp. I never paid much attention to this and this was so far never reported as an issue.

When you have manual face annotations which don't have an AI-assigned confidence (0) and no other sort criteria is used, manual faces will mix with other faces that happen to have a distance value of 0 (0 meaning 'perfect' in this instance).

There are no other "attributes" involved in sorting. If faces have the same distance value or time stamp their sort order is undefined (either can sort first) and irrelevant.

I might give this a think at some time and maybe make this a bit smarter.

[Tveloso]

Thank you Mario

[Mario]

I have made some improvements. See release note #2281 and #2280 in https://www.photools.com/release-notes/

[Tveloso]

Thank you so much Mario