Face recognition inconsistencies

[paloscsaba]

I have just purchased the 2020 license a few days ago and head-jumped into face recognition.

It made me much manual work, because it is hard to tell my children apart from some of their friends. OK.

But I had the impression that I am revisiting the same some 100-s of pictures repeatedly at least 10 times, and already confirmed faces keep tohttps://www.facebook.com/711024088965687/posts/2732963236771752/https://www.facebook.com/711024088965687/posts/2732963236771752/https://www.facebook.com/711024088965687/posts/2732963236771752/ get unconfirmedly assigned to random other people.

So I started Database Diagnostics, restarted again and again, and experienced each time 7 to 79 warnings similar to these:

    Warning: Face [12066] for file [30209] has no annotation object. Face removed.
    Warning: Face [12067] for file [30209] has no annotation object. Face removed.
    Warning: Removing 2 invalid entries from face index.
    Warning: Face region found in file[51937] with invalid face link [11474]. Face link removed.
    Warning: Face region found in file[51961] with invalid face link [11479]. Face link removed.
    Warning: Face region found in file[57487] with invalid face link [12069]. Face link removed.
    Warning: Face region found in file[57851] with invalid face link [12081]. Face link removed.
    Warning: Face region found in file[66386] with invalid face link [2928]. Face link removed.

I supposed, immediately after a diagnostics run that brings such warnings, the next one won't find any problems. But it does. The above warnings are the result of "Start again".

I understand that face recognition is a new feature, and needs time to ripe.
I know this is not a proper bug report. Do you have any suggestions what I should do differently, or what kind of information would help you fix such issues?

[Mario]

Start by explaining in detail which operations you perform.
Every detail and step you perform matters.

Which face-related features do you use?
What you exactly mean with "revisiting". Do you look at the images? Do you assign persons? Reject? Ignore? Modify?
Where? In the Viewer? Face Manager? File Window?
Do you manually create face annotations? Move or resize face annotations? Create persons on-the-fly in the person editor?
Do you copy & paste face annotation? Use the Viewer gallery?

Which operation  causes the warnings in the diagnosis, and how differs your report from the similar bug reports in the bug report board?
The more info we can gather about how to produce these warnings, the better.

See also https://www.photools.com/community/index.php?topic=22.0 and log file

[paloscsaba]

Hi Mario,
thanks for the quick reply.

Actually, I did all of them that you mentioned. The "worst" thing I did was that I also did all these manual changes while recognition was still running in the background.

Revisiting means that I repeatedly went through the unconfirmed faces and processed them using these operations:
- assignment to another person (F2)
- confirm (space)
- reject (r)
- delete (d).

The sad thing was, that already confirmed (several time) faces showed up repeatedly in the unconfirmed list of others.
I don't think I can help tracking down which operation caused what in detail, I won't have much time to experiment in the next weeks. I will report when I get to it and find something.

What I would really love to tackle now is the Database Diagnostics issue. I start the database diagnostics, and receive warnings like above. Without leaving the dialog box, I hit start again, and I get other similar warnings. I did this in series at least 6 times while finally there were no more warnings.

Is this normal, or does it mean that my database is corrupt?

Can it be that face recognition is running in the background during database diagnostics, and the two interfere with each other? I suspect this. I think it would make sense to stop background activities while in the Database Diagnostics dialog box. The whole HMI is hidden, it suggests that the database activities are suspended.

I will now try to separate activities in time, and don't interfere with the system while face recognition is running in the background.

[paloscsaba]

Now I started from a state where there were no warnings in two consecutive diagnostics runs.
I have just processed the unconfirmed faces of 3 people.
- confirmed most
- F2 assigned many
- deleted some

Then I started database diagnosis again and got again warnings. Repeatedly. There was no background activity.

Now, when I reach zero warnings again, I will only do one kind of actions: confirming unconfirmed faces. Let's see.

[Mario]

That's the way.

If you can figure out a series of steps or operations which produce the problem, and I can reproduce this here, the bug is as good as fixed.

[paloscsaba]

This time I have just done a few confirmations in the face manager's unconfirmed faces filter.
I received this warning.

    Warning: Ignored face records with confirmed/train/ignored state or with an entity oid. Corrected.

After this, some of the affected persons again had unconfirmed but correct faces.

[paloscsaba]

I get the same warning when I don't do anything else than assigning unconfirmed faces of one person to an other existing person via F2 edit.
Even if I assign one single face.

[Mario]

Just to make that clear.

In the Viewer (?) you select a face annotation. This annotation has person A assigned, unconfirmed.
You press <F2> to open the Person Selector and pick another person B.
Now you close the viewer? Or move to another image?
And then immediately run a diagnosis?

[paloscsaba]

I continue with another image. Then, after closing the face manager and chacking that the background queue is empty, I run the diagnosis.

[paloscsaba]

Now I also made some deletions in the face manager. There were some faces I never want to name, pressed d, saved the changes, closed face manager.
After doing this for a few persons, I started diagnosis:
    Warning: Face region found in file[52014] with invalid face link [11498]. Face link removed.
    Warning: Face region found in file[52037] with invalid face link [11500]. Face link removed.
    Warning: Face region found in file[52037] with invalid face link [11499]. Face link removed.
    Warning: Face region found in file[52185] with invalid face link [11569]. Face link removed.
    Warning: Face region found in file[52186] with invalid face link [12731]. Face link removed.
    Warning: Face region found in file[52186] with invalid face link [12732]. Face link removed.
    Warning: Face region found in file[52186] with invalid face link [12733]. Face link removed.
    Warning: Face region found in file[52197] with invalid face link [11574]. Face link removed.

Now I will iterate diagnosis till empty, and then try this with only one deleted face.

[paloscsaba]

Just one single face deleted (d, save changes, close):
    Warning: Ignored face records with confirmed/train/ignored state or with an entity oid. Corrected.

Some faces ignored (i,i,i,i,i,i save changes, close):
    Warning: Ignored face records with confirmed/train/ignored state or with an entity oid. Corrected.
    Warning: Face region found in file[61401] with invalid face link [12267]. Face link removed.

Shall I start over from my Imatch2019 database from a week ago again?
I invested lots of time into defining persons, but losing time with not convergent correction iterations is maybe worse.

[Mario]

continue with another image. Then, after closing the face manager and chacking that the background queue is empty, I run the diagnosis.

I don't follow. You did not answer my questions (see my post above).
Do you work in the Viewer? Because you mention the Face Manager. That's a very different thing.
It is unclear to me which operations you perform, in which sequence and with which of the features.
Please do a:

1.
2.
3.

for me.

[paloscsaba]

No, I don't work in the viewer.

1, I open People and check each persion.
2, If a person has unconfirmed faces, I open the face manager for that person (smiley icon), and try to reduce the unconfirmed faces to zero with the following procedure:
3, I go through each face (either in the thumbnails or opened per double-click) and do one of these for each one:
- confirm
- reject
- delete
- edit (either selecting an existing person or adding a new one)
4) after going through each unconfirmed face in the face manager, I save the changes, close the face manager, and proceed to the next person that has unconfirmed faces.

What I noticed:
- I see again and again the same some hundred faces as unconfirmed.
- It even happened that a person (that I did not touch) lose all its confirmed faces although these steps would not allow to lose any.

That's why I started to check the diagnostics. I found that almost any single action I execute creates new warnings.

Now I migrated my 2019 database again to have a clean start and less bulk actions. It already contained some manual face annotations, but just a few.
I first started face recognition for a small folder only, and executed the above steps almost one by one.
I keep getting some such warnings as described above.

Can you propose another process to go through years and thousands of pictures? Shall I use the viewer?

Actually, I can imagine that you won't be able to reproduce the problem, else people would complain. Possibly there is something corrupt in my database.
Does it help if I send you the database? I suppose, without the pictures it would not help at all.

[happysnapper]

This appears to be exactly the same problem as I reported here.
https://www.photools.com/community/index.php?topic=10459.0

[Mario]

Actually, I can imagine that you won't be able to reproduce the problem, else people would complain. Possibly there is something corrupt in my database.
Does it help if I send you the database? I suppose, without the pictures it would not help at all.

There seem to be a few users who experience the same issue.
I tried the whole morning with various databases, two computers, the Viewer, Face Manager and whatnot to reproduce this behavior. Added faces, moved, copied, assigned, confirmed, un-confirmed, deleted, trained, ...
Running DBDiag after each step.
This problem did not happen once!

If you can upload your database somewhere and send me a link to support email address, maybe I can figure this out.
If this does indeed depend on the database, and not on something that's 'local' to your PC...

[paloscsaba]

Hi Mario,

thanks for caring. I did not have much time to chase the problem in the meantime but I found out a few details.

1) I have found a slow but safe way that allows me to utilize the face recognition without any data loss and diagnostic warnings. For this I have to avoid executing face recognition on multiple images at once. I go through the images one by one, and execute face recognition on each picture separately.

2) The problem is not necessarily in the face manager. I already get diagnostic warnings when I execute face recognition on a folder full of pictures. The warnings seem to be linked to artifacts I call dead faces. These are faces that the face recognition positions correctly, but they remain crossed out by green diagonals. These I cannot assign to people at all, they remain the same after trying so. When I execute face recognition in the viewer for the single picture, then I get new face rectangles for the same faces, the position of the rectangles almost the same. When it can also identify the person, suddenly also the dead face wakes up and gets assigned to the same person.

I will try to reproduce this and upload the corrupt database.

[Mario]

A crossed out face annotation means that the face has been ignored. Select the annotation and press <I> to un-ignore the face.
The face recog has been designed to process thousands of files in a go. Only the user can ignore faces (and produce the X).

See also this thread for a related discussion: https://www.photools.com/community/index.php?topic=10459.0

[paloscsaba]

I am terribly sorry, I was wrong. It is not the diagonally striked out face rectangle, it is the ringed one.

The "Detect faces and create Face Annotations" running on a complete folder creates for me in some photos face annotations that have the small ring at the top and a question mark.
I thought this is impossible, because the face recognition will only put a face annotation where it can identify a face. Actually, at these places there is a real face, and if I run the detection for that single image, it really puts a correct face annotation there.

These dead ringed face annotations created automatically cannot be assigned to a person at all. As long as I have such, I get warnings in the diagnostic.
I am not sure that this is the origin of all my problems, but if we can catch and you can fix this, my life will be already easier, and I can test whether other issues exist.

I will provide you the database before and after such a detection, and also the pictures that I ran the detection on, but I need more time to prepare this. Please be patient.

[Mario]

The small ring means "manual" face annotation. A face annotation for which IMatch could not detect a face.
Usually these face annotations are created by the user, when IMatch cannot find a face. You can assign persons to manual face annotations.

Or, maybe your files already contain face rectangles in the XMP record, created by another software (not uncommon for Apple devices, for example).
IMatch by default imports these face rectangles and converts them into face annotations. It then runs a face detection to see if it can identify a face within the rectangle. If not, it creates a manual face annotation (with the circle at the top). See Working with XMP Face Regions

To check: Look at one of your files in the ExifTool Command Processor. Use the "List Metadata" preset and then search for region with the search box at the bottom.


IMatch can usually always assign persons to face annotations. Manual or not.
What happens if you select the annotation in the Viewer and press <F2>? Does the Person selector open? If cannot make suggestions (no face data), but you can search for persons or pick a person from the recently used  list).

[paloscsaba]

Yes this is the strange part, that iMatch definitely cannot assign a person to these face annotations.
If I press F2, the person selector opens. There is no suggestion on the left side. I select the person from the recent list. The person selector closes, but the face annotation remains unassigned.

[Mario]

Did you make the test for XMP regions I've explained above.
Do your files contain XMP face regions created by another software?

What happens if you just move one of these face annotations in the Viewer and commit the change with the blue button?
Can you then assign a person with F2?

Database diagnosis reports no problems?


Are you using the latest version of IMatch?
Please switch IMatch to debug logging and attach a log file from a session where you've tried this.
See log file for more information.

[paloscsaba]

Just a fast reply, did not have time now to learn logging.

Moving around the face annotations helps, they either become ignored or assigned or become assignable after confirming the new position.

There were absolutely no regions in the metadata. I did not use any other software, and it seems, Olympus does not put regions in the images either.

Database diagnosis mostly reports the above described warnings when I have such dead faces, but it already happened once that there were no warnings although the faces were corrupt (a little bit differently, I will explain later).

Please give me a week, I will deliver you a repro.

[paloscsaba]

I have just sent an email to support@photools.com with my database and step-by-step reproduction of an inconsistent state. I hope it helps.

[Mario]

My inbox is flooded with images containing 'problem metadata' and databases sent by other users for various issues. It can take a while before I can download and test yet another database.

[paloscsaba]

No problem, it is not urgent. I do have a workaround.
If I can find more details, e.g. logs that could help, I will of course share them with you. But I won't have much time to work on this in the next weeks either.

[brennock]

My experience of using face recognition in Imatch - extremely complicated and results inferior to similar software in Photoshop Elements organizer, ACDSee and Cyberlink Photo Director.  I no longer use it.
brennock

[Mario]

My experience of using face recognition in Imatch - extremely complicated and results inferior to similar software in Photoshop Elements organizer, ACDSee and Cyberlink Photo Director.  I no longer use it.
brennock

Interesting. So far the feedback was that the face recognition technology in IMatch is very good, and the associated feature set beats what is available in other software.

Which parts of the face recognition do you think are complicated?

The initial face detection via the File Window?
Assigning persons to faces in the Viewer?
The Face Manager to manage faces in bulk?
The People View?
The People Help topic?

[paloscsaba]

Just to make clear, I (the OP) am really happy with the face recognition features. I find them really thought-through and easy to use.
I suspect that the issues I intended to discuss in this thread originate from some database error in my old database, or a minor bug.
I (as the OP) don't agree with the critic above and would be happy if brennock could post his critic in a separate thread.

[Mario]

@paloscsaba

I know. I've just replied to brennock and quoted his/her post.

I'm always open to critique and suggestions. This is how IMatch is formed and how the development of future versions is steered.
I'm not sure I can improve the AI detection and recognition rate at this time. For the standard recognition benchmarks it's about 98%, which is very good and top-end.

But If brennock tells us where (s)he has problems or what he does not like, we all can learn and I can maybe improve IMatch for all users. Which is a good thing, of course.