Diagnosis log warning formatting

[cg]

I'm wondering if would be possible to change the formatting of the "has no annotation object" warning to put the File # first, to match the formatting of the "mismatch in face/anno" warning to be consistent:

Warning: File [123452] face [121287] has a mismatch in face/anno sort order.
Warning: Face [121289] for file [140400] has no annotation object. Face removed.

So the second warning would be something like:

Warning: File [140400] face [121289] has no annotation object. Face removed.

I sometimes search the file id after I get these warnings, and consistent formatting would be helpful for selecting the File #.

Thank you!

[Mario]

You should usually never see these warnings, since they indicate a problem in the database.
Can you explain after which series of operations this happens?

[cg]

I see these warnings almost every time I run a diagnosis. It's been that way since I've been using iMatch. Especially the face/anno sort order. In general, I think they appear after I've been tagging and working with people/faces. My suspicion is it has to do with creating manual face annotations. It may also have to do with linked people.

[Mario]

None of these warnings should happen.
How do you reorder faces?
How do you create face annotations.
A face without a corresponding annotation object lives in limbo. It won't show in the Viewer, but there is still a record for it in the database. Not good.

I would need more details on how to reproduce this.
I reorder faces, use manual links etc. and I don't see this warning. Which is not too perplexing because I'm the developer and I do things in a certain way. If you do things differently, something may fail. We just need to know what you do, in which order etc.

1. Do one thing with faces.
2. Run diagnosis.
3. Diagnosis runs without warnings?
  Yes: go to 1.
  No: use Help menu > Copy Logfile to make a snapshot. Use a informative file name.
4. Go to 1 until bored or no new warnings show up ion the log. ZIP and attach the logs or send (with a link to this thread) to support email address.

When I can reproduce it, I can fix it.

[Tveloso]

I see the "Face...has no annotation object. Face removed." Warning periodically also.

A Diagnosis run today, turned up 20 such warnings.  And when I went to review the files, to see if I could remember any commonality in how I had added the Face Annotations to them, I noticed that there was an incorrect person assigned in the first file I looked at, so I corrected that assignment.  Then on the next file, another incorrect person assignment was there (the Person I had  corrected in File-1 was now the incorrect label for a different face in File-2).  when I changed that assignment (in File-2), this also changed the assignment back in File-1.  In fact, there are several files that are "linked" like this.

It's as though several Face Annotations (for different people), across several files, all point to the same Face in the IMatch Database - and when the Person assignment on any one of them is changed, all of them change to that same person.

This behavior survives an IMatch Restart, and I can reproduce it at will.  Mario, I imagine that If I were to delete and re-add those "linked" Face Annotations, the issue will be corrected, but I'll keep this state for a while, in case you would like me to provide any other details on this.

I'll send a few more details and a Debug Log to the support email...

[Mario]

It's as though several Face Annotations (for different people), across several files, all point to the same Face in the IMatch Database

This cannot happen since part of the face record in the database is the unique file id. Faces are stored per-file.

When you assign a person to a face, or change the assigned person, all unconfirmed faces are re-evaluated to see if your assignment changes anything regarding the suggested person for the unconfirmed faces.

I noticed that there was an incorrect person assigned in the first file I

Was the person assigned and confirmed or unconfirmed (suggestion)?

hen on the next file, another incorrect person assignment was there (the Person I had  corrected in File-1 was now the incorrect label for a different face in File-2). 

So after you've changed the person in file 1, IMatch re-ran face assignments in the background and considered the person you had assigned a better match for a yet unconfirmed face?

when I changed that assignment (in File-2), this also changed the assignment back in File-1. 

IMatch (normally) does not re-assign persons to confirmed faces. At least it should not.
Did you duplicate the files in the database at some point? Did you copy & paste face annotations? Do you use the annotation gallery to store persons? How do you use face recognition (automatic, manually triggered)? Do you confirm faces via File Window commands or only in the Viewer?

As it happens I yesterday ran some tests, which includes an hour of adding persons, assigning persons to faces, correcting false assignments, picking good training faces for each person. I ran the database diagnosis multiple times during that hour, without issues.

The "Face has no annotation object" means that there is a "face record" for a given file, but the annotation container for that file has no annotation object referencing that face.

The Viewer supports many vector annotations, from arrows to stickers to text. Face annotations are basically a vector rectangle with a link to a face record in the database. The face may be linked to a person.

When you get this warning, there are faces in the database which don't have a face annotation object in a file annotation container. Reasons for this could be that a file (and it's annotation container) is deleted, but not the face records linked to that file. The database system should not allow that due to referential integrity rules IMatch employs.
Another reason would be the user deleting a face annotation, but IMatch fails to delete the face record for some reason.

In your workflow description above, you just change person assignments, which does not modify the annotation objects at all. Just the face record is updated with a new person id, and a background recluster is queued to re-check unconfirmed faces.

[Tveloso]

I noticed that there was an incorrect person assigned in the first file I

Was the person assigned and confirmed or unconfirmed (suggestion)?

These were all confirmed assignments.  And so far, it seems that this is happening only with Manual Face Annotations.

when I changed that assignment (in File-2), this also changed the assignment back in File-1.

IMatch (normally) does not re-assign persons to confirmed faces. At least it should not.
Did you duplicate the files in the database at some point? Did you copy & paste face annotations? Do you use the annotation gallery to store persons? How do you use face recognition (automatic, manually triggered)? Do you confirm faces via File Window commands or only in the Viewer?

No duplication of the files was done, but copy & Paste if Annotations between files, and adding them from the User Gallery was done.  The general approach for the Face Recognition phase of my workflow is:

1.) Manually trigger Face Recognition on a batch of files (Ctrl+M,F)
2.) Use Face Manager to "batch confirm" Faces
3.) Use the Viewer to add Manual Face Annotations for those not detected (sometimes this results in a real face)

Another reason would be the user deleting a face annotation, but IMatch fails to delete the face record for some reason.

I know that in at least one of the 20 Files called out with that warning, I did delete some Face Annotations, after having incorrectly pasted them.  But that probably happened in only one (maybe two) of those files.

The situation is this, while working in the Viewer:
I have added one or more manual Faces to a file that is the first of a series of photos of the same people.  Then I move to the next File to see which faces I'll need there (sometimes they are not the same ones that were missing in the prior file), and back to that prior file to pick up the needed Faces (click the first face, and Shift-click any others), then forward again to the new file, and Paste, and re-position the pasted faces.  While doing this, I sometimes accidentally press Ctrl+V back in a "completed file", when I meant to press Ctrl+C there.  So I usually do a Ctrl+Z to undo that incorrect paste operation (but sometimes also just delete the incorrectly pasted Annotations).

And I might also perform a "straight delete" in a "new File" (because I'm unhappy with how I have manipulated the pasted Face Annotation).  I like to keep the face Annotations square, so I "always" hold the Shift Key when re-sizing them.  When I forget to do that, I usually wind up with a terribly oblong rectangle.  I know that I can simply continue to resize it back to a square-ish shape...but knowing that I likely haven't made a perfect square that way, I instead delete it, and either add the face anew, from the User Gallery, or Paste again, if it was only that one face in the clipboard (I know, I know, a little nutty there).

This "linked Faces" condition among the 20 Files called out in the Diagnostics, includes more than just the two files mentioned here.  I'll check to see if it's actually all of them, and will report back with the actual OIDs of the Files and Faces involved.

[Tveloso]

Mario, it looks like the Face records that the Diagnostics reported as being deleted, might actually be needed in some cases?

Going through the group of 20 Files that the Diagnostics gave that warning about, I found that the first 11 had that "entangled state" for one of the Manual Face Annotations they contained, but File #12 did not.

I set up these variables in the VarToy App (an awesome tool!):

Code Select Expand

File.OID . . . . . . . . . . . . . . . . . : {File.OID|cast:int}
File.Annotations . . . . . . . . . . . . . : {File.Annotations}
File.Faces.OID . . . . . . . . . . . . . . : {File.Faces.OID}
Count of File.Faces.OID  . . . . . . . . . : {File.Faces.OID|count:true}
Count of File.Persons.OID  . . . . . . . . : {File.Persons.OID|count:true}
File.Faces.OID . . . . . . . . . . . . . . : {File.Faces.OID}
...and for Files 2 and 3 (that each contain 4 Face Annotaions - one of which is entangled), only three faces are being reported:

Code Select Expand

File.OID . . . . . . . . . . . . . . . . . : 135020
File.Annotations . . . . . . . . . . . . . : Collections|Annotations|Region|People
File.Faces.OID . . . . . . . . . . . . . . : 106783;106784;106785
Count of File.Faces.OID  . . . . . . . . . : 3
Count of File.Persons.OID  . . . . . . . . : 3
File.Faces.OID . . . . . . . . . . . . . . : 106783;106784;106785
For File #4 which contains two Annotations (with one entangled), only face is being reported:

Code Select Expand

File.OID . . . . . . . . . . . . . . . . . : 135032
File.Annotations . . . . . . . . . . . . . : Collections|Annotations|Region|People
File.Faces.OID . . . . . . . . . . . . . . : 106546
Count of File.Faces.OID  . . . . . . . . . : 1
Count of File.Persons.OID  . . . . . . . . : 1
File.Faces.OID . . . . . . . . . . . . . . : 106546
But for File #12, which contains 4 Face Annotations (3 manual, and 1 real face), and does not appear to have any entangled Faces, all four Faces (and Persons) are being reported:

Code Select Expand

File.OID . . . . . . . . . . . . . . . . . : 135226
File.Annotations . . . . . . . . . . . . . : Collections|Annotations|Region|People
File.Faces.OID . . . . . . . . . . . . . . : 106708;107113;107114;107115
Count of File.Faces.OID  . . . . . . . . . : 4
Count of File.Persons.OID  . . . . . . . . : 4
File.Faces.OID . . . . . . . . . . . . . . : 106708;107113;107114;107115
so it seems that for this file, the face that the Diagnostics reported as being deleted, does not appear to be creating any problems.

Could it be that precisely because there is a "true missing face" now in those prior files, the Viewer no longer sees them as Face Annotations at all?...(and is treating them as another type - the same type? - and the Label on the Annotation is not a Person Assignment at all?...and that's why when the assignment is changed on one, the "label" on the Annotations in the other files also change?)...

I have added these 20 files to a Category called Spooky Action ( ), and I'll leave that entangled state in place for a while, in case you would like for me to check anything else about them.

[Tveloso]

Just to clarify, there are only 14 files involved here, not 20.

The 20 instances of that warning:

Code Select Expand

Warning: File [134925] face [106782] has no annotation object. Face removed.
Warning: File [135020] face [106535] has no annotation object. Face removed.
Warning: File [135028] face [106542] has no annotation object. Face removed.
Warning: File [135032] face [106545] has no annotation object. Face removed.
Warning: File [135084] face [106551] has no annotation object. Face removed.
Warning: File [135086] face [106695] has no annotation object. Face removed.
Warning: File [135102] face [106554] has no annotation object. Face removed.
Warning: File [135112] face [106699] has no annotation object. Face removed.
Warning: File [135118] face [106705] has no annotation object. Face removed.
Warning: File [135122] face [106562] has no annotation object. Face removed.
Warning: File [135132] face [106563] has no annotation object. Face removed.
Warning: File [135226] face [107111] has no annotation object. Face removed.
Warning: File [135226] face [107112] has no annotation object. Face removed.
Warning: File [135292] face [107192] has no annotation object. Face removed.
Warning: File [135292] face [107193] has no annotation object. Face removed.
Warning: File [135292] face [107194] has no annotation object. Face removed.
Warning: File [135612] face [107797] has no annotation object. Face removed.
Warning: File [135612] face [107798] has no annotation object. Face removed.
Warning: File [135612] face [107799] has no annotation object. Face removed.
Warning: File [135612] face [107800] has no annotation object. Face removed.
...refer to only 14 distinct files:

Code Select Expand

134925  135112
135020  135118
135028  135122
135032  135132
135084  135226
135086  135292
135102  135612

[hluxem]

None of these warnings should happen.

Unfortunately, these things happen. I think at some point in time it was asked to assign the files to a temporary category when running the diagnosis. That would make it much easier to identify which files are involved and what was done before it happened. I agree not necessary if it would not happen, but it does. I have come to ignore the warning as it's too much effort to hunt for the small number of files. I have not seen anything which would allow me to reproduce the problem.

I just run a diagnosis after working on the database for some days.126 warnings, most of them were for missing GPS flags on files, 2 face records without thumbnail (assigned to category, nice!) and 24 of the following:

Anno Face region found in file [455516] with invalid face link [113343]. Face link removed

I checked 2 of these files, they show annotations with a question mark. It looks like a manual face annotation without the blue checkmark. These annotations had persons assigned before. I tried to reassign a person, the person selection box opens, and I can select a person. When the box closes the question mark is still there, I cannot assign a person to the annotation. Rescan with "Reload Metadata" does not change anything, rescan "Force Update" shows the annotation with the person assigned person. After the force update the annotation reacts normal and I can assign different persons.
If there is anything I can do to help track these things down, please let me know.

[Tveloso]

I just wanted to add a small update.  This morning, I ran a Database Diagnostic that turned up one instance of this warning:

Code Select Expand

Warning: Anno Face region found in file [135016] with invalid face link [106780]. Face link removed.
So a slightly different message from the ones I was seeing previously. 

The File in question here did not contain a Face Link, but a real Face Annotation, and I don't recall having done anything with that file recently...(almost certainly not between when I ran the last Diagnostics, and this one).

When I checked this file in the Viewer, the Annotation was still there, but now had the question mark beneath it (instead of the person label), and a diagonal line through the Annotation box.

I pressed F6, expecting the Viewer to put up the dialog that includes the option to remove all Faces and run Face Recognition again, but it instead just added a new Face Annotation, that looked just like the existing one (a question mark label, and a diagonal line).  So I deleted both Face Annotations, and placed a new Annotation from the User Gallery.  When I placed it over the face, it turned into a Real Face (so the Viewer did then create a Face Fingerprint).

Hopefully this behavior gives some clue...

I wonder if perhaps the Face Link feature, might somehow be "colliding with" real Face Annotations?...(since again, it was a real face in this file, and not a Face Link, as referred to in the Warning message)

[Mario]

"Real" face annotations contain a link to a face in the database.
When you see this message, the diagnosis has determined that the annotation object links to a non-existing face and then resolved that by setting the face id to zero for the annotation.

As I wrote above, something like this should not happen. IMatch should never remove a face without removing the corresponding face annotation. Actually, the database system uses referential integrity rules to prevent exactly this from happening.
But, somehow, the error shows up.

Maybe it's a bug in the annoation handling / face handling in IMatch. Maybe it's a bug in the diagnosis routine. Both possible.
I was so far unable to reproduce this. But I now have your database here with some of these errors so I can at least verify that the diagnosis is correct.

Somehow, your workflow makes this happen frequently. We still have no idea which of the operations you perform causes this error. Basically, you should do that you always do with faces, but run a diagnosis after each step (Tip: hold down <Ctrl> when clicking OK in the diagnosis dialog for much faster execution).

When you do an operation and the diagnosis afterwards shows this error, we'll know what's causing this issue. And this is the first step for me to reproduce and hopefully fix this.

Same for @hluxem, who also seems to encounter this particular problem more frequently.

[Mario]

Finally, I've figured this one out. Took only a few hours.

Re-reading all the info in this thread in order to find a way to reproduce this and looking at the sample database Tveloso has provided.

First, I double-checked that the problem is not the diagnosis itself (e.g. wrongly determining an annotation object has no face or something like that). But the sample database allowed me to determine that the diagnosis is indeed correct. Good.

After working on this for some hours, I finally found a way to reproduce this.
The culprit was that when the user presses <F6> to add face annotations to a file, the routine checked only the "Person" layer for existing faces. If, for some reason, there was a face annotation in another layer, the operation produced orphaned annotations or annotations pointing at the same face id (Telvoso mentioned that above). Because it adds new face annotations by detecting faces, but also deletes all existing face annotations in the image first.

I have now updated the check to consider all layers to see if there are face annotations.
When the user confirms the message, all face annotations are removed, not only the annotations in the Person layer. This way the annotations and the face data in the database match.
When a face annotation is copied & pasted, it is now automatically pasted into the "Person" layer when it exists. Else it is pasted into a layer with the same name as the source layer.

[cg]

Glad to hear that this issue is resolved. (As the OP, apologies for not having the time currently to do much rigorous testing to reproduce this).

I've run into the exact scenario that @Tveloso mentions with the question mark appearing. From what I understand from above, it sounds like this problem had been caused by cutting and pasting face annotations. That is something I indeed do a lot of in my own workflow (and I've seen these warnings in nearly every diagnosis).

The other thing I do frequently, which may or may not involve this error, as I haven't had a chance to test it, is draw multiple manual face boxes before clicking on each of them to assign the person.

Thanks, Mario, for diving into this and fixing it.

I DO still wonder if, for consistency's sake, either the face# or the file# should always be first in the (hopefully rarer) error messages.

[Tveloso]

Awesome!  Thank you so much Mario.
__PRESENT