Face recognition

[curtis1757]

I'm brand new to IMatch and I'm trying to get Face Recognition to work, I'm sure I'm doing something wrong, just can't figure out what.

I started with a folder with about 3,400 .jpg files.  About 3,000 of which I have used previously in Picasa so they have face meta data in XMP. IMatch shows these faces properly (about 34 unique faces).  However,  right click on a picture that has not been in Picasa and has a face that is in the People collection and I select 'Face Annotations and People->Detect faces and create annotations' and uncheck 'Ignore images with existing annotations' then click OK, it says '1 file() added to background processing queue.', OK so far so good.  I wait a bit, then check the picture and it shows the box for the Detected Face and then a '?" for the name.  It is a clear face and the person has 20 files with the face (from Picasa), No unconfirmed faces.  10 of the faces are used for training.  I deselected 5 of the 10 and repeated the process and still the face would not be recognized, only detected.  I tried the other options under Image Size in the Create Face Annotations dialog, but to no avail.  I did notice if I checked 'Ignore images with existing annotations' it skipped my selected file. So I tried on a similar picture with same face where no face had been detected yet and got same end results (although it to Detect the face with the checked 'Ignore images with existing annotations'.

In 'Preferences->Application' I have 'Automatic Face Recognition' set to Yes also.

So far I have been unable to get IMatch to recognize any previously unrecognized faces.

What am I missing???

Thanks for any help,
Curtis

[Mario]

Very strange.
I understand that face detection works, but the IMatch AI is unable to assign persons to these faces.
Does it fail for all persons in your database?
Or just the persons IMatch has created when importing Picasa data? Just in case the Picasa data has somehow polluted the IMatch AI.

Have you rejected many faces? 5 or 10 trained faces is just about right, especially when persons appear at a wider range of ages in the database.

What happens if you manually assign a person to a face using <F2> in the Viewer? Does that work?
IMatch should then automatically reprocess all unconfirmed faces in the background to see if the person also matches other faces.

To analyze this further, with IMatch to debug logging via Help menu > Support. See log file
Then run face recognition on a few files without prior face annotations.
Use the Help menu > Copy Logfile... command to save a snapshot of the logfile to disk, ZIP and attach. Maybe we can see some warnings.

Usually face recognition and person assignments "just works", within the limit of what's currently technically possible. Sometimes users overtrain persons or reject too many faces, making it very hard for IMatch to correctly assign a specific person. But that IMatch fails to recognize any person in the database, I believe this was never reported before.

Did you make any changes to the default settings for face recognition under Edit menu > Preferences > Application? Search for face to find relevant settings.

[curtis1757]

Hi Mario,
Thank you for your insights. I'm going to try and get a much smaller sample of pictures (10 or so) to test this out with your suggestions.

I'll let you know how it goes.

Thanks,
Curtis

[curtis1757]

Mario,
Yes, fails on all persons in the database.
There are no Rejected faces
If I manually assign a face in the Viewer, that works.
The only preference I changed was to set Automatic Face Recognition to Yes

I've made a small test case with 6 pictures with Picasa face data and then two that have not been in Picasa but have the face that is in the 6 pictures. I was able to repeat the issue with this small set, ie face detected but not recognized in the two pictures that have not been in Picasa.

Next, with a fresh copy of my pics I used 'exiftool -XMP:All' to remove all XMP data, so no more face data in the files.  Made new IM database with these 6 files, all faces detected, I identified/named the face in one file and IM then correctly recognized the face in the other 5 files. Then as a check I tried adding the one file that had not been in Picasa and the face was properly detected and recognized.

So, it appears to me that when IM builds its Face Recognition data from just existing XMP face data it is not doing it properly for some reason.

I turned Debug Logging on
Created new database with fresh copy of my 6 'Picasa' files (for the face in the Face Manager it showed all 6 files with the face marked as used to Train)
Added one picture with no "picasa' data (face was detected but not recognized)
I recognized/identified the face, IM took that and remembered (and made it a Trained face also)
So, I added one more picture with no "picasa' data (similar to previous one) and then IM detected AND recognized that face.

I have attached the log file for the above.

So again, it seems like when IM builds its Face Recognition data from just existing XMP face data it is not doing it properly for some reason.

I have about 100,000 .jpg files with about 200+ unique faces as detected by Picasa.  I really would like to transition to IMatch, but I need it to get my Picasa (XMP) face data into IMatch.

I can email you a GoogleDrive link to my small test case of 7 .jpg files total if that would help.

Thanks again,
Curtis

[Mario]

So again, it seems like when IM builds its Face Recognition data from just existing XMP face data it is not doing it properly for some reason.

Strange. This usually just works and there are no similar reports. I have a bunch of test files with XMP face regions written by Picasa and they just work fine. IMatch just imports the region of the face and the "tag" Picasa has set. It creates its own face fingerprint from the XMP region and if it would not find a face, it would mark the face region as "manual".

But it will assign the person based on the "tag" in any case. It will also create the person based on the tag if a person with that tag does not yet exist.
The only issue we ever had in this area was with face regions written by some Apple phones which all had a width and height of 0 (Apple bug). IMatch knows about this and sets the face regions to standard dimensions. But even in these cases the annotations were created and the persons assigned.

I will look into your test files. Hopefully I will learn something from them.
This will will take a few days, though. I get many emails every day and many emails are about "My metadata is not working right, please analyze and give advice". And such emails cause 5 minutes or 5 hours of work...

[curtis1757]

Thank you Mario, I'll email you a GoogleDrive link to my test files...

No hurry.... thanks for looking,
Curtis

[Mario]

Very good. Please include a link back to this thread so I know to which problem report your email belongs.

[curtis1757]

Mario,
One possible clue.  My Face Name tags have values like 'Case - Jessie Marie/id:C0008'. Picasa allowed the '/' and ':'.  Looking at the Face Name tags after importing into IM it removes the non-alphanumeric characters and results in 'Case - Jessie MarieidC0008'. (see screen shots below of XMP-mwg-rs value, before and after been through IM)

In IM I then saved the meta data (with the face names with no non-alphanumeric characters), copied the files to a new folder, confirmed the name tags had no non-alphanumeric characters, created new Database and added this new folder with the 'good' name tags to IM.  I then added my picture that has no Face Tag data hoping IM would now detect AND recognize the face, but no, it still just detected the face but did not recognize it.

So it appears to me the non-alphanumeric characters are not what is causing the problem... bummer.

Thanks,
Curtis

[Mario]

Since tags in IMatch are used for a wide range of purposes, IMatch only disallows certain non-alphanumeric characters. The tag managed by IMatch uniquely identifies a person, it is not to be confused with person properties like name or short name.
This is why IMatch sanitizes tags imported from XMP during import.

This makes no difference in your case, though. IMatch will just create the person with the sanitized tag and link it to the face.
I'm almost 50% down on my "Metadata or related problems" email queue so I should be looking into this in a few days.

[Mario]

Just downloaded your sample images, thank you.

I've imported them into one of my test databases.
IMatch imported the face regions from XMP written by Picasa. It created five persons.
The tag is the sanitized version of the tag written by Picasa (without the / and : ) and the unmodified tag as the Full Name.
Tip: Set IMatch to use the full name as the label under Edit > Preferences > Application. Search for label on that page.
This way you see the same label in IMatch you see in Picasa.

Although all face regions being imported and assigned to the correct person, trying to recognize the face of the young lady did not work. I could reproduce the problem you're describing, which is always a good thing.

I deleted all face annotations produced from the Google XMP record and ran face detection manually on these files.
After assigning the young lady to two of the faces detected by IMatch, I could run face recognition on the two images without prior face annotations and the faces were detected and the correct person was assigned.

So, my guess was that, for some reason, faces detected within the bounds of the Google face annotation imported from XMP and face detection within the bounds determined by IMatch worked differently. Or that the resulting "face fingerprints" were calculated differently. This is AI and complex, after all...

After some serious debugging and analyzing of what's going wrong, I've found a fringe case that might have caused this behavior (image dimensions, region dimensions etc. were involved). At least I was not able to reproduce it after making some changes anymore.

I removed the two folders containing your test images from the database. And added them again.
As before, the Picasa XMP face regions were imported correctly.
But this time, when I ran face recognition on the two images without Picasa face regions, the person was correctly assigned (the young lady).

I then duplicates the images into a new folder in Windows Explorer, removed the Picasa XMP face regions with ExifTool and added the new folder to the database. All faces were detected and the correct persons were assigned. Excellent.

I consider this bug as fixed. I will include it in the upcoming IMatch 2023.4.2 release.

See #2104 in the https://www.photools.com/release-notes/.

[curtis1757]

Thank you Mario for tracking this issue down. 

It's not clear to me, but it sounds like you were able to put some tweak into IMatch that will handle the larger Region area that Picasa makes.  (I looked at this and IM makes the face Region about 70% the size that Picasa does, even so, the X,Y center of the Face region is not always the same).

1. So, with the next release of IM, will my Picasa face data work?

2. When do you expect the next release?

3. Is there a way now (or in next release) I can cause IM to 're-detect' faces and assign the existing face Name to it?  (possibly using the Region-Area X,Y data to match up the Picasa face and the IM face and then replace the H, W and X, Y data with the IM generated face Region-Area data?)

4. If not, could I use something like Metadata Mechanic to batch process all my Region-Area H & W data to make it 70% of what it currently is?

5. Any other suggestions on how to batch process 1000,000+ .jpg files in a way that makes IM happy with the face data?

Ideally I can eventually end up with all my face data region area sizes consistent (which 3. above could do). So even if in a future release of IM it can handle the Picasa face areas, it would be best if I could batch change my Picasa face data to what IM would have generated, so future faces detected with IM are consistent with the Picasa detected faces.

Thank you again for your time,
Curtis

[Mario]

1. So, with the next release of IM, will my Picasa face data work?

Yes. At least it works here now.

2. When do you expect the next release?

Just out. If you're still on the trial, update your trial.

3. Is there a way now (or in next release) I can cause IM to 're-detect' faces and assign the existing face Name to it?

Select the files in a File Window and delete the face annotations from the right-click context menu or from the Commands menu, see Additional Commands

The press Shift+Ctrl+F5 and choose the "Reload Metadata" option. This will re-create the face annotations using the new code.
Note: If you have pending write-backs, write-back before.

If 100,000 files with Picasa XMP face regions are affected, process them in batches of a couple of thousand files.

would be best if I could batch change my Picasa face data to what IM would have generated, so future faces detected with IM are consistent with the Picasa detected faces.

They are now.
But you can also just delete the face regions imported from Picasa and run a face detection in IMatch.
After assigning persons to a few faces, IMatch should be able to assign the persons for the rest of the images automatically.
But that would be duplicate work.

Best to delete face annotations produced from Picasa XMP face region and then reload the metadata. IMatch re-imports the XMP face regions and processes them correctly.

IM makes the face Region about 70% the size that Picasa does

Yes. IMatch tries to cover only the face area without too much of the surroundings. The face regions are used for many purposes, from the People View to Events and too much surroundings just creates unwanted noise.

[curtis1757]

Thank you very much Mario, much appreciated!

Curtis

[curtis1757]

Mario,
I'm using IM 2023.4.2 (licensed) on Windows 11 laptop and still cannot get IM to resize existing Face regions from Picasa size down to IM size.  I followed your instructions above, I may be missing something though... here is my sequence:

1) Made Folder with one .jpg file with one Picasa face in it, IMG_3918.jpg which is one of the files in the GoogleDrive link I emailed you earlier, (I will email you again the link after I post this)
2) Fresh install if IM 2023.4.2
3) Start up IM, and I don't change any options
3) Create New DB in the Folder with the one .jpg
4) DragNDrop Folder into 'Media & Folders' window.
5) Thumb Nail has 'Pencil' to write the 'XMP::iptcExt\PersonInImage' tag, I click the pencil
6) DblClick Thumbnail, and it shows the Picasa size face box on the face, closed window
7)Right Click on ThumbNial->Face Annotations and People->Delete Faces->Delete All Faces
(Note: Metadata Window (Browser), still shows the Picasa size Face region in the XMP tags)
8 ) DblClick Thumbnail, and it shows no face box, closed window
9) Shift+Ctrl+F5 and choose the "Reload Metadata" option - got Info Dialog about Some Metadata Protected, I clicked OK
10) DblClick Thumbnail, and it shows the Picasa size face box on the on face, not an IM sized face.

What am I missing??

Thanks,
Curtis

[Mario]

9) Re-imports the Picasa face annotations from XMP, this time in working condition. This was the change I've made to fix the problem you have reported. IMatch does not manipulate the face region it imports from XMP. It now just creates the correct fingerprint again.

If you want a different size, remove the face annotations and run a normal face recognition in IMatch. IMatch then creates new annotations with the normal size. When you then write-back, IMatch replaces the Picasa XMP regions with it's own XMP face regions. Or resize the face annotation imported from Picasa in the IMatch Viewer.

[curtis1757]

Ok, great, I misunderstood, that does work. 

I like programming and but I have not ventured into app development yet of IM.  Would it be possible to have an IM app  run through all my .jpgs with Picasa face region data and reduce by some percentage?

Thanks,
Curtis

[Mario]

Would it be possible to have an IM app  run through all my .jpgs with Picasa face region data and reduce by some percentage?

I guess so. You can fetch the value of the region tags like any other tag and also update it from an app.
But why would you do this?

The face annotation size does not really matter that much, as long as IMatch is able to detect a face within the region.
When you manually place face annotations in the Viewer, IMatch looks within the bounds and if it can finds a face, it creates a face annotation. If it cannot find a face, it creates a "manual" face annotation.

For face annotations created based on imported XMP face regions, IMatch does the same. Except that it assigns the person identified by the XMP region tag to the annotation.