Maximizing Face Detection

[Tveloso]

I added a Feature Request on this subject:

https://www.photools.com/community/index.php/topic,14882.0.html

...that I didn't want to muddy with all the details of my reasoning, and tests performed, so I thought I'd give those details here.

I have been running Face Detection (Ctrl+M,F in the File Window) in multiple passes, using all four of the Size options IMatch provides (including the Original Size option).  This, in an effort to have IMatch detect Faces in as many files as possible, before I begin working with them in the Viewer.

Here is an example of the procedure I followed for a recent batch of 42 files:

Initial State
These are all iPhone Files, which already contain Face Regions.  But the Faces are not yet associated with any Persons (presumably because I have Automatic Face Recognition set to No, and I do not add any names to the Persons on the Phone, so the files lack the MWG::Regions\RegionsRegionListName\RegionName\0 Tag).  But even if the files did have the correct persons assigned, I would still not want to keep the imported Annotations.  Most of the time, they are offset to the upper-left or to upper-right on the face (sometimes cutting off part of one eye!), or they are too large, or too small, relative to the face.

A bit of a tangent here, that can be skipped over:
Although these files lack the RegionName Tag, they do contain a proprietary Apple Tag, that gives an integer identifier for the faces (MWG::Regions\apple-fi:RegionsRegionListExtensionsFaceID\RegionExtensionsFaceID\0).  I once thought that I might be able to make use of that, with a Metadata Template, to associate the iPhone-created Face Regions with Persons in IMatch, but abandoned that idea. 

The values delivered in that Tag did not seem to stay constantly associated with a given person...they appeared to reset with each batch of files imported from the same phone.  So while FaceID #1 in one batch of files, might identify the same person in that batch, it did not identify that same person in the next batch of files.

Also, I just noticed a small issue with that Tag...this doesn't cause any problems for me, since I have no need for that tag, but it seems that IMatch is no longer giving a list of those IDs, in files that contain multiple faces.  This seems to be because ExifTool is not delivering that tag (and a few others) as lists, but instead sends "multiple instances" of the tag, with one value each (so IMatch is showing only one value there).  Could this be an ExifTool bug?...(or is Apple maybe writing this incorrectly into the metadata?)

For example, in a file with two faces, only one Apple FaceID is being given in the MWG-Regions Group (#7 in this case):

   

...because ExifTool is emitting "multiple instances" of that tag (and others) with one value each:

Code Select Expand

[XMP-mwg-rs]        - Region Area Y                   : 0.16965803108808289, 0.78855958549222804
[XMP-mwg-rs]        - Region Area W                   : 0.18410362694300508, 0.086176165803108828
[XMP-mwg-rs]        - Region Area X                   : 0.5763834196891191, 0.58225906735751287
[XMP-mwg-rs]        - Region Area H                   : 0.24547150259067357, 0.11490155440414507
[XMP-mwg-rs]        - Region Area Unit                : normalized, normalized
[XMP-mwg-rs]        - Region Type                     : Face, Face
[XMP-mwg-rs]        - Region Extensions Angle Info Yaw: 315
[XMP-mwg-rs]        - Region Extensions Angle Info Roll: 270
[XMP-mwg-rs]        - Region Extensions Confidence Level: 122
[XMP-mwg-rs]        - Region Extensions Face ID       : 7
[XMP-mwg-rs]        - Region Extensions Angle Info Yaw: 45
[XMP-mwg-rs]        - Region Extensions Angle Info Roll: 240
[XMP-mwg-rs]        - Region Extensions Confidence Level: 633
[XMP-mwg-rs]        - Region Extensions Face ID       : 3
[XMP-mwg-rs]        - Region Applied To Dimensions H  : 2316
[XMP-mwg-rs]        - Region Applied To Dimensions W  : 3088
[XMP-mwg-rs]        - Region Applied To Dimensions Unit: pixel

Shouldn't Exiftool have delivered the above output like this?:

Code Select Expand

[XMP-mwg-rs]        - Region Area Y                   : 0.16965803108808289, 0.78855958549222804
[XMP-mwg-rs]        - Region Area W                   : 0.18410362694300508, 0.086176165803108828
[XMP-mwg-rs]        - Region Area X                   : 0.5763834196891191, 0.58225906735751287
[XMP-mwg-rs]        - Region Area H                   : 0.24547150259067357, 0.11490155440414507
[XMP-mwg-rs]        - Region Area Unit                : normalized, normalized
[XMP-mwg-rs]        - Region Type                     : Face, Face
[XMP-mwg-rs]        - Region Extensions Angle Info Yaw: 315, 45
[XMP-mwg-rs]        - Region Extensions Angle Info Roll: 270, 240
[XMP-mwg-rs]        - Region Extensions Confidence Level: 122, 633
[XMP-mwg-rs]        - Region Extensions Face ID       : 7, 3
[XMP-mwg-rs]        - Region Applied To Dimensions H  : 2316
[XMP-mwg-rs]        - Region Applied To Dimensions W  : 3088
[XMP-mwg-rs]        - Region Applied To Dimensions Unit: pixel
End of Tangent

So the starting point for the faces in these 42 files is as follows:

    File    Files    Files  Total
    Count  wo-Faces  w-Faces  Faces
      42        13      29    54

Pass #1
Select all 42 files in the File Window, and press Ctrl+M,F.  In the Create Face Annotations dialog, uncheck the Ignore images with existing face annotations Checkbox (as discussed above, I would like to discard the Face Regions the phone has created, and have IMatch create new ones), and use the Default Image Size option.  IMatch queues all 42 files for processing:

   

Afterwards, the "face stats" in these files is as follows:

    File    Files    Files  Total
    Count  wo-Faces  w-Faces  Faces
      42        18      24    32

Pass #2
After the Info & Activity Panel indicates that all files have been processed, with the 42 files still selected in the File Window, press Ctrl+M,F again, and this time check the Ignore images with existing face annotations Checkbox, and select the Optimize for images with small faces (slower) option, and IMatch now queues 18 files (indicating that the last pass found Faces in 24 files):

   

The "face stats" then went to this (faces were added to 2 more files):

    File    Files    Files  Total
    Count  wo-Faces  w-Faces  Faces
      42        16      26    34

Pass #3
Once gain, with the 42 files still selected, press Ctrl+M,F, and now select the Upscale images with small faces (slow) option.  IMatch now queues 16 files:

   

The "face stats" then show another 4 files had faces added:

    File    Files    Files  Total
    Count  wo-Faces  w-Faces  Faces
      42        12      30    38

Pass #4
Finally, select the Original size. Very slow. For special cases option, and IMatch queues 12 files:

   

...and this last pass found faces in an additional 2 files:

    File    Files    Files  Total
    Count  wo-Faces  w-Faces  Faces
      42        10      32    40

So all told, IMatch found Faces in 32 of the 42 Files.  This is better in terms of file count, than what the initial state was (what the iPhone had done), but not quite as good in terms of total Faces found.  But again, these are better quality Annotations, and they are now assigned to the correct persons.

A few more observations around this:

• Where IMatch found faces, assignments were 95% accurate (only 2 incorrect guesses across 40 Face Annotations).
• In one file with 5 faces, the iPhone had detected all 5, but IMatch found only 4 (the one not detected was wearing sun glasses - though one of the detected ones also was)
• In one photo where the iPhone detected both faces, IMatch didn't find any.  But both persons were wearing Face Masks (and during Covid, the iPhone added a feature to allow it to be unlocked via Face Recognition, even when wearing a mask, so probably that functionality was at play here?
• The opposite was also true - in a file (containing three persons) where the iPhone did not find any faces, IMatch found one face (and gave the correct assignment).  The two faces not detected were obscured, and became manual Face Annotations once added in the Viewer.

When I originally started using this "stepwise detection" procedure, I also considered going directly to a "FullSize - No Ignore" request...thinking that in the end, it might yield better results (and I would only need to run it one time), even though it would be quite slow.  But I think I decided that it didn't actually do much better overall, and the better performance of the multi-pass approach was worth maybe having missed a few faces. 

But for the sake of this testing, I also tried that "single pass at full size" approach to see how it would compare.  With that, IMatch found more faces (41), but in fewer files (29), compared with the final result of the "stepwise detection" procedure (where 40 faces were detected in 32 files).  But some of the faces detected here (at full size) were in picture frames in the background, so it was actually fewer actual faces.

Also, the contention that the Viewer does a little better job at Face Detection, than does a "batch mode" run in the File Window (even when that "batch run" was done at full size - which is what the Viewer does), seems to have come up during this test.  When I worked with these files in the Viewer, there was one file containing two faces, where I thought both faces should have been detected.  And when I pressed F6 and selected the option to Delete the existing Annotations, and run Face Detection again, both faces were now detected (and correctly assigned).

Even if it's true that something in the "Batch Mode Face Detection" is preventing it from "performing at full capacity" (and I don't know that it is - it's just a vague sense that this is happening, since the Viewer sometimes finds a face not previously detected), Face Recognition in IMatch is nevertheless still excellent!

[cg]

I have a similar process where I'll run a face detection pass on all the photos, then a second pass at a different setting to pick up faces that IMatch didn't catch on the first pass.

I've also found that my iPhone's photo regions come in I think too tight around the faces for IMatch to identify, so there are mostly ?'s under the regions. I usually just delete them all and let IMatch do the face detection from scratch. The iPhone does seem to catch more faces with sunglasses and more faces in profile.

[PandDLong]

I appreciate the detailed walk-through and findings.

I learned something - I never paid attention to the 'Ignore images with existing face annotations' checkbox and reading your note, I realize why sometimes I did a rerun of a batch at a different setting and I would get worse results.  So I stopped doing multiple passes but now I will try it again and remember the check box!

Michael

[Mario]

I've also found that my iPhone's photo regions come in I think too tight around the faces for IMatch to identify, so there are mostly ?'s under the regions.

IMatch adds some extra spacing around XMP regions when importing them. If the face regions Apple writes are too small, please send me a sample to support email address with a link to this topic.

Maybe Apple messed something up again. Some of their phones e.g. write image regions with a width and height of zero. Maybe this is yet another special case and IMatch needs to add even more room around the XMP regions imported from Apple devices.

A few more observations around this:

That's how it works. Different algorithms produce different results. IMatch may catch images Apple does not and vice versa. Face recognition is under constant development, and when I find some algorithm that is substantially better than what IMatch uses now, I will potentially upgrade.

Upgrading the face recognition will be a breaking change, since new algorithms produce different face fingerprints, basically requiring IMatch to rescan all unconfirmed/unignored faces to be able to compare face fingerprints across the database.

since the Viewer sometimes finds a face not previously detected),

The Viewer always uses full-size (full cache size image). Batch tries to improve performance by using the smallest possible image. If the image is twice as large, processing time is roughly four times slower.

For people wo work with small faces, using the "optimized for small faces" or "full size" for Batch is recommended.

It seems you are processing images with very small faces often?
The only reason I could think of for your result is that the upsizing / scaling algorithm in introduces noise in the image and that this triggers the AI to detect a face or not it did or did not at a different scaling of the image.

As always, having the actual image uses for the test would be helpful.

I test with typical portrait, family and group photos (soccer teams, choirs).