Handle Persons stored in Keywords only (without face annotation)?

[abgestumpft]

Hi,

I have thousands of files where persons are tagged with hierarical keywords, but not with a face annotation.
Because of this, in iMatch they are not listed under the Person in the People-tab.

The options I see to handle those are:

1. Ignore it (save work) and use the Category: "@Keywords - Persons - ..." to filter on Persons (but then "inconsistencies on Persons-> not all Fotos visible via Person filter)


2. Manual (painfull) way: Select all images (@All Category) and set Filter on a person via "@Keywords - Persons - ..." -> manually assign Face annotations for each person (via  copy&paste in the viewer). This will have the best end result, but will also take days of work...


3. Automatically create face annotations with a tool (exiftool?) - in case this is possible.
E.g. something like that (do once for every person -> scan through all images):
Everytime there is a tag found for the person:
<rdf:li>Persons|Friends|Firstname Lastname</rdf:li>

Add a very small face annotation box in the Metadata (of course then with wrong position):

Code Select Expand

  <Iptc4xmpExt:PersonInImage>
   <rdf:Bag>
    <rdf:li>Firstname Lastname</rdf:li>
   </rdf:Bag>
  </Iptc4xmpExt:PersonInImage>
...
<rdf:Description rdf:about=''
  xmlns:mwg-rs='http://www.metadataworkinggroup.com/schemas/regions/'
  xmlns:stArea='http://ns.adobe.com/xmp/sType/Area#'
  xmlns:stDim='http://ns.adobe.com/xap/1.0/sType/Dimensions#'>
  <mwg-rs:Regions rdf:parseType='Resource'>
   <mwg-rs:AppliedToDimensions rdf:parseType='Resource'>
    <stDim:h>2400.000000</stDim:h>
    <stDim:unit>pixel</stDim:unit>
    <stDim:w>3200.000000</stDim:w>
   </mwg-rs:AppliedToDimensions>
   <mwg-rs:RegionList>
    <rdf:Bag>
     <rdf:li rdf:parseType='Resource'>
      <mwg-rs:Area rdf:parseType='Resource'>
       <stArea:h>0.040905</stArea:h>
       <stArea:unit>normalized</stArea:unit>
       <stArea:w>0.027961</stArea:w>
       <stArea:x>0.023520</stArea:x>
       <stArea:y>0.060578</stArea:y>
      </mwg-rs:Area>
      <mwg-rs:Name>Firstname Lastname</mwg-rs:Name>
      <mwg-rs:Type>Face</mwg-rs:Type>
     </rdf:li>
    </rdf:Bag>
   </mwg-rs:RegionList>
  </mwg-rs:Regions>
</rdf:Description>

4. Other ideas?


What do you think? Any experience with that or suggestions?

[sinus]

I am in the same situation.
But for me it is clearly option 1.

I have setup a long time ago a (for me) very good category-system to "filter out" (only with categories, not the normal filter) persons (specialy family).
Hence I use (at the moment) only this for me, although the AI for people is fascinating.

[Mario]

ExifTool cannot detect faces in images.

Why don't you let IMatch create face annotations for these files?
Then train a couple of faces for each person and let IMatch assign the person to all matching faces.

Then correct false assignments, if needed.
Easy to find by selecting the person in the People View and inverse filter for the matching keyword.
Then you see all faces of "John" but without keyword "john" => false assignment. Easy to find, easy to fix.

Why do you want to do anything manual?

[abgestumpft]

ExifTool cannot detect faces in images.

The idea was to use some kind of script that just adds a face annotation tag to the images (always on same position) to make iMatch list it under the "People"-view. The face annotation is of course then in wrong position.

Why don't you let IMatch create face annotations for these files?
Then train a couple of faces for each person and let IMatch assign the person to all matching faces.
Then correct false assignments, if needed.
Easy to find by selecting the person in the People View and inverse filter for the matching keyword.
Then you see all faces of "John" but without keyword "john" => false assignment. Easy to find, easy to fix.

I have also assigned the keywords to images with a lot of side faces, and at the moment the hit-rate of AI for my kids is not very good. So when having already filtered on one person, I thought it is faster to just go through the pictures in the viewer manually and just paste in the face annotation box each time with "CTRL+V" and move it to the face.

Why do you want to do anything manual?

I don't want to do anything manual, but I guess there is no way around it (at least something needs to be done manual) 

One Idea I just got when reading your suggestion on the filtering:
1. In Categories -> @All
2. Filter - this will show all images that have the persons keyword assigned, but no face annotation
-Categories: @All -> @Keywords -> Persons -> Family -> Persons Name
-People: Select "Persons Name" and invert the result of filter
3. Optionally: Run face recognition and see how many hits we get
4. Assigning Face annotation box for all images it was missing
-create a small face annotation box somewere in the first image and assign persons name to it
-In the file browser, copy the image with "CTRL + C"
-In the file browser, select all other images and copy the face annotation box to them via: "CTRL+Shift+V" and select only "Annotations" in the "Paste Attributes and Data" Dialog
Now we have a "dummy" Face annotation tag for each of the images where it was missing before
b) Finetuning (optional): go trough all images and move the inserted face annotation box to the persons face

[Mario]

I'm not sure how using deliberately wrong (non-face) annotations will work with features I will add in the future.
If you don't need/want face annotations and face recognition, I suggest just stick to your keywords and manage your files in the Category View via @Keywords.

[Jingo]

We discussed this a few times in other threads... I had the same thought originally - create a quick app that applies a tiny "dummy" face box to link it to the person... two things stopped me from moving forward... the undocumented endpoints for Persons/annotations .. but more importantly - Mario's warning about breaking something down the road.

I have chosen to approach the Person feature as a "move forward automation" to assign the correct person keywords to new images and not try to use the feature in its current iteration for linking keywords with People.  I too would like this feature (just a simple bi-directional keyword to Person link) - but understand that this might cause issues down the road.

[hluxem]

If you don't need/want face annotations and face recognition

Can't speak for the OP, but I have the same issues. It's not about not wanting face annotations, it's about what you do with thousands of pictures without annotation. Even with the best AI, it is time consuming to update thousand's of pictures.
I had a similar issue when I started with GPS coordinates many years ago. But it was relative easy to go back and add GPS data to images I already had location categories for.

I think some of the new features for reviewing people may help to make this process more efficient. I decided to wait and add only face tags to new images at this point.

Heiner

[Mario]

The Face Reviewer feature I'm working on will make it a lot easier to deal with large batches of faces.
But it does not change the general nature of how face detection or recognition works .

I have processed tens of thousands of faces and images during the tests and in my own databases.
I've  had to correct maybe 5% of the faces so this was no big deal. An afternoon, more or less. Confirming faces requires <Tab> then <C> and then PageDown for then next image in the Viewer.
Very efficient once you've got the hang of it. Or maybe you need to <Tab><F2> if a face is wrong.

Undetected faces requiring manual annotation are more time-consuming. No way around that. Depends on your type of photography, of course.

[claudermilk]

Like many, I'm in the same boat. I am biting the bullet and going through the thousands of images. The first step was to define the people in my database from my category tree. Then, grabbing those categories & applying detection. Now to confirm the detected people. Since there's overlap, as I go through each section will get quicker with fewer people left. Like Mario said, it's been a fairly quick Tab-C-left arrow (works same as page down for me) for most images. I can hit the keys faster than IMatch is updating the database.

I look forward to the upcoming reviewer feature--I'm certain I won't be done before that is released.

For the pain, I'm looking at this as a one-time thing. After it's done, ingesting new batches of images will be quicker than in the past with manually entering each person.

[abgestumpft]

Hi,

I now did some quick tests:
A person that had 183 keyword tags for the person but no face annotations. What I did was:
1. In Categories @All I filtered to get all images of the person that have the keyword of the person assigned, but no face annotation for that person:
-Categories: @All -> @Keywords -> Persons -> Friends -> "Persons Name"
-People: Select "Persons Name" and invert the result of filter

2. Trained 4 selected images to Face AI

3. Run Face Recognition on the remaining 179 images (runtime ca. 2min)
Result was 115 images assigned to that person

4. Manual Review of those 115 images (adjusted filter) - (ca 5min)
Almost every face was correct, so Tab -> "C" was fast

5. For the remaining images (64 after corrections) I copied a "dummy" face annotation with the persons name to all of them:
First I created a face annotation on one of the images and assigned the persons name to it. Then I copied the image with CTRL+C
Then selecting all images that were left via filter (see 1.) and copied the face annotation: CTRL+Shift+"V" and select "Annotations"

6. Going through those 64 images manually and moving / resizing the face annotation to persons face (ca. 10 min)

Overall this took around 20min. Hitrate of AI was very good for my type of images: I often have small faces, from the side,...
Another test (156 images) I had created 86 person assignments by AI, 11 of them wrong. Took me around 25min to complete the process.

For Family members like my kids I currently have up to 2.000 assigned Keywords each (and until now I just loaded around 1/3 of all my pictures in IMatch).
So this could take a while to complete everything :-)
Also the hitrate of face recognition on my kids is much lower which will increase the time needed...


The Options I see are now:
1. Ignore it (save work) and use the Category: "@Keywords - Persons - ..." to filter on Persons
Maybe it makes sence to greate a dedicated category just for keywords containing person information:
@sinus: do you have some more information on your category setup? Could I create my own data-driven category on keywords?
They should just include the keywords with "Personen|..." -> did not find such a filter function on a quick check

2. Only assign dummy annotations to images that currently have person keywords assigned but no face annotations
Use the process I described above, but leave out the Face Recognition and manual checks.
This would be quick, but also dirty. It will for sure work to have all the persons assigned to a person in iMatchs People category, so filtering on Person and other actions would work.
It should also be OK from technical point of view (no corrupt metadata etc.), since it is a valid person-face annotation written by iMatch (just at the wrong position in the image)
But I'm not a big fan of this, since it might lead to other problems (though I can't think of one at the moment)

3. Do a full assignment of face annotation to people that are currently only marked by keywords
Use the full process I described above. I could do that person by person, starting with the ones that are most important to me...


For now I think I will go with (1.) and maybe play a little bit with (3.) and definetly try (3.) once the new "Face reviewer" is available.

[Jingo]

Thanks for sharing your test and workflow... certainly interesting and a possibility.. I just can't see myself doing this for the over 40,000 images I already have keyworded in my DB.

I think only you can decide how you want to use a certain function within IMatch.  For me, I'm just not sure it is worth the time/effort to go through 10,000 photos to run the AI, confirm/correct and then manually add annotation boxes (paste/resize/move/confirm) just to link known persons via keyword to the Person function.  Using Keywords, I can and have been finding people for years... knowing where they are within a photo is not all that important to my workflow.   Perhaps I'll revisit down the road.. but for now, using the AI to detect and auto-assign keywords for future images is more exciting to me and that is how I'll be using the feature.

[Mario]

Face recognition is not for every user. If you have already organized your files per person, there is no need to use it. You won't benefit yet from it.

[sinus]

Face recognition is not for every user. If you have already organized your files per person, there is no need to use it. You won't benefit yet from it.

I think, this is true for me.
I have a lot of images with familiy members. All these image (well, most of them) are tagged with a keyword.
Then of course I have also a lot of other persons, but often they appears only once or sometimes twice. A vip maybe 5-10 times.

But also these people I have tagged.
That is why I see no need to use Face recognition, although is sounds exiting for me.
If I have time enough, I will once try this feature, but if, then only with a fraction of my images on a new DB, just to "play" and see. 

[sinus]

The Options I see are now:
1. Ignore it (save work) and use the Category: "@Keywords - Persons - ..." to filter on Persons
Maybe it makes sence to greate a dedicated category just for keywords containing person information:
@sinus: do you have some more information on your category setup? Could I create my own data-driven category on keywords?
They should just include the keywords with "Personen|..." -> did not find such a filter function on a quick check

I am not sure, if this will help here, but I created some infos on another thread (to not make this here complicated):
https://www.photools.com/community/index.php?topic=9980.0

[Mario]

Back to the original question of this thread...

2. Manual (painfull) way: Select all images (@All Category) and set Filter on a person via "@Keywords - Persons - ..." -> manually assign Face annotations for each person (via  copy&paste in the viewer). This will have the best end result, but will also take days of work...

This assumes that the integrated face recognition does not work. Why else would you do manual face annotations?


When I understand you correctly you have categories for persons.
Each category has n files. No face annotations yet.

One solution would be an purpose-built app which

Processes a category hierarchy.
For each file it looks if it has one (!) existing unconfimed/unassigned face annotations (these must be created in advance with the normal functionality).
If this is the case, it would use the category name to create / assign a person.

This would be doable but would need tailoring to the specific way you have created your categories.
The app would also need to understand how it can "find" or "create" a person from the name of the category.
And all files with more than one unassigned/unconfirmed face could not be processed and the app would need 'flag' them somewhere (collection or assign a category).
...
This is nothing that could be generalized for all users. Purpose-built app.

For the upcoming major feature Face Review I have added endpoints to link persons to faces. So at least I could access that info (these endpoints are constantly changing and hence not documented and not safe to use for app programmers). If this all settles, who knows. Then such purpose-built "one user" apps can be created (if you know how to program or you know somebody who does).

[Jingo]

Back to the original question of this thread...

One solution would be an purpose-built app which

Indeed... and that was refined idea for an app...  generate annotations for imges within category (a single person @keyword), count the annotations to be sure there is only one, assign the person.

But - until we have access to the endpoints (even unofficially), there isn't much I can do but poke through existing code.  Waiting like a kid before his birthday!   

[Mario]

There are no endpoints for creating annotations (face or otherwise).

[plastikman]

Some more thoughts on this subject matter:

I see some challenge in the way People manager works currently, which is based on Faces and thus not People (as people can be in a photo without their face visible or with their face heavily obscured). Based on this, it would thus make more sense to call it either Face Manager. However, I like the way it works and therefore think it should have a way to also connect photos without a face visible / face ID with a simple toggle/filter to pictures with faces / without-faces.

E.g. if I add the gender or family of a Person via the People manager keywords, I have to manually find the pictures of the person without the face visible and still add it. If this is the case, I might as well not use the Person Keywords at all and do everything manually. So I think this aspect of the People manager needs some slight rethinking: it seems more logical that either it has a way to support non-face photos.

One way to achieve this: draw annotation/boundary boxes around every person even when they are with the back of their head towards the camera/largely obscured. Is this the way the People Manager was designed?

[Mario]

The People View shows all persons you have created.
A person is linked to a file is via a face annotation.
You cannot link a person to a file without a face annotation.
For this situation, IMatch allows you to create manual face annotations.
Use this for persons facing away from the camera. People with the back of their head to the camera. Pets. Ghosts. People outside the frame. Whatever the problem is you need to solve.
IMatch does not care. If you say "this is a face" and it's Uncle Bob, IMatch will record that info. And the file will show when you click on the Uncle Bob person. The variables and filters will work. Propagation will work.

[Tveloso]

I think that having Face Annotations can be a very valuable thing.  In my case, I'm making the effort to add them.

I'm in a somewhat unique situation, in that I started importing Face Region data in IMatch2019, but from a Data Source that was not very complete...so for an image that shows 4 persons, I might have Annotations for only two of them.  I am therefore very excited to have Face Recognition now in IMatch, as I'm needing to "re-process" all of those incomplete images.

I do understand that for the folks that have been "tagging people" for years in IMatch, via Keywording alone, and so already have well established methods, that doing the work needed to add the Annotations, might seem like a great deal of work for not a lot of added benefit, but I think that if I were one of those folks (and didn't actually "need" to add Face Annotations), I might still want to...

Having the ability to indicate which person is which can be vary valuable...both for old family group photos, where we ourselves don't actually know who is who, but also for situations where a Manual Face Annotation is needed.  The person with their back to the camera, or otherwise obscured, in an interesting "action packed" photo, might be easily identifiable for us today, but maybe a few years down the line we might be saying "who is that?", that's doing that interesting thing...(we would know if we could see their face, but maybe we can't tell anymore now).

And once Mario releases IMatch 2020.4.2, and we have the Face Reviewer, this will all be much easier!

[Mario]

and we have the Face Reviewer, this will all be much easier!

I hear that a lot. The beta groups also like it.
Maybe I should put on my greed hat, go full Apple and charge extra for it 

Sorry. I'm running low on toilet paper and this affects my mental prozesees...

[abgestumpft]

This assumes that the integrated face recognition does not work. Why else would you do manual face annotations?

Because I have also many side portraits etc. and it could be more work to review&fix + adding missing faces than doing it full manually:
And since I already have the keyword assigned it is much quicker assigning manual face annotations (copy&paste).
It's a different story for new pictures that don't have any person-keywords assignemnts yet.

When I understand you correctly you have categories for persons.
Each category has n files. No face annotations yet.

Yes, I have one hierarical keyword (-> also category) for each person, like:
Persons|Family|John Smith
But some of those files with keyword "Persons|Family|John Smith" also have face annotation "John Smith" assigned

When you check this thread of mine (Steps 1-6):
https://www.photools.com/community/index.php?topic=9976.msg70589#msg70589

I can very quickly filter on all images of a person that have the person-keyword assigned, but no face annotation yet (keeping the "John Smith" example):
-Categories: @All -> @Keywords -> Persons|Family|John Smith
-People: Select "John Smith" and invert the result of filter

Then I could just select all filtered images and copy with CTRL+SHIFT+V (and selecting annotations only) one face annotation with name "John Smith" to every image it was missing in one rush (but wrong position in the image)

I think the main problem for me (and maybe others too) is that I have tagged the images with keywords that show that person and that does not always mean the face is perfectly visible.
We have face-detection and then this face is assigned to a person. Therefore Facedetection will only find parts of the images of the persons that I assigned with keyword.

That is why I initially asked in my Feature request, if it's possible to have a keyword (like: "Persons|Family|John Smith") linked to a Person that is managed in the People-View. Then I would see in People view, when selecting "John Smith":
-Images that have "John Smith" face annotation assigned
AND
-Images that have Keyword "Persons|Family|John Smith" assigned
Since this is technically not possible I was looking for a smart way to get the face annotation assigned to the pictures I already have assigned the persons keyword...

[plastikman]

I use a variation in the form of Person|Family|Smith|John. This way I can target the smaller John Smith family as part of the larger Smith family tree.

[sinus]

I use a variation in the form of Person|Family|Smith|John. This way I can target the smaller John Smith family as part of the larger Smith family tree.

I have also something like plastikman.

Privat|Familie|Smith|77 Smith John (father, real base)
Privat|Familie|Smith|77c1 Smith Clara Beth (wife)

Privat|Familie|Smith|771 Smith Hans (child 1)
Privat|Familie|Smith|774 Smith Felic (child 4)
Privat|Familie|Smith|774c2 Finger Evelyne (wife 2 of Felic)
Privat|Familie|Smith|7742 Smith Luca Johann (child 2 of Felic)

The numbering-system I created a long time ago, form some Genealogy-book 
Each number is unique for a person.
And in a kind linked, e.g.

774 Smith Felic is a base (real base is 77)
774c2 (Evelyne) means the wife of him, and because not c1, but c2, this number 774c2 is the second wife of him.
7742 (Luca) the 2 means, that this is a child from Felic, and since this is a 2, there must be also a 1 for another (the first) child.

Hence I can search e.g. for 774 and get all family-members, in a way linked with this base (Felic), while for Felic is the base his father 77 (John), and Felic is child 4 (774) of John.
It is, I think, a simpler system that this book suggested, but for me and our family it is good enough.

I do not use this numbering-system often, but for some searches it is quite useful.
But I a know, real genealogists has sophisticated systems, but I am not a genealogist, I wanted only a system for our family, what works for me.