Face recognition feature?

[CollieDog]

Is there any plan to build a face-recognition feature into IMatch?

For my research photos, it would be immensely valuable and a huge time-saver if the program could label hundreds of people by itself!
Is building a face-recognition component extremely difficult?  Judging from the lack of effective programs out there, I'm guessing that it is.

I tried using the face recognition feature in Corel Paintshop Pro X6 to recognize faces and add names. It works in a rudimentary sort of way, but Paintshop Pro X6 does not seem to write to xmp and thus the program's face recognition ability is rather useless to me. I hope to rely on IMatch exclusively for all my cataloguing needs.

Similarly, I have not figured out a way to get Picassa to do this either.

When IMatch 5 when it was released to beta, I purchased a license for IMatch 3.5 just so that I could download version 5 and participate in testing. I was hoping to contribute more to the testing process, but haven't been able to. Too much work trying to finish my PhD dissertation and related work.

[Mario]

Is building a face-recognition component extremely difficult?  Judging from the lack of effective programs out there, I'm guessing that it is.

Unfortunately, yes. It is a million dollar technology, and the big players (Google, Facebook) have not only bought the former independent vendors of face recognition software, but also plastered the research field with tons of software patents to keep others at bay. Face recognition is a very important technology for Google and Facebook, because the resulting data (face + address + whatever you do on Google/Facebook) can be sold for very large sums to Google/Facebook clients. Companies and other parties pay a lot of money to get access to face data. Combine the data with a live camera stream and you can identify and track people anywhere - in shops, cities, airports, ... This creeps me out

Since not even Google/Facebook get face recognition 100% working using computers alone (yet), these companies have made it "social" and "cool" to "tag your friends". By asking harmless questions like "Is this Frank?" they improve the quality of their face databases continuously, using their users as free agents in that process. And all the users are excited and help these big companies to produce even better profiles.... because it's 'social'.

All this is considered as highly critical and dangerous by many data security experts, especially here in Europe.


If you use Google Picasa to detect faces, Picasa uploads the image to a Google server and performs the image recognition there. Naturally, the resulting data and identification remains on the Google servers forever. But Picasa at least can write an XMP record with Face data, and IMatch will display this record in the Metadata Panel. The data can also be searched and displayed, like all other metadata in IMatch.

I have so far found no usable and reliable open source or research project which would allow me to add this technology to IMatch. If somebody has a tip or info, let me know! As said, this is a field of research, requires massive amounts of computing power and is also plastered in software patents.

There was a company who offered the technology for a license fee. I looked into this and maybe I would have made a cost-based plug-in for it. But to use their service, IMatch would have to upload face data to their server, which is critical and against the law in most countries. Or at least a legal minefield. And since Facebook bought that company and shut them down to incorporate their technology exclusively into Facebook ("tag your friends") it does not matter anymore.


The Annotation feature in IMatch has a special "face annotation" tool which allows you to indicate faces and combine them with keywords. These annotations are grouped in a special collection as well. That's at least a good start.

My idea for a later version of IMatch is to automatically produce these face annotations from the XMP face recognition data created by other, specialized software. If a user wants to tag the faces in his images and share that data with others is then outside of the control and responsibility of IMatch. IMatch could also produce face reg XMP records from annotations created by the user in the Viewer.

[medgeek]

If you use Google Picasa to detect faces, Picasa uploads the image to a Google server and performs the image recognition there.

Are you sure about that, Mario?  I thought the Picasa client recognizes faces on a local computer; my understanding is that there is no data uploaded in that case.  If one is recognizing faces in pictures uploaded to Picasaweb (now Google+), then of course that operation takes place on the server side and Google can use that information.

[CollieDog]

Hi Mario

Thank you for your long and very informative explanation.

I recently read an article about the creepiness factor of face recognition by an authority on security. The article showed how once "they" (corporate entity) associate your name with a face, the corporation can search the entire internet for any of your photos all information associated with it.
This makes it easy to accumulate a huge database of information on people. Of course the information could be used for all sorts of nefarious purposes. The chances that such databases will be used to "improve our lives", or help social welfare etc are truly next to zero.

I went onto Facebook and removed all taggings of me, and adjusted the settings so no one can tag me either. It's probably too late and somewhere has a massive file about me.

Now with regard to face recognition in Imatch, it's unfortunate that the technology has been gobbled up by giant corporate monopolies (just like everything else) in order to monetize them. Such a tool would be very helpful in analyzing my photos when I'm writing articles - to be able to find all related photos of a particular person. Although I would never send out my photos with the photo tags imbedded, it would be useful indeed.

Using Picasa to do facial recognition seemed like a good idea until you informed us that Picasa uploads the data to Google servers which then keep the data. Very dark indeed. For me, this is good reason NOT to even install Picasa on my system. However, just for the sake of curiosity, will Picasa do the recognition and store the names in a XMP in specific places that IMatch could read and use? Do you know anything about Core's face recognition in PaintShop Pro. I will see if I can figure out if it can write XMP sidecars.

It seems strange indeed that there is not an open-source project to build facial-recognition software. But I do indeed find the commercialization/ commodification of our faces (what can be more personal?), and nearly everything about us, coupled with the ways that such data could be manipulated for poltical is an issue of serious concern. I am temporarily living in a regime that monitors everything and I find it stifling and, for lack of a better term, creepy and nefarious.

In the meantime, I'm still planning to convert all my cataloguing work to IMatch and hope that the Annotation feature will provide a good work-around for now.

[Mario]

As far as I know, Picasa uses the Google servers. I never worked with it. but that was my impression. I can be wrong, as often. But I personally don't trust Google or Facebook with personal data. What they do now and what they do in a year may be totally different things. All the big players use the "frog in the pot" principle. With every change of their "terms of service" they remove a small slice of your personal rights and protection, giving themselves more rights to use your data in more ways. And many people don't even recognize what this means for their privacy and personal lives.

will Picasa do the recognition and store the names in a XMP in specific places that IMatch could read and use?

Yes. The Metadata Working Group has introduced a set of Region tags which can be used for that purpose. To see these in IMatch use the Browser layout in the Metadata Panel, or add the MWG Region tags to one of your custom layouts. In the Tag Selector dialog, switch to Search all Tags and enter mwg in the group field to see these tags.

[jch2103]

Early versions of Picasa stored face recognition data in its internal database on your PC.  New versions can store the face region data in xml data inside your images/sidecars. You may have to adjust the Picasa settings to accomplish this and you may have to rescan images to get the xml data in your images if you were using the 'old' method.

I believe that face recognition data only gets sent to Google if you use Picasaweb, not if you only use the desktop product. But I haven't done any tests to prove it.

[medgeek]

I personally don't trust Google or Facebook with personal data. What they do now and what they do in a year may be totally different things. All the big players use the "frog in the pot" principle. With every change of their "terms of service" they remove a small slice of your personal rights and protection, giving themselves more rights to use your data in more ways. And many people don't even recognize what this means for their privacy and personal lives.

I completely agree with your cautions about personal information and I don't even have a Facebook account.  I like your analogy with the "frog in the pot" and gradual changes in terms of service over time.

[RalfC]

Personally, I think that Mario's careful approach to be wise and good.

I believe that face recognition data only gets sent to Google if you use Picasaweb, not if you only use the desktop product. But I haven't done any tests to prove it.

It starts to be off-topic but I just had a look at this and it seems that the automatic face recognition works only if you are signed-in with your Google account:

Add name tags in Picasa

Picasa's face-matching technology lets you organize your photos according to the people in them. Name tags are turned on by default, so the first time you open Picasa, it will scan your photos for new faces. This can take a while, but while it's running in the background you can start tagging them. Just follow these steps:

Make sure you're signed into your Google Account

(from https://support.google.com/picasa/answer/156272?hl=en&ref_topic=1689795)

Why would that be needed if the recognition is done locally?

I.e., if somebody enters your name to his Google contacts and tags your face on a photo Google has the info about you as well without that you agreed to it.
Taking into account that (at least some) states seem to want to know how and with whom people are communicating, it is for me hard to belief that there will be "only" the use for personalized & localized  ads (those companies like Google and Facebook have to comply with local laws in different states)

[mastodon]

what about digikam?

[Mario]

Does the software you link to provide facial recognition features? A bit more info would be helpful, especially since this is your first post.

[mgm1965]

I have found:

http://docs.opencv.org/modules/contrib/doc/facerec/facerec_api.html?highlight=face#FaceRecognizer : public Algorithm

FaceRecognizer
class FaceRecognizer : public Algorithm
All face recognition models in OpenCV are derived from the abstract base class FaceRecognizer, which provides a unified access to all face recongition algorithms in OpenCV.

If this can help.....

Marco

[Mario]

I know OpenCV this.
The face recognition is unfortunately much less reliable than anything we know from Google or Facebook. I tried it and keep an eye on changes made OpenCV...

[Frank]

Hi folks,
at the  moment using google Picasa is the best for tagging faces even google tries to upload the data to its server.
To prevent google for doing so!

1. Install with VMWare player (free) a new windows installation on your PC.
2. Install picasa on that VM machine
3. disallow network connection on that machine
4. allow usb plugin to that machine
5. start picasa and feel well

With this you will not be informed abount new versions of windows as well as picasa because there is no way for any software to make a connection HOME (software provider) 

Thats the way I work with some critical application.

Picasa wirtes the tagged faces into the picture and you can later work with the tagged faces in imatch 5

For imatch 5 it would help me if the face annotation feature could work together with the mwg face namespace. This means if I have tagged faces in the database These faces are imported into the annotation. If I tag a new face it also writes the new ace into the same namespace.
There is already a feature request out.

Currently I have over 800 different faces tagged with picasa which I group automatically within imatch 5 dynamic categories

Here are my 2 cent for that topic.

Frank

[CollieDog]

what about digikam?

Yes, Digikam has face recognition. Quite an impressive open-source project.
But how can I set it up to do face recognition on my photos that already have XMP sidecars? Will Digikam be able to read my currently existing XPM sidecars for each photo and add the face tags to the XMP without destroying or overwriting any of the other data within the XMP sidecar?

I'd be very curious to hear about other's experiences

[mastodon]

Sorry Mario for the short note. I thought you might know Digikam.
I've tried Digikam, and I found it very impressive, too. It's interface is much more usable than I-Match 3.6.
But I am using I-Match for seven years, and it is rock solid, it has been never ever stopped, failed or crashed. Digikam is not so reliable because it has too many bugs, I can't trust in it.
So I am desperately waiting for this new interface (version 5) and new features like face recognition. I found some info on Digikams site about its methode.

[Mario]

It looks like they have several people working on this, including some students doing research in facial recognition.
I won't be able to use their code because it's open source and, from what I gathered by a quick look, they are using a license which does not allow to use their code in a 'commercial' application like IMatch.

Why don't you use Picasa or digikam to fill in face data into XMP and then use it in IMatch? I don't think we need to re-invent the wheel in each application. Don't get me wrong, when I can find facial recognition code / library that I can just plug into IMatch I will do it. But as it is, there is little chance in the foreseeable future that I will spend weeks or months doing my own research in facial recognition technologies. Especially not since you can do it already in other applications and then use the data in IMatch right away. See also my comments about the annotation feature above.

I will work on feature requests once IMatch 5 is out of the door (which is my only priority at this time), and the more supporters a feature request has, the better. There are about 100 open feature requests now - which is good. Lots of cool things to add in monthly or bi-monthly updates after the initial IMatch 5 release.

[CollieDog]

Sorry Mario for the short note. I thought you might know Digikam.
I've tried Digikam, and I found it very impressive, too. It's interface is much more usable than I-Match 3.6.
But I am using I-Match for seven years, and it is rock solid, it has been never ever stopped, failed or crashed. Digikam is not so reliable because it has too many bugs, I can't trust in it.
So I am desperately waiting for this new interface (version 5) and new features like face recognition. I found some info on Digikams site about its methode.

Hi Mastodon

Have you used Digikam to do face recognition on RAW images, and then write the names to XMP sidecare files that can then be read by IMatch?
I'd really like to know your workflow.

Currently my photos are all in nested folders on an external drive. I use a folder structure like this.

2013 photos
     2013 01_Jan 14
     2013 02_Feb 11
     2013 07_July 1
     2013 08_Aug

In each folder are RAW images along with the XMP sidecar files created in Lightroom 5. The sidecars contain keywords and star ratings. I have used Lightroom 5 to do my metadata work and create my list of nested keywords and plan on replacing Lightroom with IMatch 5 when it is released.

Can I import these same folders with RAW images and their XMP sidecar files into Digikam, then run face-recognition, add the names of people, then have Digikam write this data into the XMP sidecars in a safe fashion that will not conflict with/erase the metadata in the XMP files (keywords and ratings) that were created in Lightroom?

Is there a way to do this in fashion that is reliable and safe? I would not want to damage the keyword metadata that I've assigned to each RAW in Lightroom 5.

[mastodon]

Sorry, I've just checked on about 100 JPGs, whether it works. Yes, it works good, but I did not compare to Picasa. I don't use Digikam, because I afraid of data loss.

[CollieDog]

Sorry, I've just checked on about 100 JPGs, whether it works. Yes, it works good, but I did not compare to Picasa. I don't use Digikam, because I afraid of data loss.

Thanks very much for checking! But can it do facial recognition on RAW files?

[Frank]

As I mentioned im an other Post. It is saver Not Work With RAW.
frank
Out of my mobile device

[mastodon]

I don'4 know, but I agree with Frank. Do not even try! (You might think at the beginning it works, but not in the end.)

[marco88]

Just want to say I would really love to have face recognition with auto categorization.

I now mainly take pictures of my kids so that would be massive time saver and really a big reason to buy IMatch 5.

I know there are lots of problems around that - read the thread - but just wanted to say how much it matters to people who use IMatch for classifying people/family photos.

PS: btw we could extend the request to auto categorization of any photos. Select a photo tag it select a bunch of photo select auto tagging and it goes work out which photos should have the same tags based on time / location / similarity.

PS: also I would love some basic editing but then it would become LightRoom so not going to happen.

Best a luck with the new version, I installed it it looks good it runs fast I like it already

Kind regards
Marc

[Frank]

Hi,
you can automatically Group previously tagged Pictures within imatch 5.
My naming convention is Mario_Westphal
So in imatch I usw dynamic categories.
M
.    Mario
.            Westphal
.                         Mario_Westphal
Just Tag it in picasa ans than let Imach so the Rest.
Frank

[Mario]

PS: btw we could extend the request to auto categorization of any photos. Select a photo tag it select a bunch of photo select auto tagging and it goes work out which photos should have the same tags based on time / location / similarity.

Like most automated schemata, this would only so far and you would have to weed out the false positives manually. Side note: Which is also true for face recognition, the reason for Facebook and Google trying to make "tag your friends" a socially accepted function. Only by the freely given manual labor of their millions of users they could create high-quality face databases they can use for their own purposes and sell to other companies and "services" for big money. By assigning a name to a photo on services like Facebook you are actually pointing a finger at that person and telling Facebook and the world "This is Frank, remember his face". I'm not sure that this is a good development, and George is probably rotating in his grave right now.

IMatch suggests keywords (your "tags") in the Keyword Panel, based on several criteria. Did you try that already? Just select a bunch of files and use the Recent, Favorites and Suggestion tabs to add keywords quickly to your files. This usually allows to assign keywords to large amounts of files very quickly. Fill your thesaurus as you go (you can even fill it from the keywords already in your files!) to work even faster. Clicking keywords is faster than typing and less error prone.

IMatch automatically groups your files by date and time in the Timeline View. To assign a keyword like "beach" to all files taken during your vacation, just select the corresponding week(s) in the time line, <Ctrl>+<A> to select all files in the file window and then add the keywords. Assign keywords to thousands of files in seconds. Also works for other metadata like titles, descriptions, photographer info, ...

You can organize files based on location, camera model, lens, ISO, artist and over 10,000 criteria automatically via data-driven categories.

The @Keyword category automatically groups your files based on the keywords (tags) you assign to them. Great for finding files quickly for further processing.

I could go on for a while like this but I trust you read the introductory topics in help and the time saver topic for a start. Should give you some ideas how you can make IMatch work fast for you.

PS: also I would love some basic editing but then it would become LightRoom so not going to happen.

Won't happen. I will not add any editing (except for what's in the Batch Processor). There are several or free or very cheap image editing applications available, all developed and backed by teams of programmers. From experience And most IMatch user have image editing applications anyway.

[marco88]

Thanks Mario I will read this section and will try to follow your suggestions.

I did setup a nice category tree for family and people photos but it is so time consuming to categorize.

So now I got several 1000s photos of my family to classify anything to automate would be great.

Cheers
Marc.

[Mario]

You can assign categories or keywords to any number of files at once.
IMatch automatically categorizes your files in many ways.
If you have standard face data in the XMP records of your files (e.g. added by Picasa or another spy app), you can use it for a data-driven category. May save a lot of work.