IMatch Machine Intelligence....?

[Mario]

I mentioned this in passing in the other threads discussing the Auto Tagger and potential uses for machine intelligence in IMatch (see https://www.photools.com/community/index.php?topic=8053 for example) that I was working on ways to make machine intelligence useful for IMatch users. For things like automatically adding keywords to files, maybe descriptions and face detection / recognition.

As I demonstrated in my tests regarding auto tagging (See the blog post Computer Vision in IMatch) the major players in the field produce very good results. With two (minor) issues:

1. It costs money. (Analyzing 1000 images costs about 2US$).

2. Privacy.
Most vendors operate servers in the US (or elsewhere) which makes uploading images to them legally complicated (even more so when your images show persons and you live in Europe and thus have to adhere to the GDPR). And I think many users just don't want to upload their images to a company cloud somewhere.

An alternative to these professional services would be Mario inventing an AI for IMatch. Just kidding.

This cannot happen because AI technology is a very research-intensive field, complex and costly. 'Training' models for image classification can take hundreds (!) of CPU hours, on specialized computers. Nothing you do on your notebook while browsing the web

However, as a stress relief (pushing out a massive update like IMatch 2018 generates a lot of stress, I can tell ya) I've worked a bit on my side project based on the Tensorflow AI framework. Tensorflow is  an all-purpose open source framework for machine learning. Initially developed by Google it is now used by many major companies and researchers to advance the field of machine learning. You can even rent servers designed to run Tensorflow very fast from all the big cloud vendors.

There is a lot to learn and I'm maybe at 5%.
Tensorflow is complicated to install, requires a specific runtime environment and specific graphic cards for optimal performance, etc. I don't want to bore you with technical details.

Before I spend time to figure out if and how I could make that technology easily accessible to IMatch users, I wanted to get an impression of what is possible. How fast it is etc.

There are open source projects (and pre-trained models) which show how to do image analysis and face recognition in Tensorflow.
Using one of these projects I hacked a small IMatch app which uses one of the existing image analysis models and runs it with Tensorflow to analyze images in an IMatch database.

The results are encouraging and, at a first inspection, quite often as good as the results produced by the specialized AI vendors. For other images the results are way off or even comically wrong.

This all stands and falls with the "model" used and how good it has been trained.
Training a model requires hundreds of thousands of images which already have keywords so the system can learn which images get which keywords. And training a model can require thousands of CPU hours (a computer running for thousands of hours).

Learning more about this will be the next step (when I find the time, after IMatch Anywhere 2018 is out).

This is how my crappy test app looks right now.
I ran a test with Tensorflow and files currently selected in the file window.
Note: This all runs LOCALLY, on my PC. No 3rd party cloud service needed, files stay on my PC.

[Jingo]

I appreciate your efforts in trying to bring SOME type of AI keywording into IM.... that would be a HUGE marketing and sales point regardless of the methods that are ultimately available.  I know most hardcore users are not thrilled with sharing their images/data "in the cloud"... but honestly - 90% of folks would probably not blink about it and do so currently with facebook and other social media and other backup methods because as you mentioned,.. it's free!

Looking forward to more info about these options as you make progress... and as always, please let us know if we can help with testing or whatnot!

[Mario]

Just checking the available options...

Opposite to free Google or Facebook services, AI-based image analysis in the cloud are not free.
Which is probably more of a "no-go" for many users than the potential privacy issues 

[ubacher]

I assume images with persons on it are of most privacy concern. If we could get at least local face recognition
we would be a big step forward.

My most frequent question would be: show me all pictures I have where this person (sample image(s)) is shown.

[Jingo]

I assume images with persons on it are of most privacy concern. If we could get at least local face recognition
we would be a big step forward.

My most frequent question would be: show me all pictures I have where this person (sample image(s)) is shown.

Would be nice to have I suppose if it were accurate.. of course, growing kids would need to be constantly updated... Since I need to apply other keywords to the photos anyway, its pretty easy to just include this info via the keyword panel.  Now - the auto keyword possibilities Mario has been reviewing... that has me waiting patiently!!

[Mario]

Face recognition is never 100% accurate, not even the best systems. You usually need full-frontal faces or only slightly at an angle.
And if the system is not 100% accurate you have to review the images manually afterwards anyway (except its OK for you to miss some files or have some files with wrong labels).
And if you have to review the photos anyway, you can just add the "person" keyword yourself, fairly quick and always 100% right.

The key factor behind all the big vendors investing a lot of money in face recognition is to automatically apply face recog on millions of images found on the web and in social networks. Tracking persons that way and determining "who knows who" generates most precious data for companies like Facebook or Google. The same reasons apply for developing technologies to automatically classify images (adding keywords) and detecting known landmarks. Because if they can identify persons on photos uploaded to the web or FB and they can also determine where the photo was taken, they know who you are and where you have been. This data is pure gold when it comes to creating targeted ads or profiles. And thus for making money.

For these uses it's perfectly OK to be right most of the time, with a certain percentage of misses. For the billions of images to process this is the only way to go.

For normal people like us, well, we don't want to have images of 'aunt Emma' accidentally labeled with 'Paul'. Or the keyword 'cheese' assigned to an image of a yellow dust bin.

While the AI stuff is surely exciting and can help users who face the task of somehow organizing 50,000 or 200,000 files, its still far from perfect and requires manual review afterwards. And when you setup IMatch right for your workflow (thesaurus, favorites, splashers, ...) you can assign keywords to large volumes of files very efficiently. And 100% correct.
The features for automatically detecting faces in IMatch are pretty good, and then you can quickly add a face tag with <F2>. Several hundred files per hour no problem.

[ubacher]

It is clear that AI tools can not completely eliminate manual work but it should simplify it.
If I have 10000 photos to go thru to find images of John Doe then, doing it manually is very labour intensive.
If Face recognition can present me with a few hundred potential images a lot is gained.
If one can select the certainty-factor one can trade off having many potential images presented vs.
fewer misses.

I had played a bit with Picasa's face recognition (off line) and found it quite good. The "growing kids" Jingo mentioned
did not seem a big problem at all.
It is clear that the accuracy of detection depends on the number of samples one has of the target person. In Picasa it
seemed to have used the images in which you confirmed a person as additional samples, so that recognition accuracy for the
target person quickly improved.

One of the features in Picasa's face recognition that caused annoyance  was that it wanted me to name each and every person it found!
There was no way to "ignore for now".

[Mario]

Use Picasa and make it write the face regions and tags to XMP. IMatch can then use them.

See the knowledge base article Using Google Picasa™ Face Tags in IMatch.

If I have 10000 photos to go thru to find images of John Doe then, doing it manually is very labour intensive.
If Face recognition can present me with a few hundred potential images a lot is gained.

But you will never know how many images of John Doe were missed because the face recog did not detect the face. Or detected it wrong.
Some can live with that, some don't.
I tag people while I process the files so my hit rate is always 100%.