IMatch 2018 Sneak Peak: Auto Tagger

[Mario]

I've explained about the integration of 3rd party AI / machine learning services into IMatch on a couple of occasions.

https://www.photools.com/5742/computer-vision-imatch-initial-results/

While working on the main track of IMatch 2018 / IMatch Anywhere™ 2018 I sometimes relay by working on some side projects.

One of these side projects is the IMatch Auto Tagger app which automatically adds titles, descriptions, keywords and location info to files. It does this by employing artificial intelligence services provided by companies like Google, Microsoft, Amazon, Clarifai and imagga.

I'm not sure yet if and in which form this app will make it into the end-user version of IMatch. Maybe this is one of the apps / features which are only useful and available for corporate and institutional users. Using the services does cost money, about 2 US$ per 1,000 files. So this is probably nothing that is interesting for normal users. But if you face the task to add titles and descriptions to 50,000 or 200,000 files, spending a few bucks for this may be well worth the days or even weeks of work saved.

Just for your information, this is the current state of the prototype (showing the results for the image with the yellow border):



It has many configuration options to control things like which information goes into which tag. Skip list and mapping lists for keywords. It can employ the IMatch thesaurus to automatically transform the flat "tags" delivered by the services into proper hierarchical keywords. Thresholds for returned data to automatically weed out "unlikely" keywords. Pretty useful stuff.

What would be ideal for all IMatch users would be if I could run the AI on my own server, and then provide the service for free or in a "pro" version of IMatch. There are some potential ways I could make that work, but I currently don't know enough about this and have nearly zero experience. And it would probably be impossible to achieve the same quality we get from specialized companies like Google or Microsoft with their massive computing resources and super-large training sets...

[mgm1965]

This is a feature I would pay for.

Marco

[ben]

This looks very interesting.
I wouldn't use it to tag landscapes and things for private purpose, though.
But if that's the right way towards face recognition one day, then I am very happy about it.

[Mario]

This looks very interesting.
I wouldn't use it to tag landscapes and things for private purpose, though.
But if that's the right way towards face recognition one day, then I am very happy about it.

Some of the services I support (Google, Amazon, Microsoft) offer face detection (does the image contain faces, and where). IMatch could use that to create face annotations and XMP face regions automatically.

Microsoft also has a pretty good emotion and age analysis (happy, sad, friendly, angry, laughing, male, female, age group, ...).

Microsoft and some offer also face recognition, but this requires to first build a database of known persons (in the vendor cloud) so it can later detect Paul or Emma in images. This works very well indeed.

It is technologically not very challenging to integrate this into IMatch and/or IMatch Anywhere. Thanks to IMWS and the App system and recent additions to photools.com I have now the required infrastructure and technology available. Which was one of the main tasks for IMatch generation 2018.

Two problems though: a) Privacy. b) Cost.

a) You have to trust the vendor that he does not do malicious things with the face data.
Most vendors claim that they don't store the actual face image in their database, but a mathematical fingerprint.

b) While things like automatic keywords or descriptions work on a "per use basis" (you pay, say, 2$ to automatically add keywords to 1000 files), face detection costs extra. Because all vendors require a monthly fee to store the face database of known persons required for face recognition. For example. Microsoft charges $0.25 per 1000 face records per month.

Assuming you know up to 1000 persons, this means that you have to pay 12 x 0.25$ = $3 per year to store the faces.
And when you do face recognition for 1000 faces, you pay $2 for that.

As you can see, it's not that expensive. Especially considering the time you save.
But it is too expensive for me to pay for all IMatch users.

Finding a way to offset the cost will be the real challenge.
I have accounts with all major vendors (currently for testing purposes). When I allow a user to use the Auto Tagger app, and he processes 10,000 files, I get the bill for this ($ 20).

The AutoTagger can be used by every user with IMatch 2018. But only when he is authorized for it. When Auto Tagger is first run you have to authenticate with your IMatch email address and license key (same as used for the customer portal). If you have "credits" to use the selected service, AutoTagger allows you to process files until your credits are used up.

I will use this to test things with a selected group of users. To test not only how AutoTagger works but also the authentication and credit system.
If this works, it could be the base for a "pay for what you use" system for selected features in IMatch.

If a user has no use for Auto Tagger, Face Recognition or what else we will built on that, no problem. No action required.
If a user wants to use one of the extra services, he has either to buy credits or maybe use a "subscription" version of IMatch which comes with a certain amount of monthly credits included.

As I said, this is all very fuzzy. Currently it is more important for me to integrate these modern services, because they are beneficial for many  users. Especially when combined with all the data and features already available in IMatch. And, frankly. adding keywords to files can be a real bore. Better to let the machine do it, even if it costs you a few bucks per year...

[ben]

Very interesting, thanks

Not long ago I read an article about Google.
They (maybe only research) work on neural networks that you can train offline and only share the training results. This way you don't have to upload your images.

Regarding privacy, I think Facebook is currently demonstrating very well that you cannot trust them.
Microsoft is making money not ONLY with your private data. So personally I would trust them more.
But I guess everyone has to come to his/her own conclusion

[Mario]

Running a local instance of Google's Tensorflow framework or the similar systems provided by Facebook, Apple, Microsoft requires rather beefy machines or specific graphic card models with specific setups to speed things up. This would fall under the "I run my own AI for IMatch users on my server" I mentioned above.

Running Tensorflow etc. on normal PCs is not that easy. There is (sort of) Windows version, but all frameworks clearly favor Linux environments. By all means, it would be challenging to wrap this into something that could be shipped with / included in IMatch. But things in AI and ML change fast, get better, cheaper all the time. I'm looking into several avenues.

Currently, it's far easier to just use the cloud services. They work great, are reasonably fast and get better all the time. I currently support Google, Microsoft Azure, Amazon AWS, Clarifai and imagga. All have different pros and cons, and similar price ranges. Google is unique when it comes to detect places and landmarks (based on all the info users feed them for free by using Google search, maps and their cloud). Microsoft excels at gender, age and mood detection. Clarifai has specialized models for food, sport and wedding photographers, which produce partially superior results in their domains.

Time works in our favor. Maybe there are easy to use AI/ML components in a few months. Microsoft has mentioned that they may include AI/ML directly in Windows 10 at some point. Or one or more of the cloud vendors offer turn-key AI systems ("just add your model and do some training") which I can rent for an affordable price. I could then include that in second "subscription" version of IMatch.

The key point is that the technology stack I have now in place (from the embedded Chromium browser to IMWS to all the libraries I have written to make that accessible) is the best starting point to use all the new stuff. To make asset management easier and faster. Having all the cool features in IMatch while still staying in control of your files.

[Jingo]

I'd be happy to pay for this as well.. $2 per 1000 images is not that expensive.. and we could probably save even more by only using the service for singular images and then copy the tags and other metadata from one image to other similar images from a photo shoot.  So, if I take 5 photos of a house and barn - I would only use the service for one image and then copy the data over to the other 4 photos. 

[Mario]

and we could probably save even more by only using the service for singular images and then copy the tags and other metadata from one image to other similar images from a photo shoot.

The AutoTagger already has a feature for that. There is a button that applies the results for the currently focused file (in the app) to all selected files. Basically you select a group of files with the same 'motive', perform the lookup for one of the files and then click on "Apply All" to apply the results to all selected files. One lookup, 1 to many files tagged.

[pmbvw]

and we could probably save even more by only using the service for singular images and then copy the tags and other metadata from one image to other similar images from a photo shoot.

The AutoTagger already has a feature for that. There is a button that applies the results for the currently focused file (in the app) to all selected files. Basically you select a group of files with the same 'motive', perform the lookup for one of the files and then click on "Apply All" to apply the results to all selected files. One lookup, 1 to many files tagged.

Mario, as always thought great ahead. Query once and using several times in IMATCH makes the service cheap enough to use it.

[mastodon]

Yes, I like the method using AI for tagging. Yes, I would pay for that services. Go, Mario!

[Jingo]

and we could probably save even more by only using the service for singular images and then copy the tags and other metadata from one image to other similar images from a photo shoot.

The AutoTagger already has a feature for that. There is a button that applies the results for the currently focused file (in the app) to all selected files. Basically you select a group of files with the same 'motive', perform the lookup for one of the files and then click on "Apply All" to apply the results to all selected files. One lookup, 1 to many files tagged.

Mario, as always thought great ahead. Query once and using several times in IMATCH makes the service cheap enough to use it.

The best just keeps getting better... :-)

[Mario]

I hope so!
The release notes for IMatch 2018 are a mile long (not publicly visible at this time)... 

Combining the intelligence of AI services with the smarts of IMatch is the trick...
Getting keywords from the AI is nice. But combining that with some flexible filtering, mapping and the IMatch keyword thesaurus makes this really useful.

[StanRohrer]

As a photographer that sells photos via stock photo web sites, tagging images with keywords is a significant effort in my workflow. Having an automated first attempt at keywording could help me to create tags that are relevant to the image as well as relevant to the industry. If AI tagging works reasonably well I'm all excited to give it a run. I recently did 4 days at a remote destination and came back with 2600 photos to process. Photo sales in these days are not terribly lucrative (I make 33 cents per image per year in one sales library and $2 in another - USA). So the cost of AI keywording must be very reasonable - albeit process one and copy to 100 similars is a viable strategy. I am highly interested in an Auto Tagger.

[Mario]

Thanks for sharing info about the workflow issues you need to solve. This is always helpful, because it gives me ideas and may lead to new features in IMatch.

I'm willing (and, hopefully, able) to make all this AI magic work with a few mouse clicks, allowing users to tag and describe files faster and with less effort. The real challenge will be to setup a way to recoup the cost and to find a fair and understandable license model.

One thought would be to write the Auto Tagger so users can enter their own credentials for whatever service they choose (similar to Google Maps or Bing Maps now).
This would allow users to use the free monthly credits all services offer.

But, problem is that the billing models and "developer dashboards" provided by the vendors are not aimed and "normal" people but at experienced IT folks.
The license management, billing requirements, API and key technical stuff vary between all vendors and change frequently. It would be much easier for users to let IMatch/me take care for all that and just "use" the stuff.

Especially if multiple services need to be combined.

For example, some services can detect landmarks, buildings, tourist spots (like in my example above). And in many cases they not only name the place but also provide GPS coordinates. Which IMatch could feed into a reverse geo-coding service to automatically fill in country, state, city and location in the Auto Tagger. But that's two services already.

Or, some services only offer keywords in one language (English).
Autp Tagger could send the returned keywords to an AI translation service and then apply German keywords to your files. Or maybe both.
But that's again a 2nd service to use, with separate billing and license key management.

Much easier for the user to let IMatch take care for it all, pay a few bucks extra and just use the "pro" features.

[mastodon]

Mario, what if I use different AI engines and they offer different info ex. GPS data/landmark. Is it possible to handle this data collisions in an easy way? (Detect collision and choose appropriate date)

[Mario]

You cannot use multiple services at the same time. This would complicate things a lot and you would get different sets of data for the same image, not better sets. At much higher cost.

[Lord_Helmchen]

What about your experiments to include Excire? Have you buried them?

[Mario]

What about your experiments to include Excire? Have you buried them?

I have currently no plans to integrate Excire into IMatch.

[kkiel]

I would happily pay for a feature like that, too, since it would save me hours of work. It should, however, work for (historical) illustrations as well, not for photographs only.

[Mario]

It should, however, work for (historical) illustrations as well, not for photographs only.

Just try it out. Google, Microsoft etc. allow you to upload a file and then show you the keywords for it.

https://cloud.google.com/vision/

https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/

imagga, Clarifai and AWS have most likely something similar. I just don't have the time to Google for you. Check them out as well. This will show you quickly if their AIs can do anything with your images.

[Carlo Didier]

I'm interested in the Auto Tagger, but instead of adding keywords, I'd like to have it find corresponding categories in my category tree (I'm not working with keywords at all).

[Mario]

I'm interested in the Auto Tagger, but instead of adding keywords, I'd like to have it find corresponding categories in my category tree (I'm not working with keywords at all).

Such a thing could surely be added as an option. If the AutoTagger comes to live, I'm sure I'll accept feature requests.

[Tveloso]

Such a thing could surely be added as an option. If the AutoTagger comes to live, I'm sure I'll accept feature requests.

Mario, I see that the ScreenShot for #00515 in the Release Notes for IMatch 2018, includes A.I. Vision and AutoTagger Apps...does this mean that AutoTagger will in fact be coming alive in 2018.7.2?

[Mario]

Well spotted!
No, sorry. I just did a screen shot of my system without removing the 'under development' apps first.
I have updated the screen shot to show only apps I plan to include.

In order to ship the AutoTagger I need to finish it, then setup a test group and make my mind up for how to manage the usage fees Google etc. requires for using their cloud services.
When I have figured all that out I can include the AutoTagger in IMatch and users who want to use it will have a way to enable it and pay for what they use.

[Tveloso]

Thank you Mario.

One thought would be to write the Auto Tagger so users can enter their own credentials for whatever service they choose (similar to Google Maps or Bing Maps now).
This would allow users to use the free monthly credits all services offer.

But, problem is that the billing models and "developer dashboards" provided by the vendors are not aimed and "normal" people but at experienced IT folks.
The license management, billing requirements, API and key technical stuff vary between all vendors and change frequently. It would be much easier for users to let IMatch/me take care for all that and just "use" the stuff.

So you have decided on the latter approach then, correct?...would it still be feasible at this point to include configuration options for both methods?

In any event, I'm very excited about your integration of AI Vision Technology into IMatch, and am very much looking forward to IMatch 2018...and to the eventual arrival of the AutoTagger App for it.

Thank you so much for the incredibly robust system that is IMatch!

[Mario]

I have not yet decided on the integration method to use.

I think that dealing directly with the cloud vendors is too complicated for most users.
Just buying credits from me and then using the AI features until the credit is used up is probably the safest and easiest solution for everyone.

[Lord_Helmchen]

What about your experiments to include Excire? Have you buried them?

I have currently no plans to integrate Excire into IMatch.

That's really bad news. 

[Mario]

What about your experiments to include Excire? Have you buried them?

I have currently no plans to integrate Excire into IMatch.

That's really bad news. 

You can use it in Lightroom. The keywords you create in Lr will be usable in IMatch too.