Storing AutoTagger Provider/Model/Date

[rvelices]

Hello.

This is a proposal to be able to store in a field information about the latest autotagging process ( service prover / model used / date time ).

As models will evolve, new services will be available, it would be helpful at some point to know how data was generated

Thank you

[sybersitizen]

Sure, that would be useful.

[Mario]

Fill a one or more custom AI trait tags when AutoTagger has finished a file, from a text provided by the user, with support for IMatch variables?

Add some AutoTagger-specific variables like:

{AutoTagger.AI}
{AutoTagger.ModelName}
{AutoTagger.PromptSource}   (the prompt as defined by the user)
{AutoTagger.PromptUsed} (the prompt sent to the AI, after processing variables)
{AutoTagger.Creativity}
{AutoTagger.Seed}

This would allow to record this kind of information in tags and to later search for the info, or group files based on AI, model, prompt etc.

[rvelices]

Yes Mario. This would be helpful and very flexible.
Thank you

[Mario]

Let's see how many likes your original request gets.

[mastodon]

As AI evolving very fast, that's very useful for workflow (making better prompt).

[onnod]

I wonder if a 'trial' or 'compare' mode might be useful. The ability to compare different prompts or models side by side (even if a one time option) could help users fine tune their workflow.
I am beginning to organize a 300k+ image library and it would be prohibitive to find out 50k dep that I actually prefer OpenAI to Mistral.

[PandDLong]

I think I do something similar for trialing photo editing techniques/products.   

I will duplicate a file within IMatch (sometimes several of them) and make them a version set.  I run each through its trials and make some notes in a couple of chosen tags on what I did and what was being trialed. If the the editing software produces a log I add that to the IMatch database and make it a version of the specific file for precise information.

I thought something like this would be a one-time event and the first time, I just made side notes on paper and then selected my go-to technique.  But, I found that sometimes I want to go back and check "why do I do it that way?";  "was it particular to the situation (eg. black and white vs. colour)?", "did I try out xyz?"etc.  My scraps of paper are long gone.

I have also found, I am performing comparison trials for new and different reasons and decided to go this route of saving the results through a version set.  

If these files hanging around in your database "get in the way" - you could always isolate them into a separate IMatch database for the trial or after the fact.


Hope this is helpful.

Michael

[Mario]

I wonder if a 'trial' or 'compare' mode might be useful. The ability to compare different prompts or models side by side (even if a one time option) could help users fine tune their workflow.

Unless you limit the AI very hard by setting a very low creativity value and a non-zero seed, you will get different results for the same prompt and the same image when you run AutoTagger twice. In most cases.

I doubt that a special feature is needed. You can always duplicate an image, and run AutoTagger with different prompts, same prompt and different creativity values, in all combinations you like. This will cost you a bit of money for the cloud-based AIs, but if it helps you produce better results, that money may be spent well.

Note that new models are released frequently, and that this probably invalidates previous "findings" for the older generation of the model.

In my opinion, overthinking all of this is not worth the effort. AI models, at this time, are by nature "unreliable" and "non-deterministic". Unless you use seed = <number>  creativity = 0, you will (very often) get different responses for the same prompt (and image). This makes "testing" quite impossible. And this is why I added <Alt>+<F7> to quickly re-run selected files to get new results.

Studying some of the prompt engineering blog posts and tutorials out there (for the AI and model you use) is also helpful if you want to coerce very specific results for your specific type of images.


I try not to overthink this.
I have a small set of prompts for different motive types. I select a bunch of files and run AutoTagger with the best prompt for the motive.  I use variables to add context and guide the AI when the images have useful content like location or person names (I use the trick shown in the help to resolve the variables to "nothing" when no data exists).

The results are usually OK for my purposes. I have a File Window layout which displays headline, description, keywords and the traits I use. I can quickly check the results and re-run the images where AutoTagger did not produce something I like.
Just select and <Alt>+<F7>. Or I manually edit the odd headline or description in the Metadata Panel.

I usually add a couple of manual keywords before I run AutoTagger, and then use these base keywords in the prompts.

I have now maybe processed 100,000 files over he last months (testing and for real) and AutoTagger beats writing descriptions and keywords by hand by a mile.

And I always keep in mind that all AIs out there sometimes hallucinate and produce nonsense. Nothing beats us humans yet in precision. But the AIs beaut us in speed and doing boring tasks for us.

[Mario]

I've pondered how to best implement this feature request (and other, similar requests voiced in the General Discussion board) in the past week.

I've came up with several ideas, like a special "No AI Prompt" which just sets a tag from a variable, to some "Extra" options for AutoTagger to set one or more tags from data...all workable but not satisfying.

On Wednesday I finally though: "Why not add a feature to AutoTagger to run a Metadata Template after processing a file?" - ant that was it. Because it is the most flexible solution.

This was part 1.
Part 2 is a set of new AutoTagger variables (IMAT.*) which, within the context of AutoTagger, allow the user to access information about the current AutoTagger run. For example:

{IMAT.Name} returns the name of the AI used ("OpenAI", "LM Studio", ...).
{IMAT.Model} returns the name of the model used ("Gemma 3 12B").
{IMAT.ResponseTime} returns the time in milliseconds the AI needed to process the request.
...
{IMAT.Prompt.Description} returns the prompt as provided by the user.
{IMAT.Prompt.Description.Parsed} returns the prompt after resolving embedded variables, which is what the AI actually saw
...
{IMAT.AI.Tokens.Input} returns the number of input tokens reported by the AI
{IMAT.AI.Tokens.Output} returns the number of output tokens reported by the AI
...

These variables only exist after AutoTagger has processed a file, but they can be accessed by the Metadata Panel configured for AutoTagger.

This means a user could use a variable like:

{IMAT.Name} with {IMAT.Model}
{Application.DateTime|format:YYYY-MM-DD}
{IMAT.ResponseTime|cast:int;math:div,1000}s
{IMAT.AI.Tokens.Input}, {IMAT.AI.Tokens.Output}

in the Metadata Template and store it in a standard tag like XMP-dc:Instructions (this ensures the data is also stored in the image) or in an AI tag, if the data should only exist in the database.

Furthermore, AutoTagger runs the Metadata Template after updating metadata in the database. This means that the metadata produced by AutoTagger is accessible in the Metadata Template.

Which allows for things like validating, copying into other tags, or, as asked for in this thread, restore Person keywords replaced by AutoTagger when the corresponding option was enabled.

I quite like this concept. It worked for the first time last night, and it is very useful indeed.
If and which Metadata Template is run is part of the AutoTagger setting, which means you can use different Metadata Templates for different settings.