Question about "Merge XMP keywords"

[banzai]

Hi there,

I have recently activated the option "Merge XMP keywords" for my version relation. Now I experience an unexpected doubling of keywords in my JPGs:
This screenshot shows the situation before propagting, only the master has been assigned some keywords:


This screenshot shows the situation after writing back metadata for the master and the two versions:


Only the version relation for the JPG has this option activated, not the relation for the DNGs (if I activate it also for this second relation, the same thing happens for the DNG).

Now my question is if this behavior is to be expected and I just don't understand this option correctly or if this is a bug?

Ciao,   banzai

[Mario]

ExifTool should avoid this duplication, at least that's what IMatch is telling it to do.
Do your files contain flat keywords? Hierarchical keyword? Legacy IPTC keywords? Different keywords in these tags?
What exactly do you relocate in your relation (what do you have checked)?
In my tests this has never happened, so I wonder. More details required.

[banzai]

Do your files contain flat keywords? Hierarchical keyword?

For this test, I first deleted all keywords from all three files.
Then I added "aa" and "bb" to the master, which was then in the hierarchical keywords and also in the flat keywords.
After propagating, the duplication only happens in the hierarchical keywords (containing "aa; bb; aa; bb"), the flat keywords are OK (only "aa; bb").

Legacy IPTC keywords?

Uhm, not really sure, not an expert for that. When I switch to metadata layout "4 IPTC Metadata", I can see only values in Location related tags (City,State,Country, GPS, etc.)

What exactly do you relocate in your relation (what do you have checked)?

- Categories
- Annotations
- Attributes
- Rating
- Label
- XMP without Camera Raw data
- Don't copy XMP orientation
- Merge XMP keywords

Let me know if you need more info, like output from Metadata analyst or whatever

[Mario]

I could so far not reproduce this using your exact replication settings.

Neither flat keywords (top-level) nor hierarchical keywords are duplicated when writing back the master or manually  propagating using F4,P.

When I add some of the keywords assigned to the master also to a version and then write-back the master, only the keywords not already in the version are assigned. No duplication.

I've used IMatch 2023.1.18 for my test.

I used a File Window layout that lists all keywords to make this easier to test and track.

Any additional idea, steps etc. that allows me to reproduce this?
Are all files in your database affected by this or only some?

I've tried with the classic "RAW Master", "JPG and TIFF Version" but also with a fully manual version where I've just versioned some JPEG files. It works correct in both scenarios.

[banzai]

Oh, sorry, nearly missed this post.

I've used IMatch 2023.1.18 for my test.

Same here.

Are all files in your database affected by this or only some?

I used some other files today (.CR3 as master and .DNG as version), same result, so I don't think it's connected to the files (ofc I just tested some files, so who knows...)

Any additional idea, steps etc. that allows me to reproduce this?

Not really, no idea why the same settings are no problem when you try it.
But I noticed something that might help:

If I use just
"XMP Keywords (Merge)"
instead of the combination
"XMP without Camera Raw data" and "{+} Merge XMP keywords"
then the problem does NOT happen. But I don't want only keywords, so this doesn't help yet.

But also the combination of all three options does work for me, no duplication in this case.
This would be a viable workaround for me, assuming that still all XMP stuff is propagated.
Ofc I wouldn't mind if you manage to find/fix this

PS: the curly brackets above should be square brackets, but that always turned into some list formatting

[banzai]

But also the combination of all three options does work for me

Oh, I just noticed I cannot really select all three at the same time. When I select either "XMP Keywords (Merge)" or "{+} Merge XMP Keywords", the other one gets deselected.
Didn't notice that earlier.

So the workaround is the combination of
"XMP without Camera Raw data"
and
"XMP Keywords (Merge)".

[banzai]

Sorry, I have to correct myself once again.
Now that I wanted to continue my workflow with this workaround settings, it turns out that keywords, that I have only assigned to a version, are overwritten on metdata writeback of the master, so no merge happens...

[Mario]

I tried this again today.
Propagation:

XMP without camera raw data
Merge Keywords

Used a RAW master (M), a JPG (J) and TIFF version (T).

1. Test:
- add keywords to M.
- J already has some keywords, none of them assigned to the master.
- T has no keywords.

Write back.

M has all keywords.
J has the same keywords as master, plus the keywords it had before
T has the same keywords as the master.

Mark the master as modified and write back again. Same result.

2. Test
Add two new keywords to T.
Remove some master keywords from J
Add a new keyword to M

Write back.

M has all keywords, including the newly added one.
J has the same keywords as M again, plus its own keywords
T has the same keywords as M plus the two keywords added in Test 2.

Mark the master as modified and write back again. Same result.

I've also tried "XMP without camera raw / LR" and "XMP All Data" in combination with "+Merge Keywords".
It seems this something specific with your database (?) or other settings.
Do you use any non-standard Metadata 1 / Metadata 2 settings?
Protection of some metadata?

No idea at the moment what else I can do to reproduce this here.
No similar reports from other users either.
If you check Help menu > About, the ExifTool version must be 12.64.

[banzai]

Sigh, I'm really at a loss here. No idea why this is only happening for me.

I created a new test database, imported one folder with CR3 files and one DNG created out of the first CR3, same result, duplication happens. So it seems it is not a database related setting (I kept default settings as far as possible, I only activated the CR3 version rule that comes by default and activated the propagation).

After that I installed IMatch 23.1.20 on my laptop where IMatch was not installed and did the same, used the same folder (a copy of it), same default version rule activated, changed only the propagation settings. Still got the duplication.

One probably very stupid question though: You are looking at "hierarchical keywords", correct? I noticed that I never mentioned this, because I never care about the flat keywords and only work with hierarchical keywords.
But in the flat keywords, the duplication is not happening, only in the hierarchical keywords.
Would it make sense to provide the output of exiftool when propagating? For me the output looked inconspicuous, but perhaps you could would notice something..

[Mario]

Yes, propagating keywords means hierarchical keywords (and flat keywords, in the end).
Ive tried with deep hierarchical keywords and simple "a", "b", "c" keywords.

Do you maybe have a "cycle" in your relation rules, where a version is also a master and then propagates to another version that is also propagated to be the versions master?

If this does not depend on the files or database, I should be able to reproduce immediately. But so far I could not.
Not sure what's going on.

Maybe prepare a database which holds only one folder with two files.
If this database allows you to produce the problem, send me the database and the two files.
Maybe this allows me to reproduce this and fix it, if it can be fixed.
Also send me your IMatch settings database "C:\ProgramData\photools.com\IMatch6\config\imatch.pts".
This allows me to reproduce your environment.

ExifTool has recently got a "no dupes" feature, which is what IMatch is using to avoid duplication.
Maybe I'm using it wrong or something.

[sinus]

I tried also to reproduce this, without success.
And the only thing, what I had in mind also, that maybe another rule is involved?
I have only one rule, hence it is simple.

[banzai]

I don't think it's a rule cycle, I have no rule that declares a .DNG file as master.
And I can reproduce this with a fresh database where I have only enabled the default provided version rule for .CR3 masters, all other default rules are still disabled.

I will play around with exiftool on the commandline and try to analyse the output from the exiftool output panel, perhaps this gives me an idea of what specialty I have in my setup...

[banzai]

I had a look at the exiftool commands that are run when I write back metadata. This is the command that is run for propagation to the .DNG file:

Code Select Expand

  ----  Propagate:  ----
-overwrite_original_in_place

-charset
FILENAME=UTF8
-xmp:all=
-tagsfromfile
@
-xmp-dc:subject
-xmp-lr:hierarchicalSubject

-tagsfromfile
C:\Users\bka\h\Bilder\bk\pics\incoming\2023-07-21\Pic_2023_07_21_1988.CR3
-xmp:all
--xmp-crs:all
--xmp-lr:hierarchicalSubject
--xmp-dc:subject
-xmp-lr:hierarchicalSubject+=aa
-xmp-dc:subject+=aa
-xmp-lr:hierarchicalSubject+=bb
-xmp-dc:subject+=bb
-m
C:\Users\bka\h\Bilder\bk\pics\incoming\2023-07-21\Pic_2023_07_21_1988.dng
-execute
-overwrite_original_in_place

-charset
FILENAME=UTF8

-tagsfromfile
C:\Users\bka\h\Bilder\bk\pics\incoming\2023-07-21\Pic_2023_07_21_1988.xmp
-xmp:all
--xmp-crs:all
--xmp-lr:hierarchicalSubject
--xmp-dc:subject
-xmp-lr:hierarchicalSubject+=aa
-xmp-dc:subject+=aa
-xmp-lr:hierarchicalSubject+=bb
-xmp-dc:subject+=bb
-m
C:\Users\bka\h\Bilder\bk\pics\incoming\2023-07-21\Pic_2023_07_21_1988.dng
-execute

-execute9999
    1 image files updated

    1 image files updated
The first command (all up to first -execute) copies tags from the master (except keywords) and manually adds the keywords of the master to the version. Affter that command, my version file looks completely fine.

But the second execute step now copies from the master's XMP sidecar (again except keywords), and again manually adds the keywords of the master to the version. This seems to be where my duplication is coming from.

In now wonder how this normally works (apparently in other environments the commands are not run like for me) and what setting in IMatch might have an influence on how this exiftool command line is built?

[Mario]

Does your CR3 file contain an embedded XMP or legacy IPTC record?
It should not contain XMP. For RAW, XMP should only be in the sidecar file.

[banzai]

When I run 

& 'C:\Program Files\photools.com\imatch6\exiftool.exe' -a -G1 -s .\Pic_2023_07_21_1988.cr3

and filter for XMP, all I see is this:

[XMP-xmp]      Rating                          : 0

But in IMatch settings, I have set "Ignore rating of embedded XMP record" to yes.
And this is how the image got out of the camera, I have not assigned a rating yet.

[banzai]

And as additional info: In my older .NEF file, where I first experienced this problem, I find this with XMP in the NEF file:

[XMP-x]        XMPToolkit                      : Public XMP Toolkit Core 3.5
[XMP-microsoft] RatingPercent                  : 0

[Mario]

Many cameras write, for no reason, a rudimentary/incomplete XMP record with a hard-coded rating = 0.
Which causes all kinds of issues, which is why the setting in IMatch exists to ignore that.

IMatch copies the tags from the file / xmp, except keywords:

Code Select Expand

--xmp-lr:hierarchicalSubject
--xmp-dc:subject
and then adds the keywords stored in the database one by one.
ExifTool should de-dupe these automatically.

Send me the database and the test files. I will look into this when there is time.

[banzai]

I have sent a mail to the support email address with all necessary info.
I'm not sure you need my database for reproduction cause I just used a freshly created one with nearly everything set to default, but you got it now.

ExifTool should de-dupe these automatically.

I'm not so sure about this when I read this and the linked FAQ entry:
https://exiftool.org/forum/index.php?topic=8092.0

It suggests to use "-=" in combination with "+=".

[Mario]

This problem has been resolved for IMatch 2023.1.22.

The duplication only occurred when there was a master with an XMP side car file.
When the "Merge Keywords" option is enabled, IMatch cannot longer use ExifTool for propagation alone - it has to propagate keywords in a separate step "manually" to avoid duplicate keywords.
This step sometimes failed to recognize that the current propagation involved multiple source files (the master ant its XMP sidecar file) and in these cases added the to propagate keywords two times.

[banzai]

Today I finally found the time to continue to work on my photos (updated to .22)

And I can confirm that the duplication no longer happens. That's Good.
But unfortunately, the merge no longer happens.

Master: aa; bb
Version: xx

After writeback:
Master: aa; bb
Version: aa; bb

Before the fix after writeback the version had "aa; bb; aa; bb; xx".

[Mario]

Works here just fine, even using your CR3+XMP and DNG file.
Also tried manual relations and PSD->TIFF/JPG version sets.

Master m1,m2
Version v1,v2
After writing back the master:
Version m1,m2,v1,v2

My rule propagates rating, label, XMP w/o Lr + camera raw data, don't copy XMP orientation,  + Merge Keywords.

[rolandgifford]

My rule propagates rating, label, XMP w/o Lr + camera raw data, don't copy XMP orientation,  + Merge Keywords.

An aside for clarification in my own mind.

If I specify 'Merge Keywords' and I delete a keyword from the master, this deletion won't be propagated?

[Mario]

If I specify 'Merge Keywords' and I delete a keyword from the master, this deletion won't be propagated?

Not with merge, which means "merge keywords from master into version".

[banzai]

My rule propagates rating, label, XMP w/o Lr + camera raw data, don't copy XMP orientation,  + Merge Keywords.

I tested this and this combination indeed works.
But my original setting was "XMP without Camera Raw", not "XMP w/o LR and CR". And my original combination does not work (no merge).
I thought hierarchical keywords are part of LR specific XMP metadata, therefore I wanted to include it.

But your combination of propagation settings seem to work for me (although I would have expected that hierarchical keywords are not part of the set of propagated metadata), so it's a viable workaround for me.
Just wanted to let you know that there still is a combination which does not work (as far as I understand these settings).

[Mario]

Hierarchical Keywords are part of the XMP Lightroom metadata and hence when you copy this entire namespace, hierarchical keywords are copied, basically overriding the merge option. Which comes "too late".

Yes. Metadata is complex. And not even IMatch or I can deal with every possible combination of every possible set of metadata and workflow.

[Mario]

I have thought about this specific problem.
Combining a set of tags which copies all XMP data (or all XMP data sans Crop/Camera RAW Settings) in combination with "merge keywords".

The problem in this case is that dc:subject (flat keywords) and lr:hierarchicalSubject are part of the XMP data that is copied in this situation. Adding a merge keywords on top does nothing, since the keywords have already been copied by ExifTool when the "merge" step is applied.

To deal with this particular combination of settings, I've implemented a special treatment. Yes. Another one. On top of the already mind-boggling complexity of propagating metadata between files.

Note to users: Stick to the simple tag-wise propagation right at the top of the list of options. Copying entire tag groups like All XMP, All XMP without CRS etc. is for experts only and should be used with great care. XMP contains a lot of EXIF metadata copies, from image dimensions to orientation to color-related stuff. Copying that between files is usually not a good idea.
I shall add another red warning box to the corresponding help topic. Sometimes users copy large sets of metadata between master and versions without truly understanding the outcome. Metadata propagation is an advanced IMatch feature and copying entire tag groups is even more so only designed for expert users.

My recent addition was to extend the configuration file IMatch uses to implement and drive the propagation to add options to apply when the "Merge Keywords" add-on option is enabled.

IMatch now performs a quick check to see if the "Merge Keywords" option is enabled before it builds the command list for ExifTool. If this is the case, it injects additional commands declared by me in the configuration file to prevent dc:subject and lr:hierarchicalSubject contents to be propagated. This step is then delayed to the "Merge Metadata" step which deals with copying keywords from the master to versions and de-duplication.