How to propagate @Keywords?

[pajaro]

I have successfully set up versioning for my workflow. Most of my masters are jpgs (I rarely shoot raw), and versions are jpgs processed in Photoshop. So, creating version stacks works without any problem. But I cannot figure out how to propagate @Keywords to versions. I read the Help file and search the forum but cannot find the answer. While searching the forum I came across the old thread (https://www.photools.com/community/index.php?topic=234.msg1177#msg1177) where Mario says that propagation of @Keywords may be dangerous and that this option was removed from IMatch. Now, what can I do? The main reason why I am trying to propagate @Keywords is that I am processing in Photoshop the files that already have @Keywords assigned and thus creating versions. So, it would help me a lot if the processed files have automatically @Keywords assigned. But if there is a reasonable work-around (apart from manual @Keywords assignment to the versions) I can use that.

Thanks for any help.

Pavel.

[Ferdinand]

Keywords are stored in XMP, so if you propagate XMP you will also propagate Keywords.

What Mario was referring to was selecting @Keywords to propagate like any other category.  @Keywords may look like categories, and in many respect behave like categories, but in fact they're not normal categories.  They are in fact a special sort of data-driven category, and so to propagate them you must propagate the data that they're based on, which is XMP.

One thing you therefore can't easily do is to propagate only Keywords and not other XMP.  I have a script that solves that special case and one or two others, should it be needed. 

[pajaro]

Keywords are stored in XMP, so if you propagate XMP you will also propagate Keywords.

What Mario was referring to was selecting @Keywords to propagate like any other category.  @Keywords may look like categories, and in many respect behave like categories, but in fact they're not normal categories.  They are in fact a special sort of data-driven category, and so to propagate them you must propagate the data that they're based on, which is XMP.

One thing you therefore can't easily do is to propagate only Keywords and not other XMP.  I have a script that solves that special case and one or two others, should it be needed.

Thanks, Ferdinand, for the info. I also tried to propagate XMP, but it did not help...

The script would be perfect 

Thanks,

Pavel.

[Mario]

Keywords are stored in XMP (hierarchical in the lr:hierarchicalSubject and flat keywords in dc-Subject) so when you propagate XMP, keywords will under all circumstances propagated - unless you use the XMP with no LR option. The only way that this can fail is when you also propagate legacy IPTC data and you have also configured IMatch to not propagate XMP to legacy IPTC. Please give more details.

[pajaro]

Keywords are stored in XMP (hierarchical in the lr:hierarchicalSubject and flat keywords in dc-Subject) so when you propagate XMP, keywords will under all circumstances propagated - unless you use the XMP with no LR option. The only way that this can fail is when you also propagate legacy IPTC data and you have also configured IMatch to not propagate XMP to legacy IPTC. Please give more details.

Mario, actually keywords were propagated (I could see them in the Keywords panel), but @Keywords were not - I did not see them in the Categories panel. And this is what I want...

[Mario]

@Keywords is a data-driven category that is filled from the actual data in your files. So it should update automatically after the write-back.
Need more details. Is @Keywords shown as pending? Did you try to refresh it?

[Ferdinand]

Script is here.  Note that I haven't run it in a long time so use with caution:
https://www.photools.com/community/index.php?topic=3086.0

But you shouldn't need it.  What you want to do should just work.  I strongly suggest that you try to use the native functionality if at all possible.

[pajaro]

@Keywords is a data-driven category that is filled from the actual data in your files. So it should update automatically after the write-back.
Need more details. Is @Keywords shown as pending? Did you try to refresh it?

Mario and Ferdinand, thanks for your help. Turns out it's all my fault    In the latest IMatch version I excluded data-driven categories from the automatic update. So even though I rescaned the modified file (Shift-Ctrl-F5), I did not refresh @Keywords. After I did that, the assigned @Keywords appeared 

One question: I set keywords to be saved in xmp files; all my image files are read-only. Now, if I propagate with version files set as read-only I get an error message and keywords are not propagated. Once the image files are not read-only, the propagation works and keywords are written to xmp files. So, it seems to me that the read-only status does not really matter as the keywords are written to sidecar files anyway. Is this normal behavior?


Ferdinand: I tried your script but it failed with this highlighted row:

Database.Categories(dlg.targetcat).UnassignAll

I do not know what it means, but I think I am fine with the propagation implemented in IMatch.

Thanks again.

Pavel.

[Mario]

This depends on your file format, embedded XMP or not, if existing IPTC/EXIF/GPS data needs to be updated in the original file. We don't have sufficient info...which file formats are you using, which metadata settings, MWG compliance enabled etc...?

For example: If you propagate XMP, IMatch also updates existing IPTC/EXIF/GPS data in the original file (if enabled and the options are right).
If you make the original file r-o and it contains IPTC/EXIF/GPS data, you are creating a mess because you actively prevent IMatch from synchronizing metadata across metadata formats. And then you end up with messy out-of-synch metadata and wonder why this happens...

[pajaro]

This depends on your file format, embedded XMP or not, if existing IPTC/EXIF/GPS data needs to be updated in the original file. We don't have sufficient info...which file formats are you using, which metadata settings, MWG compliance enabled etc...?

For example: If you propagate XMP, IMatch also updates existing IPTC/EXIF/GPS data in the original file (if enabled and the options are right).
If you make the original file r-o and it contains IPTC/EXIF/GPS data, you are creating a mess because you actively prevent IMatch from synchronizing metadata across metadata formats. And then you end up with messy out-of-synch metadata and wonder why this happens...

Mario, a vast majority of my master files are jpgs and I do not use embedded XMP even though I know it is recommended by the MWG. I save all metadata in sidecar files, even for jpgs. I know it is a non-standard approach but I have my reasons for that. I attached screenshots of my metadata and metadata2 settings and configuration of selected file formats. Not sure if it is enough or if you need more info. But I do not want to waste your time. I can copy keywords to Photoshop-processed files using the Keywords Panel which is not as convenient as propagation but it is a reasonable work-around. So, if my problem would be difficult to solve, don't worry about it.

Thanks,

Pavel.

[attachment deleted by admin]

[Mario]

You did not include information about which metadata you propagate.
I assume you propagate JPG to JPG?
You force IMatch to maintain XMP for JPEG files in sidecar files?
Does any XMP metadata get propagated to the versions?
Which XMP set to you propagate?

I've made a quick test with "XMP All Data" and some of the other XMP propagation sets (except XMP No LR, which skips keywords) and it works.

Do this:

1. Open the ExifTool output panel
2. Change a keyword for one of your master files
3. Write back the master file

Copy/Paste the text in the ET output panel into a text file and attach. This shows us what IMatch / ExifTool are writing to which files.

[Ferdinand]

Re the script, I think I can see the problem.  Silly me.  I'll fix it when I have time or when someone needs it, whichever comes first.

[pajaro]

You did not include information about which metadata you propagate.
I assume you propagate JPG to JPG?
You force IMatch to maintain XMP for JPEG files in sidecar files?
Does any XMP metadata get propagated to the versions?
Which XMP set to you propagate?

I've made a quick test with "XMP All Data" and some of the other XMP propagation sets (except XMP No LR, which skips keywords) and it works.

Do this:

1. Open the ExifTool output panel
2. Change a keyword for one of your master files
3. Write back the master file

Copy/Paste the text in the ET output panel into a text file and attach. This shows us what IMatch / ExifTool are writing to which files.

Mario, attached is the file with the ExifTool output.

Regarding your questions: yes, I do propagate jpg to jpg, I force IMatch to maintain xmp in sidecar files, some of the xmp metadata definitely gets propagated (e.g. xmp exif, but due to the amount of data that shows in the Metadata Panel I cannot find an easy way how to compare masters and versions) and I selected to propagate all xmp data.


Thanks,

Pavel.

[attachment deleted by admin]

[Mario]

Output looks good, tons of keywords are written to

...yDSC-RX100-vlaky\DSC00133.xmp

Since you use non-standard settings, I suspect a side effect of that. Make sure that the JPEG file associated with the XMP has no IPTC data or disable all MWG-compliant processing in IMatch to avoid unexpected side effects of your forced non-standard handing.

[pajaro]

Make sure that the JPEG file associated with the XMP has no IPTC data or disable all MWG-compliant processing in IMatch to avoid unexpected side effects of your forced non-standard handing.

Ok, thanks a lot. I'll try that and will let you know if it helps.

Pavel.

[Mario]

Please understand that I support non-compliant and non-standard workflows like yours only up to a point. It's just too time consuming to try to figure out what side effects such particular workflows cause in conjunction with the dozens of involved options and settings.

[pajaro]

Please understand that I support non-compliant and non-standard workflows like yours only up to a point. It's just too time consuming to try to figure out what side effects such particular workflows cause in conjunction with the dozens of involved options and settings.

Yes, I 100% understand and as I said before, I do not want you to waste your time. I am happy with the way copying keywords works now, propagation would just make things easier...

I know that I use a non-standard workflow, but on the other hand I find the practice of storing xmp into jpg very inconvenient in my particular case. My database in constantly developing and I often change keyword assignment which results in write-backs to hundreds or thousands of files. If I would let IMatch to write the keywords to jpgs, a large amount of data would have to be saved every time I change the keywords (this also includes my backups). With sidecar files things are much easier as they are obviously substantially smaller than jpgs. But I understand that as a software developer you have to comply with industrial standards...

I appreciate your help, you saved me several times in the past 

IMatch is a perfect piece of software and  I enjoy working with it.

Pavel.

[Mario]

Unless you really have to write-back the files every time, IMatch can keep all the data safely in the database without writing back at all. That's probably the best approach when making experiments. Just disable background write-back and you're good. You can write back all or selected files (e.g. for a delivery or upload) at your convenience.

If you keep working with XMP data for JPEG files, your metadata is incompatible with almost all other software. I'm pretty sure standard software expects sidecar files for JPEG or is flexible enough to be configured to support this. I surely wrote that already.

Also, if you prevent IMatch from updating your JPEG files, it cannot synchronize changes from XMP back into IPTC, EXIF and GPS (which is probably the reason for your non-propagating keywords). You have to make sure that the JPEG files do not contain IPTC data (at least, assuming you don't change EXIF data in XMP as well). Or you'll end up with two non-synchronized sets of metadata and when IMatch re-imports metadata, non-synchronized XMP/EXIF/IPTC/GPS data in your JPEG files may wipe out changes done to XMP stored in the sidecar files. Alway tricky...

[philippe28]

This depends on your file format, embedded XMP or not, if existing IPTC/EXIF/GPS data needs to be updated in the original file. [..]
If you make the original file r-o and it contains IPTC/EXIF/GPS data, you are creating a mess because you actively prevent IMatch from synchronizing metadata across metadata formats. And then you end up with messy out-of-synch metadata and wonder why this happens...
[..] IMatch can keep all the data safely in the database without writing back at all

I've all metadata set on master images, but not written in files. And as you explain that prevents to propagate the xmp data to version images.
If I write-back the metatada to master files the propagation is effective.
I don't want to overwrite jpg original images (master) and I don't need neither to keep xmp on them.
But as you say also, all data are safely kept in the database...
Is there any way to overcome this limitation ?
Is it possible to have all metadata in categories and map them to xmp data (+GPS) when files must be written ?
Thanks for help
Philippe

[Mario]

I'm not sure that I understand what you want to do.

Keywords are stored in:

1. XMP hierarchical keywords.
2. XMP flat keywords (XMP-dc:Subject)

if your files have legacy IPTC data, also in

3. IPTC:keywords

[philippe28]

I'm not sure that I understand what you want to do.

Sorry for not being clear.
So far I'm in on the following state.
I've my original images ingested in IMatch. I've set metadata as I wanted (keywords, Title, Description, Rating, GPS data, ...).
I've also setup some "File Relations" to link Master and Version images.
I see two next steps:
1. propagate Master images metadata to Version images
2. write (selected) metadata to Version files (following the images' destination chosen metadata may be different)
I'm stuck on the first one and I've understood that is because I've not written the metadata to Master files (and I want to avoid because I have no use of xmp on Master files, and I don't want to overwrite jpg Master files).
So, is there any possibility to define metadata on master images and keep them in database (without writing them into files), propagate these metadata to Version images and eventually write (only) as needed metadata into Version files ?

[Mario]

Metadata propagation is performed via ExifTool. After writing metadata to the master, IMatch runs ExifTool again to copy data from the metadata to the versions. This is the only way to do it.

If you don't let IMatch update your master files for whatever reason you cannot use metadata propagation.

[philippe28]

If you don't let IMatch update your master files for whatever reason you cannot use metadata propagation.

I understand.
In case I remove from propagation rules every option marked with *, does propagation still use ExifTool ?
Would I be able to propagate only Categories for example without have write-back on Master files ?

[ubacher]

You are making it difficult for yourself by fighting the system.
If I understand it correctly you want to write different metadata to a file on manual initiation.

I can see the need - for instance I sometimes want an English description for files I give away, other times I want the description in German.


One way I see is to store your metadata in attributes, and then write it to metadata when wanted - and then initiate a write-back.
(If you then want to undo it it gets more involved - but can be done with the exiftool processor. You probably would do the metadata write on a copy of
the file and thus just delete the copy after giving the file away.)

I am not sure if there is a way of using the exiftool to directly write attributes to metadata - this would allow you to skip a step or two.

[philippe28]

You are making it difficult for yourself by fighting the system.

I quite agree, one should not fight with the system !
That's why I'm trying to understand the fondamentals (and because there are tons of other interesting IMatch features).
Your suggestion with attributes makes sense from my way of thinking but throws away the basic keywords workflow for example.
Another way to think about propagation which would fit my needs, based on a script:
- select Version files I want to update metadata to
- run a script which gets Master images metadata and updates Version images accordingly (in database).
Is there a way to find the Version file's master file ? (I've seen some File.Relations variables but not this one).

[Mario]

You can find the master using the corresponding command from the right-click content menu if versions.

If you only want to propagate categories, no write-back is required.

[ubacher]

You might want to work with categories instead of keywords. When time comes you convert the categories into
(same name) keywords on the version file. Not sure if this conversion would need a script.
You lose the use of the keyword panel and the thesaurus - but maybe you can live with this.