Some thoughts and questions about versioning

[Schmidtze]

Hello,

I'm just playing around with the versioning option in IMatch. A very very cool feature!!! As I'm not absolutely sure how to manage my situation, I'm asking here for some help. Maybe somebody has an idea or tip in which way to use IMatch for my purpose. Maybe I'm doing it the right way already...

On my harddisk I have 2 base folders, the folder "RAW" and the folder "Final", both with many subfolders (with identical hirachy structure). Both folders contain the same image files, in folder RAW usually DNG files, but also some JPEG files (e.g. shot with mobile phones etc.). The folder "Final" contains all the images contained in folder "RAW" as exported JPEG files (created by Lightroom).

I added both root folders ("RAW" and "Final") now to IMatch. Then I created 2 versioning items, one for DNG and one for JPEG. Honestly I didn't expect the versioning for JPEG files will not work, but it seems that it does :-) For each versioning item the "Where to Search"-folder is set to folder "Final" with 5 levels down.

Question: As the subfolders of "Final" has got exactly the same folder hirarchy as the subfolders of "RAW", is there maybe a better solution by using a pattern with {p0}, {d0} etc. for the "Where to Search"-folder? In pattern dialog there's an example path. Maybe it would be a good idea to use a real example from the database there instead of showing a fantasy example. 

By now I didn't find an option to propagate keywords from masters to versions. I only see this possibility for categories. Question: Do I perhaps miss something?

My main question: Do you think I'm going the right way with my folder structure as I used it for years now? Or does maybe something argue against it? For me it seems that it will work as expected and I will be very happy with it 

Regards
Friedemann

[claudermilk]

A quick example of your folder structure might help.

I can at least give an example of how I have my versioning set up.

First is the folder structure. The base structure is <drive>:\Photography\YYYY\YYYYMM\ --however that part is irrelevant as you will soon see. Within that structure, I have separate directories for each shoot thus:
|_YYMMDD - <shoot name>
  |_RAW
  |_Working
  |_Done
    |_Web

My versioning definitions follow this pattern (using my CR2 one):

• Link via: File Name
• Master Expression:
  
  • \.(cr2)$
• Replacement Expression: ^_*//
• Link Expression: ^(_*{name})([+\-_]*[0-9|a-z]*)*\.(jpg|jpeg|psd|tif|tiff|dng)$
• Where to search: List of folders
• Direction: Specified Folders
  
  • {p1}\RAW
  • {p1}\Working
  • {p1}\Done
  • {p1}\Done\Web
  • {p1}\Web
• Versioning/What to Propagate: Categories, Rating, Label, All Metadata

Note that last line--that is what propagates the  data. Those settings are on a separate tab in the Preferences panel--it's easy to miss.

[Schmidtze]

Hello,

many thanks to your answer! My folder structure is this

|_RAW
| |_YYYY
|   |_YYYY-MM <shoot name 1>
|   |_YYYY-MM <shoot name 2>
|     |_location 1
|     |_location 2
|_Final
  |_YYYY
    |_YYYY-MM <shoot name 1>
    |_YYYY-MM <shoot name 2>
      |_location 1
      |_location 2

As you can see, same folder structure in "RAW" and in "Final". The problem is maybe that I sometimes (not very often) have subfolders under "YYYY-MM <shoot name>", so it can't be processed by only one simple pattern. I think I have to use 2 patterns.

Versioning/What to Propagate: Categories, Rating, Label, All Metadata

Honestly I'm a little bit afraid of using "All Metadata" for propagating only the keywords. Perhaps the master files contain data which will overwrite same existing data in version files which contain a different content (e.g. color information, whatever). I'm wondering why "@Keywords" will not be considered, because "Categories" is enabled in "What to propagate". The tree in "Categories to propagate" do not contain "@Keywords". In the help file it looks different. Do I have to setup another option maybe elsewhere to consider "@Keywords" also in propagation?

Best regards
Friedemann

[Schmidtze]

Just tried it, these 2 patterns will work for my folder hirarchy

{p3}\Final\{d1}\{d0}
{p4}\Final\{d2}\{d1}\{d0}

But still don't know how to propagate the keywords. Does this anybody know???

Best regards
Friedemann

[jch2103]

Check the Versioning tab in Preferences > File Relations. It gives you options of what to propagate from master to version.

[Schmidtze]

Hi jch2103,

Check the Versioning tab in Preferences > File Relations. It gives you options of what to propagate from master to version.

thank you! But you didn't read the whole thread, did you? Of course I know where to set up the propagation options. But I don't know what to enable there as "Keywords" or "@Keywords" are not available. The option "All Metadata" seems to be a little bit oversized for it...

Regards
Friedemann

[jch2103]

Hmm. There's a discrepancy between the current IMatch Versioning and what Help shows for this topic. The Help example shows that @Keywords and @Builder check boxes should be available under 'Categories to Propagate', but they're clearly not...

I don't know if this was by design or omission.

[claudermilk]

I was simply providing my setup as an example. Take a look down the list of what you can propagate, you should be able to get to what you want. There's a lot of options; for my testing, I simply told it to propagate all the EXIF/IPTC/XMP data to keep from getting too complex in the often-rebuilt databases.

Oh, and as an aside: if you get these relations set up before importing any files, all your versioning stacks will be created when you ingest the test images.

[Ferdinand]

Hmm. There's a discrepancy between the current IMatch Versioning and what Help shows for this topic. The Help example shows that @Keywords and @Builder check boxes should be available under 'Categories to Propagate', but they're clearly not...

I don't know if this was by design or omission.

So there is.  I did encourage Mario to provide more granularity in what can be propagated, but he didn't want an endless list.  But perhaps this is an oversight. 

Keywords are stored in the file.  If they were only in IPTC then propagating IPTC should do it.  But IMatch 5 is based primarily around XMP so propagating XP should do it.  If you only want partial propagation then you'll need to make a feature request, and you will also need a bit of luck in getting the existing list of options increased.

However there is a bug in propagation IMHO (Mantis 714).  I am finding that XMP is propagating correctly, but this isn't then being populated in IPTC.  If you're supplying images to clients or uploading them to websites you will usually want this information in IPTC.  What I will be doing is entering it in XMP for the master and will want it to spread to IPTC in the versions.  I find that it's not.  I wouldn't  mind someone else trying this out.

[jch2103]

I tried a test of IPTC propagation in a small test database.

I copied a NEF file and derived jpg into a test folder, then deleted all metadata in the jpg using the ExifTool Command Processor (Delete All Metadata). Next, I set up NEF versioning and set 'What to propagate' to 'Classic IPTC Data (IIM)' and refreshed relations. (I left the default check marks: Bookmarks, Flag, Dot, etc.; the only other option selected was 'Classic IPTC'.) Finally, I made a small change in the master (added an entry in the Description Writer tag) and used the 'pencil' to write the change to the master.

It appears that the IPTC data was successfully copied to the jpg. I've attached a small version of the jpg so you can check to see if I'm referencing the right set of tags.

[attachment deleted by admin]

[Schmidtze]

It appears that the IPTC data was successfully copied to the jpg. I've attached a small version of the jpg so you can check to see if I'm referencing the right set of tags.

Yes, it contains IPTC, but not the keywords in XMP data, which would be very important regarding hierarchical keywords...

Regards
Friedemann

[jch2103]

Right. For purposes of responding to Ferdinand's issue re IPTC copying, I excluded XML (and hence keywords) from the versioning 'What to propagate' for this little test.  If I add 'XML All data' then the keywords are copied too.

[Schmidtze]

 jch2103,

If I add 'XML All data' then the keywords are copied too.

What a surprise  I already mentioned this also above, did you read it???  Of course the keywords will be copied then, but also all other data will be copied also. Do you really know, if all data from the RAW file makes also sense  in the JPEG file???  I don't! That's why I'm a little bit concerned about using "XML All Data" or "All Metadata".

[Schmidtze]

To make it clear again: In my opinion keywords are the most important data (besides some others of course) which can be maintained very very comfortable with IMatch. That's why I would expect that there's also an option for propagating them, without propagating maybe millions of other data... I hoped that I'm doing maybe something wrong or that I miss something. Propagating all metadata or all XMP data is in my eyes not a good option.

Best regards
Friedemann

[jch2103]

https://www.photools.com/community/index.php?topic=234.0

[Mario]

Currently IMatch allows you to propagate (also in combination)

All (suitable) Metadata
Classic IPTC data
EXIF data, with and without orientation, safe tags only
All XMP data
All XMP data with the special "disabled crop"

plus some additional data like maker notes, PDF data, etc.

I'm not sure that I got everything said in this thread, but propagating only keywords (which keywords?) may cause irritation.
For example we may have to deal with several sets of keywords: Classic IPTC, XMP, LR hierarchicalKeywords and maybe even (some day) MWG hierarchical keywords. So even if only keywords would be propagated, fresh IPTC and/or XMP records need to be created in the output file.

The next user then comes and only wants to propagate the headline and maybe a copyright and author statement. And just like that we have to implement a full-fledged "per-tag" propagation rule system which complicates things for all users, and may solve some specific problems or wishes from some users.

For most users, just ticking classic IPTC and All XMP is usually sufficient. Copying EXIF is possible but often causes trouble because EXIF was not designed to be copied between files, especially not the file formats typically used in versioning. Copying EXIF from a RAW to a JPEG creates a lot of invalid information, for example.

If more is needed, I'm open to suggestions. We discussed this in the Pre-Beta group to some extent and then I postponed this to the public Beta to get more feedback from users. The feature request board exists for these purposes.

But I don't really like the idea that I add more and more "tag sets" or special cases to the propagation list, or even implement a "per-tag" selection. Because picking individual tags can lead to invalid metadata, if tags need to be copied in pairs or sets but the user is not aware of that.

[Schmidtze]

Hi Mario,

hmm, honestly I don't really understand your point of view. I can understand regarding "more and more "tag sets" or special cases", but not regarding keywords.

For example we may have to deal with several sets of keywords: Classic IPTC, XMP, LR hierarchicalKeywords and maybe even (some day) MWG hierarchical keywords. So even if only keywords would be propagated, fresh IPTC and/or XMP records need to be created in the output file.

You are right, technically the keywords are stored in several different data areas. But you do already a synchronization. So what's the difference between adding a keyword by using the keyword panel or by using the @Keywords node in categories list compared to propagating (let's also say copying) a keyword from one image file to another? In my opinion, you did a great job regarding keywords. It's a pleasure to assign and maintain them using IMatch. For the user it doesn't play a role where IMatch (ExifTool) stores them, in Classic IPTC, XMP, LR hierarchicalKeywords or whatever. All data areas will be consistent. IMatch does the job already to synchronize them. That's for me, and maybe a lot of other users, a very very cool feature! That's why I don't understand, why just keywords will not be propagated. In my eyes propagating the keywords between masters and their versions is a must! You also included the option for propagating ratings. That's in my opinion exactly the same problem, as the rating also have to be saved in xmp:rating and also in xmp:ratingpercent.

Of course, it is possible to propagate by using "XMP All Data" and "Classic IPTC Data". And I think I will use it when there won't be another solution. But as I said, I'm not sure if this will maybe propagate XMP data of RAWs which will not make any sense in JPEGs. Then the JPEGs will contain data which will perhaps confuse other software.

Best regards
Friedemann

[Ferdinand]

Right. For purposes of responding to Ferdinand's issue re IPTC copying, I excluded XML (and hence keywords) from the versioning 'What to propagate' for this little test.  If I add 'XML All data' then the keywords are copied too.

I don't want to hijack Friedemann's thread.  I am very glad that he is here and asking these questions, as I have been struggling in this general area for a very long time, and he understands it better than most.

But the quote above doesn't quite capture my problem.  If there is IPTC in the master image and I configure IMatch to propagate IPTC then it is propagated.  The same for XMP.  This works so far.  But under MWG rules the XMP that is propagated to a version should also be transferred to IPTC, and for me it's not.

Consider a RAW file that someone has made read-only (a common approach).  In IMatch 5, the metadata including keywords resides in the RAW file's XMP sidecar.  So propagation initially is to XMP in the version, but if MWG is on it should also be populated in IPTC (after writeback).  Mantis 714 also highlights other odd behaviour.

I mention this in this thread because IMHO there are a number of propagation bugs to be ironed out.

[Erik]

Work has been a nightmare lately, so I've been away from IMatch and most of my photography hobby (regretfully since it is usually a nice stress release).

Anyway, reading through this thread, it would seem a potential solution would be to have an option to allow propagation of @Keywords.  Then there wouldn't be so much of a question of what keywords should propagate as the logic would be similar to a manual assignment through an @Keywords panel.  I'm sure that sounds easier than it really is.

As for Ferdinand's question, I guess it seems that if you have an image as read-only, then I'm not sure IMatch should map the XMP to the IPTC since the image is read-only.  I'm not sure if this is the problem as I thought your problem was more generic to any file (i.e. those that are not read-only).  Then, all I can think is that the logic is such that Mario is expecting that if you want IPTC to propagate you can select it in addition to XMP (I'm only speculating).

Lately, I've just selected IPTC to get that propagation to occur in addition to having XMP selected.

[Ferdinand]

As for Ferdinand's question, I guess it seems that if you have an image as read-only, then I'm not sure IMatch should map the XMP to the IPTC since the image is read-only.  I'm not sure if this is the problem as I thought your problem was more generic to any file (i.e. those that are not read-only).  Then, all I can think is that the logic is such that Mario is expecting that if you want IPTC to propagate you can select it in addition to XMP (I'm only speculating).

This doesn't quite capture my problem.  See thread 404, and let's leave this thread to propagation of keywords.