Problem syncing keyword with Photo Mechanic and I-Match.

[happysnapper]

I wish to apply keyword in PM then view them and sometimes edit in IM.
I thought I had it sorted
I can apply a string of hierarchical keywords in PM eg 1|2|3  they appear OK in IM as 1|2|3
I can then apply additional keywords in PM eg 4;5  and as the function is set to append keywords these also are added OK in IM so I now have 4;5; 1|2|3
Now the problem, if I add another keyword in PM eg.6 it does NOT appear in IM.  keywords in IM remain as 4;5;1|2|3 there is no 6

I fully understand that any keywords edited or added in IM will only show in PM after metadata write back is applied either by clicking pencil or selecting real time write back.

If I now add a keyword in IM eg X then write back metadata the keywords do indeed appear in PM  as X;4:5;1|2|3  the 6 is now removed.

I'm a photographer not a programer so please be gentle with me if I've overlooked something    I'm pretty sure the problem is with IM as all changes to keywords in PM show up OK in Capture One my  Raw converter so confirms PM is updating the xmp file correctly.

I may be naive but it is as though IM sets a limit, not on number of keywords but on the number of times it will read the xmp file.

[Mario]

I don't know if PM applies the rules of the Metadata Working Group for keyword mapping or if your files contain only XMP data but also legacy IPTC data. This would be important to know. Your files may contain three sets of keywords: legacy IPTC, XMP flat keywords, XMP hierarchicalKeywords (in the Adobe namespace).

IMatch fully implements the rules and recommendations of the Metadata Working Group (Adobe, Apple, Microsoft) and this means it maps keywords between XMP hierarchical keywords, XMP flat keywords and legacy IPTC keywords (if existing). This ensures that keywords and other metadata written by IMatch is compatible with the widest range of applications and systems.

IMatch performs the mapping every time it reads a file after it has been updated in external applications (when it detects a change in the image file or the XMP sidecar file - there is no limit).

If keywords get garbled or lost when updating metadata in multiple applications, the usual case is that one or more applications have their own 'ways' for updating metadata, don't follow the MWG rules, don't synchronize metadata properly or similar.


Tip: You can see all keywords in your files quickly in the IMatch ExifTool Command Processor (<F9>,<E> or Tools menu). Use the "List Keywords" Preset:

-G1
-iptc:keywords
-xmp-dc:subject
-xmp:hierarchicalSubject
-a
{Files}

If you work with external XMP files, replaces {Files} in the last line with {XMPFiles} to see the data from the XMP file as well.
All the keywords you see should be identical and synchronized after PM has updated the metadata in your files.

It would be helpful if you can upload a sample image and an XMP file somewhere - files which produce the wrong result when they are imported in IMatch after you changed them.

[happysnapper]

Thanks for your quick response Mario.

I checked all your points and still don't know what is happening.

In PM I first applied string Quick|Brown|Fox which then appeared in IM keyword panel ok.
I then in PM applied jumps;over as flat keywords which also appeared in keyword panel ok.
I then in PM applied lazy;dog as flat keywords  as noted before these keywords did not appear.

I then followed your F9 E instructions. To me it looks like PM has done things right as all the applied keywords are there, I think in the right places.

https://www.dropbox.com/sh/87i9f3ayt1bdkie/AADHnLhQIau0J1oYUzPBV-xCa?dl=0

At this dropbox location is a raw file, the associated xmp. file, a screen grab showing both the keywords panel and the exiftools panel and a text file of the data from the xmp file read by exifviewer.

I hope this helps Mario.

Cheers Graham

[Mario]

I looked at your XMP file with the following ExifTool command (works also in the ExifTool Command Processor in IMatch):

exiftool -G1 -xmp-lr:hierarchicalSubject "IMG_9821.xmp"

The result is:

[XMP-lr]        Hierarchical Subject            : Quick|Brown|Fox
[XMP-dc]        Subject                         : Quick|Brown|Fox, jumps, over, lazy, dog

which shows the problem. The keywords "jumps", "over", "lazy" and "dog" are only in the flat keywords, but not in the hierarchical keywords. The keywords are not properly synchronized. It should look like this:

[XMP-lr]        Hierarchical Subject            : Quick|Brown|Fox ,jumps, over, lazy, dog
[XMP-dc]        Subject                         : Quick|Brown|Fox, jumps, over, lazy, dog

If this is output generated by PM, it is wrong.

IMatch by default assumes that if a file has hierarchical keywords that these keywords are properly synchronized with the flat keywords. This is a reasonable assumption and also according to the Metadata Working Group rules. IMatch uses only the hierarchical keywords on import.

You can override this behavior by disabling the option Don't replace existing hierarchical keywords in Edit > Preferences > Metadata. This will merge existing hierarchical and flat keywords contained on your files when IMatch imports them (into the IMatch database) and when you later write the files back, IMatch will synchronize flat and hierarchical keywords properly.

You may use this as a work-around for an initial import of your files. But when you continue updating metadata in multiple applications and some of them mess up your metadata, you will be in trouble. I recommend only using IMatch or asking the PM developers to fix this.

Note: Disabling the option should be done on a needed-only basis because it can play havoc with your keywords if you combine it with thesaurus lookups to map flat keywords into your keyword hierarchy on import. This probably won't affect you in this case, so this is just a thing to keep in mind...

[happysnapper]

Hi Mario, sorry for delay getting back.

[IMatch by default assumes that if a file has hierarchical keywords that these keywords are properly synchronized with the flat keywords. This is a reasonable assumption and also according to the Metadata Working Group rules]

My personal assumption would be that hierarchical keywords are written to hierarchical subject and flat keywords just to subject. I spent all last night trying to learn more about MWG rules but failed to find the rule you refer to. Not saying it's not there just I couldn't find it. :-)

IMatch uses only the hierarchical keywords on import.

I can see that but it also appears to recognise just the 1st flat keyword (or string of keywords) keywords that PM have only written to dc:subject.

I  passed on your comments to Photo Mechanic and here is my reply and my response to it.

Graham,

I'd have to check the standards again, but I don't see why non-hierarchical keywords should be represented in the hierarchical keywords field.  We put both in the normal keywords because for quite some time most applications would only ever look in the dc-Subject field for keywords (really just in the legacy binary IPTC record at that.)

PM reads and manages both fields.  Why can't I-Match do the same?  Just make a union of the keywords (removing duplicates) before putting them into the database.

-Kirk


Thanks Kirk for your fast response.
As a mere photographer not software  programmer I too could not see the logic so spent the whole night googling for xmp. standards without a lot of success, but I do know that Mario is a very clever chap. I have a feeling it has something to do with the way Lightroom handles files, I'm not a Lightroom user so can't experiment.

I will pass on your comments to Mario, I'm sure this problem must have a solution. There must be hundreds if not thousands of Pro Togs using both your applications.

Cheers Graham

[Mario]

My personal assumption would be that hierarchical keywords are written to hierarchical subject and flat keywords just to subject.

This is not how IMatch sees this. You should not have separate lists of keywords for hierarchical and flat keywords. IMatch does not support this, many tools will break this (e.g. Adobe products). Always assume the mental model that each of your files has exactly one set of keywords. You cannot have cow, dog in flat keywords and then cat and burger in hierarchical keywords. A keywords like "beach" is a hierarchical keyword - on the topmost level.

I wondered what you mean when you set "I entered hierarchical keywords there and flat keywords there"...now I know what you did.

IMatch works like this for the past two years. The mechanism to synchronize flat and hierarchical keywords are very sophisticated and leave users many options. The keyword editor, @Keywords and all other keyword-based features are based on hierarchical keywords, which are the superset of flat keywords.

If you really want to maintain separate keywords, you can disable MWG compliance and then you can put in any metadata wherever you like, even managing separate sets of keywords in XMP and even in legacy IPTC.  This allows you to store 3 sets of totally unrelated keywords in your files. IMatch will always work with one set of keywords (hierarchical) and import flat keywords - or not. Dunno, never really tried to do such a thing - having separate keywords...don't know how the implementation treats this.

[TomS]

I too have looked at various documents available on the MWG's website and can't seem to find any guidance that would indicate that all keywords fields should be synchronized based on what is in the adobe lr:hierarchicalSubject tag. With all due respect, I think the IMatch enforces rules based on IMatch's "interpretation" of the MWG's recommendations. If I am mistaken, I would welcome a link to MWG's guidance on this specific issue.

I'll admit that the issue of how to handle hierarchical keywords is very complex and causes a lot of problems when using various software applications to manage file metadata. Unfortunately, if all programs won't agree on how this metadata is to be managed, we can never expect our files' metadata to be "compliant" based on each developer's definition of compliance. Frankly, it's a huge headache. I have 30,000 files that I've already tagged using other software and IMatch will not correctly interpret my keyword hierarchies because that software developer doesn't agree with IMatch on what the standards say.

I, for one, would welcome the wide spread adoption of the MWG's hierarchical keyword recommendation, which is the create an XML based structure for storing both the hierarchy and the flat keywords, which, based on my interpretation, don't have to be the same. The MWG clearly understands the higher levels of the hierarchy are meant as grouping tools and are not meant to be broken into flat keywords and written to the dc:subject tag. This is where IMatch seems to disagree with the MWG. IMatch insists that if you want your higher level keyword hierarchy to be used for grouping only, then that hierarchy does not get written to the file at all. That means that if I wanted to take my files to another program to manage them that the metadata in the files would not be sufficient to allow the other program to recreate my hierarchy. Following the MWG's specifications, that would not be the case.

I do also disagree with the response from PM and agree with Mario about putting all keywords in the lf:hierarchicalSubject. If you're going to use that tag then you should put all of your keywords in it. If they don't belong to a hierarchy then, as Mario stated, they are just the top level of the hierarchy.

This is a huge subject and one that there is obviously no consensus on but I think the MWG has a pretty good grasp on it and if all metadata management tools would adopt it, we'd all be a lot better off.

[Mario]

That IMatch maintains one synchronized set of keywords for files was a design decision made about three years ago, when the first Alpha versions of IMatch 5 were shipped to testers and agencies. The Adobe hierarchical XMP Lightroom keyword namespace was (and still is) outside of the MWG recommendations, and the MWG just recently added their own idea about hierarchical keywords, which again differs from what Adobe does in their products. This opens a fourth mine field with keywords, and I have to say I'm fed up with metadata.

About 50% of the total development time for IMatch 5 was spend dealing with metadata, trying to come up with options to allow IMatch to handle all the various metadata schemes different applications come up with, the many problems we encounter in metadata produced by applications over the past 10 or 20 years, the various partial implementations of IPTC and/or XMP support. Even when writing this I can feel my blood pressure rise because of all the months I have spend with figuring all this stuff out and coming up with a standard way of dealing with it that works for the majority of the user base (and did so very well, over the past two years). And IMatch also has options to allow users to configure it to handle even the crudest and most common metadata-related scenarios. Including disabling MWG-based metadata mapping altogether - if one understands the consequences of maintaining non-synchronized sets of IPTC, EXIF and XMP in a file.

If your workflow requires that you keep separate lists of keywords in IPTC, XMP, and hierarchical keywords, IMatch may not be the software for you. I could add options to not synchronize keywords on import, export and updates fairly easy...

But 2 hours later some users would request that the Keyword Panel can be switched between non-hierarchical and hierarchical keywords, that the thesaurus maintains separate sets of keywords for flat and hierarchical, that the @Keywords category can be switched to operate either on flat or hierarchical keywords and so on. I would roughly estimate several full weeks of development time to implement this. And all this for maybe 5 users world-wide who want to manage different sets of flat and hierarchical keywords for their files? No, sorry. I think that if a user needs that, IMatch may not be the software he needs. There are other DAM products out there which may fit his workflow better.

[Ferdinand]

The MWG clearly understands the higher levels of the hierarchy are meant as grouping tools and are not meant to be broken into flat keywords and written to the dc:subject tag. This is where IMatch seems to disagree with the MWG. IMatch insists that if you want your higher level keyword hierarchy to be used for grouping only, then that hierarchy does not get written to the file at all.

I'm not sure I follow all your post about who and what you're agreeing and disagreeing with.  But I wanted to point out that your comments seem to relate mostly to the "Group" option in the thesaurus.  There is also the "Exclude" option.  If you mark a node as Exclude rather than Group, then the full hierarchical keyword is written to lr:hierarchicalSubject but not all the nodes are.  Only the non-excluded nodes are.  So if I have "Who|People|Family|Fred" as a keyword and mark the first two nodes as Exclude, then only "Family" and "Fred" will be written as flat keywords.  If you think that this option will help you then read the help file for more information.  Note also that this option is not consistent with how Adobe s/w handles things, and you may find that it you open such images in Adobe and then again in IMatch, IMatch will want to correct the flat keywords to match your preferences.  This is not a major problem, just something to be aware of.

[TomS]

Thank you Ferdinand. That is what I would like to accomplish and it mirrors what I've done in the past. I guess the only issue is that my files are already tagged in that way, with complete hierarchies stored in the lr:hierarchicalSubject but only certain levels of those hierarchies stored as flat keywords in the dc:subject tag. The only problem seems to be that on initial import, all of the levels will get written to the dc:subject tag because there is no way to tell IMatch which levels are to be excluded before importing the files, unless of course I attempt to recreate my entire keyword hierarchy first, which I don't think would be that easy.

Maybe I'm mistaken and there is an easy way but this has me questioning whether or not it's a good idea to exclude any levels. From the standpoint of trying to make my metadata easily interpreted by other programs it seems best to include all levels of my hierarchy in the flat keywords. Do you have any recommendations for me in this respect.

I'm sorry for any confusion I may have caused with my earlier post. I guess all I was trying to say is that the standards for handling keyword hierarchies are ambiguous at best and open to interpretation. The MWG has proposed new tag structures that are aimed at removing that ambiguity but I'm not holding my breath while waiting for software vendors to adopt them. Mario seems to think that the MWG proposal creates even more problems and he is certainly more qualified to know that then I am. So, in the meantime, I guess I need to figure out the best way to use what we already have.

[Mario]

I'm quite confident that you won't find any software out there that gives you so many options to control when, which and how keywords are written, propagated or consolidated.

On import, IMatch maps flat keywords into hierarchical keywords. This process can be controlled and disabled via Edit > Preferences > Metadata. To map flat keywords (or flat keywords you have produced from hierarchical keywords somehow by removing one or more levels...) can be mapped back to hierarchical keywords by utilizing your thesaurus. There is no other way IMatch can know that the flat keyword "Daytona" actually is "Locations|Beaches|Daytona".

I have written a ton of info in the corresponding help topics and explain how the options work, how you can control flat -> hierarchical mapping on import, hierarchical -> flat on export. How the groups and exclude levels in the thesaurus allow you to setup levels only for your convenience or levels which tell IMatch which parts of the hierarchical keywords to skip.

If you work with multiple applications on your keywords, you may end up in tears. Some applications don't do hierarchies at all. Some don't synchronize IPTC and XMP (or only in some of their products, or only in some versions of their products, or differently in different versions of their products) so keep that in mind and do some tests.

And yes, the approach to handle hierarchical keywords proposed by the MWG is so bloated and complex that I closed the PDF and cried for a while afterwards. Thinking about the typical IMatch user with 2,000 to 20,000 keywords and hierarchies up to 10 levels deep - doing that in the MWG style will not only be really hard, but probably produce XMP records which are larger than the actual image...they have users in mind who work with maybe 200 keywords on three levels.

I have postponed this for a long time, unless there is really demand for such a thing. Many other things to do which are beneficial for more than a handful of users.

[Ferdinand]

I guess the only issue is that my files are already tagged in that way, with complete hierarchies stored in the lr:hierarchicalSubject but only certain levels of those hierarchies stored as flat keywords in the dc:subject tag. The only problem seems to be that on initial import, all of the levels will get written to the dc:subject tag because there is no way to tell IMatch which levels are to be excluded before importing the files, unless of course I attempt to recreate my entire keyword hierarchy first, which I don't think would be that easy.

Well, that's what I did.  Are you coming from IMatch 3.6 by any chance?  I have a script that can create the thesaurus in advance, which you can then import and mark the various nodes as Exclude before you import the images. 

if not, there are still solutions to this, it's a question of how much work you are prepared to go to, and what s/w you've used previously.   At the worst, you can import the images into an IMatch 5 DB, populate the thesaurus from the metadata, remove all the images, mark nodes as Exclude, and then reimport the images again.

As Mario just said, there are metadata preferences options to control this and you need to have them set correctly.  I can provide mine if you are interested, since they should match what you want.

[TomS]

I guess the only issue is that my files are already tagged in that way, with complete hierarchies stored in the lr:hierarchicalSubject but only certain levels of those hierarchies stored as flat keywords in the dc:subject tag. The only problem seems to be that on initial import, all of the levels will get written to the dc:subject tag because there is no way to tell IMatch which levels are to be excluded before importing the files, unless of course I attempt to recreate my entire keyword hierarchy first, which I don't think would be that easy.

Well, that's what I did.  Are you coming from IMatch 3.6 by any chance?

No, I'm coming from Photo Supreme. I really like Photo Supreme and will be hard pressed to make a change. The interface is great and the feature set includes some built-in functionality that IMatch doesn't seem to have, at least not that I've discovered yet. Photo Supreme uses categories and "labels" to manage metadata. Labels can be configured to automatically create keywords (or not) when assigned to files and can also be mapped to other metadata tags. This allows me to have one big hierarchy that includes locations, descriptive tags, people, etc. and decide which specific labels get written to which metadata fields including keyword fields (including lr:hierarchicalSubject).

I don't want to change the topic of this thread to a comparison of features, which is why I haven't mentioned Photo Supreme by name until now. My reason for trying IMatch is to see how well it works. I want to determine if I can come up with a workflow using IMatch that is as efficient as the one I'm currently using and to see how reliable the software is. My only complaint with Photo Supreme is that there are some minor bugs that never seem to get resolved completely and some quirks in the behavior of the user interface. Even though the developer will correct many of the bugs promptly, they often end up getting reintroduced with later updates or new ones are created whenever new functionality gets added. I sometimes feel like I've been acting as a bug tester for them for the last 3 years. I wanted to see if using IMatch will be the same or whether Mario's attention to detail would result in less of those situations. I have nothing but praise for the developer of Photo Supreme and his response to requests for bug fixes and new features but I feel I owe it to myself to see if there is a better option for me. After some searching, I'm convinced that IMatch is the only serious competitor to Photo Supreme.

I haven't quite figured out how to design my workflow yet in IMatch to accomplish what I want to accomplish so it's way too soon to see if everything works as well as I'm hoping. I haven't had much time to experiment so I don't want to start making any assumptions yet or asking too many stupid questions. I've spent several hours reading the help file and I think I have a general understanding of how much of IMatch is designed to work but I am struggling putting it all together. I should probably start a new thread before going into specifics. I've learned much from this thread already.

[Ferdinand]

In relation to the topic of this thread, if you need more help on the Exclude option then just say so.

On the workflow comparison with Photo Supreme and its predecessor, I'm not in a position to comment. There are people here who made the switch and probably can.  My only point would be that it's probably a little unrealistic to expect to switch from one program to another and replicate an exact workflow.  You've got a workflow based around PS and it's capabilities, and we here have something similar based around the capabilities of IMatch.  A new thread is probably the best approach to exploring this issue.

[TomS]

Thank you Ferdinand. I will let you know if I find that I need help implementing the exclude option in my keywords.

I will definitely create a new topic to address any general workflow questions I come up with.

On this topic, I regret that nothing I've contributed to this thread has helped to resolve the incompatibility issues between IMatch and Photo Mechanic. It seems that what I've learned from this thread has just reinforced what I already believed. Best practice seems always to be never use more than one program to manage the metadata in your files. Unfortunately, that is easier said than done.

Perhaps happysnapper can find a workaround. Is there anyway to force PM to interpret flat keywords as being part of a hierarchy and add the keyword to the lr:heirarchicalSubject tag? PM must parse the keywords entered to determine whether or not they are hierarchical. What happens if you put the bar "|" symbol in front of the flat keyword you enter in PM?
Or even put the bar before and after it. It's worth a try.

Or better yet, why not create a top level hierarchical keyword in IMatch, like "FlatKeywords", and configure it with the grouping option and then when you enter your flat keywords in PM, enter them as "FlatKeywords|*keyword*"? If I'm not mistaken, this would cause PM to create an entry in both the lr and dc tags. Then when IMatch sees this change it will rewrite the keywords in both the lr and dc tags without the "FlatKeyword" included.

[Ferdinand]

Best practice seems always to be never use more than one program to manage the metadata in your files.

That's certainly my approach and experience.

Or better yet, why not create a top level hierarchical keyword in IMatch, like "FlatKeywords", and configure it with the grouping option and then when you enter your flat keywords in PM, enter them as "FlatKeywords|*keyword*"? If I'm not mistaken, this would cause PM to create an entry in both the lr and dc tags. Then when IMatch sees this change it will rewrite the keywords in both the lr and dc tags without the "FlatKeyword" included.

Good luck with that!

[sinus]

Best practice seems always to be never use more than one program to manage the metadata in your files.

That's certainly my approach and experience.

Mine too. I for me personally see no reason to change this. I use Metadata only with IMatch.