Keywords - flat not hierarchical

[graham1]

My workflow involves editing my RAW images in Lightroom, then exporting the images as JPEG files.  At the stage of export, I have some keywords assigned by Lightroom, and these are included in my JPEGs on export on a non-hierarchical basis. 

I want then to be able to manage my keywords in the JPEGs in IMatch.  These are kept in an entirely separate folder and I have set up a separate database for them, which only contains the exported JPEGs.  I have an extensive hierarchical list of keywords, which I have imported into the thesaurus for this database.

What I want to do is this, on a per image basis:  (1) add keywords manually, and (2) add keywords from the thesaurus, all as flat keywords only.  What I do not want is for my images to contain hierarchical keywords separated by pipes (or whatever separator you choose in preferences).  I just want my JPEGs to have flat keywords.

This sounds as if it should be simple, but it does not seem to work like this.  The keywords included in the original Lightroom export seem to be automatically matched to the thesaurus, and IMatch then seems to create the hierarchical keywords (in accordance with the Lightroom standards).  And the same for keywords from the thesaurus: I would like the entire keyword branch and synonyms to be available to save to my files, but as flat keywords: I do not want to preserve the hierarchy, and if I want to delete some of the intermediate child keywords from part way along the branch or a particular synonym as flat keywords, I want to be able to do this by deleting the relevant flat keyword.

I think I have tried every available combination of tick-boxes in the metadata tab in preferences.  Pressing the Propagate arrow in the Keywords panel seems to make no difference. 

How can I configure IMatch to respect writing only flat keywords, without adding unwanted hierarchies to my JPEG images?

All advice welcome.

Graham

[Mario]

1. Unless your thesaurus has only one level or all except the leaf-level are marked as group-levels, you are working with hierarchical keywords.

2. If you don't want IMatch to match flat keywords against your thesaurus on import, disable the corresponding option under Edit > Preferences >Metadata.

3. Of course IMatch allows you to do all that. You can even have hierarchical keywords and push only flat keywords into IPTC and XMP keywords, this is controlled via your thesaurus and the settings you make under Edit > Preferences > Metadata. Click <F1> when this dialog is open to read all the detailed information in the IMatch Help System.

4. Using only flat keywords is pretty uncommon these days, and very limiting. But whatever works for you is good. And IMatch does not impose a workflow or a way to deal with keyword upon you.

5. Note that Lr always works with hierarchical keywords, but may sometimes do real funny things when mapping them to legacy IPTC and regular XMP keywords.
Best to check some of your JPEG files in the The ExifTool Command Processor in IMatch to see the keywords they actually contain. This will also help you to setup your keyword settings and thesaurus in IMatch.

[graham1]

So far as I can see, the output into the JPEG file is always both the flat and the hierarchical keywords.  In the keywords Metadata panel the content of the Hierarchical Keywords top sub-panel seems to be included even though the two lower keywords sub-panels show only flat keywords.

Another query, if I may.  I have set the metadata preferences to write path elements and to assign synonyms.  The only synonyms assigned, however, are the leaf synonyms.  What I want is the entirely of both the path elements and each of the path elements synonyms.  So far as I can see, entering new keywords in the top sub-panel of the Keywords panel, or entering a filter expression in the bottom sub-panel, only results in the list of available hierarchy branches without their synonyms except at leaf level.  If you click on a branch, it is added to keywords, including only the leaf synonyms. 

If you want the branch synonyms, it seems you have to go back to the unfiltered thesaurus list and manually check each level along the branch in order to include the respective synonyms.  But clearing the filter does not leave you with the thesaurus list focussed on the keyword you have just selected - you have to go through the whole process of clicking through the various levels of keywords to find the branch you were looking for so that you can check the individual levels.  This is a very time-consuming process with a large thesaurus with multi level branches. 

Is there no way of either (or both) choosing to have synonyms at every level along the branch automatically included, or at least having the thesaurus list left focussed on the branch last selected with the filter?  Unless I am missing something, it is a very tedious process having to find the branch and then check every level to get the synonyms into the keyword list.  I would much rather have all the synonyms included by default, then if there are any I do not want, it is easy enough to delete them.

Thank you for any further help you can give.

Graham

[Mario]

IMatch only works with hierarchical keywords internally, because they are the superset of all keyword standards.
IMatch of course always writes flat and hierarchical keywords in your files. And even flat legacy IPTC keywords if your files contain legacy IPTC data. IMatch implements the rules and recommendations of the metadata working group and common industry practices.

If you add only flat keywords to your files, all keyword variants contain the same content. I see no problem there.

A synonym is an IMatch concept, not a metadata standard.
A synonym is always on the same level as its keyword. Synonyms are assigned when you use the Keyword Panel. You immediately see the results.  What you see in the Keyword Panel is what ends in the hierarchical keywords in the file. How the hierarchical keywords are flattened into the simple keywords can be controlled with great flexibility via Edit > Preferences > Metadata.

Once added to a file, synonyms become normal keywords.
IMatch can still tell synonyms from normal keywords via your thesaurus.

Keywords are always assigned with their full path, from the root. Synonyms as well. Unless you use group levels or excludes.
All this is explained in detail in the corresponding help topic, with examples.

[graham1]

Mario, thank you for your reply.

You say that:

IMatch only works with hierarchical keywords internally, because they are the superset of all keyword standards.

I understand this.

If you add only flat keywords to your files, all keyword variants contain the same content. I see no problem there.

Generally, I agree.  My reservation is that if keyword lookup is enabled in Preferences > Metadata, even the flat keywords entered are converted into the hierarchical tree. 

But often you want this, but even if you want the tree and its synonyms as flat keywords, this gives rise to a number of issues.

I can give you an example.  I want to assign a keyword "sidmouth".  This is a small UK south coast town.  The attached screenshot shows how this appears in my thesaurus.

I started off with no keywords at all. Then I took the following steps.

(1) I enter "sidmouth" in the top part of the Keywords panel.  I click on it to make sure that only the flat keyword is entered when I press enter, not the entry suggested from the thesaurus.  So far, so good, I have a single word flat keyword.  This is what shows in all 3 windows of the Metadata Keywords panel.

(2) I press the green tick in the Keywords panel to accept.  Nothing changes (as expected).  I then write the metadata to the image by pressing the yellow pen on the image thumbnail.  This updates the Keywords panel to show the hierarchy.  This also shows in the 3 Metadata panels: screenshot Metadata update 1.  All fine so far: I actually want this branch in full, and its synonyms (although only as flat keywords).

(3) This update causes the update yellow pen to appear again on the thumbnail (because so far it has only written the single word "sidmouth" as a flat keyword).  So you would think that pressing this again to write the keywords to file would do only that, writing the keywords as then shown in the Metadata Keywords panel.  But the writing of the metadata also repeats the query of the thesaurus.  If you have any keywords in the branch you have just accepted, these too are searched for and added.  Sidmouth is on the south coast of England.  It so happens that I have elsewhere in the thesaurus the keyword "south coast" in a branch which refers to Cuba.  So now I find that the (totally unwanted) Cuba south coast branch is added: screenshot Metadata update 2.

Clearly this is generating nonsense/unwanted keywords, because I have the same keyword twice in my thesaurus, in different locations, all resulting from the entry of my single word flat field keyword "sidmouth". In other words, it gets into a loop of write : query : update : write etc. until no more matches for any of the keywords in the branch are found elsewhere in the thesaurus.

So the obvious answer is to disable Keyword lookup in the metadata options settings.  But the reason this is enabled is actually to get the path elements for this keyword.  I also want the synonyms that go with those path elements.  The only option I can see to have these included, is to have both keyword lookup options enabled in Metadata Settings. 

You say in your previous reply:

Keywords are always assigned with their full path, from the root. Synonyms as well. Unless you use group levels or excludes.

This only happens if keyword lookup in enabled, which has the unwanted consequences described above.  Is there any option whereby you can include the hierarchical keyword branch you want, with all its synonyms, without having the thesaurus then search for all branch keywords in the original branch?

You might say that it is not good practice to have repeated keywords, but that is how I have set up my hierarchy.  And, for example, I found another nonsense addition when I was keywording a train.  Naturally I wanted to include "rail", and was puzzled why moorhen and its Latin name were included, until I realised that IMatch was picking up from the thesaurus rail as the class of birds which includes moorhens.  Clearly it is impossible to have, on any sensible basis, a single keyword "rail" that you use both for trains and for birds!

Finally, the Metadata Keywords panel shows all three keyword variants, hierarchical and flat fields.  The second and third sub-panels show exactly the flat keywords I want to see, derived from the hierarchy seen in the top panel.  But I only want the contents as shown in the second or third panel written to file, not the top panel with the hierarchy separators.  But the hierarchical version is also always saved to file (I have checked the actual contents as exported by IMatch in other applications).  I have read the help files and your email and have tried (I believe) every combination of metadata preferences in Preferences, but once I have used hierarchical metadata from the thesaurus, I cannot write to file without this, so as to write only the flat keywords.  For my purposes, which include keywording for stock photography, I cannot have the hierarchy written in the JPEGs, I need just the flattened comma separated keywords.  Is this possible?  Would it be possible to select only one of the three metadata keyword variants shown in the metadata keywords panel to be updated, or is there a fundamental reason why all three tag fields must be updated together in the file as exported?

Many thanks for your help.

Graham

[Mario]

even the flat keywords entered are converted into the hierarchical tree. 

This is the sole reason for the lookup. To map flat keywords contained in your files and written by whatnot software into your thesaurus. This is what most people expect and why they like IMatch so much.
If you don't want this, just turn it off. This solves all the problems you are describing in your very long posts.

When you write the flat keyword beach (even when intentionally preventing the actual hierarchical keyword from being written) your flat keywords contain "beach" afterwards. But when you write back and IMatch re-imports the file, your thesaurus mapping, if enabled, is performed. Mapping "beach" to "WHERE|location|beach" or whatever you have defined in your thesaurus. This is how this works, how people want it. Don't like it, turn mapping of. For 99% of the users, all this "just works" as it comes out of the box.

[Jingo]

even the flat keywords entered are converted into the hierarchical tree. 

This is the sole reason for the lookup. To map flat keywords contained in your files and written by whatnot software into your thesaurus. This is what most people expect and why they like IMatch so much.
If you don't want this, just turn it off. This solves all the problems you are describing in your very long posts.

When you write the flat keyword beach (even when intentionally preventing the actual hierarchical keyword from being written) your flat keywords contain "beach" afterwards. But when you write back and IMatch re-imports the file, your thesaurus mapping, if enabled, is performed. Mapping "beach" to "WHERE|location|beach" or whatever you have defined in your thesaurus. This is how this works, how people want it. Don't like it, turn mapping of. For 99% of the users, all this "just works" as it comes out of the box.

I must admit - this is one of the main reasons why I stick with hierarchical keywords even though I technically do not need such complexity any longer... the flexibility it provides is nice especially when "beach" has more than one meaning... "Places|USA|Massachusetts|Cape Cod|Sandwich|Sandy Neck Beach"  is very different from "Clients|Photo Shoot|Hansen|Beach Wedding" ... choosing the specific (or expanding your hierarchy) keeps things unique.  For display purposes, I still only show the leaf - but the hierarchy is there and available in my keywords and thesaurus to keep it all tight.  It helps that I only use IM to add keywords but there was some cleanup/re-mapping work that was needed when adding "legacy" images that contained flat keywords.

[Mario]

IMatch supports you either way.

You can have 200 flat keywords all on the top-level in your thesaurus.
Or you organize them with group levels, which do not become part of the keyword written to the file => still flat keywords but comfortable organization in your thesaurus.
As you said, depending on how you tag your files, hierarchical keywords are the only way to make it possible to differentiate between a photo of the car "Tesla" and a photo of the man "Tesla" 

[graham1]

Mario, thank you for your reply.  I am sorry you thought my post was too long: I thought it best to describe the exact issue.

I will keep my reply short. My requirements include easily writing existing keywords (and their synonyms) out of the thesaurus into a selected file. Enabling the lookup gives rise to the issues I described. Disabling it means that synonyms along the branch are not written to the image file (unless you go along and manually click each check box, which is too tedious for me).

I am pleased that the way this works is what you and the vast majority of your users want. Unfortunately it does not work for me. Being able to write all the keywords and their synonyms from a single branch of the thesaurus tree in a DAM application via a single click/selection does not seem a big ask to me, but I guess my requirements are different from those of most others.  Thanks anyway for your comments.

Graham

[Jingo]

I still am a bit confused how "Carribean|Cuba|south coast" was added to the image.  I just set up my own example - metadata preference are identical.

I have the thesaurus setup with:
  Pets|Dog
  Animals|Dog

I add Animals|Dog to the image, write-back the file, and only get Animals, and Animals|Dog as keywords
I added Dog directly the the IPTC keywords field, filed and did a write back and didn't get any keyword updates.



** NEVER MIND ** - just realized this is only in play when you IMPORT the files for the first time... :-) 

[Mario]

Enabling the lookup gives rise to the issues I described. Disabling it means that synonyms along the branch are not written to the image file (unless you go along and manually click each check box, which is too tedious for me).

I don't follow.
The thesaurus lookup is performed when you import files into your database. it maps flat keywords to hierarchical keywords using your thesaurus.
This has nothing to do with synonyms.

When you have a keyword A (flat or hierarchical) with synonyms a,b,c and you assign the keyword A to a file, the synonyms a,b,c will also be assigned. And written to the file during write-back. These synonyms are treated like any other keyword. You will end up with A,a,b,c in both hierarchical and flat keywords. Absolutely no problem.

What issue do you have with that? This is usually nothing a user ever need to think about. All this just works.

[Mario]

I still am a bit confused how "Carribean|Cuba|south coast" was added to the image.  I just set up my own example - metadata preference are identical.

I have the thesaurus setup with:
  Pets|Dog
  Animals|Dog

I add Animals|Dog to the image, write-back the file, and only get Animals, and Animals|Dog as keywords
I added Dog directly the the IPTC keywords field, filed and did a write back and didn't get any keyword updates.

Please do not cross-post about different topics / issues / problems in the same thread. Open your own thread and describe your own problem. We then can discuss what is happening.
Please also detail your Metadata and Metadata 2 settings, your thesaurus (group and exclusion levels, the keyword contents of your image before you assigned your keywords, if you use versioning with metadata propagation etc.) All this may impact how IMatch deals with your keywords. Especially important is how you flatten keywords (what ends up in XMP/IPTC and how this then maps back to your thesaurus).

[Jingo]

I still am a bit confused how "Carribean|Cuba|south coast" was added to the image.  I just set up my own example - metadata preference are identical.

I have the thesaurus setup with:
  Pets|Dog
  Animals|Dog

I add Animals|Dog to the image, write-back the file, and only get Animals, and Animals|Dog as keywords
I added Dog directly the the IPTC keywords field, filed and did a write back and didn't get any keyword updates.

Please do not cross-post about different topics / issues / problems in the same thread. Open your own thread and describe your own problem. We then can discuss what is happening.
Please also detail your Metadata and Metadata 2 settings, your thesaurus (group and exclusion levels, the keyword contents of your image before you assigned your keywords, if you use versioning with metadata propagation etc.) All this may impact how IMatch deals with your keywords.

Sorry Mario - I'm not having the problem... GRAHAM mentions in his post that he doesn't understand why "Carribean|Cuba|south coast" was added to his image when he added "sidmouth" so just trying to help him make sense of this process.  Not sure when he added it (if upon import - I suppose because the entire hierarchy was added, broken apart (including south coast) - other "south coast" items are added?). 

[Mario]

"Issues" with keyword mapping are usually caused when the user breaks apart the individual levels into individual keywords and then the re-mapping of these keywords via his thesaurus creates new keywords on different levels. This has to be coordinated else it can produce unexpected results.

The thesaurus is clever, but it cannot handle every possible use case.
If true flat keywords are needed, mapping during import should be off or the thesaurus should only contain one level (optionally with a group level above).

[graham1]

OK, I will try to be as brief as I can. I set up a new database with a thesaurus limited to just a very few top level keywords.  I attach an export of this (as text). Then I imported into my database a single file with no keywords at all.  My goal was to write to this all the keywords and synonyms in the Sidmouth branch, no more and no less.

(1) I checked (in Metadata Preferences) both check boxes under Keyword Export and Keyword Lookup. In the Keywords Panel, "Enter new keywords here", I entered "Sidmouth".  Each time I clicked the metadata pencil in the thumbnail, it refreshed and added more of the thesaurus keywords, not only synonyms, but also from the branches that do not contain Sidmouth.  Result: lots of unwanted, irrelevant keywords added to the image.

(2) I deleted all keywords, and (with the same settings) I entered Sidmouth in the filter ("Enter a filter expression here...").  This entered no keywords immediately.  I checked Sidmouth at the leaf end of the thesaurus panel.  This added all the keywords in the Sidmouth branch, but none of the synonyms in it. The image thumbnail showed unwritten metadata: clicking it then led to the same result as in (1), namely multiple updates until most of the keywords in the thesaurus had been added.

So keyword lookup checked in Metadata Preferences retrieves a whole lot of keywords on export to the file that are not intended.  My next step was to uncheck both check boxes in Keyword Lookup, and then repeat the steps in (1) and (2).

(3) - repeating steps in (1) - I get Sidmouth as a single flat word and can select the Sidmouth branch.  Path elements and hierarchical keywords from the correct branch are written but no synonyms.

(4) -repeating steps in (2) - filtering on Sidmouth produces a view of the branch which has Sidmouth as a leaf.  No keywords at all are added for writing to the file unless Sidmouth is manually checked.

For the sake of completeness, unchecking the Keyword Import options made no difference to writing keywords and synonyms.

To summarise, if the thesaurus contains synonyms between the top level and the leaf level, I cannot see that there is any way of writing to a file metadata comprising only the keywords and synonyms in the branch in which you are interested, unless you manually go and tick every single check box for a keyword which has a synonym, right along the whole length of the branch.

I suppose one solution might be to have an option that when a branch leaf is selected, all superior levels are also ticked, so that their keywords and synonyms are exported to the file.

That is the best I can do to explain this, and I hope that the upload of my thesaurus I have used in this example will help others understand the issue, even if they do not agree it is a problem for them.  This started off as a query about flat as opposed to hierarchical keywords being exported, but this seems to me to be a lesser issue that that of being able easily to export synonyms at all.  If, as is quite likely, I am doing something wrong, hopefully this will enable someone to tell me how I can do what I want to do, adding keywords and synonyms to images that do not already contain them.

If nothing else comes from this, at least Sidmouth has had a lot of exposure to people who might not otherwise have heard of it.  It is a delightfully charming seaside town, well worth a visit!

Thanks to all concerned for their input.

Graham

[Mario]

When trying to import your fragment thesaurus file, I discovered a problem with the ISO language list in the IMatch resources. This caused the import to fail with a "Invalid argument" or similar error message. I have fixed this for the next release. Thanks for bringing this to my attention.


After importing your thesaurus, I assigned the keyword Sidmouth to a file. This produced the hierarchical keyword

Europe|United Kingdom|England|southern England|south coast|Dorset|Sidmouth

which is correct. On the first level of your thesaurus, the [Travel & Geography] level is marked as a group level so this does not become part of the hierarchical keyword. It is for organizational purposes only.

IMatch assigns synonyms on the keyword level, not all synonyms assigned to any of the other levels resulting to the keyword. So no synonym is assigned, because Sidmouth has no synonyms in your thesaurus. I guess you expect otherwise? If so, this feature is not available in IMatch.

If you would assign the keyword south coast, this would also assign the synonyms coastal and seaside.
Handling this differently would probably end up with 3 hierarchical keywords being assigned when you assign Sidmouth

Europe|United Kingdom|England|southern England|south coast|Dorset|Sidmouth
Europe|United Kingdom|England|southern England|coastal|Dorset|Sidmouth
Europe|United Kingdom|England|southern England|seaside|Dorset|Sidmouth

or, actually, since you have many more synonyms on higher levels, this would result in an entire bunch of hierarchical keywords, one for each unique combination of synonyms leading to your leaf-level keyword Sidmouth.

[graham1]

Thanks, Mario.

IMatch assigns synonyms on the keyword level, not all synonyms assigned to any of the other levels resulting to the keyword. So no synonym is assigned, because Sidmouth has no synonyms in your thesaurus. I guess you expect otherwise? If so, this feature is not available in IMatch.

Yes, this is exactly what I was hoping for.  I need all the synonyms at all levels to be there for my stock photography keywording.  Thank you for clarifying that this is not possible.  I am looking to export the synonyms with the keywords along the length of the branch - otherwise, what is the point of including synonyms? I have a lot of wildlife pictures, and when I export them I need to see not only the scientific names of the particular animal (which are included with the common name at leaf level) but also the scientific names which are synonyms further up the classification branch (e.g. mammal, synonym mammalia).

this would result in an entire bunch of hierarchical keywords, one for each unique combination of synonyms leading to your leaf-level keyword Sidmouth.

The discussion goes full circle! It was precisely because I do not want lots of hierarchical keywords saved to the file that I asked whether it was possible simply to have only flat keywords assigned to a particular image.  The hierarchy is there for organisational purposes, but when I assign keywords for export, I only need the flat keywords in the images I am going to post elsewhere.

A suggestion, if I may. In the Metadata Preferences, would it be possible to add under "Keyword Export" a third check box (maybe as a sub-box of "write path elements") called "Write synonyms", or something similar?  Maybe this is technically not possible, but to include the option here to write synonyms along the whole of the path elements, instead of just the path elements alone, would largely solve these issues.

This would in turn lead to a solution to the issue of including paths as well as flat keywords, which I have discovered during my endless happy hours researching this in IMatch.  The fix is this.  Having assigned and written all the keywords with their multiplicity of paths, the keywords are shown in flat form in the Metadata Panel third sub-panel (XMP).  Although not obvious, the contents of this sub-panel (the flat keywords, with no repetition from the many paths) can be copied to the clipboard. and then pasted to replace the contents of the top Hierarchical Keywords sub-panel.  Alternatively, the Keywords Panel can be used, to delete all keywords and replace them by those in the clipboard, so that they become flat keywords (with thesaurus look-up turned off!).  I can therefore deal with the issue of the multiple hierarchical paths, albeit in a somewhat clumsy way, if only the issue of there being an easy method for getting the synonyms along the entire branch could be solved.

This is not possible at the moment, as you explained, but hopefully the collapsing of the hierarchical structure to flat keywords will be useful to someone interested enough to be following this thread.  And I am glad that this discussion has at least resulted you being able to squash a bug, albeit I suspect a fairly minor one.

Graham

[Mario]

The functionality you need/want does not exist.
When you assign a keyword, IMatch assigns all synonyms of that keyword. It does not assign other keywords or synonyms, from any other part of the keyword hierarchy.
Since you only need flat keywords, why do you bother to produce a complex and deeply nested hierarchy in your thesaurus? Why not just add all keywords on the top-level (maybe with one or two group levels for easier handling) and then assign as many synonyms to each keyword as needed. Other stock photographers do it that way.
Or they use IMatch Attributes for multi-language / multi-agency keywords and a Metadata Template or Batch Processor template which combines the keywords from the Attributes and puts them into the output file.

add under "Keyword Export" a third check box (maybe as a sub-box of "write path elements") called "Write synonyms", or something similar?

This would not really work in general (maybe it would work for your precise use case).
If IMatch would add synonyms placed on all path elements, this could cause a lot of trouble during re-import and re-mapping to the thesaurus. I have not thought this through, mind, but I foresee many edge cases and problems. Keyword management across 3 or even 4 metadata standards is a bitch as it is, and I really don't want to complicate this further.

If all you need are flat keywords with many synonyms, setup your thesaurus accordingly. You may even want to look into the thesaurus element option ("Use sub-elements") as well.
And of course into Category Favorites for @Keyword categories, which allow you to add 2, 5 or 50 keywords with a single click or key press.

That said, as always, I want to hear about ideas and desired features. Feel free to write a feature request in the Feature Request board. This allows other users to see your request and to comment. If there is sufficient demand, I will consider it for a future release.