Hierarchical Keywords out of sync between exif, microsoft XMP and keywords panel

[JonesD14]

I have hierarchical keywords that are not in sync between the keyword panel (which always matches the Dublin Core Keywords and XMP Adobe Lightroom XMP Hierarchical Keywords) and the exif XP-Keywords and Microsoft XMP Last Keyword XMP. In some cases the exif XP-Keywords and Microsoft XMP Last Keyword XMP are out of sync with each other.

The keywords involved originated in three parts of iMatch: the Reverse Geocoding, the People Manager, and the Thesaurus. All of the keywords are hierarchical. No keywords were typed into the keyword panel.

The Reverse Geocoder generated WHERE keyword by default from the time I started using iMatch until I figured out where the WHERE keywords originated a few days ago.

I created my own controlled vocabulary in the thesaurus. Two keyword groups are relevant to this issue:

1) A {Party}|{Person}|{Person Name}|Name hierarchy that I created to store names of people (there are other keywords in the Party tree that aren't in any of the files with problems).

2) A {Place} name hierarchy that I created to record place names because Location in the Geo data stores street name and number

After I figured out where the WHERE keyword was coming from I made the following changes:

1) I renamed the Place keyword to Place Name, hoping to avoid conflicts when I changed the WHERE keyword to a Place keyword. When I changed it I responded to the prompt to tell iMatch to update all affected files (there were about 16 with place names).

2) I changed the WHERE keyword in the Geo & Maps Preferences to Place and added a properly formatted level after Country for the State, matching the syntax and naming of the other levels in the hierarchy

I then ran the Reverse Geocoder on individual files to generate the Place keyword. This worked as expected. It did not remove the WHERE keyword, so I deleted it manually in the keyword panel for all files with GPS data.

After that I went to the People Manager and tried the Keyword section of the Person Editor, at first on a single person. I set the keyword to generate to the thesaurus entry for Party|Person|Person Name|Janice Jones by entering text then selecting the keyword from the thesaurus entries that showed up in the list. iMatch proceeded to generate a Janice Jones keyword for the small number of files that had her with a face annotation but no party name keyword. I don't know if it replaced or ignored the files which already had the keyword.

I waited for a while until things seemed to be done processing in the background, then closed the application and shut down the computer.

The next day when I went in iMatch went immediately to updating files with metadata to write back. It then almost immediately hung. After no entries showed up in the application log for an hour and a half I looked at the log to determine the file it was working on when it hung. I selected that file, clicked on the icon indicating that the database was being updated and told iMatch to abandon the update for that file. This almost immediately cleared the update that were being done. There were still 204 updates in the write back queue, however, so I told iMatch to do those updates.

When these were done I had forty-some files still in the metadata write back queue. After a considerable amount of research and checking files, I found that a number of the problem files had issues with the generated Party Name keyword not being in sync across the metadata elements, while others still had the WHERE attribute (and it was out of sync) and some had issues with the Place Name being out of sync.

I turned off the Person Name keyword generation on the 15 or so people I had put it on. This took off all party name keywords (or at least most) in all files that had a party name, generated or manually input. There were still some files with party names out of sync, however. Most of these were able to be resolved by putting back Party Name keywords manually in the keyword panel, but some remained out of sync.

I then began working on each file individually to see what might fix the out of sync keywords. I discovered the Metadata2 preferences and turned off the Protect Existing XMP and Keep Existing XMP preferences, figured out how to delete the tag contents of the exif XP-Keywords and XMP Microsoft Last Keyword XMP and updated the keyword panel to have the necessary tags in Party Name and Place Name. Updating the metadata fixed some of the files.

In a number of cases I had to add the WHERE keyword back to the keywords in the keyword panel to make things match, and/or delete the generated Place keyword (not my preferred solution). After experimenting with different combinations of the metadata2 preference I managed to get things down to just a few problem files.

I did try redoing the reverse geocoding with no keyword generated to see if that would take out the WHERE keyword, but that didn't seem to work.

Eventually all of the keyword issues appeared to have been fixed, so I began replacing Party Name keywords from the thesaurus using the keyword panel. After somewhere between 30 and 60 minutes, two of the files reappeared in the metadata write back collection. Something had put the prior erroneous metadata back in the exif and Microsoft XMP keywords. This had been happening on a regular basis as I was figuring out how to fix the problem, but the delay was only a minute or two.

Nothing I had tried would work for either of these files, so I deleted them, rescanned them, and reentered the metadata. They are fine so far.

At some point along the way I had upgraded iMatch to the latest version. That appeared to work fine, but had no impact on the keyword issues (as expected).

I left the application running all night after that. It had nothing in the metadata write back queue at that point. When I checked on it the next morning there was still nothing in the write back queue, but only for a few seconds as another file came back with the original out of sync metadata.

I tried fixing it, but every time it came back with the incorrect metadata. After the initial metadata update the exif and Microsoft XMP keywords were not in the browser view and the exif log showed the metadata being cleared, then set to matching values in the XMP and DC keywords. Then the file updated again, with the exif and Microsoft keywords out of sync again. There were no entries in the exif log when this second refresh of the file happened. The options to protect and preserve XMP metadata were still set to No. I assume this was the read of the file after the metadata update to refresh the File Window, but I have no idea why it would sometimes take hours to refresh the window and metadata. It seems like some background process is running that finds the old metadata somewhere and puts it back in, regardless of the metadata2 settings, but just for certain files.

I also noticed that the files in the file window, which had the index number showing in the second row in the right half, were displaying  the index number left aligned when the definition of the file window specified it being right aligned.

At this point it looked like something weird was going on with the iMatch app itself, beyond the metadata sync issues, so I shut down iMatch. It asked if a wanted to run the backup. I said Yes, but the pop-up to run the backup never appeared and iMatch shut down abruptly.

I waited a few minutes, then clicked on the iMatch icon to reopen the app. I got a message that an instance of iMatch was already running, even though there was nothing indicated on my screen. I found an iMatch Application process running in Task Manager and closed it. I waited a few minutes, then rebooted the PC to clear any temporary queues.

When I rebooted and started iMatch it indicated it had shut down imcorrectly. I chose to restart the app with the last copy of the database. What came up was exactly like what it was before I shut it down. The issue with the index number in the file window seems to be fixed now, but the metadata issues are of course not.

Am I missing something in iMatch that would help me fix these issues (synchronize metadata, get rid of the remaining WHERE keywords and some stray people keywords that keep reappearing)?

Any ideas on how I got two instances of iMatch running?

So far the only fix that has appeared to work for the really problem files is deleting the file, rescanning, and reentering metadata.

There is a previous post from today that asks about Clearing Entries from the Country, State/Province, City and Location Fields (by moviemaker445) that doesn't concerns keywords, but does appear to have issues with inconsistencies across the exif and XMP metadata that may be similar to the issues I'm experiencing in trying to fix the inconsistencies in my metadata.

Thanks for any ideas or suggestions.

[Mario]

Welcome to the community.
I understand you have some problems with keywords.

and the exif XP-Keywords and Microsoft XMP Last Keyword XMP. In some cases the exif XP-Keywords and Microsoft XMP Last Keyword XMP are out of sync with each other.

IMatch only works with hierarchical keywords internally. During write back it syncs hierarchical keywords into "flat" XMP (dc:Subject) and, if the file contains legacy IPTC data, into IPTC keywords. If and how keywords are flattened is controlled via Edit > Preferences > Metadata.

IMatch ignores the proprietary and long ago abandoned XP keywords and Microsoft XMP Last Keyword , some versions of Windows used in the past. If you want to reuse these for some purpose, you'll have to transfer them via a Metadata Template or the Metadata Mechanic.

For helpful information about common keyword problems when dealing with out-of-sync keywords, see Keyword Problems in the Metadata Problems and Pitfalls section of the Metadata for Beginners help topic for more information.

Your post is quite long and I could not follow it entirely. You often write "Did not work", but not really show us what the actual result was.

It is also very important to see the metadata of the files causing the issue, so we can see which metadata mess they contain. Providing screen shots and sample images for download is always helpful for all things related to metadata.
Problems caused by an existing metadata and keyword mess in files during write back can only be analyzed by having the actual files.

After I figured out where the WHERE keyword was coming fr 

Create keywords from this expression

The process is quite simple. IMatch takes the expression you specify and writes the resulting keywords to the file.
Running reverse geocoding again with a different expression is additive, it does not remove keywords that may have been created by a previous reverse geocoding operation with a different keyword expression.

The same is true for keywords you associate with persons.
They are applied when you link the person to a face annotation or a file.
They are removed when you remove the link of the person to the face or file.

Changing the keywords associated with a person does add the new keywords, but not remove keywords previously added. IMatch does not maintain keyword histories for persons.

The thesaurus also plays an important role when keywords are imported (initially or after a write-back).
See Keyword Mapping during File Import

The easiest way to globally clean up keywords, rename them, or remove them, are the Thesaurus (Updating Keywords in the Database from Thesaurus Changes) and the @Keywords Category.

Tip: when IMatch marks a file as pending for write-back, pointing the mouse cursor at the write-back icon shows the first 10 tags IMatch needs to write.

If this includes keywords (Subject) and the pen comes on again after write-back, the problem are unresolvable out-of-sync keywords, and you need to apply one of the strategies given in the Pitfalls help topics (link above).

If you describe keyword issues, it always helps to
list the before keywords and after keywords, provide the corresponding thesaurus contents, include screen shots, describe what you did etc.

I discovered the Metadata2 preferences and turned off the Protect Existing XMP and Keep Existing XMP preferences, figure

Just make sure you understand the meaning and consequences of these features before you turn them off.
For 99% of the user base, the defaults work just perfect.

At this point it looked like something weird was going on with the iMatch app itself, beyond the metadata sync issues, so I shut down iMatch. 

"Something weird" is not helping us much, sorry.

IMatch has extensive built-in logging capabilities. See log file
Switch IMatch to debug logging and use Help menu > Support > Copy Log File to make snap shots of the log file after you encounter a problem. ZIP log files to reduce them by 90% before uploading and attaching to your posts.
Log files show us what IMatch is doing, if there are any problems, warnings from ExifTool, how the system is performing and more.

Any ideas on how I got two instances of iMatch running?

You cant.

There is a previous post from today that asks about Clearing Entries from the Country, State/Province, City and Location Fields (by moviemaker445) that doesn't concerns keywords, but does appear to have issues with inconsistencies across the exif and XMP metadata that may be similar to the issues I'm experiencing in trying to fix the inconsistencies in my metadata.

Totally unrelated. EXIF does not even contain location data.

As I said above, provide a minimum of information, some sample images, your thesaurus (export it and attach it) and IMatch log files when "something weird" happens.
This gives us a minimum of data to work with and help you.

[JonesD14]

Mario,

Thanks for the fast and thorough response.

I don't need to send additional information. I figured out the problem and the culprit is not iMatch. It's OneDrive. I had both the staging area I scan photos into to work on and the master photo collection in the OneDrive folder. OneDrive was adding the exif and Microsoft XMP keywords on updates after the initial input of the file to OneDrive, which created the infinite metadata write back loop. I moved the staging area to a folder outside of OneDrive (which deleted the file from OneDrive), fixed the issues there, and moved the files back to OneDrive when I was done with them. Fortunately, iMatch makes this easier than it sounds.

Most of the time, having the master photo collection in OneDrive doesn't cause the metadata write back loop. I'll look for a cloud archiving solution (as opposed to backup and restore) that doesn't insert out of date metadata, but for now my modified workflow will work.

Thanks for creating iMerge. It seemed like the obvious best choice when I was researching data asset management software, and it has exceeded my expectations. The documentation and support is unmatched in my experience in software that isn't enterprise level.

[Mario]

OneDrive was adding the exif and Microsoft XMP keywords on updates after the initial input of the file to OneDrive,

Hm, are you sure about this?
I would expect OneDrive to store data I upload unmodified and to return it to me as I've uploaded it.

I use OneDrive as an convenient tertiary backup service.

But I never upload anything that has not been locally encrypted. For very good reasons.
I live in Germany, and the EU and Germany privacy laws are very strict.

Servers hosted in the US are not considered as a secure and legal storage location under the GDPR.
By encrypting data before uploading to OneDrive, my data stays secure and legal. This also prevents OneDrive with messing with the images, videos, databases, source codes etc. I upload, since all it "sees" are encrypted archives.

I hence don't know if OneDrive really modifies the metadata in images one uploads. This is an irritating idea.
I know they are legally required to scan all images and videos uploaded for "illegal" content (based one laws of the US state the data center resides in - no problem with that), but modifying user data or image metadata is something totally different.