Metadata Write-back issue

[Trashcan Mike]

I am seeing strange behavoiur when trying to write-back metadata.

I change a couple of images with keywords and I write back the metadata on all files. I get a message saying there are over 1000 pending files.

Why?

I hit OK.

Wait...... and wait

Finished.

Choose write back metadata again....

It says I have ~1500 pending files.

Why?

Hit cancel.

Choose write back metadata again....

It says I have ~2500 pending files.

WHY??

Something is clearly wrong here.

Please help.

Thanks

Mike

[Mario]

I change a couple of images with keywords and I write back the metadata on all files.

How exactly do you do that?
Do you select the files and click the pen?
Or do you use one of the Write-back commands from the Commands menu?

When you look at the Pending Write-back collection in the Collection View, how many files does it show?

Was IMatch still importing files in the background, thus generating new files with pending writeback?
Do you use versioning with metadata propagation?

When you import files fresh into your iMatch database, IMatch will create "new" metadata. This is because of the mapping between legacy IPTC, EXIF, GPS and XMP that is performed during import. This means that files added for the first time will all have pending write-backs until written at least once. Maybe this is what confuses you?

[Trashcan Mike]

Hi Mario,

Thanks for the quick response.

Some answers to your questions - please remember I am quite new to the application.

How exactly do you do that?
Do you select the files and click the pen?
Or do you use one of the Write-back commands from the Commands menu?
The latter. I wait until they are all done and then if I choose that command again, it throws up thousands more.

When you look at the Pending Write-back collection in the Collection View, how many files does it show?
AFTER I write back all the files it shows 2774 still pending.


Was IMatch still importing files in the background, thus generating new files with pending writeback?
No

Do you use versioning with metadata propagation?
Not sure what this means, sorry.

When you import files fresh into your iMatch database, IMatch will create "new" metadata. This is because of the mapping between legacy IPTC, EXIF, GPS and XMP that is performed during import. This means that files added for the first time will all have pending write-backs until written at least once. Maybe this is what confuses you?
I don't think so . I have not imported files in a week yet every time I open iMatch I have huge numbers of pending files to process.

I was really hoping that this was something I did not understand but you are making it sound more serious.

[Trashcan Mike]

Is there any way to "clear" the pending metadata. It would be a useful starting point for problem solving.

[Mario]

Open some of the files in a file window.

Point the mouse cursor at the pen icon in the thumbnail panel. This will show you the first 10 tags IMatch wants to write. Maybe this gives a clue.
Often, with reoccurring write-backs, the problem is divergent metadata in the file, e.g. keywords which have not been properly synchronized between IPTC, XMP and hierarchical keywords.
Which applications did you use before IMatch?

Please attach one of the files which exhibit this behavior, ZIPpped.
If the file is larger than 2 MB you can send it to my support email address (https://www.photools.com/support/) with a link back to this topic.

You can clear pending write-backs by clearing the processing queue via Database > Tools.
But when IMatch thinks it needs to write-back data, it is right in 99.99% of all cases and marking the files as "current" may lose metadata.

[Trashcan Mike]

Hi Mario,

There only seems to be one tag (see image) which is the same for every image I looked at. It won't write back if I click the pen icon. WHat is a "dc\Subject " anyway and how do I get rid of it?

I have sent you the file and XMP. It's almost 10mb.

I was using Photo Supreme before and I have a bit of a mess of keywords that are in the XMP but not in the Thesaurus. could that be it? If need be I am happy to dump all the old keywords if it gets me on the right track. Please let me know how.

I appreciate the help. Everything else seems to be going smoothly. Just need to stop this write-back palava.

M

[Trashcan Mike]

This worked!

You can clear pending write-backs by clearing the processing queue via Database > Tools.

Yay!

[Mario]

IMatch wants to write back the XMP keywords. This indicates that there is a problem with mismatching keywords in the legacy IPTC record and the XMP hierarcicalKeywords.

[Mario]

Thanks for sending the file.
As I've assumed, the problem is the embedded metadata record in your NEF file. But only  partial record, looks like something written by Nikon Capture or a camera?
Since IMatch (for good reasons) manages XMP metadata for RAW files in a separate sidecar file, your files end up with two competing metadata records.
This will always cause problems.

I recommend to remove the embedded XMP record from your NEF files via the ExifTool Command Processor in IMatch.
Use the Delete XMP Metadata preset. This will remove the embedded XMP and from then all XMP data will only be managed in the XMP sidecar file. This avoids any problems makes your files compatible with all major applications.

Notes:

1. Make sure you have backups of your files!
2. Try with a few files first (select them in a File Window, then press <F9>,<E> to open the ExifTool Command Processor.
3. Select the "Delete XMP Metadata" preset and press <F9> or click on the "Run" button.

[Trashcan Mike]

Hi again,

Thanks for the help. Running the delete XMP now. It left the files I tested it on unchanged but I think the previous action is actually what solved things. Will remember in future.

Cheers

M

[Mario]

Which software did you use before?

The only software I know that insisted on embedded XMP into NEF files was Nikon Capture. This caused a lot of problems.

[Trashcan Mike]

Yes, I used NX along time ago. Since then Aperture until Apple hosed it. Tried Photo Supreme but it was awful.

[Mario]

This explains that. The decision of Nikon to embed XMP into NEF files (as the only software) caused a lot of workflow grief.
Many Capture users did not recognize this while they stayed in the Nikon World but fell rather hard when switching from Capture to other software.

IMatch can handle the embedded metadata, but there are obstacles like competing different data for the same tag in both the embedded XMP and the sidecar.
By definition, IMatch considers metadata embedded in the image as 'more important', naturally.

The best in these cases is to clean up the NEF files by removing the embedded XMP after IMatch has created a proper XMP record after import/write-back.
For the same reason I have deleted legacy IPTC data from my images long ago.

[Stoffel]

It's the same with me here. Or at least very similar.

I have imported 35.000+ pictures into a new iMatch database. After the import I wrote back all the pending metadata, leaving about 7000 pictures still pending. I tried to write all metadata back by selecting all pictures from the "pending metadata write-back" collection and by selecting the menu "Comands --> Metadata Write-back --> For all pending files"

Saving a single picture by clicking on the pen sometimes is sucessfull.

When I hover over the pen, it always says:
IPTC::AplicationRecord\Keywords
XMP::dc\Subject

Do you have any idea, how to handle this?

[Mario]

This most likely means that the keywords in the legacy IPTC record in your image file differ from the keywords in the XMP subject (flat keywords) and XMP hierarchicalSubject.

Please select one of the problem files in a file window and then run the ExifTool Command Processor via <F9>,<E>.
Select the "List Metadata" preset at the top and then run it with <F9>.
Select the text in the result pane on the right, copy into a text file and attach that text file to your reply.
You can also upload a problem image somewhere or send it to my support email address.

[ad1965]

Dear Mario,

I do seem to have a similar issue. I haven't really used IMatch a lot over the last 1 or 2 years. However, whenever I did it was always writing or reading metadata, hence, being very slow..... It didn't really bother me, because I hardly used it. But now I re-started with IMatch and I want to get it back into working condition. The "pending write-back" collection seem to re-fill again and again. Some files are resistent for the write-back command. They look the same as described here.

When I hover over the pen, it always says:
IPTC::AplicationRecord\Keywords
XMP::dc\Subject

Can I just send you the file with the metadata list of one of those files?

Thanks in advance
Axel

[Mario]

This means that IMatch wants to write-back legacy IPTC keywords and flat (non-hierarchical) XMP keywords.
We would need a sample file, and also a screen shot of the settings you have made under Edit > Preferences > Metadata.
If you use a thesaurus, we also need that - because fi you map flat keywords via the thesaurus to hierarchical keywords on import, the thesaurus may be the source.

Just attach this information to your reply and upload a sample image somewhere. Or attach it, you can attach files up to 2 MB.

A quick look at the existing keywords in your file as described above would probably already enough to figure this out.
Try to delete the IPTC keywords of a sample file to see if this clears up the mess. Often users have used different software on their files, and in many cases this software updated only some of the metadata records, causing a mess and different sets of keywords in IPTC and XMP. Deleting the IPTC keywords an then letting IMatch write-back and synch everything once is usually all that is needed to get a clean and standard-compliant set of metadata.

[ad1965]

Update.
I noticed the following:
After each cycle of writing back metadata the collection of pending files gets smaller. It started with some 5,000 pending files. After around 10 times telling the system to write back metadata I am now down to 89 files. Those seem to be resistant, the number doesn't get smaller.
Why does the system need several attempts to write the metadata? I was expecting that it does it in one go?

Thanks in advance
Axel

[Mario]

That IMatch sometimes requires two passes per file (see also the corresponding help topic) is also caused by not properly synchronized data in the image. After writing back for the first time and the following re-import, IMatch again produces "new" keywords (usually) and the second write-back is then able to properly synchronizes everything.

This is usually a problem when users have "touched" their files with to many sub-standard or old applications over the years. This often leaves a mess in the files and the users recognize this only when they look at the metadata of their files in an application like ExifTool or IMatch, which try to produce properly synchronized and standardized metadata.

[ad1965]

Thanks, Mario!
Attached please find a sample of the "resitant" files
In another post I will put a screenshot of the metadata preferennces (too large for one post).
Axel

[ad1965]

And here the screenshot.

[Mario]

This file does not seem to have any metadata. Please ZIP it before uploading it again (in case the community software stripped metadata after processing the flle).
I could write title, description, some keywords, rating and label to this file without any problem.

[ad1965]

This file does not seem to have any metadata. Please ZIP it before uploading it again (in case the community software stripped metadata after processing the flle).
I could write title, description, some keywords, rating and label to this file without any problem.

It does have metadata, which probably got lost in transmission.... I zipped it and attached again.

[Mario]

Now I see the metadata, thanks.
This file has a ton of keywords and it would be mind boggling to figure out the differences.

What I did to fix the mess:

In the Keyword Panel I switched to the text edit mode (button on the right side of the input field) and copied all keywords into the Windows clipboard.
Alternatively you can switch to a Metadata Panel layout that shows hierarchical keywords (e.g. the Browser layout) and copy from there.
Then I deleted all hierarchical keywords and triggered the write-back. This erased IPTC and XMP keywords from the file.
Now I pasted the keywords back and then let IMatch write-back the file again. IMatch re-ceated IPTC and XMP keywords, but now all are in sync. Problem solved.

I recommend you do the same with the problem files. This is quick to do and apparently works just great.

[ad1965]

Many thanks!