Write-back of Metadata

[rienvanham]

Hi Mario,

In the past week I have updated all my (personal) photo's with IMatch and I found it (for the first time in many years) the perfect tool for doing this work. I did a lot of keywording (made all normal keywords hierarchical). Now I'm writing back the keywords into the files but a strange, small problem is that the "pencil" keeps coming back in 85 of my 14.500 photo's. The popup states "The file has unwritten metadata .... "XMP::dc\Subject". The file-extension is ARW. It is not an ARW-specific problem because I have many more and there the metadata was written normally.

Am I doing something wrong? (I am quite new (come from Lightroom and Photo Mechanic Pro - what a relief is iMatch... ;-) )).
Thanks in advance,

Rien van Ham

[Mario]

The typical reason for this is out-of-sync keywords in legacy IPTC metadata and XMP => Different keywords in IPTC and XMP. Usually caused by applications not synchronizing keywords properly.

When the problem goes away when you write back the file 2 times, all is well. IMatch was able to synchronize the keywords.
If the problem stays, you need to manually check the keywords in the file and fix them. This may also be influenced by your Edit > Preferences > Metadata 1 settings (how you flatten keywords) and your thesaurus.

Run the Metadata Analyst for one of the problem files, then use the green Copy Results button to copy and paste the results into your reply.

[rienvanham]

Metadata Analyst Results. Version 2020.14.2. 3/26/2021 4:16:42 PM
File analyzed: E:\Bestanden\Originele Foto's\2019\20190904\20190904_144222_Fietsvakantie.arw
Errors: 2
Warnings: 14

Warning: [System] File has unwritten metadata (pending write-back).<br/>The metadata loaded from the image and the data in the database may not match.
Warning: [GPS] Altitude missing.
Warning: [Legacy IPTC] Character Set Encoding: unspecified.
Warning: [XMP] Embedded XMP record (Image::ExifTool 11.54) and XMP sidecar file (photools.com IMatch 20.14.0.2 (Windows)) found.
Warning: [XMP] Embedded XMP rating is 0.
Warning: [XMP] XMP-exif GPS Altitude missing.
Warning: [XMP] XMP-exif GPS Altitude missing.
Warning: [XMP] [ExifIFD]:CreateDate not mapped to [XMP-xmp]:CreateDate (embedded).
Warning: [XMP] [IFD0]:Artist not mapped to [XMP-tiff]:Artist (embedded).
Warning: [XMP] [IPTC]:By-line not mapped to [XMP-tiff]:Artist (embedded).
Warning: [XMP] [IFD0]:Orientation not mapped to [XMP-tiff]:Orientation (embedded).
Warning: [XMP] [ExifIFD]:UserComment not mapped to [XMP-dc]:Description (embedded).
Warning: [XMP] [ExifIFD]:UserComment and [XMP-dc]:Description (sidecar) mismatch.
Warning: [XMP] [IPTC]:Caption-Abstract not mapped to [XMP-dc]:Description (embedded).
Error: [Keywords] Different XMP keywords in embedded XMP record and sidecar file.
Error: [Keywords] Different keywords in IPTC and XMP (embedded).

[Mario]

1. The ARW file has embedded XMP data. How come?
IMatch by default stores XMP data for RAW files always in sidecar files, not in the RAW.

2. Warning: [Legacy IPTC] Character Set Encoding: unspecified.
It seems the ARW also has a legacy IIM3 IPTC record.

3. Error: [Keywords] Different XMP keywords in embedded XMP record and sidecar file.
Error: [Keywords] Different keywords in IPTC and XMP (embedded).

This means that the file not only has a superfluous (and most likely outdated) XMP record, but also different keywords in the embedded XMP and the sidecar and in the embedded legacy IPTC and the other keywords from XMP.

My recommendation:

1. Delete the legacy IPTC record from the RAW. This data format has been discontinued 15 years ago.
2. Delete the embedded XMP record in the RAW. IMatch will not update it and having two sources of truth for metadata is bad.

The ExifTool Command Processor has presets for deleting legacy IPTC and XMP metadata.

The usual practices apply:
- only work on a copy of the file first
- make sure it still works in whatever applications you use
- make backups of the files you treat that way
- keep these backups for several months

[rienvanham]

Hi Mario,

I followed your suggestions and I could only say: Wow, perfect, what a great support!

It works like a charm: all information is now written and the pencil is not coming back!

Thanks a lot, Danke schön, Dankjewel!

Rien.

[Mario]

Very good.

[rienvanham]

Hi Mario,

I don't know if this gives enough information:

iMatch couldn't write back the information to a file:

04.10 11:48:37+23984 [1934] 01  W> ETWARN:Warning: Error rebuilding maker notes (may be corrupt) - E:\Bestanden\Originele Foto's\2018\20180510\20180510_112818_Weekend Duitsland.jpg
Warning: Error rebuilding maker notes (may be corrupt) - E:\Bestanden\Originele Foto's\2018\20180510\20180510_112818_Weekend Duitsland.jpg
  'V:\develop\IMatch5\src\IMEngine\PTETWrapper.cpp(3514)'
04.10 11:48:38+ 1688 [1934] 02  I> E:\Bestanden\Originele Foto's\2018\20180510\20180510_112818_Weekend Duitsland.jpg loaded in 297ms. Result: 2
04.10 11:48:38+   62 [1934] 02  I> E:\Bestanden\Originele Foto's\2018\20180510\20180510_112818_Weekend Duitsland.jpg loaded in 297ms. Result: 2
04.10 11:48:41+ 2922 [2D3C] 01  W> ETWARN:Warning: [minor] Unrecognized MakerNotes - E:\Bestanden\Originele Foto's\2018\20180510\20180510_112818_Weekend Duitsland.jpg
  'V:\develop\IMatch5\src\IMEngine\PTETWrapper.cpp(2241)'
04.10 11:48:42+  625 [1934] 02  I> E:\Bestanden\Originele Foto's\2018\20180510\20180510_112818_Weekend Duitsland.jpg loaded in 312ms. Result: 2
04.10 11:48:42+   63 [1934] 02  I> E:\Bestanden\Originele Foto's\2018\20180510\20180510_112818_Weekend Duitsland.jpg loaded in 312ms. Result: 2

But after a restart it could!

Perhaps this helps.

Regards,
Rien.

[Mario]

The Error rebuilding maker notes usually indicates a damaged maker note (mostly caused by an application saving the image but not removing the maker notes or not updating the maker notes to match the new file).
Probably ExifTool could fix that during the first write-back, which is why the second write-back worked.

That's the problem with all the proprietary and undocumented maker notes your camera vendors stuff into your images. They contain absolute byte offsets to find data, and these offsets are no longer correct if the file is saved or converted. ExifTool knows about this and handles it (or skips the file). But so much other software gives a sh*t and just copies the maker notes and breaking them.

If camera vendors would finally abandon the 30 year old EXIF format and use XMP instead, all these problems were solved immediately. But they give a sh*t because users don't know or care (so, no pressure) and it would also cost them a few dollars to update their firmware to produce XMP instead of EXIF. So we drag along, still dealing with 30 year old EXIF formats and undocumented maker notes...

[rienvanham]

Hi Mario,

One correction: it wasn't the second time iMatch succeeded. I tried it many times, all without success. Only after a restart of iMatch it succeeded (in the first try).

Thanks!

[Mario]

Strange. Virus checker?

[rienvanham]

mmmm, I'm using (already for many years) Bitdefender Total Security.

I added exiftool.exe and imatch2020x64.exe to the exceptions.

Any other suggestions?

[lbo]

I added exiftool.exe and imatch2020x64.exe to the exceptions.

just a note about this: The Activestate+PAR/pp based ExifTool shipped with IMatch runs the real ExifTool Perl environment in the temp directory. Exiftool.exe is a self extracting archive.

My version of ExifTool ("alternate installer" mentioned on exiftool.org) is cleaner in this respect but I don't suggest to fiddle with your IMatch installation since it's not clear whether this is the cause of your problem.

Oliver

[Rene Toepfer]

The Error rebuilding maker notes usually indicates a damaged maker note (mostly caused by an application saving the image but not removing the maker notes or not updating the maker notes to match the new file).
Probably ExifTool could fix that during the first write-back, which is why the second write-back worked.

That's the problem with all the proprietary and undocumented maker notes your camera vendors stuff into your images. They contain absolute byte offsets to find data, and these offsets are no longer correct if the file is saved or converted. ExifTool knows about this and handles it (or skips the file). But so much other software gives a sh*t and just copies the maker notes and breaking them.

I have this issue with DNG of my Google Pixel 3XL. The maker notes cannot be fixed by EXIFTOOL (used parameter -F at command line). Even it is not the best solution, I skip such files. This is much better than getting corrupt metadata.

[Mario]

If this happens for unmodified, out-of-camera, DNG files: I'm sure Phil Harvey from ExifTool would like a sample.
If you have processed the files, the tool you have used to process them probably did something to the maker notes.

[Rene Toepfer]

OK, do you think that he is open for such issues? That would be great. Of course I still have the unmodified DNG.

[Mario]

Sury. You can reach him via the ExifTool forum: https://exiftool.org/forum/