Need hlep with metadata write-back failing for 181 files

[Robert Wheeler]

Metadata write-back is failing for 181 of my 57,542 images. Although this is a small percentage, I would like to resolve the problem for these 181 files. I began using iMatch this month. My thesaurus is a newly cleaned-up hierarchy that seems to perform very well as I use it to add keywords.

When I try to use metadata write-back for any of these 181 files, the yellow pencil icon goes away briefly but then quickly reappears. It does not matter whether I click the icon with one file selected or with all these files selected. Using the menu command for metadata write-back produces the same result whether issued for selected files or all files needing write-back.  Performing write-back twice does not correct the problem. The files remain in the Collection of files needing metadata write-back.

The write-back process does work correctly for other files (when I add keywords, for example).

Running ExifTool Command Processor to delete legacy IPTC (IIM) metadata for these files generates a notice that the deletion was successful, but the files remain in the Collection of files needing metadata write-back.

Antivirus software (Malwarebytes) has been told to ignore the iMatch database folder and my image folders. That did not correct the issue.

I ran Database Optimization, which reports: "The database has no errors."

The iMatch Dashboard shows 35 files as having problems writing metadata. That will be a good group to address as a separate issue, but none of these are in the Collection of 181 files needing metadata write-back.

When I show the Collection of 181 files as a tabular list, I can efficiently hover over the yellow pencil icons for each image to see the "List of Tags to Write." In every case, only one tag is listed per image.
•   For 177 of the files, the tag listed is XMP::dc\Subject, and all of these are Sony awr files. I have other awr files that do not have the problem of metadata write-back failing.
•   Four other files (one jpg, three tif) need to write the tag XMP::xmp\CreateDate.

Here is the Medatdata Analyst report for one of the arw files identified as needing the Subject written:
----------------------
Metadata Analyst Results. Version 2023.7.2. 2/13/2024 10:23:55 AM
File analyzed: F:\Photos\Images\2023\202306\20230606s_IslandDay3\Selects\20230606_1840_1537_v0.arw
Errors: 2
Warnings: 24

Warning: [System] File has unwritten metadata (pending write-back).<br/>The metadata loaded from the image and the data in the database may not match.
Warning: [Metadata] Warnings: 'IPTCDigest is not current. XMP may be out of sync'
Warning: [Legacy IPTC] Character Set Encoding: unspecified.
Warning: [XMP] Embedded XMP record (XMP Core 5.6.0) and XMP sidecar file (ILCE-1 v1.32) found.
Warning: [XMP] Embedded XMP rating is 2.
Warning: [XMP] [ExifIFD]:DateTimeOriginal not mapped to [XMP-exif]:DateTimeOriginal (embedded).
Warning: [XMP] [IFD0]:Copyright and [XMP-dc]:Rights (embedded) mismatch.
Warning: [XMP] [IFD1]:Copyright and [XMP-dc]:Rights (embedded) mismatch.
Warning: [XMP] [IPTC]:CopyrightNotice and [XMP-dc]:Rights (embedded) mismatch.
Warning: [XMP] [IFD0]:Artist not mapped to [XMP-tiff]:Artist (embedded).
Warning: [XMP] [IPTC]:By-line not mapped to [XMP-tiff]:Artist (embedded).
Warning: [XMP] [IFD0]:Orientation not mapped to [XMP-tiff]:Orientation (embedded).
Warning: [XMP] [ExifIFD]:UserComment not mapped to [XMP-dc]:Description (embedded).
Warning: [XMP] [ExifIFD]:UserComment and [XMP-dc]:Description (sidecar) mismatch.
Warning: [XMP] [IPTC]:Caption-Abstract not mapped to [XMP-dc]:Description (embedded).
Error: [Keywords] Different XMP keywords in embedded XMP record and sidecar file.
Error: [Keywords] Different keywords in IPTC and XMP (embedded).
Warning: [Detailed Validation] Entries in SubIFD are out of order
Warning: [Detailed Validation] Tag ID 0x011a XResolution out of sequence in SubIFD
Warning: [Detailed Validation] Tag ID 0x7000 SonyRawFileType out of sequence in SubIFD
Warning: [Detailed Validation] Tag ID 0x0111 StripOffsets out of sequence in SubIFD
Warning: [Detailed Validation] Tag ID 0x74c7 SonyCropTopLeft out of sequence in SubIFD
Warning: [Detailed Validation] Tag ID 0x0102 BitsPerSample out of sequence in SubIFD
Warning: [Detailed Validation] Tag ID 0x0142 TileWidth out of sequence in SubIFD
Warning: [Detailed Validation] Non-standard format (undef) for IFD0 0x8649 PhotoshopSettings
Warning: [Detailed Validation] IPTCDigest is not current. XMP may be out of sync

----------------------
Metadata Analyst Reports on several other files are similar, with variations in details of mismatched fields.

I see a variety of things that may need to be corrected. However, I am not confident in how to make the corrections. I also am unclear which corrections would be most important to accomplish first.

I have iMatch and Photo Mechanic as my most comfortable tools for modifying metadata. I also have Capture One as an alternative. I have Adobe Bridge and Adobe Lightroom, but am not certain that they would put correction into the necessary fields. I have ExifTool, but I am very much a beginner with that (have only used to examine metadata).

I would welcome guidance about which corrections should receive priority and how to make the corrections. Happy to send more information if that would help.

Thank you,

Robert Wheeler

[Mario]

Metadata write-back is failing for 181 of my 57,542 images.

What is failing? Does ExifTool refuse to write to a file with damaged metadata? Is there a file system issue?

The Metadata Analyst results just tell us that the metadata in the file is a mess and XMP metadata is out-of-sync with native EXIF/GPS data. Which is pretty common when users use non-pro software or just software that does not care to modify metadata in their files. Most of this can be fixed by IMatch/ExifTool during write-back.

To see what the actual problem is:

Open the ExifTool output panel via View menu > Panels > Output Panel

Switch IMatch to debug logging via Help menu > Support
Write back one or two of the problem files
Make a copy of the log file via Help menu > Support > Copy...
ZIP and attach. See log file

Ctrl+A in the Output Panel, paste into a new text file (Notepad)  save and attach.

This will show us if ExifTool is reporting warnings or errors and what IMatch is trying to write to the files.

[Robert Wheeler]

Thank you for your prompt response and detailed instructions.

By "fail" I simply meant to report the symptom that any attempt to perform write-back on the problem files results in the yellow pencil icon briefly going away then reappearing, with the file remaining in the collection of files needing metadata write-back.

I followed the instructions to enter debug logging mode. I selected one problem file where the hover menu over the yellow icon listed the required write-back as XMP::dc\Subject. I used the menu to request metadata write-back for the selected file. I then repeated the process for another problem file where the hover menu over the yellow icon listed the required write-back as XMP::xmp\CreateDate. In both cases, the yellow write-back icon briefly went away then promptly came back.

I have attached the requested log file in zip form. Looking forward to next steps.

Robert Wheeler

[Mario]

By "fail" I simply meant to report the symptom that any attempt to perform write-back on the problem files results in the yellow pencil icon briefly going away then reappearing, with the file remaining in the collection of files needing metadata write-back.

This can be normal. Try a second write-back.
See Files Pending Again After Write-Back in the Metadata Problems and Pitfalls section in the Metadata for Beginners help topic.

Users run into the same problems over and over and I'm a bit tired to explain it every time in community posts. Hence the "Metadata Pitfalls" chapter in the help.

In the MD analyst output, look at these lines:

Error: [Keywords] Different XMP keywords in embedded XMP record and sidecar file.

Error: [Keywords] Different keywords in IPTC and XMP (embedded).

This means that your ARW file has both embedded XMP data (bad) (written by XMP Core 5.6.0 ) and XMP data in a sidecar file (written by SONY ILCE-1 v1.32).

The camera/software you have used to work on this image created two sources of truth, and they don't match.
And there is even legacy IPTC metadata in the file, which is a 20 year old metadata format that usually only gets in the way. In this case, the XMP keywords embedded in the ARW and the keywords in the IPTC embedded in the ARW don't even match!

Your file has keywords in the embedded XMP, the XMP sidecar file and legacy IPTC, and these were written by two applications and they are out of sync. There is no way for IMatch to fix this mess by itself.

RAW files should not contain XMP metadata, they should only have XMP in XMP sidecar files.
IMatch sees the embedded XMP data in the image and prioritizes it over the XMP it finds in the sidecar file.
IMatch writes XMP data for RAW files only into the XMP sidecar file. At this point, latest, there will be two sources of truth

Remove the embedded XMP metadata from the ARW and, if you don't really, really need it, the legacy IPTC data too. Then there is only EXIF/GPS in the RAW and XMP in the sidecar file, which is ideal.

Make backup copies of all the problem files.

Use the The ExifTool Command Processor with the "Delete XMP Metadata" and "Delete IPTC (IM3) Metadata" presets for one of the files with the problem. Wait until IMatch as re-ingested the file. Verify metadata for completeness. Now write back, which should work just fine.

In case of success, select the other files with the same problem and rinse and repeat.
Keep the backups for a couple of months, just in case.

[Robert Wheeler]

The help topics you mention provide excellent information. They guided my initial efforts before I resorted to this forum. The forum posts on this topic were also informative. I had tried second write-back and had also tried deleting legacy IPTC IIM metadata, both without success. I understood that mismatched metadata could prevent write-back from working, but I was unclear about the optimal way to make corrections to resolve the issue. Your additional instructions helped.

Adding the "Delete XMP Metadata" to the process worked. Perhaps I missed that suggestion in the help resources.

I began by selecting two of the problem files, one after the other. For each, I executed "Delete XMP Metadata" and "Delete Legacy IPTC (IIM) Metadata." I then issued the command "Metadata Write-back for Selected Files." For the first file, the yellow write-back icon went away, and the image vanished from the collection of files pending metadata write-back. The second file required an additional write-back, which then produced the same happy result.

I was not able to find an ExifTool Command Processor preset for "Delete IPTC (IM3) Metadata." But the preset for "Delete legacy IPTC (IIM) metadata" was present. Since that one is mentioned in the help resources, I assume that is the one intended (favorable result in any case).

I will gradually work through the remaining files to finish the cleanup process.

I totally agree with your statements about where metadata should live. My few problem files mostly relate to use of software in the past that did not adhere to metadata standards. I am delighted that iMatch adheres to the metadata standards and that it has tools to identify and correct issues like this.

[Mario]

Very good

I mention that in Multiple XMP Records but don't give the name of the preset. The name may change and it will be named different in different IMatch user interface languages.

My few problem files mostly relate to use of software in the past that did not adhere to metadata standards.

You would be amazed how many of the modern and popular image editors and cloud-based photo storage providers don't give a sh*t about metadata. As long as your images are in their system, you won't notice. And when you switch to another product, you're no longer their customer and they care even less. It's just sad.

[Robert Wheeler]

Since this thread has attracted a fair number of views, I will report how well the rest of the database cleanup process went.

All 181 files mentioned earlier have now gone through the metadata correction steps successfully. There are no files remaining of the list of files pending metadata write-back.

The was a separate group of 35 files that iMatch could not write to (dashboard showing "there was an error while writing back metadata to this file. The file may be write-protected, the format unsupported or the file itself may be corrupted." The problem mainly was file corruption, which iMatch would not be able to correct. I think it is excellent that iMatch classified these files as having a different type of problem than the other files where metadata write-back was the issue.

Twenty five of the 35 files were tif files that could not be opened in Photoshop – generating an error message about an "unexpected end-of-file was encountered." That typically happens with corrupted files. I have the original raw files available (and working correctly in iMatch), so the best solution was to delete the defective files rather than waste time on looking into file repair tools.

Six of them were tif files that could not be opened in Photoshop but generated a different error saying "could not complete your request because of a program error." That can happen for many reasons, including file corruption. Again, all of the source files were available, so deletion rather than repair was the most efficient solution.

Four of them were tif files that Photoshop was able to open without producing an error message. I performed "save as" to have Photoshop store these files under new names. Sometimes opening then saving a file can correct glitches. iMatch did a good job of noticing the newly saved/renamed tif files and made them available in the database with no problems identified. I was able to use iMatch to add new keywords to these newly-saved files without any problem.

In conclusion, my iMatch database has been cleaned of problem files within a short time of my starting to use the program. The tools and capabilities of iMatch are quite impressive. And the technical support is outstanding.

[Mario]

Very good result!

It is not uncommon that users notice problems with their files when switching to IMatch. Because this usually means that files are read which have been created many years (often decades) ago and processed by a wide variety of software over the years.

Metadata issues, corruption caused by bitrot or hardware/software problems are detected when IMatch ingests files and the user is notified. Then the cleanup must begin, which requires individual actions, depending on the problems found.

About 200 problem files is quite good. I have seen much worse.