The headache and the mystery of metadata

[jmsantos]

Again testing IMatch. I am a Lightroom user always looking for a DAM software that improves the management of photographs, both of an institutional Archive and of my personal and professional production. I have already written other times in this forum about the problems I encounter with managing RAW file metadata (NEF and RAF generally), especially with the sidecar that Downloader Pro generates and when I combine the two applications, IMatch and Lightroom. I have not found a satisfactory solution yet.
In this new test I have focused only on IMatch. The Metadata and Metadata 2 tab are set to their default values, so that everything is in accordance with MWG compliance. The process is the following:

- Download the card with Windows File Explorer to a folder managed by IMatch.
- Renamed in IM with the YYYYMMDD_HHMM_NNNN.EXT template
- I add author, Copyright and other metadata with Metadata Template.
- Write Back for IMatch to embed the metadata and generate the XMP sidecar.

At this point the Author field disappears 

I've read tons of help information, forum posts, Metadata Analyst and Exiftool reports, I follow the advice about not changing anything that affects MWG Compliance, but I can't find an explanation.

I have had the same problems with NEF, especially when Lightroom has modified the metadata entered in IMatch; just by saving the processing data, Author and Copyright disappeared. It doesn't matter if you set IM by default or activate Favor XMP sidecar for some extension, at some point you lose something. I am sure that many IMatch users will work together with LR, and I would like to know what options they use to avoid problems.

But in this test that I have done, LR has not intervened at all, only IMatch. You can download the ZIP file with the following content:

- DSCF9495.RAF original RAW image of the card, without modifications.
- 20210424_1144_9495.RAF renamed image with metadata added in IMatch.
- 20210424_1144_9495.xmp sidecar from the previous image.
- ExiftoolPanel.txt copies the Exiftool Output Panel when adding metadata.
- imatch_log20210429 debug log file.
- MetadataAnalyst.txt Metadata Analyst report after write back.
- Metadata Analyst Results - 24.json full report of Metadata Analyst.
- Set Copyright Info JMS.immdt Metadata Template.

https://1drv.ms/u/s!AqOdfCDSkB4YsGKHXpqSC-GBHNCw?e=BhuuT5

Thanks

[Mario]

I add author, Copyright and other metadata with Metadata Template.

Which tag do you set in your template?
Note that author is a tag that exists (under various names) in several metadata standards.

What happens if you change the author in the Metadata Panel and then write back. Does it also vanish?
Since I'm really busy and I already have a queue of over 10 tickets, all with files to check for metadata issues, it will help you probably quicker when you check this.
It will take several days for me to look at your files and tell you what the problem is.

[jmsantos]

I understand that you are busy, there is no rush.

My template the Author tag is XMP :: dc \ Creator (I attach image).
I have added the Author data in the Metadata Panel again and then write back, ok. But then I add data in Headline and Description in the same panel, and when write back, Author disappears again. I attach image with the Metadata Analyst info.

Thanks

[Mario]

Please check if your RAW file contains an XMP record (via the ExifTool Command Processor).
If so, remove it (also via the ECP). There should be only one XMP record for RAW files, in the sidecar. Else you deal with two sources of truth and IMatch only updates the XMP sidecar file for RAW files.

[jmsantos]

I don't see any XMP records in the RAF file.
I attach the ECP List Metadata output.

[Mario]

No XMP data in the file so no issues there.

Do you have changed any of the default Metadata settings (especially the MWF compliance)?
The file contains an image description "Panel informativo sobre las Termas de Munigua" and also a EXIF user comment "Panel informativo sobre las Termas de Munigua" and both tags are also mapped to the description field in XMP.

This should just work. Analyzing this will take time, and probably it's not IMatch in the end, as so often. I will look into this next week.

[Mario]

I have downloaded the ZIP file, thanks.

It contains two RAF files:

A) 11290\20210424_1144_9495.RAF with a matching XMP sidecar file.
B) DSCF9495.RAF without a sidecar file.

I first added metadata in the Metadata Panel for file A. Author, description, headline, title, keywords.
The data was correctly stored, no data was lost. I then changed the rating and label and performed a second write-back.
The data is still OK. The XMP file looks good, the existing EXIF data in the RAF has been updated correctly. OK.

For file B) I could reproduce the mysteriously vanishing Author tag. Very strange.
Because the XMP file created by IMatch has the correct dc:Creator tag with the correct value (this is what Author is called on the XMP level).

I then checked the metadata in the RAF. It has two separate "IFD" sections, both with an "Artitst" tag. This is the tag that must be synchronized with the dc:Creator tag in XMP (both ways).
The metadata in the file looks like this after the write-back:

[IFD0]          Artist                          : IMATCH-Author
[IFD1]          Artist                          :

The first IFD artist has been updated correctly when ExifTool mapped XMP back to EXIF. But the second IFD was not. This is the IFD for the thumbnail image contained in the RAF, probably. It reports a minimum of info, like 72 DPI and 8970 bytes length.

It seems that the duplicate Artist tag in the second IFD overrides the EXIF artist seen by IMatch during import (during import IMatch maps EXIF data back into XMP). I have not seen a similar case and I will set this aside for some later time, when I have a day or two to dig into this. This is just how ExifTool has implemented the XMP => EXIF mapping.

I fixed the problem by by removing the unexpected secondary Artist tag from the RAF. You can do that quickly using the ExifTool Command Processor (ECP).

Copy & paste the following arguments into the ECP and save them under Delete IFD1 Artist or similar.

# im-warn
-overwrite_original_in_place
-ifd1:artist=
-charset
filename=UTF8
{Files}

NOTE: Test with a copy first.
When you run this, it will delete the empty Artist in IFD1 from all selected files.
Making changes to the Author tag afterwards then "stick".

Another example of the typical metadata mess encountered frequently. I would put that into the same category as a camera embedding a minimal XMP record in the image, with only the "Rating=None" value. Just to mess up things.

[jmsantos]

Wow! Thank you, Mario. It works perfect.

Now I have to investigate why that double tag is written, as in some photos it is present and in others it is not.

Would there be any way to filter in IMatch the ones with that IDF1:Artist tag?

[Mario]

Probably your camera writes the extra empty Artist tag. Cameras do all kind of weird stuff with metadata, depending on make, model, firmware version and competence of their developers. Metadata is not something camera vendors care much about - else they would have stopped using EXIF 10 years ago in favor of XMP.

Would there be any way to filter in IMatch the ones with that IDF1:Artist tag?

No, IMatch does by default not import this tag so it leaves no trace in the database.
It only causes problems during XMP->EXIF mapping during write-back.

Maybe use a normal metadata value filter to find all files without a value for Author?
Or use the ExifTool command line tool to scan entire folders for files with or without a given value? There are examples for in the docs https://exiftool.org/exiftool_pod.html

[jmsantos]

Well, I have confirmed my Fuji X-E3 camera writes double Artist tags in [IFD0] and [IFD1], both in RAF and JPG files, checked with Exiftool in the cases I was able to inspect.
I created the category "No Author" to filter out test images written with my template. Applying with ECP the "Delete IFD1 Artist" arguments you suggested, the Author data that was hidden in [IFD0] Artist and in XMP::dc\Creator appears. Perfect, solved!
Now I have to adjust the workflow with Fuji files. I don't know if it is possible (and if it would be safe) to run ECP when indexing.

I already wrote on this forum in 2019 and 2020 about the problem I had with images with Downloader Pro and Lightroom metadata. I couldn't find a solution and that's why I couldn't decide to buy an IMatch license. The source of the problem was that I was using Downloader Pro to download, distribute into folders and write only a few metadata, thus creating an XMP sidecar with this content:

[XMP-dc] Title : 20190510_131403_4698
XMP-dc] Rights : © 2019 José Manuel Santos Madrid
[XMP-dc] Creator : José Manuel Santos Madrid
[XMP-photoshop] Authors Position : Fotógrafo
[XMP-xmpRights] Marked : True

In IMatch at some point Author, Copyright or both disappeared in my RAW files from different Nikon and Fuji cameras. Usually this happened after developing in LR and saving the changes. The solution of changing the setting to "Favor XMP sidecar" didn't convince me and didn't completely solve the problem. But reviewing all this, ubacher's message (https://www.photools.com/community/index.php?topic=9039.msg63488#msg63488) and Exiftool, I think I have found a possible solution.
The idea is to transfer to EXIF tags the data that I know exists in the sidecar file (but IMatch does not present in the metadata panel) by running in ECP the following arguments:

# im-warn
-overwrite_original_in_place
-charset
FILENAME=UTF8
-m
-EXIF:Copyright=
-EXIF:Artist=
-EXIF:UserComment=
-IFD0:Copyright=
-IFD0:Artist=
-tagsfromfile
%d%f.xmp
-@
C:\Program Files\photools.com\imatch6\arg_files\xmp2exif.args
--make
--model
{Files}

I don't know if this will be very safe, but it works. (One minor problem is that Windows shows strange characters, as seen in the image.) Would it be possible to do this with a Metadata Template? Would it be possible to do this with a Metadata Template?
I will continue testing before I definitely move my entire 112,000+ photo archive to IMatch. And then recommend IMatch in the Institutional Archive where I work.

Translated with www.DeepL.com/Translator (free version)

[Mario]

Not sure that I can follow.

IMatch automatically builds a 'rich' XMP record during ingest by importing existing EXIF, GPS and legacy IPTC data into the XMP record. And writing it back as needed.
IMatch implements the XMP standard and follows the rules and recommendations of the MWG.
The XMP data IMatch produces with the help of ExifTool is rich, standardized and complete.

There is usually no need to involve metadata templates, directly fiddle with ExifTool or the ECP.

[jmsantos]

Sorry, I may not have explained myself well. I already know that IMatch complies with XMP standards and MWG recommendations. But the example with Fuji RAF indicates that sometimes we have to resort to ECP to solve some problems, even if these problems are not caused by IMatch.

The issue I am now referring to was raised by me two years ago.  Here your explanation: https://www.photools.com/community/index.php?topic=9039.msg63492#msg63492. A year later I revisited the issue, hoping that it had been resolved with the release of IMatch 2020: https://www.photools.com/community/index.php?topic=8951.msg62907#msg62907.

The usual indexing flow does not correct the discrepancies in my specific cases. It is clear to me that the source of those discrepancies is not in IMatch and apparently it is not a usual case. That is why I have raised that a possible solution could be to resort to ECP with the arguments I have put forward. I wanted to know if it is considered safe. I will research further. I don't want to waste any more of your time.

Translated with www.DeepL.com/Translator (free version)