Metadata discrepancies IM-LR

[jmsantos]

Hello again.

I have been a former user of IMatch 3.6, which I left to focus on Lightroom because using two applications made my asset management too complicated. Now I'm looking for a DAM application, I'm going to try IMatch 2019 again and I see that it has improved a lot in these years. One of my surprises is the decision to centralize the metadata in XMP, which was one of my demands at that time for the discrepancies that were with the Adobe software.

However, I have problems again with the metadata. The IMatch reading of metadata does not reflect some fields introduced in Lightroom. For example, in RAW images with XMP sidecar, something as simple as the fields "Copyright" and "Creator" written with a template in LR are empty in IM.
On the other hand, LR keeps the field "xmp: CreatorTool =" with the name of the camera and firmware version, while modifying metadata with IM that field changes to "xmp: CreatorTool =" photools.com ... ". Is there any option to avoid this?

Of course, I've read carefully the IMatch documentation and the IM and LR joint workflow, as well as comments in the user forum.

Thank you (and sorry for my English).

[Mario]

Hi, welcome to the community.

IMatch uses the renowned ExifTool software since version 5 to read and write over 14,000 metadata tags, including the full XMP, IPTC and IPTCExt tags.
About 50% of the user base of IMatch uses (or used) Lr together with IMatch, including myself.

if the data you see in Lr does not show up in IMatch, make sure Lr has written the XMP data to the file. If the XMP data only exists in the Lr database, IMatch cannot access it.
By default, Lr does not write XMP data to the sidecar file unless triggered manually via the right-click context menu.

Tip: You can use the ExifTool Command Processor in IMatch (Tools menu) to see all the metadata in your RAW files and the side car file. Use the "List Metadata" preset to see the data in the RAW file. Use the "List metadata in XMP files" to see all data Lr has written to the XMP fioe.

Lr and IMatch work great together and many users use Lr only for editing but leave the metadata management to IMatch, which does a much better job.

[jmsantos]

Hello Mario.

Thanks for the welcome.
The XMP data is written in the sidecar, I have activated that option in LR and, just in case, I saved manually with Ctr + S.

The Author and Copyright data appear in the Exiftool command:
[XMP-dc] Creator: José Manuel Santos Madrid
[XMP-xmpRights] Marked: True
[XMP-dc] Rights: © 2019 José Manuel Santos Madrid

Of these three, the only thing that matches IMatch is the Marked field; Author and Copyright are empty.

I have no doubt that Imatch and Lr can work very well together, that's why I am surprised by this discrepancy.
Thank you

[Mario]

Please show us the output of the ExifTool Command Processor and provide a sample image and sidecar.

I'm sure thar your files have data in legacy IPTC and/or EXIF which overrides the XMP data during input.
IMatch maps metadata between EXIF/IPTC/XMP based on the recommendations of the metadata working group and if your files have out-of-sync metadata elements, this may happen.
Also, IMatch recognizes XMP data embedded in your RAW file, which is usually ignored by Lightroom. All this may cause the problems you describe. Impossible to see what Lr has done this time without having an image.

I use Lr Classic CC myself every day together with IMatch.

[jmsantos]

I attach the Exiftool Command "List Metadata in XMP files". An image and the sidecar in zip file can be downloaded in:
https://www.dropbox.com/s/240i7psf512oe0j/20190316_122810_3484.zip?dl=0

I also use Lr Classic CC.

[Mario]

Your RAF file contains an EXIF record with an empty copyright notice:
On import, IMatch maps this value to the corresponding XMP tag. When you change the XMP copyright in IMatch, it writes it to the XMP sidecar file and updates the corresponding EXIF tag in the RAW - to ensure that all related tags are properly synchronized between XMP and EXIF/IPTC/GPS. Apparently, Lr did not do this and now you have a copyright value in the XMP file which does not match the EXIF data.

The same is true for the other tags you mention. Lr metadata mess again.
IMatch now has metadata in the sidecar file and corresponding metadata in the EXIF record embedded in the RAW. By default, it considers XMP/IPTC/EXIF/GPS embedded in the original image as more important and overrides the corresponding fields found in the sidecar. Since IMatch always synchronizes metadata this problem can only happen if you use an application which does not synchronize metadata in your workflow or you mix multiple applications when adding/modifying metadata.

Since the data in your files is out-of-sync and the EXIF data thus replaces some of the XMP fields Lr has updated but not synchronized back, you need to override the default "Data in image is more important than data in sidecar file" in IMatch.

Go to Edit > Preferences > Metadata 2.
Click on Configure File Formats
Scroll down until you see RAF and change the settings as follows:



If you use other RAW formats with the same problem, configure them accordingly.

Then select the files with the problem in a File Window and press Shift+Ctrl+F5. In the dialog, choose "Rescan Metadata".
This reloads the metadata using the new options. Now the data in the sidecar is used and corresponding EXIF/IPTC/GPS data in the image itself is ignored.

[jmsantos]

Thank you, this has solved the problem!
However, I have a question about how this solution affects the workflow from now on. I explain:

I have about 120,000 photographs managed with LR. I imported that catalog to IMatch with the import tool. IMatch indexes the data with the option that Mario suggests: "Favor XMP sidecar file" for my RAW files (NEF, ORF and RAF).

If I decide to change my DAM workflow in favor of IMatch and keep LR to process the RAW images, from now the process will be reversed: IMatch writes metadata, Lightroom reads and saves again with the development data. How does the "Favor XMP sidecar file" option affect this pipeline?

[Mario]

By default IMatch favors metadata embedded in the image (EXIF, IPTC, GPS and optional XMP). It considers embedded metadata as more important than sidecar data.
When initially reading an image, IMatch produces a rich and high-quality XMP record by mapping EXIF/IPTC/GPS => XMP,. according to the MWG and industry standards. When writing back metadata, IMatch performs the reverse operation, mapping from XMP to EXIF/IPTC/GPS embedded in the image.

IMatch of course also reads and writes XMP sidecar files, depending on the file format.

On import, by default, metadata extracted from the image overrules the corresponding metadata from XMP (for example, some EXIF and IPTC tags which may both exist in the image itself and in the sidecar as a copy in the XMP). IMatch assumes that the metadata in the image and the sidecar are properly synchronized and contain the same values. Or that the EXIF/IPTC/GPS data in the image itself contains the original and thus more true value (for tags which may exist both in EXIF/GPS/IPTC and as a copy in the XMP record).
This is not the case in your files, where Lr modified the EXIF data in the XMP, but not in the image itself. Hence you need to tell IMatch to favor the data in the sidecar file over the data in your images. This only affects the initial ingest, and is required because your files have different metadata for the same tags (in the image and the sidecar). Which is unfortunate and can produce problems in your workflow. Adobe of course does as they please, and users only notice such problems once they leave the "Adobe world".

IMatch always synchronizes between XMP/EXIF/IPTC/GPS on read and write so once you have written back metadata in IMatch, the data in the image and the XMP sidecar will contain the same data for the same tags.

IMatch is flexible enough to overcome this problem in your images with the favor sidecar option. I cannot guarantee that this will work for all files and all software products out there. Some of the minor RAW processors make a real mess of XMP, strip metadata from files during save etc.

XMP metadata was only  pro.-forma designed as a metadata exchange format. It is usually best when you modify metadata only in one application - preferably one that does synchronize metadata properly both ways.

[lbo]

By default IMatch favors metadata embedded in the image (EXIF, IPTC, GPS and optional XMP). It considers embedded metadata as more important than sidecar data.

Mario, if you find the time to retrace the reasons for this design decision, could you please let me know them? I don't want to miss important reasons not to change the defaults!

I'm still checking the pros and cons for different scenarios and would like to discuss them in a constructive way (please!).

Likely the most important question to determine the right settings is how companion software (e.g. the used RAW developer) handles metadata: Where does it write, which data is used/preferred on read?

jmsantos' posting strengthens my information about Lightroom: LR writes metadata changes only to the XMP sidecar but not to the RAW file. As a result, LR must also "favor" the sidecar data over the RAW data.

Oliver

[Mario]

This is why IMatch gives you the choice.

If you use IMatch to create the initial XMP data, you're all set with the defaults. The same is true when you use software which properly maps between EXIF/IPTC/GPS and XMP, both ways.
If you use LR and don't modify metadata that exists both in EXIF and IPTC and XMP (like copyright, date and time, caption/description) you are also safe with the defaults.
It's only when you modify, for example, the copyright (which exists in EXIF and IPTC and XMP) and your other software only writes the new copyright to the XMP when you get into trouble when you afterwards add the file to IMatch. IMatch then performs the default handling of trusting the embedded metadata and merging it with an optionally existing XMP sidecar file to produce the rich initial XMP record during ingest.

If you use software which only updates the XMP, or use a software/device which insists in embedding the XMP in the RAW file, or your software only writes a partial XMP record (some software does not copy existing EXIF/IPTC/GPS data at all into the corresponding XMP fields, but just creates a fresh XMP record out of thin air, with a minimal set of data) you have options in IMatch to handle that and to make IMatch do what you want. Even on a per file-format basis. See above.

I know you are referring again to the other thread and I guess is you want me to add an switch that allows the user to change the default behavior to "ignore embedded metadata in the image if there is an XMP sidecar file". Globally. For all file formats.

This would be doable and would make Lr users happy which a) process their RAW files in Lr  first and b) update metadata in Lr which has counterparts in EXIF and legacy IPTC and c) afterwards add these files to IMatch.

But it also has many shortcomings because IMatch is then unable to "update/merge-in" metadata that does not exist in the XMP side sidecar file - this is just how ExifTool and the argument files work and how IMatch merges the data (no single-tag merge). IMatch produces a much richer (better quality and complete) XMP record than Lr (even more true for the newer LR CC) so I'm strong in favor of the current default behavior and giving the user the choice.

As always, if there are sufficient feature requests or affected users by this Lr behavior, I will do what is best for users.

[lbo]

I know you are referring again to the other thread and I guess is you want me to add an switch that allows the user to change the default behavior to "ignore embedded metadata in the image if there is an XMP sidecar file". Globally. For all file formats.

your guess is wrong. It's neither about "all files" nor about a new switch.

In your own posting you considered changing the defaults just for raw files, I cite again: "But I wonder if I should switch IMatch to default to "favor sidecar / ignore embedded" for RAW files in the next version."

But it also has many shortcomings because IMatch is then unable to "update/merge-in" metadata that does not exist in the XMP side sidecar file

I never asked for a setting with this behavior.

But your reply made me clarify my questions from https://www.photools.com/community/index.php?topic=8936.0 (there I added the question whether "force" and "favor" works on a "per tag" basis or on "all or nothing" basis).

You have written that it may take a few weeks before you find time to search for the details, so I propose to suspend the discussion until then.

Oliver

[jmsantos]

Thanks for your detailed response, Mario. This help is very useful.

XMP metadata was only  pro.-forma designed as a metadata exchange format. It is usually best when you modify metadata only in one application - preferably one that does synchronize metadata properly both ways.

I agree, all in one would be ideal. But if we need software for DAM (IMatch) and other software to develop (Lr), the latter writes the processing metadata (and something else?), as suggested in the workflow "Using Adobe Lightroom® and IMatch together".

It's not easy for me to adopt IMatch in that workflow, I still have to tie up a few loose ends. I'll have to keep doing tests, before the IMatch trial period is over  .

[Mario]

This is not caused by a problem or misbehavior in IMatch.
It's just caused by Adobe not properly synchronizing metadata between XMP and EXIF (although Adobe was one of the founding members of the Metadata Working Group).
I use Lr myself for developing RAW files, Photoshop for editing, Affinity Designer for vector work. IMatch has no problems at all to deal with the metadata produced by these applications.

Just remember to not change the copyright info/description in Lr when you want to use IMatch with the default settings. Or use the "Favor sidecar/ignore metadata in image file" option.

The exact behavior of Lr regarding metadata and what it writes depends on the Lr version, too.
If you also use non-Adobe products or a mix of different Adobe products in different versions (e.g. Photoshop for real editing and Lr) you might want to check the metadata produced by that application mix as well.  Especially older version of the Adobe XMP toolkit had 'issues' with XMP data management and mapping.

If your image collection has older images produced by older Adobe products or other software, you will need a software like IMatch to point out the problems and allow you fix them.
Getting the metadata straight and standard-conform is usually one of the biggest challenges when migrating to a true DAM software. As long as you stick to the Adobe world you might never notice any problems...

[jmsantos]

Getting the metadata straight and standard-conform is usually one of the biggest challenges when migrating to a true DAM software. As long as you stick to the Adobe world you might never notice any problems...

We have an expression for that in Spain: eyes that do not see, heart that does not feel.

There are several issues here. Lightroom is easy to use as DAM, I've used it for years, but it's getting slower and slower, and it does not properly comply with the Metadata Working Group's specifications (by the way, the MWG seems to have disappeared, at least its website is not operational). We need a good DAM software: let's go back to IMatch.

IMatch is very good DAM, follows the MWG standard better than Adobe and has many (many, many) functions missing from Lightroom. But starting to work with IMatch is more difficult, at least it takes a lot of time to learn.

Time is the issue. Do I waste my time with Lightroom even if it is not according to the standard or I use time to learn IMatch?

There was another question in my first post:
LR keeps the field "xmp: CreatorTool =" with the model of camera and firmware version, while modifying metadata with IM that field changes to "xmp: CreatorTool =" photools.com ... ". In this case, I prefer the Lr option. Is there any option to avoid this?

[Mario]

This can be discussed, but there is currently no option for the user to change what IMatch writes into this system tag.
As always I'm open to suggestions. But keep in mind that many users manage non-image files (not produced by anything with a firmware/model/make)...

IMatch considers itself as the CreatorTool of the XMP record it produces and writes to the file or sidecar.
XMP is not for images only and IMatch also writes XMP for video files, audio files, Office documents and other file format types.

When I've implemented this in 2015, I checked what other software was doing (including Adobe software at that time).
All used their product name and version to indicate which software and which version of that software produced that XMP record. And this made sense to me and hence I've adapted it to IMatch as well.  This info can come very useful later, with migrating the metadata to later editions of XMP, for compatibility checks, fixing broken data written by a certain version of the Adobe XMP toolkit etc.
Which camera and firmware produced the image to which this XMP record belongs (and which could be a totally different image anyway - consider versions) is not so important I think.

The name and model of your camera and the firmware version (often) is recorded in the EXIF metadata of your image anyway. Make/Model are also copied into their corresponding XMP counterparts (at least by IMatch). IMatch / ExifTool are often able to populate the XMP:aux Firmware tag as well, so you have direct access to that info in XMP (depends on your camera model and EXIF version). You have access to all EXIF data, including maker notes, in IMatch (Lr does not do that).

Lr might or might not show you all that info, depending on where the camera vendor has stored it (inside the official EXIF tags or in maker notes).
In IMatch you can see this data, search for it, sort by it, display it in the metadata panel, in the File Window.
You can even combine make/model/firmware version and store it in a XMP tag using a Metadata Template (title or object name are sometimes used for that).

Some agencies require make/model as a keyword, which is also easy to do with a simple Metadata Template in IMatch.
Why you would care for the firmware version of the camera, I don't know. Any reason for that?

Comparing Lr and IMatch is comparing apples and oranges. Lr is an all-purpose RAW developer with some rather 'reduced' image management capabilities plugged on. This may still be good enough for many, and that's fine. After looking for a couple of hours at the "Lightroom CC Cloud" product, my impression is that this is even more limited than Lr Classic CC? More mass-market perhaps?

If you want to judge the complexity and learning curve of the IMatch DAM, you may probably get a better impression by comparing it with other DAM products from companies like Canto, FotoWare, AssetBank, DAMinion, Widen, Bynder, Extensis etc. Like IMatch, they aim at DAM users not SOHO users who use a software like Lr to casually process RAW files and somehow 'manage' them. This is not really the user type for DAM.

[jmsantos]

Thanks for your advice, Mario.
I do not want to make a strong defense of that issue. Anyway, I will follow your suggestions on that matter, but I expose my point of view:
The XMP specifications suggest for xmp:CreatorTool tag "the name of the first known tool used to create the resource". Therefore, as I understand it, CreatorTool of a RAW file would be the camera model and firmware version. The software generates a derived from the original RAW image would be the CreatorTool of this new image.

That's right, it seems that Adobe has a tendency from time to time to make Lightroom a mass product. However, every time they have done something in that sense, voices have emerged from more professional sectors that have criticized it and Adobe has sometimes backed down. As a professional photographer, I consider myself closer to that critical sector with some issues that can be improved in LR, not only in terms of DAM. That's why I look for alternatives and not exactly in the SOHO 

(I feel my difficulty with English, I write with the help of Google Translator.)

Jose Manuel Santos

[Mario]

Jose,

IMatch produces an XMP record from scratch usually, by importing and mapping existing legacy IPTC, EXIF and GPS data, forming a MWG-compliant XMP 'super-record' in the database. It sets the CreatorTool during that process with the exact IMatch version which has produced that record. This is important info, especially considering potential later migration of the data record to newer XMP version or IMatch versions. This step is performed for all file formats, not just images taken with a digital camera. And not all cameras record firmware versions etc.

This is in place since IMatch 5 (around 2014) and no user ever has complained or even mentioned that. And older Adobe products also recorded that info, not sure when Adobe moved to recording a duplicate of the camera model/brand and firmware version on this tag. You can always access this data using the proper metadata tags (IMatch allows you to access them, Lr may not). So, at this time, I see no need to change this. There is also no real other place to record this important data.

As usual, your mileage may vary. Feel free to post a feature request in the feature request board so other users can see your request and comment.
I'm always open to suggestions and ideas. This is how IMatch develops and becomes better all the time.

Your posts are perfectly understandable. I'm a native German speaker and English is only a secondary language for me. Many users here have English as their second language.
For online translations, I can recommend DeepL (free). It produces much better translations than any other service currently available (I pay for it so I can use it in the IMatch Translator app!).

https://www.deepl.com/translator