iMatch 2021 and DXO PhotoLab3 - Geometry correction problem

[Travels]

I use DXO PhotoLab 3 for processing my photos (Sony ARW).
After renaming a file with iMatch, the geometry correction in PhotoLab doesn't work anymore for that renamed file.
Comparing the original and the renamed ARW-file with WinMerge (compares files and shows differences), there are some changes at the beginning of that two files (see screenshot).

Any hints?

Martin

[JohnZeman]

I also use PhotoLab (version 4) to process my files.

Are you using file relations in IMatch?  If so you might want to make sure the buddy .dop file was also renamed by IMatch.  If that's ok then in PhotoLab you may need to reload the sidecar (.dop) file assuming you have PhotoLab set to use sidecar files.

I always rename a file in IMatch before it ever goes to PhotoLab and I have never had this problem.

[Mario]

This sounds like you renamed only the file but not the .DOP file which contains all the settings for that file?

As John said above, you need to tell IMatch that your ARW files have buddy files with the .DOP extension in form of a Buddy Files relation (follow the link to learn more).
Only then IMatch can know that it should look for and rename / move / copy / delete the .DOP file when you rename / move / copy / delete the ARW.

Renaming a file in IMatch will of course not change any metadata.

There is a default rule for .ARW which includes the .DOP extension. Did you enable it during the initial setup (IMatch asks).
If not, go to Edit > Preferences > File Relations.

[Travels]

The "File Relations" rules had been enabled from the beginning. But as I didn't made any photo processing and just viewed the photos in PhotoLab there are no buddy files (like DOP). As PhotoLab also stores (as far as I understand it) a copy of the .DOP information in it's own database (.../AppData/Roaming/DxO/...) I deleted this database also before using iMatch for the ARW file -- just in case ...
In iMatch 2021 I just marked the file, hit "Ctrl-Alt-S" (even not renaming the file this time) and the ARW files where changed like in the screenshot of the initial post. From then on PhotoLab didn't make the geometry corrections with this ARW file.

When I worked with iMatch 2019 two years ago (processing 50000 files) I didn't had this problem.
Is "Ctrl-Alt-S" and something in my 2019-preferences the problem?

[Mario]

Ctrl+Alt+S writes back metadata you have changed to the file and/or sidecar XMP file.
So it is pretty normal that the image file is changed afterwards. The metadata in the file will have been updated with your changes, timestamps will have changed, checksums and digests etc.

If your photo software now has problems with the file, I recommend you contact their support and ask what problem their software has.
Updating metadata in files with ExifTool is super-safe.

The only occasions where this caused issues with other software in the past was always a failure in the other software.
Like, relying on hard-coded offsets or even camera or make metadata tags with a specific number of superfluous blanks...
Millions of users use ExifTool daily to modify metadata in files. IMatch uses ExifTool since IMatch 5.

I have never used PhotoLab but I'm sure there are plenty of IMatch users out there which use it. And there are no problem reports from other users.
Maybe your workflow is unique or you uncovered a bug in PhotoLab...

I'm not sure why you deleted the database containing the settings - does this not wipe out anything? Maybe you should have made the software "write" the database contents into the DOP files so the settings survive the deletion of the database?

[jch2103]

Like @JohnZeman, I'm a DxO PhotoLab user. I'm not clear from your description what sequence of steps you've taken.

PhotoLab's image corrections are stored in a .dop sidecar file or in the PhotoLab database (or both) depending on your PhotoLab (PL) settings. But PL only applies changes to the output images, not the original raw file. Of course, processing changes made in PL and stored in the .dop sidecar or database are displayed in PL so you can see what processing you've done before you create output images.

As Mario and John have pointed out, if you rename your ARW file but not the .dop file, then PL won't be able to see the adjustments you previously made in PL. If your adjustments were instead stored in the PL database and then you renamed the ARW file, the database of course won't know about this name change and PL won't be able to apply those earlier corrections to the newly renamed ARW file. And if the corrections were only in the PL database file that you deleted, then of course PL won't have any record of them.

Is this what's happened?

[Travels]

Thanks all of you for your help.

1. As I pointed out I just viewed my untouched ARW in PhotoLab so no DOP-file has to be created (and consistently was not created). Geometry corrections did work in PhotoLab.
2. I then imported the ARW in iMatch 2021, marked it and pressed Ctrl-Alt-S (of course normally also doing renaming, adding geodata etc. to the file ...).
3. When I opened the ARW in PhotoLab again the geometry correction didn't work anymore.

I did a little research with EXIFTool to see what went wrong: When iMatch 2021 stores the file (Ctrl-Alt-S) it does not write back the "exif:Software" tag -- in my case "DSC-RX100M3 v2.00".  This seems to be necessary for PhotoLab to do the geometry correction because when I insert this tag manually with EXIFTool (after iMatch had written the metadata) the geometry correction was working again!

So my wish to Mario: Can you please fix iMatch so that the exif:Software tag is not deleted?

[Mario]

I don't know why this tag should go missing.
The only place IMatch uses this is when the Batch Processor outputs a new image and writes a new EXIF record.
I didn't even recall this tag until you mentioned it.

Please provide some sample images for download so I can have a look. Send the link to support email address.

The EXIF:Software tag is linked to the XMP-xmp:CreatorTool tag.
When you look in the Metadata Panel (in Browser mode) for your files, does this tag contain a value?
Because during write-back, XMP-xmp:CreatorTool is mapped back to Exif:Software via the ExifTool arg file so there should be no "loss" - until something goes wrong.

[Mario]

I have downloaded the files you have provided.

When I import them into IMatch, I see for the original file

the value DSC-RX100M3 v2.00

in these tags:

Exif::Main\305\Software\0:
XMP::tiff\Software\Software\0
XMP::xmp\CreatorTool\CreatorTool\0

ExifTool has mirrored the tag value from EXIF into the corresponding XMP tags which must mirror the data.
But I see two IFD records in the original file, which indicates that it has two sets of EXIF metadata? Why?

When I look at the "After IMatch" ARW file, I see the value in these tags:

Exif::Main\305\Software\0
XMP::tiff\Software\Software\0
XMP::xmp\CreatorTool\CreatorTool\0

(from the XMP file) and when I look at the native metadata in the ARW I see

[IFD1]          Software                        : DSC-RX100M3 v2.00

which is the EXIF record, but not longer in the additional IFD0 record.

When I copy the ARW to a new name (to unlink it from the XMP) and add it again to IMatch,
I see the same values as for the ORIGINAL file, including the EXIF tag, the XMP::tiff and XMP::xmp tags.
ExifTool has found the EXIF data and has properly mapped it to the corresponding counterparts in XMP.

This all looks good to me, except for the fact that ExifTool rolled the two separate IFD records into one.
I'm not that deep into the EXIFTool inner workings to comment on that. But I usually believe that Phil knows very well what he is doing and he has daily feedback from users and developers working with ExifTool with a wide range of files.

Since the EXIF record in the "modified" file still contains the EXIF::Software tag after IMatch has written it and, in addition, the properly updated and synchronized XMP data, I see no problem.

[Travels]

1. The download link was send to you via PM (before you edited your last post).
2. In the iMatch metadata browser in both the original and iMatch-saved file the software-info "DSC-RX100M3 v2.00" can be seen (but PhotoLab doesn't uses this tag as far as I see).
3. The exif:software value can be found in the original ARW under IFD0 and IFD1. After iMatch only the value in IFD1 exits.

By the way: The "geometry correction" is not a manually done correction by the user (which will result in a DOP-file) but the correction done automatically by PhotoLab using their specific camara/lens profiles.

EDIT (10:33): Just saw your above reply. Maybe I copy an older version of EXIFTool into the iMatch folder as I hadn't this problem two years ago ...

[Mario]

Do not willy-nilly copy ExifTool versions around.
ExifTool has introduced many breaking changes over the past year, and the old version no longer matches that and may wreak havoc in your IMatch database or your files.

[Travels]

OK, not that good an idea ...
Maybe EXIFTool can batch-copy the IFD1 value to IFD0. As I have no further experience with EXIFTool maybe you can give a hint for the command line that will do the trick?
Nonetheless I will contact Phil on that matter.

Thanks by the way for your great and fast support!

Martin

[Mario]

As far as I understand this, the EXIF::Software tag in the file is what your other software should use. If it looks for tags only in one IFD, they may have to look into their code perhaps.

[Travels]

If my information is right, IDF0-tags are for the main picture and IDF1-tags are for the embedded thumbnail.
As PhotoLab handles the main raw file and not the embedded jpeg file the IDF0-tag for the software/firmware is essential for them. DxO can't be blamed if they don't use the (irrelevant) thumbnail information (IDF1) for the raw picture information (IDF0) though in this case they might be identical.
I also checked my older photos which were handled by iMatch 2019 -- they have both the IDF0 and IDF1 tags.
So to my eyes in iMatch 2021/EXIFTool important information is lost during saving the file after processing. Maybe it's not DxO who may look into their codes.
Please correct me if I am wrong.

[jch2103]

Can you clarify? It sounds like you made metadata changes to the ARW raw file, not to a .xmp sidecar file. PhotoLab is very picky about information in raw files; changes to the raw file can prevent PL from properly recognizing and working with the image. I've been using PL for a long time and carefully avoid making changes to the raw file except for very narrow exceptions.

[Mario]

I have explained what I have found.
I know that the EXIF::Software tag is mapped to the XMP tags described above, and during write-back these XMP tags are mapped back by ExifTool's argument files.
Nothing has changed how IMatch lets ExifTool map between XMP and EXIF for a long time. At least nothing I can see in my code.
So this should just work, as it did before as you say.

IMatch does not manipulate IFDs of EXIF data directly during write-backs.
IMatch writes the XMP tags modified in the database (!) and then runs the ExifTool args files which map XMP2EXIF, XMP2GPS, XMP2IPTC (if the file has legacy IPTC).

I see the correct software in both XMP tags for the software. Why EXIFTool may remove or re-arrange EXIF during writing I don't know. You may need to ask Phil.
Maybe this is something specific to your ARW files only. No idea. I'm not that deep into metadata binary representation to tell you more about why ExifTool works that way.

[abgestumpft]

I recently also had the problem with one file that DXO did not apply any lens related corrections (like: CA, distortion, vignetting corrections and lens sharpening). But nothing iMatch related -> iMatch was not involved at all.
In my case the LensID information was lost during editing and saving in another application.
You can try to copy all Exif data with exiftool like that (that helped in my case):
exiftool -TagsFromFile source.jpg target.jpg -All:All

Maybe you can fix the renamed file like this?
Please test with a copy on a save place, just to be sure 

In general I use iMatch and DXO Photolab 4 a lot:
I select the photos I like to edit in iMatch and then open DXO via iMatch Favorites. I edit them in DXO and export them into the filesystem. After a short time iMatch picks the up and adds them to the DB.
Never had any issues with that... (except that I have to save Metadata to files before, otherwise Star ratings etc. are lost, because DXO is of course not aware of that).

[jch2103]

In general I use iMatch and DXO Photolab 4 a lot:
I select the photos I like to edit in iMatch and then open DXO via iMatch Favorites. I edit them in DXO and export them into the filesystem. After a short time iMatch picks the up and adds them to the DB.
Never had any issues with that... (except that I have to save Metadata to files before, otherwise Star ratings etc. are lost, because DXO is of course not aware of that).

That's my workflow as well. Also no issues.

[Aubrey]

Same workflow as John. Absolutely no issues for me. (Nikon D500 and Canon powershot g16)
Nef is master, xxx_dxo. dng is version. I have Nef pick up view from dng.

Aubrey

[Travels]

Thanks for your comments. I had no such problems using iMatch2019 with EXIFTool version (11.6.2.0) with several workflows (also like you above mentioned).
Nonetheless I posted my problem on the EXIFTool Forum (https://exiftool.org/forum/index.php?topic=12884.0).
With their help I found a solution:
In the XMP2EXIF.args file [last edited by PH on 2018/05/07] which is provided with iMatch 2021 (see folder "arg_files" in iMatch program folder) I removed the "-" before "XMP" in the line
-EXIF:Software < -XMP-xmp:CreatorTool
(Phil will fix that in the next release.)
As Mario pointed out he made no changes in the XMP2EXIF.args file in the last two years which I can confirm when I looked at my previous installations of iMatch. So it's still mysterious to me why it worked until I changed to iMatch 2021.

[Mario]

These args files are part of the ExifTool installation and I ship them unmodified 'as-is'.
IMatch has its own args file for special purposes and I only ever make changes there.

Good that you have figured this out!
What difference an extra - can make sometimes.

Many people help with ExifTool, from decoding maker notes or lens info and sending the data to Phil to finding bugs.
All this improves ExifTool, which is a good thing for everybody.

Since Microsoft, Adobe and especially the camera/device vendors do such a shitty job in making metadata accessible and documented, ExifTool is the solution and one of the best things since sliced bread!

[lbo]

yet another confirmation of my strategy not to write raw files anymore

[Mario]

yet another confirmation of my strategy not to write raw files anymore

The problem with that is that you deliberately let the embedded EXIF/IPTC/GPS data differ from the copies of this data in XMP. This can lead to other problems along the way.

Note that this particular ExifTool issue caused a problem in one specific software product which apparently relies on the EXIF::Software tag for checking whatever. Maybe they somehow have reverse-engineered a specific firmware version and hence cannot use only the make and mode and lens tags to determine the precise nature of the image for processing. They also need the version number of the firmware?. Not sure if there is other software out there which relies on that...else the superficial - in the XMP2EXIF.args configuration file would have caused issues earlier.

Other RAW processors use make/model/lens id to determine what they are working with, as far as I know. Which is why this data is read-only in IMatch.

[lbo]

yet another confirmation of my strategy not to write raw files anymore

The problem with that is that you deliberately let the embedded EXIF/IPTC/GPS data differ from the copies of this data in XMP.

Therefore the RAW file should contain as little human-generated information as possible.

Although rare, there are occasional incompatibilities when writing RAW files. And sometimes you don't notice them until later.

[Mario]

Maybe some day camera vendors should stop embedding EXIF and GPS data into their undocumented and proprietary RAW formats, switch to DNG (which does all the proprietary RAW formats do) and XMP metadata. Problem solved.