Version 2025 - Conversion problem - Sony ARW and XMP-data

[Stenis]

Hi Mario!

I have just made a test import of all my processed pictures to Imatch 2025. Most of it worked fine but two of my most precious historical picture folders with pictures from Afghanistan taken in 1972 and 1978 was a disaster since both the visible RAW-pictures was gone replaced by a DXO Photolab placeholder that can be used to open the picture in Photolab. The original RAW could be seen in Quick View but the thumpnail in the "contact sheet" itself was replaced and so for all the RAW-files strangely enough.

These are repro photographed color slide pictures. 

So I warn people not to make a full scale import because it also will ruin your original XMP-metadata for these files. So no full scala import/conversion withour a proper cackup fellows.

I have managed to correct this problem by just copying all these RAW-files and their corresponding DXO .DOP-files and XMP and replace these corrupted files. I have checked and in my case it seems like it is just these two folders that seems to have been affected. I have no idea why this happened, have you?

[Stenis]

JPEG-files seems to be fine both when it comes to the dosplay of the pictures and the metadata.

[Mario]

since both the visible RAW-pictures was gone replaced by a DXO Photolab placeholder

This is IMatch's last resort fall-back when it cannot process a RAW image with WIC or LibRaw.
Did you keep the log file (see log file) from that session. Because it will contain more detailed information about why IMatch could not produce a thumbnails and cache image for these files.

1. Please ZIP and attach the log file. This gives us a minimum of information to work with.

2. Select one of the failed files in a File Window and press Shift + Ctrl + F5. In the dialog choose "Force Update".
Does the image show now?

3. If not, open Edit menu > Preferences > Application and search for photools. Disable "Prefer photools.com RAW processing" and then select the failed files in a File Window and press Shift + Ctrl + F5 > Force Update.
Does the image show now?

If 2. worked, there was probably some kind of "overload" are stress issue when IMatch processed the files. IMatch processes many files in parallel, and if the computer has some stability issues under prolonged loads, sometimes this happens.

If 3. worked, LibRaw was unable to read your RAW files.

4. Select one of the failed files and go to Help menu > Support > WIC Diagnostics.
Save the results to a text file and attach.
This will show us which WIC codecs are installed on your system and if they and/or LibRaw could process your RAW files.

So I warn people not to make a full scale import because it also will ruin your original XMP-metadata for these files.

Why? When importing an image, IMatch extracts EXIF, GPS, IPTC and existing XMP metadata and produces a "super" XMP record in the database. This does not modify the image or the XMP sidecar file.

Or did you change and write-back metadata manually and then the effect happened?

and replace these corrupted files.

I assume these files were corrupted before you added the file to IMatch. How were they corrupted?

Billions of RAW files have been read and updated by ExifTool and IMatch over the past decade - without any problem.

If you have images which become "corrupted" just by indexing them in IMatch database, please upload some samples to your cloud space and post a link. Or send them to me via support email address, with a link back to this topic.

[hamishr]

For what it is worth, I no longer use ARW files for long-term archiving. I do the initial RAW edit in Adobe RAW and then use Adobe Bridge to convert the ARW files to DNG files, making sure to specify in the Camera RAW preferences that metadata is embedded in the file. After the conversion, I go to Windows Explorer to delete the sidecar files because the metadata is in the main file and the sidecar files are likely to cause problems through being duplication of what are in the main files. In preferences, I also specify a full resolution jpg to be included in the DNG.

In my experience, DNG files without sidecar files work much better in IMatch that RAW files and are a more stable archival format. The only problem I have found with them is that saving changes to a RAW edit takes much longer than with an ARW file with sidecar. This is why I do the RAW edits before the conversion. I haven't specifically tested this but IMatch metadata write-back probably also takes longer. By the way, I notice that the metadata writeback is indeed considerably quicker in IMatch 2025 - very nice! So write-back is not such a big deal any more.

[Mario]

No issues with RAW files NEF, SRF, NRW, OFF, CR2, CR3 in IMatch, as long as either WIC or LibRaw can read them.
RAW files are considered unsafe for archival by most library institutions for good reasons (proprietary, undocumented, ever-changing).

DNG is an ISO standard.
That being said, Adobe changes DNG whenever they want or need something for a new version of one of their software products. Like, recently, switching to JPXL for the embedded previews, which no other software supported, neither WIC nor LibRaw.
I've had to put in a lot of work to combine the Adobe DNG SDK and LibRaw (and the author of LibRaw too) to make these new DNG variants usable in IMatch 2025.

[hamishr]

Mario, thanks for keeping up with the dng changes - the effort is well worth while.

[Stenis]

since both the visible RAW-pictures was gone replaced by a DXO Photolab placeholder

This is IMatch's last resort fall-back when it cannot process a RAW image with WIC or LibRaw.
Did you keep the log file (see log file) from that session. Because it will contain more detailed information about why IMatch could not produce a thumbnails and cache image for these files.

1. Please ZIP and attach the log file. This gives us a minimum of information to work with.

2. Select one of the failed files in a File Window and press Shift + Ctrl + F5. In the dialog choose "Force Update".
Does the image show now?

3. If not, open Edit menu > Preferences > Application and search for photools. Disable "Prefer photools.com RAW processing" and then select the failed files in a File Window and press Shift + Ctrl + F5 > Force Update.
Does the image show now?

If 2. worked, there was probably some kind of "overload" are stress issue when IMatch processed the files. IMatch processes many files in parallel, and if the computer has some stability issues under prolonged loads, sometimes this happens.

If 3. worked, LibRaw was unable to read your RAW files.

4. Select one of the failed files and go to Help menu > Support > WIC Diagnostics.
Save the results to a text file and attach.
This will show us which WIC codecs are installed on your system and if they and/or LibRaw could process your RAW files.

So I warn people not to make a full scale import because it also will ruin your original XMP-metadata for these files.

Why? When importing an image, IMatch extracts EXIF, GPS, IPTC and existing XMP metadata and produces a "super" XMP record in the database. This does not modify the image or the XMP sidecar file.

Or did you change and write-back metadata manually and then the effect happened?

and replace these corrupted files.

I assume these files were corrupted before you added the file to IMatch. How were they corrupted?

Billions of RAW files have been read and updated by ExifTool and IMatch over the past decade - without any problem.

If you have images which become "corrupted" just by indexing them in IMatch database, please upload some samples to your cloud space and post a link. Or send them to me via support email address, with a link back to this topic.

Well Mario, I was absolutely wrong about blaming iMatch for all these problems. It is true though that something went really really wrong with the conversion of two folders of my most precious historical folder from Afghanistan 1972 and 1978. BUT, it was I who did not understand not to use "Delete" instead of "Remove from Database". My fault BUT I have never EVER seen anything like a DELETE command that have the capacity to remove all my near 20 000 postprocessed from within a a program like this. Since I didn´t expect that I did not at all expect this to happen.

A "Remove" in Photomechanic just removes a "virtual" folder in the database and not the originals on the disk. Just a question, why have a "Delete" funktion without type "three levels of warnings to prevent people from doing mistakes like the one I did? - if you should have a function in that place at all? It has really the potential to create disasters.

Well, no problem since I had redundance backups. The database was corrupted of some reason but I managed to repair it with the "Database Diagnostics" and some of my redundace data, so I´m on trac again. No problem. Everything is fine after a couple of "Relocate" of these two folders.

All in all this was a good practice after all :-).

For now I´m pretty sure I will migrate in the beginning of Mars.
I have just one BIG concern and that is how slow the initial import/indexing of all the images is compared to my other softwares I have used but luckily this is not a big problem since you do it very seldom.

DNG has its benefits but there is no general problem with ARW-files so don´t even think on running on that ball Mario.

Thanks for your support!
Very responsive as always.

[Mario]

My fault BUT I have never EVER seen anything like a DELETE command

I know that people sometimes press the wrong key or use the wrong command. Has happened to me (multiple times), and will happen again in the future.

IMatch supports Undo (Edit menu > Undo) for many operations and commands.
And IMatch makes accidentally deleting files or folders hard, for that very reason (no undo possible).

IMatch by default does not delete files when you press <Del> or use the "Delete File" from the context menu in File Windows. It just marks them as rejected instead.You can undo the delete easily by pressing <Del> again or setting the rating to a non-rejected value.

There is a special command (with an extra "are you sure?" confirmation dialog) that deletes all rejected files in the current scope or database-wide. After displaying a confirmation prompt.

This makes accidental deletion impossible. Unless you use the dedicated "Delete know without questions" keyboard shortcut or you disable the prompt dialog via the application settings.


When you press <Del> for a folder, or use the "Delete" command from the context menu, IMatch displays a warning/confirmation dialog box:



This dialog explains that this command will really delete the folder and all sub-folders and files contained within from the drive. Only if you confirm this with Yes, IMatch will perform delete.
I consider this as "safe enough".

A "Remove" in Photomechanic just removes a "virtual" folder in the database

I find it dangerous to make assumptions like that similarly named commands do the same in different applications. I have been burned by that myself, and learned my lesson many years ago.

Tip: IMatch has no virtual folders either. You can use categories for a similar purpose, though.

IMatch uses "Delete" and a trash can (same as Windows Explorer) and "Remove Folder from Database" with a matching icon to differentiate the two commands in the folder context menu.

[Stenis]

For what it is worth, I no longer use ARW files for long-term archiving. I do the initial RAW edit in Adobe RAW and then use Adobe Bridge to convert the ARW files to DNG files, making sure to specify in the Camera RAW preferences that metadata is embedded in the file. After the conversion, I go to Windows Explorer to delete the sidecar files because the metadata is in the main file and the sidecar files are likely to cause problems through being duplication of what are in the main files. In preferences, I also specify a full resolution jpg to be included in the DNG.

In my experience, DNG files without sidecar files work much better in IMatch that RAW files and are a more stable archival format. The only problem I have found with them is that saving changes to a RAW edit takes much longer than with an ARW file with sidecar. This is why I do the RAW edits before the conversion. I haven't specifically tested this but IMatch metadata write-back probably also takes longer. By the way, I notice that the metadata writeback is indeed considerably quicker in IMatch 2025 - very nice! So write-back is not such a big deal any more.

I don't use DNG because DXO Photolab and Capture One have problems to interchange them. ARW is no problem for me.