Why do / don't you write metadata to raw files?

[lbo]

Hi all,

please share your settings for writing metadata to raw files. Did you change the IMatch default settings, and why?

What is the benefit of writing raw files at all?

One reason for "write raw" is that you can use "Windows Search" to find raw files by metadata. But IMatch users don't need Windows Search to find raw images, correct?

Reasons against writing raw:

• Compatibility, e.g. http://owl.phy.queensu.ca/~phil/exiftool/#problems
• Performance e.g. for backups

What did I miss?

Oliver

[Mario]

What is the benefit of writing raw files at all?

RAW files will contain the EXIF metadata, optionally GPS, legacy IPTC and (more often now) XMP metadata.
If you configure IMatch to not update the RAW (definitely not recommended) the EXIF/IPTC/GPS data in the XMP and image will differ, which causes a massive amount of problems.

Note that there is no reason not to write metadata to RAW files. All RAW files are based on the TIFF file structure, and the metadata records are clearly separate from whatever RAW data your camera has stored in the RAW.

If you want to "protect" your RAW files for whatever reason (there is an urban myth that claims "Never write to RAW files) I recommend you make direct binary copies of all files on a secondary long-term archival medium, before you touch the RAW file with any software.

Note that some camera vendors have started to include partial XMP records in the RAW, usually with some tracking code and a hard-coded "rating=none" value. This often needs correcting to process your RAW files in normal workflows and with the default "XMP in sidecar" data scheme.

[lbo]

If you configure IMatch to not update the RAW (definitely not recommended) the EXIF/IPTC/GPS data in the XMP and image will differ, which causes a massive amount of problems.

Mario, could you give some details about the "massive amount of problems" you mention?

[Mario]

If you configure IMatch to not update the RAW (definitely not recommended) the EXIF/IPTC/GPS data in the XMP and image will differ, which causes a massive amount of problems.

Mario, could you give some details about the "massive amount of problems" you mention?

I have written tons on Metadata Working Group recommendations and why letting the copy of the EXIF/IPTC/GPS data in the XMP become different from the EXIF/IPTX/GPS in the RAW is bad and causes synchronization issues, vanishing changes, cross-application issues in the corresponding sections in the IMatch help system. I have explained all this here in the community over the years over and over. I'm tired of all this metadata mess. #

Check all the info the IMatch help and then stick to the standards.
If not, don't forget to mention that whenever you run into metadata-related problems. I have wasted so much time in the past with such problems, I won't bother anymore. Life is to short to spend time with "Should I write to my RAW files or not" discussions. Stick to the standards and have more time to make more photos.

[lbo]

for today, just one question:

Does Lightroom write to raw files or only to XMP sidecars?

I'll try a more detailed reply tomorrow.

Oliver

[Carlo Didier]

In my case, all metadate is written to the raw files because I have a complete DNG workflow. I find it simpler to not have to deal with sidecar files. Also, the DNGs I create in ACR contain full size JPG previews which saves me from configuring visual proxies and such stuff.
Of course, if your workflow is different, everything else changes too.

[Mario]

Does Lightroom write to raw files or only to XMP sidecars?

I don't know. It should, because all changes done to data which is both in the RAW and in the sidecar file becomes out-of-sync when it changes values.

[lbo]

In my case, all metadate is written to the raw files because I have a complete DNG workflow. I find it simpler to not have to deal with sidecar files. Also, the DNGs I create in ACR contain full size JPG previews...

Carlo,

to my knowledge, DNGs created in ACR are container files containing the original raw file (hopefully) plus metadata plus alternate formats.

So strictly speaking, you don't write metadata to the raw file, but to the container.

Oliver

[lbo]

Check all the info the IMatch help and then stick to the standards.

of course, I've been reading the IMatch help, and also the MWG guidelines. Thoroughly.

If you write "stick to the standards", you should be able to name the source of the requirement for data duplication.

Pointers to the relevant places in the IMatch help are also welcome. Maybe I overlooked something.

Oliver

[Mario]

Check all the info the IMatch help and then stick to the standards.

of course, I've been reading the IMatch help, and also the MWG guidelines. Thoroughly.

If you write "stick to the standards", you should be able to name the source of the requirement for data duplication.

Pointers to the relevant places in the IMatch help are also welcome. Maybe I overlooked something.

Oliver

Why? What am I, a metadata authority? Do whatever you like.
I don't have the time to fight metadata wars or "my settings are better than yours". Read the XMP documents. Read the IPTC documents. Read the EXIF standard. Read the Metadata Working Group recommendations. Then decide for yourself. This also depends on your workflow and tool chain of course. Maybe you don't use Lr anymore next year?

IMatch offers many options to deal with metadata, and numerous ways to shot yourself in the foot.
Don't want to synchronize metadata between the XMP and the RAW? Disable MWG compliance and turn off  all other options via Edit > Preferences > Metadata 2 : File Formats.
Be aware of the consequences, though.

If you run into problems later, don't forget to mention that you use non-default settings.
Whatever works for you is good. I recommend that all other users keep the default settings, because they worked well for so many years and in so many environments.

[Carlo Didier]

In my case, all metadate is written to the raw files because I have a complete DNG workflow. I find it simpler to not have to deal with sidecar files. Also, the DNGs I create in ACR contain full size JPG previews...

Carlo,

to my knowledge, DNGs created in ACR are container files containing the original raw file (hopefully) plus metadata plus alternate formats.

So strictly speaking, you don't write metadata to the raw file, but to the container.

Oliver

True, but that's a technical detail which is irrelevant to the average user 

[lbo]

Mario,

Why do you answer so aggressively and polemically? It's not about "fighting" or offending someone.

Please let's discuss technical questions constructively.

Questions:

1. What would be the drawback of this setup:

• If no sidecar exists, a new sidecar is created using metadata from the raw file only
• All edits go to the sidecar, and sidecar metadata has higher priority than raw metadata

I know about the "Nikon Capture(NX)" incompatibility.

2. Which settings are needed to get the behavior described above?

• "Favor XMP sidecar file" or better "Force XMP sidecar file"?
• Is disabling "Write EXIF" and "Write IPTC" in the respectiv file format setting sufficient to keep IMatch from writing the raw files?

Anything I forgot?

Oliver

[Mario]

If you don't sync from XMP to EXIF/IPTC/GPS your changes to XMP will be lost if IMatch has to re-import the file, e.g. after a change in an external program. IMatch produces XMP from EXIF/IPTC/GPS and other metadata contained in the original image on import.

I don't have tested all possible combinations of disabling MWG compliance and per-file format metadata overrides.
I also don't know how all the other applications out there will handle EXIF/IPTC/GPS data in the XMP that does not match the EXIF/GPS/IPTC in the original image.
The defaults have been designed to work for the majority of all users. All the "off" switches and per-file format configuration only exists for very specific use cases or to solve very specific problems with specific files.

I suggest you read all the the information I have written in the help and then make some experiments with your favorite with your other apps.
Then stick with what you like.

[lbo]

Thanks, Mario, I will do some experiments and report the findings.

Oliver