[IMA2020.14] Disadvantages of embeded Metadata in Sony or Canon RAW?

[Rene Toepfer]

Will there be any disadvantages if metadata will be embeded in Sony or Canon RAW? I have read Embed XMP in file but for me it is still not clear. In some older threads (from 2013) it is mentioned that some applications could have problems if they write metadata into a sidecar file. This could cause inconsistencies due to embeded metadata and sidecar file. Is it still an issue?

[Mario]

I'm not sure I understand your question.

The XMP standard and documentation and industry standard focuses on using XMP in sidecar files for RAW formats.
Some camera vendors (who usually suckfail at everything software, so don't get your hopes up) started embedding rudimentary XMP records in the RAW files.
Why? I don't know. The don't document this.

Usually these sad XMP records only contain a "rating = none". Sometimes also the vendor name or a make or date. That's so far all I have seen. Even in modern RAW formats from Japan.
Smart phone DNG files usually do much better, but then, DNG is designed to use embed XMP. And EXIF. And GPS. Different thing. And the software is written by software companies, not camera companies.
But I digress.

So, some cameras, depending on model and firmware, embed a bit of XMP in the image. But they don't create a complete and standard-compliant record.
Many fields mandatory are missing. Why? I don't know.

You'll end up with an incomplete XMP record embedded in the RAW.
Which does not contain any EXIF data.
And you still have EXIF data in a format 30 years old - with all the problems caused by character sets, field length limits, time-zone issues and more.
And of course undocumented and proprietary maker notes on top.
Why do they do that? I don't know.

Why don't they do the right thing and write all the EXIF data (also) into the XMP record? Which can hold everything legacy EXIF does, but without the problems.
I don't know.

If RAW processors or image editors or DAM software detects and uses the embedded XMP record depends.
Some may use it. Most don't. Some may give it a higher priority than XMP found the sidecar files. Others don't.
Some may merge the contents. Some don't.

Nothing of this is standardized or documented.
Please refer to the latest XMP standard in the ISO documentation and on the Adobe web site. Contact the customer support of your camera vendor and RAW processor vendor for more information about why and which XMP data your camera vendor embeds in your files. I'm sure they will be happy to assist you.

IMatch tries to make the best of it, as detailed in the IMatch help. Which you probably have read.
So you know already that IMatch is aware of the embedded XMP and gives it higher priority than sidecar files.
That IMatch merges embedded XMP and data in sidecar files during ingest automatically.
That there is a dedicated setting which by default makes IMatch ignore the irritating "rating=none" in the embedded XMP.
(Why camera vendors write that, I don't know. Neither do they, probably).
That IMatch follows the ISO standard and writes XMP data into sidecar files for RAW files. So other software can use it.
It does not update the embedded XMP record. Unless forced.
There should be only one source of truth for metadata. So no partial XMP record embedded in the RAW and all the other XMP data in the sidecar file.
I have explained all that at length in the metadata and XMP-related sections of the IMatch help. There is nothing I could add here.

What is your question, precisely?

[Rene Toepfer]

Thanks for you explanation.

Currently I use sidecar files with IM but I am thinking if it is really necessary to have a lot of small files which slow down any copy operation. Furthermore IM changes Sony and Canon RAW while propagating metadata anyway even if sidecar files will be created (by default). Therefore I am thinking about to change the setting to force IM to write all metadata into the RAW.
The help and forum were not very detailed to me to figure out (dis)advantages of embeding metadata into RAW files.

With your explanation I understand there is no standard for embeding metadata into RAW but sidecar file. Then the decision is clear: sidecar files for a.m. RAW.

[thrinn]

In my opinion, sticking to an existing standard is preferable - even if not all software are standard compliant. And this means: sidecar files for RAW, embedded metadata for DNG and JPG. Because I am sure that different products will ignore the standard in different ways, and most of it is not nearly as configurable as IMatch. Too  much trouble ahead...

Regarding your question:

it is really necessary to have a lot of small files which slow down any copy operation.

Be aware that embedding metadata into RAW files would maybe the reduce the number of files. But every metadata change would need to update the RAW file which is much, much bigger than the XMP sidecar file (in  my case, a typical RAW file has about 20.000 kB, an XMP file about 6 kB). Thus, your typical backup software has to backup much more "bytes" (by a factor of 3000!) for the same change. Overall I doubt that there would be any performance gain at all - quite the contrary.

[sinus]

In my opinion, sticking to an existing standard is preferable - even if not all software are standard compliant. And this means: sidecar files for RAW, embedded metadata for DNG and JPG. Because I am sure that different products will ignore the standard in different ways, and most of it is not nearly as configurable as IMatch. Too  much trouble ahead...

Regarding your question:

it is really necessary to have a lot of small files which slow down any copy operation.

Be aware that embedding metadata into RAW files would maybe the reduce the number of files. But every metadata change would need to update the RAW file which is much, much bigger than the XMP sidecar file (in  my case, a typical RAW file has about 20.000 kB, an XMP file about 6 kB). Thus, your typical backup software has to backup much more "bytes" (by a factor of 3000!) for the same change. Overall I doubt that there would be any performance gain at all - quite the contrary.

That is exactly, what I think also.

[Mario]

IMatch performs checks at write-back time to determine if writing the sidecar file only is sufficient. And when this is the case, write-back is 100 times faster than writing the sidecar file and the image file.

In the past, users complained that XMP is not embedded in video files.
Now some video formats support embedded XMP, and users complain that it takes so much longer to modify a 2GB video image compared to a 5KB XMP sidecar file.
Especially when the video file has no room for more data and it needs to be spliced and re-combined to make room for extra XMP data.
There is no free lunch...

Today we've got a problem report about ExifTool being unable to update a 7 GB (!) video file, due to lacking large file support. I don't even know if this is supported on Windows.
Stuff like this should be handled in the application that produced the 7GB video file. Not later in an asset management system.

I have wasted so much time with all the metadata mess users create with the applications they use.
Or metadata produced by applications which give a shit for standards and just do what they want. Like writing sidecar files for JPEG images. Or camera vendors creating rudimentary XMP records in RAW files with only "rating=none" but nothing else. Not even the minimum number of tags required by the XMP standard.

And the first time users´realize their mess is when they switch to IMatch and wonder where all the data is gone.
Why characters look strange. Why data is missing (stored in the catalog somewhere of their previous software). Why they have different keywords and data in legacy IPTC and XMP. Or why XMP has none or just a part of the EXIF in the image. So many problems caused by other software and then I have to deal with it.

I'm so sick of everything metadata. I can fully understand why other software companies don't really bother, and invest their time into things like better UI and flashy new features.
Because that sells and makes money. Proper metadata handling does not. And when users complain, they let their complaints die in their "ticketing" systems in their support forums. Just too expensive.
Something to think about. What I could do with all the time lost analyzing and explaining metadata problems over and over again.

XMP files usually fit into a handful of sectors on disk. Which allows "imaging" backup software like Macrium Reflect or TrueImage to operate very efficiently and fast.

[Jingo]

I'm so sick of everything metadata. I can fully understand why other software companies don't really bother, and invest their time into things like better UI and flashy new features.
Because that sells and makes money. Proper metadata handling does not. And when users complain, they let their complaints die in their "ticketing" systems in their support forums. Just too expensive.
Something to think about. What I could do with all the time lost analyzing and explaining metadata problems over and over again.

XMP files usually fit into a handful of sectors on disk. Which allows "imaging" backup software like Macrium Reflect or TrueImage to operate very efficiently and fast.

And there is only a handful of users that even USE metadata... most folks simply don't care or use very very simple tags.

There is the benefit of not writing back to your image files too... though I have yet to corrupt a file, the possibility exists... writing to a sidecar is thus safer.

[Rene Toepfer]

Thank you for all of your comments. It was helpfully to me to make the decision to use sidecar files also for future images.