Metadata with iMatch

[LateJunction]

As part of my ongoing self-education on iMatch, working with DarkTable as an effective replacement for LightRoom 6, for my environment, I came across this assessment of iMatch in a post on the PIXLS.US site:

"Before migrating from Lr to dt I've studied different DAM: digikam, daminion, idimager and iMatch. The one I preferred for its power and flexibility was iMatch. I didn't move to iMatch because it was just totally dependent on XMP (no way as with dt to disconnect the XMP companion files). A good backup of the database is secure enough not to suffer the burden of these files all the time."

I cannot make sense of it. Is it saying that iMatch and Darktable are not mutually compatible? Are there any 'negative' implications in using DarkTable with iMatch? Feedback from iMatch DarkTable users would be most welcome.

[DigPeter]

It would seem that the critic has a hang-up with XMP files.  I cannot think what he is getting at.

[bekesizl]

Although I haven't tried IMatch and Darktable in cooperation, I was trying out Darktable end of 2018.

I guess, that IMatch should not have problems with the XMP files from Darktable, as long as you don't want to add/read metadata in Darktable.

The standard naming for XMP files is (used by IMatch and normal programs):
filename.raw
filename.jpg
filename.xmp

Darktable uses another convention:
filename.raw
filename.raw.xmp
filename.jpg
filename.jpg.xmp

I don't know about metadata, but the raw development settings are saved in the xmp files.

So my guess is, that if you define the ".ext.xmp" files as buddy files to your files and don't try to read/write metadata in darktable, it should be able to work together.

You can try it out on a small set of your files copied to an independent structure.

[Mario]

I have no real idea either.

XMP is the industry standard for storing metadata in image and other files (or as separate XMP files in case of RAW and other files). Images. Video. PDF. Office. ...
XMP is also a superset of the older standard IIM3 IPTC, EXIF and GPS and the Metadata Working Group defined rules for how older metadata has to be migrated into XMP, and back.
IMatch does all that.

That IMatch is so good at working with XMP is actually a massive benefit for you.

Because it means that your metadata is stored inside your images, not only in the IMatch database.
The IMatch database servers as a powerful cache and aggregator - but your precious metadata is stored where it belongs: in your image files. In a standardized and widely supported format.
This makes you independent from IMatch and allows you to see and work with your metadata in all other applications that also support XMP. Which are basically all professional and commercial products, ExifTool, many open source projects.

I have never actually worked (processed image files) with DT. But I've checked it out last year to see if and which image metadata it supports - and I was not impressed.
If memory servers it supports only a small subset of XMP. No migration between legacy IPTC/EXIF/GPS.
It also uses a proprietary file naming scheme that is not covered by the XMP standard, which makes the sidecar files it uses invisible / unusable in other applications.
Actually, if you have RAW files with an XMP sidecar file already, it produces another XMP sidecar file, with only a small portion of the original metadata. And this is what you have to work with.
It even produces sidecar files for image formats like JPEG/TIFF/... which must use embedded XMP as per XMP and industry standards.

DT uses file names like _DSC123456.raw.xmp instead of the standardized _DSC123456.xmp (XMP files must have the same name as the linked image, but the extension .XMP). This is what all standard applications in the imaging market expect. See the post from bekesizl above (our posts crossed).

I would be really careful not to add or edit metadata in DT. Use IMatch if you have it. This ensures rich and standard-compliant metadata.
For image editing, DT is surely an alternative for some other other RAW processors. Depending on your workflow and demands of course. Each RAW processor has its strengths, and not all RAW processors work well with all RAW formats or shooting conditions.

[LateJunction]

Sorry to keep on asking questions about this topic (the implications of managing metadata added by both iMatch and another programme, such as DarkTable), but at this stage of my learning  my ignorance far exceeds my understanding.

I have just read this comment, which is a part of a response to a question I posed to the DarkTable users list:
"...DT exports to the JPG only the metadata it knows of. If you use an external DAM, you'll need for it to independently write the metadata into the JPG, not only to a sidecar file."

Bearing in mind that it is generally acknowledged that DT makes limited use of metadata, I expect to add a significant amount of metadata via iMatch. However, my understanding is that iMatch will write this data to the associated .xmp sidecar (which DT ignores) and NOT into the .jpg image file. Can I set iMatch to write this extra metadata (especially keywords) into the image file, after it has been created (by export) by DT ?

Again, I would welcome feedback from DarkTable users. Or even RawTherapee users, as I think the same situation applies.

[thrinn]

However, my understanding is that iMatch will write this data to the associated .xmp sidecar (which DT ignores) and NOT into the .jpg image file. Can I set iMatch to write this extra metadata (especially keywords) into the image file, after it has been created (by export) by DT ?

In fact, IMatch by default embeds metadata in JPG files as this is the recommended standard. Just to clarify: It depends on the file format if metadata should be embedded into the image file (e.g. JPG, DNG) or written to a separate XMP sidecar file (e.g. for raw file formats).

But if Darktable reads metadata from the JPG itself, what does the non-standard filename.jpg.xmp that you mentioned above contain? Development settings, but no metadata?

I do not use Darktable, so I am speculating here.

[Mario]

I can only recommend again the extensive information about metadata and storage in the IMatch help: Metadata Storage It even has pictures 

Where the metadata is stored is indeed specified and supported (less or more) by all standard applications in the imaging and video field.
JPG, PNG, TIFF, PSD, PSB, DNG, GIF, PDF and some others use embedded XMP data - in the image file itself.

For RAW formats or other file formats which don't support embedded XMP, XMP metdata must be stored in the associated XMP sidecar file.
Note that the RAW may still contain EXIF, IPTC and GPS data - if set by the camera.

This is not optional. This is not application specific. And neither is the naming scheme for XMP sidecar files.
All XMP-aware application must stick to the rules.

XMP is the super-metadata format. It can contain everything that is in EXIF, IPTC and GPS, and lots lots more.
This is why IMatch (with the help of ExifTool) produces a rich XMP metadata record when it ingests a file - by importing EXIF, IPTC and GPS data from the image into the XMP metadata record IMatch maintains for each file.

During write-back, IMatch not only writes the modified XMP data (like a rating, title, description, date and time...) but also maps this data back into the (when in the file) EXIF, IPTC and GPS records. Many XMP files have counterparts in EXIF (like date and time) IPTC (like title or description) or GPS.
This way, all the data is always in sync. And applications which don't support XMP or implement only half-but solutions can at least see the EXIF and IPTC data.

In general: Creating a partial XMP record, not mapping between metadata standards and using a custom naming scheme for the XMP sidecar files is not a good idea.
And this is what I've learned from looking at Darktable to see how it handles metadata. As I said, I was not impressed.

I have never used it to develop RAW files, because my workflow is different and I could not use DT.
If DT gives you excellent results for your images compared to other RAW processors - by all means, use it. And it's free!

But I would recommend to use only IMatch to handle your metadata. And best after you are done with the image in Darktable - else metadata created and managed by IMatch may get lost or stripped by Darktable.

[LateJunction]

But if Darktable reads metadata from the JPG itself, what does the non-standard filename.jpg.xmp that you mentioned above contain? Development settings, but no metadata? .

Getting concise answers from the DarkTable subscription list is not easy, but the impression I have is that this file contains development settings and some metadata which DT decides it wants to write. I think it is this latter data that I lose when using DT with iMatch (and, obviously, that is not caused by iMatch). Those development settings are unknown to and unusable by iMatch, I assume - just as they would be for any raw processor I care to use in conjunction with iMatch.

[Mario]

IMatch will not touch these files, not read or write metadata to them. Because they use a custom naming schema. IMatch does not see these files.
You may want to make "\.ext.xmp" a buddy file of \.ext in  File Relations: Buddy Files so IMatch keeps these proprietary DT files together with the image when you move, copy, rename or delete files.