Enable Embedded XMP for file format

[monochrome]

Is there a way to enable embedded XMP for any file format? I've added a file format to IMatch that is supported by ExifTool (and ExifTool can write embedded XMP to it) and would like it to enable embedded XMP for it, but the file format options under Metadata 2 don't include that option. Am I looking in the wrong place?

[Mario]

IMatch automatically embeds XMP in all file formats for which a) the MWG specifies embedded XMP and for b) ExifTool reports that it can write (-listwf must include the extension) and c) the file format options under E > P > M: File Formats does not force a sidecar file.

It would have helped if you at least gave us a hint about the name of the file format, the application, file extension ...

[monochrome]

IMatch automatically embeds XMP in all file formats for which a) the MWG specifies embedded XMP and for b) ExifTool reports that it can write (-listwf must include the extension) and c) the file format options under E > P > M: File Formats does not force a sidecar file.

It would have helped if you at least gave us a hint about the name of the file format, the application, file extension ...

I've added a custom file type to ExifTool: RDF. It's based on XMP and with the mime type application/rdf+xml. Using ExifTool from the command line does what you'd expect - update the file in-place.

a. It's XMP, so by the XMP guidelines "it is recommended that XMP metadata be embedded in the file that the metadata describes." The MWG guildelines do not, as far as I can see, contradict this.

b. It does show up in the list when I run ExifTool with the -listwf option from the ExifTool command processor.

c. There's no such option.

[Mario]

When I run -listwf I don't see .RDF listed as a supported format for write-back.

Writable file extensions:
  3G2 3GP 3GP2 3GPP AI AIT ARW CIFF CR2 CRW CS1 DCP DNG DVB EPS EPS2 EPS3 EPSF
  ERF EXIF EXV F4A F4B F4P F4V FFF GIF HDP ICC ICM IIQ IND INDD INDT INX J2K
  JNG JP2 JPE JPEG JPF JPG JPM JPX JXR M4A M4B M4P M4V MEF MIE MNG MOS MOV MP4
  MPO MQV MRW NEF NRW ORF PBM PDF PEF PGM PNG PPM PS PS2 PS3 PSB PSD QT RAF RAW
  RW2 RWL SR2 SRF SRW THM TIF TIFF VRD WDP X3F XMP


Your RDF files, I assume, contain resource description framework data. These files don't contain XMP, they are XML files following a specific syntax. XMP is derivative of RFD, using a specific flavor cooked up by Adobe 10 years ago. Usually you edit RDF files in a text editor, or in a special XML editor which supports the RDF variant you use.

[monochrome]

When I run -listwf I don't see .RDF listed as a supported format for write-back.

Writable file extensions:
  3G2 3GP 3GP2 3GPP AI AIT ARW CIFF CR2 CRW CS1 DCP DNG DVB EPS EPS2 EPS3 EPSF
  ERF EXIF EXV F4A F4B F4P F4V FFF GIF HDP ICC ICM IIQ IND INDD INDT INX J2K
  JNG JP2 JPE JPEG JPF JPG JPM JPX JXR M4A M4B M4P M4V MEF MIE MNG MOS MOV MP4
  MPO MQV MRW NEF NRW ORF PBM PDF PEF PGM PNG PPM PS PS2 PS3 PSB PSD QT RAF RAW
  RW2 RWL SR2 SRF SRW THM TIF TIFF VRD WDP X3F XMP

To quote myself: "I've added a custom file type to ExifTool: RDF". When I run -listwf here, RDF is included.

Your RDF files, I assume, contain resource description framework data. These files don't contain XMP, they are XML files following a specific syntax. XMP is derivative of RFD, using a specific flavor cooked up by Adobe 10 years ago. Usually you edit RDF files in a text editor, or in a special XML editor which supports the RDF variant you use.

Why should the relation between XMP, RDF and XML is matter in this case? Let me rephrase the question and let's get to the core of the question here: How do I add a custom file type with embedded XMP to IMatch?

[Mario]

A RDF file is an XML file. You cannot embed XMP in RDF. That would be like embedding a text file in a text file.
What you want to make IMatch do is to embed an XMP file into an RDF file? How can this work?
You cannot write arbitrary XMP metadata into an arbitrary RDF file.

Please explain what you did, exactly. If you have added a custom file format to ExifTool via a config file or anything (I have no idea how to do this) this does not mean that IMatch will pick that up or make all this work by magic.

[monochrome]

A RDF file is an XML file. You cannot embed XMP in RDF. That would be like embedding a text file in a text file.
What you want to make IMatch do is to embed an XMP file into an RDF file? How can this work?
You cannot write arbitrary XMP metadata into an arbitrary RDF file.

Please explain what you did, exactly. If you have added a custom file format to ExifTool via a config file or anything (I have no idea how to do this) this does not mean that IMatch will pick that up or make all this work by magic.

Mario - let's just focus on this question, because it is the one I want answered: How do I add a custom file type with embedded XMP to IMatch?

[Carlo Didier]

Mario - let's just focus on this question, because it is the one I want answered: How do I add a custom file type with embedded XMP to IMatch?

I think you did not understand what Mario explained: RDF files do not and can not contain XMP metadata.
From what I gather from Wikipedia (http://en.wikipedia.org/wiki/Resource_Description_Framework), RDF files can (but not necessarily need to) use the XML syntax. But XML is not equal to XMP.

[Mario]

Mario - let's just focus on this question, because it is the one I want answered: How do I add a custom file type with embedded XMP to IMatch?

Please answer my questions from above. How did you make ExifTool recognize your custom format as a writable file format first? IMatch usually gets the writable formats from ExifTool, and then sends the XMP data into the file if ExifTool reports it as writable, else it writes a sidecar file.

[monochrome]

Mario - let's just focus on this question, because it is the one I want answered: How do I add a custom file type with embedded XMP to IMatch?

Please answer my questions from above. How did you make ExifTool recognize your custom format as a writable file format first? IMatch usually gets the writable formats from ExifTool, and then sends the XMP data into the file if ExifTool reports it as writable, else it writes a sidecar file.

Ok, I edited imatch_et.config (http://www.sno.phy.queensu.ca/~phil/exiftool/config.html) and added a user-defined file type to it.

%Image::ExifTool::UserDefined::FileTypes = (
    RDF => { # <-- the extension of the new file type (case insensitive)
        # BaseType specifies the format upon which this file is based.
        # If BaseType is defined, then the file will be fully supported,
        # and in this case the Magic pattern should not be defined
        BaseType => 'XMP',
        MIMEType => 'application/rdf+xml',
        Description => 'RDF XML file',
    },
);

Does the job. With that, I get RDF as a writable format, and command-line ExifTool works as you'd expect it to.

[Mario]

I understand you manually edited the IMatch system config file imatch_et.config by hand? Why do you fiddle with IMatch config files? This is not supported and future IMatch updates will replace the file anyway and thus erase your changes.

And after manually adding a user-defined file format to the ExifTool configuration file you expected IMatch to somehow magically pick that up and from now or 'know' that RDF files are just XML files and that IMatch should instruct ExifTool to write XMP metadata to these files? I doubt that this will work and I honestly never tried such a thing. You are probably the only user in the world who wants writing XMP metadata into RDF files - for whatever reason.

If you want IMatch to support this and me to look into this, I suggest you write a feature request. I can then, given that some users support this, allocate a time slot to reproduce your experiment here and see where I need to make changes in the metadata processing and file format handling in IMatch in order to support this. And if the effort is worth it. Currently there are no metadata options for user-defined file formats, which means that IMatch considers the file as an entity and does not write IPTC, EXIF, GPS or XMP into the file.

I'm still not sure why you want to write XMP metadata to RDF files as well. RDF is a XML-based WC3 standard for describing resources on the web. Adobe choose the RDF syntax as the basis for XMP (with some derivations maybe), but RDF is not XMP and vice versa. A RDF file may contain anything, from a list of web sites to a web shopping cart or even a calendar or a library catalog.

Don't forget to backup your custom imatch_et.config file before installing IMatch updates. This is a system file and not designed to be changed by users. Updates will replace the file with the one I ship with IMatch.

[Mario]

One change:

I gave this a quick look, puzzled by the fact that IMatch did not display the XMP option in E > P > Metadata 2 : File Formats for user-defined formats.

I have changed this as follows: If a user-defined format is selected, IMatch asks ExifTool (by looking at the cached output of -listwf) if the custom file extension is in the list of writable formats. If this is the case, IMatch now also offers the XMP sidecar options, which allows you to configure IMatch to write XMP into the file itself instead of a sidecar (which is the default for user-defined formats).

This solves half of your problem. It does not solve the problem that .RDF is not a format natively supported by ExifTool. Changing the IMatch system configuration file is not the proper way to add this.

Another problem with RDF files is that they can contain 'anything': shopping lists, library catalogs, lists of spare parts. When ExifTool is extracting data from these files, it maybe produces tons of unknown tag groups and tags, which IMatch does not know. When a database is created, IMatch asks ExifTool for a list of tag groups and tags, and builds database structures based on that. If a new ExifTool version is detected, this process is repeated. IMatch is prepared to add unknown tags from known groups, but not to add both unknown groups and tags when the metadata of a file is processed.

The file you play with happens to use XMP namespaces and descriptors, so most of the groups and tags are known to ExifTool and IMatch. If you use other RDF files, e.g. the sample files from WC3 or arbitrary RDF files found on the web, IMatch will not import any RDF data at all. So, while this may work for your very specific tiny niche case, it will not work in general anyway.

[monochrome]

I understand you manually edited the IMatch system config file imatch_et.config by hand? Why do you fiddle with IMatch config files?

Because it is the only way to add a new file type to IMatch with full metadata support. I can sit around and wait for ExifTool to get updated, and then wait for you to release a new IMatch version that maybe, maybe, maybe will include the file format I want to use - or I can add a couple of lines to the tool IMatch uses for metadata input/output and have it now.

Really, is this so hard to guess? That a user of a digital asset management tool would like to extend its functionality to new asset types?

IMatch does a really great job of managing assets and metadata. But both asset types and metadata are continuously changing. Being able to reconfigure the tool to manage not just new assets but also new metadata (by being able to define XMP tags) should be possible - and is possible with ExifTool. Being able to define new metadata tags is the purpose of XMP, that's why it is called "eXtensible". Why is it so strange that I would want IMatch to do the same?

And after manually adding a user-defined file format to the ExifTool configuration file you expected IMatch to somehow magically pick that up and from now or 'know' that RDF files are just XML files and that IMatch should instruct ExifTool to write XMP metadata to these files?

Pretty much, yeah. Just as IMatch magically picks up WIC codecs, I figured that as long as ExifTool could load and save metadata, IMatch would just treat the file as it treats every other file.

One change:

I gave this a quick look, puzzled by the fact that IMatch did not display the XMP option in E > P > Metadata 2 : File Formats for user-defined formats.

I have changed this as follows: If a user-defined format is selected, IMatch asks ExifTool (by looking at the cached output of -listwf) if the custom file extension is in the list of writable formats. If this is the case, IMatch now also offers the XMP sidecar options, which allows you to configure IMatch to write XMP into the file itself instead of a sidecar (which is the default for user-defined formats).

Thanks, Mario! That's awesome and just what I need.

One final question: How do I trigger IMatch to update its tag database from ExifTool? I know this is triggered on a change of the ExifTool version string relative to the version recorded in the DB (so I think I can force it by zapping the value in the DB), but it would be nice if there was a button for it somewhere.

[Mario]

As I said above, if you want any change in this area, file a feature request. Allowing users to define their own XMP tag via ExifTool configuration files, implementing all the related logic in IMatch etc will be **a lot** of work - for maybe 3 users in the world who will ever need this. If you add a feature request other users can Like it so we see if there is demand for other users. Or not.

IMatch triggers a re-import of tags only when the version number of ExifTool changes. You cannot trigger this manually. See above.

[monochrome]

Just wanted to hop in here with a big Thank You! The 5.4.2 update works perfect.