How can I link JPG image to XMP file on import? (Capture One workflow issue)

[jmhdassen]

Hi,

I am new to IMatch, doing a trial evaluation. I saw all the Videos and liked most of what I saw.
But when actually trying it I can not even get the information I want loaded into IMatch.
I have hundreds of Capture One sessions that I want to catalog (CO catalog is an abomination).
I do not really want RAW files in IM. Although they can be referenced I do not want to use embedded image or embedded metadata.
I was hoping (and trying) to import processed JPG image with associated data stored in XMP file. JPG itself has no embedded metadata.
I have tens of thousands of images like that and reprocessing JPG to include metadata is not an option for me.

So far I have not managed. I can see the JPG but no sign of any metadata.
I read the manual several times and I can not find anything explicit about metadata in XMP files. Everything seems to refer to embedded xmp.
Maybe I am trying to do something which IM has not been designed to do?
For clarity: file names of NEF, JPG and XMP files only have different extensions. The base file names are the same (original camera file name).

So: is it possible, and if so how, to import JPG and get metadata from "associated" XMP file???
Without that, or an easy workaround,  IMatch is not a viable option for my image collection.

Cheers, JD

[Mario]

Note: This board is not for asking questions. It is for giving other users tips and for posting workflow tutorials.
Always read the board description before posting. Don't just post in the first board that shows up on screen. If in doubt, post in General Discussions.
I will move your post this time into the appropriate board.

The XMP standard requires JPEG files to have embedded XMP metadata. Same as with PNG, GIF, PDF, PSD, TIFF, DNG.
XMP in sidecar files is designed to be used for for RAW files and other file formats.

IMatch follows the XMP standard and the rules and recommendations of the Metadata Working Group when processing metadata.
This also ensures optimal interoperability with other applications and services.

You can force IMatch to use the XMP data in the sidecar file via Edit > Preferences > Metadata 2: Configure File Formats.
See https://www.photools.com/help/imatch/#rmh_config_metadata2.htm?dl=h-11

When you enable "Favor XMP sidecar" for JPG files there, and then select your JPEG files and force a reload of the metadata by pressing Shift+Ctrl+F5, IMatch will import the XMP data from the sidecar file. And also write metadata you add or change in IMatch into the XMP file. Not into the JPG file.

But I would not recommend it. Because it will make the way you work with metadata different from how everybody else works. And it may create trouble down the road, when you try to process your JPEG files in other applications and these don't support XMP metadata stored outside the JPEG file.

How do you create these JPEG files? Does C1 not copy the metadata from the RAW and what you enter into the JPEG when you create it?
If C1 cannot do that, you could either

a) Use ExifTool to copy metadata from the RAW into the JPEG, where it belongs or
b) Add your RAW files to IMatch, setup a file version rule, and let IMatch synchronize metadata from the RAW to the JPEG

[jmhdassen]

Sorry I was not aware I was on the wrong board. I was reading the posts on Capture One Workflow and wanted to tie into that.

I am well aware of metadata in images and in XMP files. I have had many trial and errors using metadata and I consciously chose to keep metadata out of images and put it either in XMP files or sqlite where appropriate. This I have done for many years now successfully with Capture One XMP files. I know very well that CO metadata is incomplete and inconsistent, but that is what I get. CO can put metadata into jpg but I chose not too for various reasons.

So with your suggestions I did get the data from the XMP file to load. Kind of............
It works if the xmp is in the same directory as the jpg. But I was wondering if we can have a lateral relation as well. I see this stuff about Master Folder and Direction in the File Relations, but without examples that is a bit obscure to me.

This is what I have with a Capture One Session:
SESSION FOLDER   -- The folder name is the name of the session and I want that in the file Tree
-- CAPTURE FOLDER - (images that I do NOT want)
-- SELECTED FOLDER - RAW and XMP for the images that I want.
-- OUTPUT normally empty
-- TRASH normally empty
-- WEBCONTACTSHEET
------ PHOTOS
---------- PREVIEWS       JPGs that I want to display, matches the RAW files in SELECTED
---------- THUMBNAILS   JPGs that I do NOT want
------ SCRIPTS  stuff that I do not want in iMatch
------ THEMES   stuff that I do not want in IMatch

So with this, I can import the session into iMatch, and use the preview JPGs but I get the metadata only if I copy the xmp into the previews folder.
I was wondering if there is a ways to relate  JPG  ./PREVIEWS/DSC1234.JPG to ./SELECTED/DSC1234.XMP, maybe via DSC1234.NEF ??

I could write a script that would copy the xmp but that is a bit ugly and creates redundancy. Or can I hook a kind of pre-loading script to IMatch ??

Another question related to this: Can I hide the folders that I do not want to see? CAPTURE, OUTPUT, TRASH, THUMBNAILS, SCRIPTS, THEMES.
Can I prevent CAPTURE, TRASH and THUMBNAILS from loading?? Or script a "remove from database" ??

I know there are a lot of questions, but I expect a bit of flexibility on the loading end of a Catalog. And I hope to use it for all images.

Cheers, JD

[Mario]

It works if the xmp is in the same directory as the jpg.

By definition (XMP Standard): The XMP file must be in the same folder as the corresponding image and must have the same file name.
This is easy and standardized. If your rather unique workflow distributes XMP files all over the place, IMatch cannot use them.

Maybe a standard DAM is not the best solution for you. I mean, if you deliberately don't follow established metadata standards, move XMP files where they don't belong, tolerate incomplete metadata, store metadata in SQLite databases...maybe you need a custom collection of scripts which implement your workflow.

DAM is all about adhering to standards. Standard metadata in standard formats and standard storage locations. Not something you want, from your descriptions.

Another question related to this: Can I hide the folders that I do not want to see?

IMatch only adds folders to the database you explicitly add (optionally including sub-folders).
This usually keeps the riff raff out of the database.
It is also common DAM practice to keep a clean ship and move temporary folders and similar out of the folder hierarchy managed by the DAM.

If you absolutely need to add sub-folders recursively even if you don't want them in IMatch, you can use the skip folder option in the Indexing options.

Indexing

Again, my impression is that you are working actively against all common metadata standards and concepts. This will probably fall on your feet in the long end.
At least it will make working with a standard DAM like IMatch much harder, because you always will have to find ways to prevent standard operations from happening, make you DAM somehow find the place where you store the metadata actually belonging into the image etc.

IMatch works great for many people. It may not work great for you.
Your workflow and metadata handling is to special. I would give this a big rethink. And if you have reasons to stick to this rather personal way to deal with metadata, I recommend to consider that IMatch may not be the software you need. I foresee problems in the future, and probably a ton of support cases...

[jmhdassen]

Thanks, Mario.

The indexing preference is what I need.
About the XMP positioning I had a rethink about it. It would actually be good to keep CO xmp separate from IM XMP. I can build a small tool that can copy and compare the XMP's. That way I could, if I wanted to, selectively put back metadata changes into CO. Or at least monitor/compare them.

Don't worry about my workflow. It is the result of years trial and error.
I have 30 years professional Data Management experience. I have learned a lot in those years, especially in interfacing systems that do not follow the same standards.
My university professor already told me: The nice thing about standards is that there are so many to choose from.

I think it is very good the way IMatch implements standards. I am impressed by the way the Metadata is covered. But you can not impose your standards on the outside world without limiting the usefulness of IMatch. Capture One is an excellent image processor, but they have no clue of data management. I still want to use both, one as processor, one as Catalog.
[I am/was building my own Image Catalog using Full Stack web technology. I could never find anything suitable for my needs until someone pointed out IMatch. So now I paused my development and are hoping Imatch can do the essentials of what I need.]

Thanks for your help and fast response.

JD

[Jingo]

I still want to use both, one as processor, one as Catalog.
JD

I suppose this all comes down to how much you like/dislike using multiple software to manage your photos.  I personally only use C1 as a RAW file editor... and once my images leave C1 - I hardly ever use C1 to touch them or view them again.  They go into C1 as RAW - get exported from C1 as JPG and then imported into IMatch as JPG.  Total management of the images are then done from IM only....  if I ever need to edit a RAW version of that JPG again, I just pop back into C1, find the image in the catalog and edit/export a new copy.  Easy to find the image in C1 because my RAW folder structure and image name 100% match my JPG folder structure and image name.

IMO - C1 is an EXCELLENT RAW editor... IMatch is an EXCELLENT Data Manager.. and I use each product's strengths as such.

[medgeek]

They go into C1 as RAW - get exported from C1 as JPG and then imported into IMatch as JPG.  Total management of the images are then done from IM only.

Hi Andy. Just a quick question. I assume C1 can export a clean jpg without metadata. After importing into IMatch, do you then start management of the jpg de novo, or, as a starting point, do you sync the jpg's metadata with the original raw file straight out of the camera? Thanks.

--George

[Jingo]

They go into C1 as RAW - get exported from C1 as JPG and then imported into IMatch as JPG.  Total management of the images are then done from IM only.

Hi Andy. Just a quick question. I assume C1 can export a clean jpg without metadata. After importing into IMatch, do you then start management of the jpg de novo, or, as a starting point, do you sync the jpg's metadata with the original raw file straight out of the camera? Thanks.

--George

Hi George - yes.. I use C1 with no exported metadata so it is as clean as it can be.  I then do ALL management solely on the JPG image in Imatch... keywording, geolocating, rating... even culling!  I choose not to link the RAW and JPG in IMatch via versioning because in my workflow, I'm rarely going back to the original RAW image (I consider the RAW files like a true negative... process into a "print" and store them away just in case).  My old workflow using older versions of IM was to use versioning.. and I found I really never did anything with the RAW's... so just abandoned them altogether!  The nice part of this process.. the DB is smaller, no need for cache since the JPG's load fast, no WIC issues, etc...

Hope this helps! - Andy.

[medgeek]

Hi George - yes.. I use C1 with no exported metadata so it is as clean as it can be.  I then do ALL management solely on the JPG image in Imatch... keywording, geolocating, rating... even culling!  I choose not to link the RAW and JPG in IMatch via versioning because in my workflow, I'm rarely going back to the original RAW image (I consider the RAW files like a true negative... process into a "print" and store them away just in case).  My old workflow using older versions of IM was to use versioning.. and I found I really never did anything with the RAW's... so just abandoned them altogether!  The nice part of this process.. the DB is smaller, no need for cache since the JPG's load fast, no WIC issues, etc...

Hope this helps! - Andy.

That does help, Andy, thanks. What you describe is certainly streamlined.

I'm not sure I want to abandon all the metadata in the original raw file. To that end, I'm considering a workflow from the point you have a clean jpg as you mentioned. Then one could import the metadata in the original raw file, before raw processing programs have a chance to wreak havoc with the info. Of course, camera manufacturers also do a lot of nonstandard stuff with metadata, so even that could be problematic.

--George

[Jingo]

Hi George - yes.. I use C1 with no exported metadata so it is as clean as it can be.  I then do ALL management solely on the JPG image in Imatch... keywording, geolocating, rating... even culling!  I choose not to link the RAW and JPG in IMatch via versioning because in my workflow, I'm rarely going back to the original RAW image (I consider the RAW files like a true negative... process into a "print" and store them away just in case).  My old workflow using older versions of IM was to use versioning.. and I found I really never did anything with the RAW's... so just abandoned them altogether!  The nice part of this process.. the DB is smaller, no need for cache since the JPG's load fast, no WIC issues, etc...

Hope this helps! - Andy.

That does help, Andy, thanks. What you describe is certainly streamlined.

I'm not sure I want to abandon all the metadata in the original raw file. To that end, I'm considering a workflow from the point you have a clean jpg as you mentioned. Then one could import the metadata in the original raw file, before raw processing programs have a chance to wreak havoc with the info. Of course, camera manufacturers also do a lot of nonstandard stuff with metadata, so even that could be problematic.

--George

Hi George - I should add - I don't "remove" the metadata from the images upon export to JPG... all EXIF and standard camera metadata within the IPTC is transferred along to the JPG... I just don't ADD any metadata (such as ranking, copyright, keywords) to the images in C1 to pass along either within the file or as sidecars.  After the JPG's are added to IMatch, I run metadata templates to add general image info such as copyright.  To me - it is the best of all worlds... a catalog of RAW edits in C1 (my negatives) and a catalog of metadata within IMatch and exported out into the JPG's (my prints).

[Carlo Didier]

Just to give you one more view, here's my workflow.

I index raws (90% DNGs until recently) and raws+jpgs (since I started using C1). In the latter case, I set iMatch to uise the JPG as the visual proxy to see my edits of the raw (what did the integrated previews in the DNGs).
As opposed to Andy, I have those proxies only for display purposes in iMatch. When I look for images, I always go to the raw files and then develop them for the use I have then.
You see, there are many possible approaches.

As to your workflow, from my perspective it looks like you are unnecessarily complicating your life. I respect your 30 years experience, but maybe that long time also got you stuck in a mindset that is closed to new ideas and possibilities, even if they could be better or beneficial. To use an analogy, just because you crossed the road for 30 years on the red light without harm is no guarantee that you will not be struck by a truck tomorrow ... 

I know that mentality from the company I work fort (in IT since 1987 ...). When we decided to replace paper forms by electronic forms, the new electronic ones had to look exactly the same as the old paper ones (meaning like from the 50s ...!). Fortunately, that mindset has changed over the last years.

As Mario set, you are working against the system and I'm pretty sure that any professional DAM out there will cause the same "problems" for your workflow.

[medgeek]

Hi George - I should add - I don't "remove" the metadata from the images upon export to JPG... all EXIF and standard camera metadata within the IPTC is transferred along to the JPG... I just don't ADD any metadata (such as ranking, copyright, keywords) to the images in C1 to pass along either within the file or as sidecars.  After the JPG's are added to IMatch, I run metadata templates to add general image info such as copyright.  To me - it is the best of all worlds... a catalog of RAW edits in C1 (my negatives) and a catalog of metadata within IMatch and exported out into the JPG's (my prints).

Thanks again, Andy. I see I was mixing up your remarks with JD's (the original poster).

[jmhdassen]

Thanks for everybody's comments,

I have more or less finished my evaluation of the IMatch trial and I will go ahead to use Imatch as my catalog (and Anywhere as viewer).
My workflow is under control. Everyone has their own way and I have mine. I just have to work with several hundreds of Capture One Sessions and close to a hundred thousand images which already have the essential metadata. I am not going to start from scratch.
Moreover Sessions are an ideal tool for backing up RAW plus adjustments. I will keep using sessions.

Unfortunately CO is not very good with metadata. It has not controlled value lists to speak of.
IMatch does all this very well.
Now there is one thing I am missing (or could not find) in IMatch:
--> If I import data which already has metadata, can I "validate" this metadata against IMatch Thesaurus lists?

That would be very useful to me. Maybe I have not found the tool yet ??

JD

[Mario]

There is no feature to validate metadata against thesaurus entries.

I guess you would like to check if the metadata tags in your externally modified metadata contain one or more? values also fond in your thesaurus?
This is something rather unusual to do and I have encountered this only during migration projects with big metadata mess cleanup tasks in commercial and institutional environments.

It is fairly easy to write a custom IMatch app which understands what kind of validation you need for your files. Including thesaurus look-ups etc.
This is what makes IMatch and the integrated IMWS so great. You can extend the functionality of the application with your own apps or via scripts. If you can program in any computer language, changes are that you can write an app/script/program which does this.

If you cannot program, you can maybe hire somebody who writes custom metadata validation routines for you.

[jmhdassen]

OK Mario,

I was already expecting to have to write a Panel App. Should be no problem for me.

I am kind of surprised that this is not something that comes up regularly. Many RAW processors have the ability to add some metadata and may have different or usually even no value lists. I am not talking about EXIF data, but things like Location/City/State/Country (very important for me) and things like Scene, Genre and even persons shown. CO is a mess for that and darktable is very limited. I am glad that IMatch can solve this. My situation is indeed a migration from legacy data. I suppose most users start with legacy data.

I will look at writing a validation app.

(What would be nice to have also is a preprocessing hook on which I can hang a node or python script)

JD

[Mario]

No hooks or similar need.
You can access IMatch database data, including metadata, from Python directly. Or any other programming language which can utilize a web service as provided by the IMWS embedded in IMatch. It is fully documented and language-agnostic.

Only when you want to run a real app inside IMatch, you need to use HTML and JavaScript.

See the IMatch Developer Center for more information.