XMP Sidecars for Jpgs = bad idea (Mario) -- how integrate LR without them

[WebEngel]

Here is what I do:

I always shoot Raw+Jpg
Imatch is my DAM
Jpg is Master, Raw is buddy
Both Master and Buddy are always in the same directory
Some Raws are processed in LR; the resulting Jpg then replaces the original Jpg
In case I am done with LR (either processed or no processing necessary), I delete the raw file

The applications I use for generating metadata are:

Imatch
GeoSetter
FastPictureViewer

So far, I have done everything in XMP files.  This seems logical to me: The metadata is valid for both the Jpg and the Raw file (at least all data that I am using), so both should share it, and the sidecar seems perfect for that.

But then...

Since you force IMatch to use sidecars for XMP (very bad idea, IMHO)

How is LR going to get the metadata if I am not using sidecars?

LR does not care about metadata in Jpg files (I tested it) when loading the Raw file with the same name.  This means that LR does not know the metadata.  As soon as I export the Jpg, the existing Jpg with the metadata is overwritten, and all metadata is gone forever.

The only solution I could come up with is a kind of script that permanently copies relevant metadata from the Jpg to a sidecar as soon as the Jpg is changed.  Is that how I am supposed to integrate LR?

[pajaro]

Since you force IMatch to use sidecars for XMP (very bad idea, IMHO)

The only solution I could come up with is a kind of script that permanently copies relevant metadata from the Jpg to a sidecar as soon as the Jpg is changed.  Is that how I am supposed to integrate LR?

I do not use LR so I may not help you with your question, and I know that Mario does not encourage us to use sidecar files for xmp. Nevertheless I just wanted to say that I changed Metadata 2 settings so that IMatch writes to and reads from sidecars even for jpgs and this setup works quite flawlessly for me (I use raw rarely, so most of my files are jpgs). So you may try to allow use sidecars for jpgs and see if it works for you...

[Ferdinand]

I don't understand why the JPG is the master.  I make the RAW the master, the info goes into the sidecar (which really only relates to the RAW, not the JPG), you can propagate the metadata to the JPG if you like, LR can find it and will export it to any developed images or you can propagate it from the RAW if you prefer.  Where's the problem?

I guess the problem is that you delete the RAW, which seems an odd approach to me.  But you can propagate from the RAW sidecar to the JPG prior to deleting the RAW I guess.

[WebEngel]

So you may try to allow use sidecars for jpgs and see if it works for you...

Well, that's what I have been doing for years.  Imatch 5 had a couple of bugs for that configuration, the approach is not MWG-compliant, and Mario discouraged me to use sidecars for Jpg.  That's why I was rethinking my approach.

[WebEngel]

I don't understand why the JPG is the master.

The Raw is only there if I decide that both
-the Image needs processing and I have not done so
-The image is so good that I want the option to re-process it
This is only for 10% of my images.

So why should a temporary file that is only needed by LR be the master?

Also, for all tagging operations, it is much easier and faster to work off a standard Jpg format rather than a proprietary Raw file.  So FPV, GeoSetter, and Imatch itself only need to look at the Jpg when I tag an image.  The free FPV that I am using does not handle Raw files anyway.

info goes into the sidecar (which really only relates to the RAW, not the JPG)

Not sure what you mean.  In my case, the information in the sidecar is valid for both the Master and all buddies.  There may be very few data elements like processing status that only refer to the Raw, but things like rating, location, country, description, copyright go for both.

I guess the problem is that you delete the RAW, which seems an odd approach to me.

Well, for all images but the outstanding ones, I process the Raw file once, and that's it.  I doubt that I will ever have the time and the motivation to re-process mediocre images, so I decided not to keep the Raw files.  That saves me up to 80% HD space.

  But you can propagate from the RAW sidecar to the JPG prior to deleting the RAW I guess.

Basically, I have two options:
1) Continue with sidecars for Jpg and Raw
2) Embed XMP into Jpg (MWG compliant) and propagate metadata to a sidecar for files that I am planning to import into LR.

The first option seems to be simpler to me as the additional propagation is not necessary and metadata in sidecars is easier to backup.

[Mario]

I will move this to General Discussion.
his post contains no Tips & Tricks or Tutorial helpful for other users, and was posted in the Tips & Tricks board.

[Mario]

How is LR going to get the metadata if I am not using sidecars?

LR loads metadata for (most) RAW files from the associated sidecar file. For JPEG/TIFF/DNG/PSD files, LR expects embedded XMP data. LR does not look in JPEG files to find metadata for RAW files.

The default settings (MWG compliance) in IMatch work exactly like LR works, and most other RAW processors.

[Nytewulf]

How is LR going to get the metadata if I am not using sidecars?

LR does not care about metadata in Jpg files (I tested it) when loading the Raw file with the same name.  This means that LR does not know the metadata.  As soon as I export the Jpg, the existing Jpg with the metadata is overwritten, and all metadata is gone forever.

The only solution I could come up with is a kind of script that permanently copies relevant metadata from the Jpg to a sidecar as soon as the Jpg is changed.  Is that how I am supposed to integrate LR?

LR does indeed read and write metadata to jpg files.  While you can force it to write metadata to a sidecar file, LR itself will only read the metadata embedded in the jpg.

If you're treating RAW+JPG as one file (which is the LR default), then you're editing the RAW and overwriting the original jpg when exporting (and the original metadata) unless you rename it.

John

[Ferdinand]

So why should a temporary file that is only needed by LR be the master?

Because it is a convenience that will enable you to work the way you want to work, that's why, if I understand your preferred workflow correctly.

So in situations where you want to keep the RAW, you will have in the folder the RAW file and a JPG file, where the JPG has been processed and replaced the OoC JPG.  In situations where you don't keep the RAW, then you just have the OoC JPG.  Have I got this right?

In the case where you don't keep the RAW, the JPG can't be a master if it's on it's own.  A master must have a version.  If you delete all versions then the master ceases to be a master to IMatch, even if you still regard it as one (which I would and in fact I do for my RAW files without versions).  So in one sense it doesn't matter which one is the master in situations where you only end up with one file, since once it's on its own it's neither master nor version.

In my suggested workflow, you mark the RAW as a master and enter your metadata.  You edit the file in LR and produce a version, which replaces the OoC JPG.  This should have the metadata you entered for the RAW, provided that your LR export settings are to include metadata in the exported version.  If not, you can propagate the metadata to the version easily enough in IMatch.  You want to delete the RAW?  No problem, so long as you haven't defined the JPG to be a buddy file - you delete the RAW master and now you have the orphaned version which is no longer a version, since there's no master.  But if wouldn't be a master at this point anyway if you had previously made it one, because there's no version any more. 

The point of making the RAW the master and entering your metadata for it initially is:
.  LR will read the metadata;
.  IMatch can propagate it, if required
.  Writing metadata to a RAW file's sidecar is much faster than embedding it in a JPG, since the sidecar is only a text file. 

I haven't found tagging operations faster for JPGs than RAW files, since it's faster to write to a sidecar than a JPG, and I don't use sidecars for JPG since it's non-standard and not compatible with Adobe s/w.  I use the same s/w as you, although not a free version of FPV (I wasn't aware that there was one).

[Nytewulf]

In the case where you don't keep the RAW, the JPG can't be a master if it's on it's own.  A master must have a version.  If you delete all versions then the master ceases to be a master to IMatch, even if you still regard it as one (which I would and in fact I do for my RAW files without versions).  So in one sense it doesn't matter which one is the master in situations where you only end up with one file, since once it's on its own it's neither master nor version.

While I normally shoot RAW, with one camera I shoot RAW+JPG and usually end up working with the jpg but often keeping the RAW in the event I want to make a large print.  IMatch has the flexibility  to handle these versioning scenarios -- it's simply a matter a adopting a workable file-naming convention.  In this situation I designate the jpg the master the RAW a version and all files derived from the original jpg as versions.  If the RAW is subsequently removed, it's simply the deletion of a version -- it has no effect on the rest of the version structure.

John

[Ferdinand]

While I normally shoot RAW, with one camera I shoot RAW+JPG and usually end up working with the jpg but often keeping the RAW in the event I want to make a large print.  IMatch has the flexibility  to handle these versioning scenarios -- it's simply a matter a adopting a workable file-naming convention.  In this situation I designate the jpg the master the RAW a version and all files derived from the original jpg as versions.  If the RAW is subsequently removed, it's simply the deletion of a version -- it has no effect on the rest of the version structure.

I can see that that works.  But the OP wants something different.  He wants to set the JPG as master, enter the metadata into it, then develop the RAW in LR and replace the original JPG with the developed one, and sometimes / mostly delete the RAW.  Keeping the metadata when you replace the JPG is tricky.  If LR reads it and includes it in the developed file you'd be ok, but for that to happen LR has to get it from the sidecar for the RAW.  Alternatively IMatch can propagate the metadata to the developed JPG if it's stored in another file.

It's replacing the file that is the metadata store that causes the issue for the OP IMHO, and the workflow I described is how I would do this if it was my workflow.

[WebEngel]

Thanks, Ferdinand!

So in situations where you want to keep the RAW, you will have in the folder the RAW file and a JPG file, where the JPG has been processed and replaced the OoC JPG.  In situations where you don't keep the RAW, then you just have the OoC JPG.  Have I got this right?

Almost.  The third situation is that there is a processed Jpg and no Raw anymore.

For example, the Jpg is just 2 stars but had a wrong WB.  I correct that in LR, export the Jpg, and delete the Raw, because for a 2-star image, I don't need the option to re-process it later on.

In my suggested workflow, you mark the RAW as a master and enter your metadata.  You edit the file in LR and produce a version, which replaces the OoC JPG.  This should have the metadata you entered for the RAW, provided that your LR export settings are to include metadata in the exported version.  If not, you can propagate the metadata to the version easily enough in IMatch.  You want to delete the RAW?  No problem, so long as you haven't defined the JPG to be a buddy file - you delete the RAW master and now you have the orphaned version which is no longer a version, since there's no master.

Ok, but on which files should I work in Imatch ("work" means use for tagging, view, show to friends, export for web page) according to your workflow?

Raw files?  Well, first of all this would require raw caching in Imatch despite the fact that there are Jpgs available.  Ok, tolerable, just a memory thing.  Even worse, I would be looking at the unprocessed image if there is a developed Jpg

Jpgs?  Well, the OoC Jpgs don't have any metadata, as everything is in the sidecar and I assume I would set Imatch to MWG compliance.  So I would not see the metadata.  Also, once I change metadata in Imatch when working on the Jpg, it would change the Jpg and not the sidecar, so LR would miss the changes.

So in order to work with Jpgs, I would need some kind of background job that permanently and immediately propagates all sidecar changes to the Jpg and vice versa.  Is that what you are suggesting?

Mixture of Raw and Jpg: I cannot come up with a simple filter setting that would show the right Jpgs and the right Raws; this would be too complex and end up in metadata inconsistencies.

I haven't found tagging operations faster for JPGs than RAW files, since it's faster to write to a sidecar than a JPG.

Misunderstanding.  I was saying that it is faster to use the Jpg as the image source for tagging rather than the Raw file (I mean the file that I am looking on to determine whether it is 3 stars or 4 stars).  I agree that writing a sidecar is faster than embedding into a Jpg.

and I don't use sidecars for JPG since it's non-standard and not compatible with Adobe s/w. 

I would love to use a standard approach as well (that's why I started this thread in the first place), but so far I have not found a way to do so with my workflow.

Martin

[Ferdinand]

Ok, but on which files should I work in Imatch ("work" means use for tagging, view, show to friends, export for web page) according to your workflow?

Raw files?  Well, first of all this would require raw caching in Imatch despite the fact that there are Jpgs available.  Ok, tolerable, just a memory thing.  Even worse, I would be looking at the unprocessed image if there is a developed Jpg

Jpgs?  Well, the OoC Jpgs don't have any metadata, as everything is in the sidecar and I assume I would set Imatch to MWG compliance.  So I would not see the metadata.  Also, once I change metadata in Imatch when working on the Jpg, it would change the Jpg and not the sidecar, so LR would miss the changes.

So in order to work with Jpgs, I would need some kind of background job that permanently and immediately propagates all sidecar changes to the Jpg and vice versa.  Is that what you are suggesting?

Mixture of Raw and Jpg: I cannot come up with a simple filter setting that would show the right Jpgs and the right Raws; this would be too complex and end up in metadata inconsistencies.

The OoC JPGs will have the metadata if you propagate it.  So yes, that's what I'm suggesting.  It can be done automatically.

I would use the RAW files for tagging and metadata input.  For the 80% of these that you delete, you propagate the metadata to whichever JPG you have - OoC or developed - before you delete the RAW.  I'd use the JPGs for view, show to friends, export for web page.  That's what I do, I can't see that a complex filter is needed. 

Just because a file is a master, it doesn't mean it's the final, best version.  Master is just a concept for versioning, so that you can find related files and propagate if needed.  If in some cases you're left with just the JPG (OoC or developed), then that file is not longer a master, so you can't use the master collection as the basis of viewing, showing exporting.   You'll need some way to filter the "final" images. 

It occurred to me that there may be another way that you could work.  If you set the JPG to be the master, enter any metadata and propagate it to the RAW before you send the RAW to LR.  When you develop a JPG, be sure to set the LR export template to include all metadata in the developed image.  LR should get this from the sidecar, because it has been propagated there.  Then when you replace the OoC JPG master with the developed master, it has all the metadata.

[Erik]

So the JPG files out of the camera don't have their own metadata embedded (or at least those from the RAW+JPG)?

If that is the case, you are in one of these situations where at least initially two images are sharing an XMP file.  With regard to file management, this was a topic that showed up as an issue not long ago (search the forums as I can't remember where it might have been).

This issue of one XMP file for two images could end up problematic, especially as soon as you change one of the files (i.e. replace the JPG).  LR will potentially edit the XMP file to reflect develop settings, etc, but I'm not sure what it will do when you replace the JPG.  IM and LR may independently get confused or rather look in the wrong places for data. 

It really might be worth thinking about your strategy for file management as you continue to develop your workflow.  Deciding on your Masters and Versions is important, but I don't think it matters that much which is which as long as you are consistent with it.  My suggestion would be that you work out a method to name the Master and Versions differently for the root of the file name (not just extension) so that if you need this all external XMP file workflow, that there will be an independent XMP file for each image file. 

I believe this will make errors for metadata propagation less likely as data will be copied from one file to another.  This will also make the removal of any files from your database simpler as it is clear which XMP file belongs to which.  I would not recommend having a Master and Version be buddies directly with the workflow you are using only because you could inadvertently delete all files in a version set when you may only intend on removing one.

[WebEngel]

I would use the RAW files for tagging and metadata input.  For the 80% of these that you delete, you propagate the metadata to whichever JPG you have - OoC or developed - before you delete the RAW.  I'd use the JPGs for view, show to friends, export for web page.

That won't work for me: Viewing and Tagging are one, I cannot separate the two processes.  Yes, I normally do an initial run for tagging only and tag all pictures.  But then, when viewing the pictures later on, I may find "this pic is crap" and assign a "red" label (= delete).  Or I may like the image better and increase the rating from 3 to 4 stars.  Or I may find that I did not do a good job in LR and assign a yellow label (in my scheme stands for "reprocess").  Or I may want to add a specific description to a few pictures (while during mass-tagging, I had more generic descriptions for an entire set)

So I think I will need to work with Jpgs only in Imatch.  Which brings me back to my last statement:

So in order to work with Jpgs, I would need some kind of background job that permanently and immediately propagates all sidecar changes to the Jpg and vice versa.

I had already created the ExifTool commands in Imatch to do that weeks ago.  But doing that manually will be prone to errors, and I fear that I will end up with inconsistencies.  So I will look into automated background jobs now.

[Erik]

So I think I will need to work with Jpgs only in Imatch.  Which brings me back to my last statement:

So in order to work with Jpgs, I would need some kind of background job that permanently and immediately propagates all sidecar changes to the Jpg and vice versa.

I had already created the ExifTool commands in Imatch to do that weeks ago.  But doing that manually will be prone to errors, and I fear that I will end up with inconsistencies.  So I will look into automated background jobs now.

While I can understand that sentiment, I think your best bet will be to name the photos differently during the initial download process you do...

For instance

File1.JPG    --- Master
File1_raw.RAW   --- Version

Then let IMatch sync everything that needs syncing through its version propagation routines.  I think this would save you a lot of hassle, and you could still remove the RAW file when you are ready to.  Each image could have its own independent XMP file:

File1.XMP (for the JPG)
File1_raw.XMP (for the RAW file).

It would solve a lot of your problems and be automated.  You still may have some details to work out with LR, but I think this would be simpler than having to find some external app to do background processing.

Of course you can adjust things as you prefer.  However, you still have to watch out for losing the XMP information when creating the new JPG to replace the old one. 

[sinus]

Here is what I do:

I always shoot Raw+Jpg
Imatch is my DAM
Jpg is Master, Raw is buddy
Both Master and Buddy are always in the same directory
Some Raws are processed in LR; the resulting Jpg then replaces the original Jpg
In case I am done with LR (either processed or no processing necessary), I delete the raw file

The applications I use for generating metadata are:

Imatch
GeoSetter
FastPictureViewer

So far, I have done everything in XMP files.  This seems logical to me: The metadata is valid for both the Jpg and the Raw file (at least all data that I am using), so both should share it, and the sidecar seems perfect for that.

Only some general remarks from me:
Why is Jpg the Master and Raw the buddy? This seems to me completely wrong. Not logical.

You write for example "Some Raws are processed in LR; the resulting Jpg then replaces the original Jpg" and here you let work the Raw like it is natural: as a Master and a Raw should be a Master, IMO.

And further you replaces with this created jpg, created from RAW(!) the jpg, YOUR Master.
In your term this means, you have a Master (jpg), you takes the buddy (RAW), creates a new jpg, and replaced the original Master with this created, new jpg.

Sorry, this is completely not logical.

And also this statement "The metadata is valid for both the Jpg and the Raw file" is not logical. It is also against the original rule of Metadata and xmp: Metadata are stored in sidecars for RAWs, but Metadata are stored inside a jpg.

Of course, IF THIS WORKFLOW WORKS FOR YOU: FINE.
But if it does not work, and you faces some problems, then maybe you should rethink your workflow and what you are doing.

I bet, you will not find a lot of photographers, who does have such a workflow like you.
But, ok, swimming against the big flow is not always good, but in this case - my opinion - it makes a lot of sense.

But of course I wish you good luck, so that you can solve your problems!

[cytochrome]

This discussion, and several similar ones, all point to the fact that IMatch is implicitly written for photographers who:
- shoot raw
- derive jpg/tiff
- use versioning with raw=master and jpg/tiff=versions
- use Adobe or Adobe compliant programs = write metadata for the raw in a sidecar and metadata for jpg/tiff in the image
- sync metadata between raw and jpg/tiff with IMatch's versioning mechanism

It is implicit but also explicit since most examples in the Help apply to this schema.

Because IM has so many possibilities and is very flexible one can take different routes and develop other workflows, but then one has to experiment. As Sinus (and Ferdinand, and Mario, and... ) remark, the jpg/master | raw/version is an extreme deviation. Maintaining sidecars for jpg/tiff which are formats designed to encapsulate metadata (and exiftool will do it of course)  demands a strong discipline in the order the operations are applied and of course in the naming scheme (raw and jpg/tiff clearly differentiated) and also in the folder tree (much easier if different file formats are not mixed in one folder). It is certainly feasible, but it is worth the trouble?

I have also a somewhat exotic workflow in that I avoid sidecars at all. The raw is my master, it gets fully cataloged, then all metadata is written to the raw files (Nikon and Panasonic) and Imatch takes care of writing whats needed to the jpg/tiff versions. Very simple if one is not using Adobe and its continuous fabrication/updating of sidecars...

Francis

[Erik]

Francis,

You make some good points about the requirement of discipline when your workflow becomes "unusual" enough.  A bigger issue becomes how other software works with your workflow.  IM has flexiblity, very few other programs do, especially ones that focus on editing (not on cataloging). 

Of course, I guess it doesn't take much to be exotic with one's workflow.  If embedding xmp data is exotic, then I guess I am, although it's somewhat by virtue of my camera having an option to shoot in DNG (and subsequently converting earlier raw files to DNG for uniformity). 

As Sinus points out, the OP's workflow is quite a bit more unusual, but I'm certain I've seen others utilize a JPG as master type of workflow in the past.  I think the bigger challenge, which Sinus alluded to, is that the OP is replacing the Master with a derivative of his Buddy File/Version.  The problem in IM is that this essentially replaces the original file with a new file.  I'm not sure that IM would think that this JPG is the same as the old one.  IM might remove information associated with the original JPG file and replace it with information from the new JPG file.  In essence, the Master file is replaced with a new and different Master.  This is kind of the reverse of what would normally be done since propagation won't work in reverse.

Since the idea of Master and Version (or Buddy) could be reduced to just a name, it might be best for the OP to use the RAW files as masters even if there is no intent on keeping them in the long run.  Once you remove the RAW file and you're only left with the JPG, then the idea of Master and Version are removed as far as IM is concerned.  But, as long as the RAW is there, it would be easy to keep the metadata in sync.  The new JPG would receive the metadata from the RAW file automatically after which point the RAW file could be removed.  I think this could stick with the OP's overall intent. 

Care still has to be taken with file names if the goal is to stick with sidecars for all files.

[cytochrome]

Erik, you make me think !! (big deal  )

I took for granted that for Imatch an image is a name (complete with extension). But to you "I'm not sure that IM would think that this JPG is the same as the old one.  IM might remove information associated with the original JPG file and replace it with information from the new JPG file.  In essence, the Master file is replaced with a new and different Master. "

How would IM differentiate between two jPGs with the same name? by analyzing the metadata context in the db? seems very complicated. Of course Mario can do it and more...

With a well designed naming scheme and coherent version rules it should work, but still.. Of course the raw as master is the simplest by far, iM will love it.

Francis

[WebEngel]

So the JPG files out of the camera don't have their own metadata embedded (or at least those from the RAW+JPG)?

Well, they have of course EXIF like aperture and so on.  They don't have description and geo coords.

If that is the case, you are in one of these situations where at least initially two images are sharing an XMP file.  With regard to file management, this was a topic that showed up as an issue not long ago (search the forums as I can't remember where it might have been).

Well, you name it "sharing".  My intention is: It is the SAME image.  Two files, one image.

This issue of one XMP file for two images could end up problematic, especially as soon as you change one of the files (i.e. replace the JPG).  LR will potentially edit the XMP file to reflect develop settings, etc, but I'm not sure what it will do when you replace the JPG.  IM and LR may independently get confused or rather look in the wrong places for data.

LR will only look in the sidecar.  It will correctly store all metadata changes in the sidecar (assume I change a rating in LR because the WB correction I had in mind when doing the initial rating does not work).  It will unfortunately also store the Develop settings in the sidecar (a design flaw, I think, because these are proprietary settings that belong in the internal LR DB and not in a standard metadata sidecar).  But anyway, everything works well.

My suggestion would be that you work out a method to name the Master and Versions differently for the root of the file name (not just extension) so that if you need this all external XMP file workflow, that there will be an independent XMP file for each image file. 

There seems to be a misunderstanding.  The intention of storing the metadata for the Jpg in a sidecar is NOT that I need the XMP information for the JPG in the sidecar.  I need the XMP info for the Raw in the sidecar.

If I split/duplicate the XMP information (two metadata records, one for the Jpg and one for the Raw), I could as well embed the record for the Jpg in the Jpg and remain MWG-compliant.

My entire idea was to have just ONE metadata record for the image, not two records!

[WebEngel]

And further you replaces with this created jpg, created from RAW(!) the jpg, YOUR Master.
In your term this means, you have a Master (jpg), you takes the buddy (RAW), creates a new jpg, and replaced the original Master with this created, new jpg.

Sorry, this is completely not logical.

Well, my idea was: I do not have a raw file for all Jpgs, and I also delete 80% of my Raws, 50% of them without having used them at all.  So Raws are only temporary auxiliary files for LR to enable better image corrections in some rare situations.  There are only three reasons why Imatch should actually know about the Raws: To rename them along with the Jpg, to possibly move them to another folder along with the Jpg, and to delete them based on a color scheme.

You say it is not logical to replace the master.  Were the Raws the master, I would delete the master and make the former buddy the master or the only file.  Sounds also a bit illogical to me.

That's why I considered it to be more logic for the Jpg to be the master.

I am open to change my mind, but in this thread; however, it turned out that it does not really matter.  What does matter is how many XMP records I have: just one in the sidecar or two -- one in the sidecar and one in the Jpg.

Grüße aus dem großen Kanton
Martin

[WebEngel]

Because IM has so many possibilities and is very flexible one can take different routes and develop other workflows, but then one has to experiment. As Sinus (and Ferdinand, and Mario, and... ) remark, the jpg/master | raw/version is an extreme deviation. Maintaining sidecars for jpg/tiff which are formats designed to encapsulate metadata (and exiftool will do it of course)  demands a strong discipline in the order the operations are applied and of course in the naming scheme (raw and jpg/tiff clearly differentiated) and also in the folder tree (much easier if different file formats are not mixed in one folder). It is certainly feasible, but it is worth the trouble?

From your comment about the naming scheme and folders, I conclude that there still seems to be a misunderstanding with regards to my intention.

I want the following
A.Jpg
A.Raw (if existing)
A.XMP

The XMP should hold the XMP metadata for both the Jpg and the Raw.  There is just ONE XMP record.  That's why both files MUST NOT be separated and MUST NOT have different names in my workflow.

This approach does NOT require discipline.  Any application changing metadata should ONLY change the sidecar.  So nothing can get mixed up.

The approach suggested in this thread in a nutshell is: Two XMP records, one in the Jpg, one in the sidecar; and Imatch permanently propagating metadata between the two.  This does require that Imatch be up and running all the time and pick up all changes and propagate them.  If Imatch misses a change, there will be inconsistency.

For example: I change the city and sublocation in GeoSetter and embed the information in the Jpg.  Assume Imatch does not propagate that to the XMP sidecar (because maybe I forgot to launch Imatch).  Then I will process the Raw in LR.  LR does not see the city and sublocation, will export the Jpg and thereby kill the city information in the existing Jpg.

[WebEngel]

I think the bigger challenge, which Sinus alluded to, is that the OP is replacing the Master with a derivative of his Buddy File/Version.  The problem in IM is that this essentially replaces the original file with a new file.  I'm not sure that IM would think that this JPG is the same as the old one.  IM might remove information associated with the original JPG file and replace it with information from the new JPG file.  In essence, the Master file is replaced with a new and different Master.  This is kind of the reverse of what would normally be done since propagation won't work in reverse.

This is a good point.  In previous Imatch versions, this caused trouble: If I exported the Jpg from LR, Imatch 3 would loose the information associated with it like categories.  So I had to shut down Imatch 3 before doing the export in LR.  When I re-opened Imatch 3 or the DB, it picked up the new image but retained the categories.

Imatch 5 does not have this problem.  During the replacement of the Jpgs, the image becomes greyed out in Imatch and shows the "Refresh arrows", but Imatch keeps all the information.

[WebEngel]

How would IM differentiate between two jPGs with the same name?

Not sure if I understand you correctly.  In my workflow, there are never two Jpgs with the same name, as the derived Jpg replaces the OoC.

Anyway, to your question: I belive that Images in Imatch have a unique Id, so you can have the same file.ext unlimited times (though not in the same folder, but this is a Windows thing).  But Mario can answer that better.

As mentioned in my last post, Imatch does recognize abc.jpg as the same image even if it is overwritten or changed by external applications.  So it seems to retain the Id, retain the categories, just recreate the thumbnail and so on.

[WebEngel]

Guys, I am notorious for being stubborn and a pain in the *** in discussions.  Despite my sometimes harsh comments, I do appreciate your comments and suggestions a lot.

And even if it sometimes does not show, I DO think about them!

Thanks!

[Ferdinand]

This thread is now very long and I don't have the time or mental energy to comprehend it all.  But I will make some concluding comments.

IMatch is very flexible and allows you to configure it to work in may ways.  Mario has gone to a lot of lengths to provide these options, often at the cost of a lot of time to code them and also the cost of complex code and options.

So I don't agree that " IMatch is implicitly written for photographers who use Adobe ..." - anyone can use it almost any way they like.

LR on the other hand is highly inflexible.  It's the program that insists on sidecars for RAW and embedded for standard file formats.

The problem that people have in using nonstandard workflows IMHO is not caused by IMatch, it's caused by LR.  If you want to use Adobe s/w as well as IMatch then you have to follow their standards.  Simple.  If you do then IMatch fits the workflow well, but if you try to buck Adobe's approach (e.g. sidecars for JPG), it's not IMatch's fault that there are issues.

There is an old joke about the difference between a pure mathematician and an applied mathematician.  An applied mathematician can find the solution to any problem, whereas the pure mathematician can find the problem with any solution.  (People with tertiary maths training will appreciate this the most.)

I feel like the applied mathematician, trying to find a solution that works.  I am sure that a workflow could be found that wasn't too complex.

[Carlo Didier]

Guys, I am notorious for being stubborn and a pain in the *** in discussions.  Despite my sometimes harsh comments, I do appreciate your comments and suggestions a lot.

And even if it sometimes does not show, I DO think about them!

Thanks!

Thinking is very hard at times, but often very useful 

Apart from that, I'd like to offer my two cents about your workflow. Most people find it illogical because of this analogy:

A raw file is like a negative. And a negative is the original source of your image, so logically a master from which all prints are derived. You will never derive a negative from a print
(ok, you could photograph a print and get a negative, but that's not the "normal" way). The whole definition of master and version is that a version is derived from the master and that's where your workflow goes the other way around.

I don't have all these problems because my whole workflow is DNG based, but that's not for everyone either.

As others posted, but you apparently didn't understand, is that not IM is the problem when you use one xmp file for both the raw and the jpg. It's all the other applications, like LR. You think LR gets the metadata from the XMP for both, but that's not the case. LR reads and writes metadata for the raw into the XMP file but completely ignores the XMP file for the jpg (in the default configuration). If you deviate from this standard, you are calling for trouble and essentially causing yourself more problems and loosing more time than if you'd stick with the traditional ways.

I hope you can sort this out without loosing too much hair and time.

[sinus]

Grüße aus dem großen Kanton
Martin

ot:
Auch Grüsse zurück, hm, grosse Kantone müssen ja auch viel Geld bezahlen an den Schweizer Staat, viel mehr als die kleinen - deshalb haben Hoeness, Schwarzer und Konsorten schon längst begonnen damit 

[sinus]

There is an old joke about the difference between a pure mathematician and an applied mathematician.  An applied mathematician can find the solution to any problem, whereas the pure mathematician can find the problem with any solution.  (People with tertiary maths training will appreciate this the most.)

I feel like the applied mathematician, trying to find a solution that works.  I am sure that a workflow could be found that wasn't too complex.

again ot, sorry:
Lately I read a sentence on a shirt:
"If this it the solution, than I want my problem back!" 

[Carlo Didier]

"If this it the solution, than I want my problem back!" 

Excellent!

[WebEngel]

As others posted, but you apparently didn't understand, is that not IM is the problem when you use one xmp file for both the raw and the jpg.

I disagree.  I have posted quite a few issues already in the bug forum here. There is one open.

It's all the other applications, like LR. You think LR gets the metadata from the XMP for both, but that's not the case. LR reads and writes metadata for the raw into the XMP file but completely ignores the XMP file for the jpg (in the default configuration).

Not really.  I don't think that "LR gets the metadata from the XMP for both".  I think that LR ignores the Jpg completely.

LR does exactly what I want: It treats Raw and Jpg as ONE IMAGE and then ignores the Jpg, as it has the higher-quality Raw file, so why should it bother with the Jpg.  It reads the metadata from the XMP file.

If you deviate from this standard, you are calling for trouble and essentially causing yourself more problems and loosing more time than if you'd stick with the traditional ways.

I am still not convinced that the standard is less trouble.  As probably agreed in this thread: Using the standard requires permanent metadata propagation from sidecar to Jpg and back, and if that fails, there will be inconsistent data.

You are right when it comes to loosing time.  The problem is that I am the only one using and testing Imatch this way, and this is extremely time-intensive.

[Carlo Didier]

I don't think that "LR gets the metadata from the XMP for both".  I think that LR ignores the Jpg completely.

LR does exactly what I want: It treats Raw and Jpg as ONE IMAGE and then ignores the Jpg, as it has the higher-quality Raw file, so why should it bother with the Jpg.  It reads the metadata from the XMP file.

Wrong: LR gets the metadata from the XMP for the raw file only! It does in no way associate the XMP file and its contents with the jpg.
LR does not treat them as one image, but as two associated images, one derived from the other and both having their own seperate metadata in seperate places, the raw file in the XMP sidecar and the jpg in itself.

[WebEngel]

I don't think that "LR gets the metadata from the XMP for both".  I think that LR ignores the Jpg completely.

LR does exactly what I want: It treats Raw and Jpg as ONE IMAGE and then ignores the Jpg, as it has the higher-quality Raw file, so why should it bother with the Jpg.  It reads the metadata from the XMP file.

Wrong: LR gets the metadata from the XMP for the raw file only! It does in no way associate the XMP file and its contents with the jpg.
LR does not treat them as one image, but as two associated images, one derived from the other and both having their own seperate metadata in seperate places, the raw file in the XMP sidecar and the jpg in itself.

I disagree.  The setting in LR is "Treat JPEG files next to Raw files as separate photos"; by default it is false, so LR will treat them as one photo.  Since there is only one photo, there is only one metadata record, and that's the one from the sidecar.  There is no option in LR to view the Jpg's metadata.

Let's not get into a philosophical discussion about LR details like "one ore two photos",  whether "ignore" is the right term for what LR does, or speculate how exactly LR represents the Jpg in the DB. The point is: LR's behavior is as if the Jpg were not there.

Sorry to insist, but this is absolutely key to the question in this thread.  Everybody says my existing workflow is exotic and LR does not support it nor do other applications.  Now this is not true, LR perfectly supports this workflow.

[WebEngel]

Why is Jpg the Master and Raw the buddy? This seems to me completely wrong. Not logical.

By the way, I just remembered the main reason why I decided some time ago that the Jpg MUST be the master and the Raw the version.  It is all about deleting files in Imatch:

When I delete a Raw file, I only want to delete the Raw file (I have decided against future image processing but want to keep the Jpg image)
When I delete a Jpg, I want to delete the Jpg, the Raw, and the sidecar. (I want the entire photo to be deleted)

With Jpg as master, this works easily with the default settings.  With Raw being the master, this could turn out to be very difficult.

[Ferdinand]

When I delete a Raw file, I only want to delete the Raw file ... With Raw being the master, this could turn out to be very difficult.

The fact that the RAW is the master and the JPG the version should not cause a problem.  I have done this once or twice, but test first.  What would cause a problem would be if the JPG was a buddy of the RAW file - then the JPG would be deleted at the same time.  Buddy relations and version relations are different. 

[Mario]

IMatch does not delete versions when you delete the master. But it deletes buddy files when you delete the master. That's the purpose for buddy files: To move, copy, rename and delete them together with their master file. If you don't want this, don't use buddy files.

[sinus]

Why is Jpg the Master and Raw the buddy? This seems to me completely wrong. Not logical.

By the way, I just remembered the main reason why I decided some time ago that the Jpg MUST be the master and the Raw the version.  It is all about deleting files in Imatch:

Well, but for deleting, a jpg MUST NOT be the master and the Raw the version. This is FOR ME no reason.

When I delete a Raw file, I only want to delete the Raw file (I have decided against future image processing but want to keep the Jpg image)

I do the same, sometimes, for some clients.
I have a RAW, have processed them and delivered the jpgs to the client.
After some time, I do not want hold the RAWs, because I use them never more. If the client wants some pics again, well, then I have still the jpgs.

So, beeing the RAW the master, and the jpg a version, delete the RAW is no problem. I select the RAW, delete it (or severals) and voila: the RAW INCLUDING the sidecar (xmp) will be deleted, only the jpg without sidecar is still here. And also logical: the version-symbol of the jpg disappears.
For me: fine.

When I delete a Jpg, I want to delete the Jpg, the Raw, and the sidecar. (I want the entire photo to be deleted)

Sometimes I want hold the RAW too, because I think, one day I wants to develop it again (better). But the jpg I do not more use, so I want to delete only the jpg.
Easy: select the jpg (the version), delete it and it is done: RAW and sidecar will be stay. And also logical: the master-icon from the NEF disappears.
Fine

If you say, when you delete a jpg, you want delete also the RAW and the sidecar. Well, what if you want one day ONLY delete the jpg and hold the RAW?

The solution (and the more intuitive workflow for me) is simple and for me more logical, like it is one of the main-guide of IMatch: most of the commands are based on the selected thumbs.

When you wants delete the JPG AND the RAW (in your words "I want the entire photo to be deleted): simply select both and both photos will be deleted and also the sidecar. Your "entire photo" is deleted.

With Jpg as master, this works easily with the default settings.  With Raw being the master, this could turn out to be very difficult.

While your first sentence may be true (I did not test it, but of course I believe you), I see not the problems you are seeing with your second sentence.
With RAW beeing the master, you will not have troubles and it works smoothly.

And - not to underestimate - this way it the native way of IMatch and hence with this "natural" workflow (Raw is the master), IF problems will arise, a lot of users will have them and Mario will try to solve them.
If you have a workflow, what is a kind of "counter the native way), if problems arise, you will be more "alone".

But, don't get me wrong: IF you are happy with your workflow and you have no troubles: go ahead.

[WebEngel]

By the way, I just remembered the main reason why I decided some time ago that the Jpg MUST be the master and the Raw the version.  It is all about deleting files in Imatch:

I made a mistake here; it should be "Buddy" instead of "version".  I was referring to the old Imatch 3.5 days.  To be honest, I have never used versions in Imatch.

The fact that the RAW is the master and the JPG the version should not cause a problem.  I have done this once or twice, but test first.  What would cause a problem would be if the JPG was a buddy of the RAW file - then the JPG would be deleted at the same time.  Buddy relations and version relations are different.

You are absolutely right.

[WebEngel]

OK, new episode: I created a new test DB to try out the new settings:
-MWG-compliant
-Raw is Master, Jpg is version, as many her suggested.

As mentioned earlier, I need to tag based on Jpg data.  I could do that in Imatch with Proxies, so view the Jpg and tag the Raw file by writing to the sidecar.   But other applications are not as smart.  So assume all tagging applications are MWG compliant, so they will embed their data into the Jpg.  How do I propagate the data then into the sidecar?

Imatch by default only propagates from master to version.  This leaves a couple of options:
Manual propagation via Exiftool-Job from within Imatch
Other Imatch background scripts that I don't know of?
Make Jpg the Master in Imatch?
Set up the other tagging applications to write sidecars for Jpg?
Any other solution?

What is your suggestion?

Sorry, I still don't fully understand the workflow proposed here.

[Ferdinand]

This thread became far too complex for me along time ago.  Nonetheless, I will try one more time, based on what I think your desired workflow is.

.  Edit your versioning relations to make the RAW the master and the OoC the version.
.  Edit your versioning relations so that the metadata fields you want in the JPG are propagated from the RAW, e.g. XMP without Camera Raw data and without orientation.
.  Edit your buddy file definitions so that the JPG is not a buddy file of the RAW.
.  Automatic Propagation on in Background processing

.  Enter the metadata in the RAW, and commit the writeback.
.  If automatic propagation is on then it should happen automatically.  Check that this is the case.
.  If it didn't, then manually propagate from the RAW master to versions.  There are a number of ways to do this, e.g. select all files and F4,P.
.  If you're happy with the JPG image then you can delete the RAW.  Since the JPG is not a buddy file, it should remain and become neither a master or version, since it's on its own.
.  If you're not happy with the JPG then open the RAW in LR.  Edit the file and export it, replacing the OoC JPG if you like.
.  Make sure that your LR export settings are to include all relevant metadata in the developed JPG.  Since the RAW has it all, LR should insert it in the JPG.
.  Now the developed JPG should have all the metadata you entered.
.  If it doesn't, you can propagate it from the RAW.  Select the RAW and F4,P or select the JPG and F4,T.
.  You can now delete the RAW if you like.  Again, since the JPG is not a buddy file, it should remain and be neither a master or version, since it's on its own.
.  You want to view all files?  No problem, filter on JPGs, since they're your final images.
.  If you want to change metadata at this stage, it will get a little tricky but still possible.  You'll need to create a category that includes all master files and all JPGs that are neither master or version.  It would be possible to create such a formula category, and use it as a filter.  These files are the ones that you need to enter the metadata in - if there's a RAW it will propagate to the JPG and if there's only a JPG then that's the one you enter metadata into.

This is based on my understanding of your workflow.

In this post:
https://www.photools.com/community/index.php?topic=3304.msg21691#msg21691
I sugggested another workflow that you may prefer.
"It occurred to me that there may be another way that you could work.  If you set the JPG to be the master, enter any metadata and propagate it to the RAW before you send the RAW to LR.  When you develop a JPG, be sure to set the LR export template to include all metadata in the developed image.  LR should get this from the sidecar, because it has been propagated there.  Then when you replace the OoC JPG master with the developed master, it has all the metadata."

I'm not going to spell out all the steps again.  Hopefully you can get the idea based on the first part of the post, but it is very similar to what I suggested.

[WebEngel]

This thread became far too complex for me along time ago.

I'm not going to spell out all the steps again.  Hopefully you can get the idea based on the first part of the post, but it is very similar to what I suggested.

Thanks for your patience.  There was maybe a misunderstanding, as I had already implemented a new DB to test the steps you describe in your mail.   So that part worked.  I just had some very specific questions to remaining open issues.  Particularly, how to automatically propagate from version to master.

So I guess that this is not possible, and I would need to manually run ExifTool jobs.

.  If automatic propagation is on then it should happen automatically.  Check that this is the case.
.  If it didn't, then manually propagate from the RAW master to versions.  There are a number of ways to do this, e.g. select all files and F4,P.
[...]
.  Now the developed JPG should have all the metadata you entered.
.  If it doesn't, you can propagate it from the RAW.  Select the RAW and F4,P or select the JPG and F4,T.
[...]
If you want to change metadata at this stage, it will get a little tricky but still possible

Thanks.  This workflow gets now very complicated.  I am going to further test it in my new test DB.

In this post:
https://www.photools.com/community/index.php?topic=3304.msg21691#msg21691
I sugggested another workflow that you may prefer.

Thanks, I did not overlook it.  It is just that everybody says the Raw should be the master and everything (including what I do currently) is illogical.  That's why I deferred this approach in the beginning.

Martin

[Ferdinand]

Particularly, how to automatically propagate from version to master.

Not possible.  At least not using standard file relations. 

I have a script in the script gallery that copies label and rating from a version to the master and all other versions.  I wrote this because I often find it easier to rate final versions rather than RAW masters.  This script could be hacked to propagate other metadata, but I don't propose to do this myself.