IMatch and Lightroom: Sidecar necessary

[sinus]

Sorry, folks,
I think, we have discussed SOMEWHERE round about this scenario, but I could not find this specific question.

I want only to be sure, hence comments would be helpful for me:

------
I work with RAWs (NEF) and jpgs.
I do ALL my DAM-work in IMatch.
I do use Lightroom/Photoshop for my RAW-editing.

I am still not sure, should I go with sidecars with my RAWs or not.

Are these infos correct, sorry, only to be sure:

1) If I work NOT with xmp-labels and xmp-rating, I could do my work WITHOUT sidecars, because
a) I can hold all Metadata in IMatch itself (in the db of IM)
b) All my RAW-edition-work I can hold in the DB of Lightroom

2) If I want have my RAW-edition in LR with my images, then I must go with the sidecar, so I would have a sidecar, what travels with the RAW, in this case, I could be without hold these informations (editing) in the DB of Lightroom, because these informations goes with the sidecar.

So I think, when I do think right, I have to decide, want I have the RAW-editing-Information with the image or in the LR-database.
If I go with sidecars, then in IMatch I could see the cropping. But I would have an xmp-sidecar for each RAW-image.
If I trust the LR-DB, then I could be without the sidecar, but have not informations in a sidecar, hence no cropping in IMatch.

I hesitate, because I have never worked with sidecars, and I am a bit anxious to work with suddenly 100'000 images (xmp) more.
I could simply avoid this, if I would trust the DB of LR (and IM of course).

And finally, if I go with sidecars, what is best: let create them in IMatch or Lightroom?

Or did I miss someting basically? (good possible)

Sorry, for such a lot of questions, and a bit "uncoordinated", and we have touched such questions, but I could not find this here.
And on the other hand, I am sure, there are a lot of users out there, who has such a scenario: use IMatch and use Lightroom and RAWs, and do not know, should they use Sidecars or not.

[Ferdinand]

You and I had a long discussion about this a while back.  If you can't find it then I'll try and find it next time I have internet access.

My view is to use sidecars.  But you must be careful to writeback any pending changes in IMatch before you do anything in LR.  When you switch to LR you must reload the metadata.  When you have finished in LR you must ensure that it has written out its settings to the sidecar (there is a catalog setting to automate this) before you return to IMatch.  Back in LR you need to reload the metadata.  In short, you must be careful to follow a strict sequence.

[sinus]

You and I had a long discussion about this a while back.  If you can't find it then I'll try and find it next time I have internet access.

My view is to use sidecars.  But you must be careful to writeback any pending changes in IMatch before you do anything in LR.  When you switch to LR you must reload the metadata.  When you have finished in LR you must ensure that it has written out its settings to the sidecar (there is a catalog setting to automate this) before you return to IMatch.  Back in LR you need to reload the metadata.  In short, you must be careful to follow a strict sequence.

Yes, Ferdinand, thanks a lot, I will try to find it again (I did, but ok, I searched not long enough).

A strict sequence, I think, this will not be a problem, this I will do.
I can remember, according to your informations, I thought, I will got this sidecar-line, I hesitate only, because I have not experience and because it is a lot of xmp-files more.

But ok, I will not realize them, I think. I have only to care also, first what you saide about LR and IMatch and for example for backup.

Thanks, hey, have a lot of success and fun on your travel!

[sinus]

And, Ferdinand,

Here I found 3 links, where the subject is also about xmp-sidecars! 

https://www.photools.com/community/index.php?topic=2607.msg16935#msg16935

https://www.photools.com/community/index.php?topic=650.msg4347#msg4347

https://www.photools.com/community/index.php?topic=596.msg3478#msg3478

[Erik]

I'm being lazy (not looking at the links), but my thought would be to see what Lightroom does by default.

If you don't want XMP files and LR and IM support embedding XMP data in your file formats, then stick with the embedded information.  I've found that LR especially can be quite picky because it does expect metadata in specific locations, depending on the file type.  LR is not nearly as flexible as IM is.

However, with embedded metadata, you still have to be as careful about your sequencing when using LR as Ferdinand stated, especially on the LR side.  i.e. make sure you scan when you start LR and make sure your develop settings are written to XMP (if that is what you want).  I have that set up to run automatically.

[Jingo]

Just curious.. are you exporting the JPG's from your NEF files in LR and using only those to view in IMatch?  That is the method I am going with now... JPG files only (which are the fully developed files from LR) in IMatch with the NEF file  remaining in the LR database.  No metadata in LR for the NEF... if I need to edit the image again down the road, I will find it by filename in LR using IMatch as the DAM.  I find this method a) much faster because IMatch is only dealing with JPG's   b) easier because I don't need to worry about versions and metadata conflicts  and   c) Ideal because I see the actual edits in IMatch the way the photo is supposed to look (the NEF wouldn't show me this and I"d need a JPG or convert to DNG in any event.

I thought about DNG but it is just too darn slow with 24MP ARW files.... the conversion takes forever, the system is slow handling these large files and it seems like a big inconvenience just to maintain a single file.

Hope this helps!

[sinus]

I'm being lazy (not looking at the links), but my thought would be to see what Lightroom does by default.

If you don't want XMP files and LR and IM support embedding XMP data in your file formats, then stick with the embedded information.  I've found that LR especially can be quite picky because it does expect metadata in specific locations, depending on the file type.  LR is not nearly as flexible as IM is.

However, with embedded metadata, you still have to be as careful about your sequencing when using LR as Ferdinand stated, especially on the LR side.  i.e. make sure you scan when you start LR and make sure your develop settings are written to XMP (if that is what you want).  I have that set up to run automatically.

Hi Erik
I do use NEFs (Nikon) and I do not want embedd Metadata in the files. I think, I will go the same line as Ferdinand does, with sidecars.

[Erik]

I'm being lazy (not looking at the links), but my thought would be to see what Lightroom does by default.

If you don't want XMP files and LR and IM support embedding XMP data in your file formats, then stick with the embedded information.  I've found that LR especially can be quite picky because it does expect metadata in specific locations, depending on the file type.  LR is not nearly as flexible as IM is.

However, with embedded metadata, you still have to be as careful about your sequencing when using LR as Ferdinand stated, especially on the LR side.  i.e. make sure you scan when you start LR and make sure your develop settings are written to XMP (if that is what you want).  I have that set up to run automatically.

Hi Erik
I do use NEFs (Nikon) and I do not want embedd Metadata in the files. I think, I will go the same line as Ferdinand does, with sidecars.

Sounds good... I had thought you said you didn't want them.  I think you'll have no problems.  After you practice the process it goes well and by instinct.

[Mario]

LR uses MWG-based settings (I think, Adobe does not document all this in detail). But in general, Adobe expects embedded metadata (XMP) for all files they list in their XMP specification (including JPEG, TIFF, DNG, PSD,...) and they expect and handle only sidecar files for RAW formats. If you stick to that, LR and IMatch work well together.

Make sure you configure LR to export XMP data immediately to the image file / sidecar and not to keep it in the catalog only (then the data is invisible for other applications and IMatch). In the Catalog settings this is the Automatically write changes to XMP option.

LR writes data back at some time (in the background) so you may have to wait a bit when you have modified many files. If you modify RAW files, the update is usually fast because only XMP files have to be updated. If you work with DNG or PSD it can be slower. The bulk of what LR writes to the metadata are it's own proprietary settings. LR does not allow you to modify many XMP fields, mostly standard IPTCCore fields anyway.

When you close LR while it still writes metadata in the background you will see a warning and you should wait.

IMatch will re-import the files automatically as soon as LR has finished writing them, picking up the changes made and updating your database, re-creating data-driven categories, collections, running metadata templates etc.

When you have LR and IMatch running at the same time and you make changes to metadata in IMatch and you write it back, LR usually picks up the changes automatically. Sometimes this does not happen then you need to manually Synchronize the folder.

It is generally not a good idea to create situations where you change the metadata for the same files at the same time in both LR (or another application) and IMatch. This can cause race conditions where one of the applications either does not see the changes made by the other application, or even erases the changes made by the other application by overwriting the metadata of the other application with it's own metadata.

Adobe did not really design XMP to be used as a reliable metadata exchange format for multiple applications from different vendors. There are no locking mechanisms etc. to prevent concurrent updates of the same XMP file etc.

But with a bit of common sense, LR works well with IMatch. The same is true for other RAW processing applications or audio, video and Office software. I work with Photoshop, LR and IMatch 5 at the same time in my usual workflow.

[Ferdinand]

When you have LR and IMatch running at the same time and you make changes to metadata in IMatch and you write it back, LR usually picks up the changes automatically. Sometimes this does not happen then you need to manually Synchronize the folder.

I have never seen it happen automatically.  I have always had to manually sync.

I suggest the above post be copied to a FAQ.

My recollection is that at one stage Markus was thinking of working with embedded XMP in IMatch and sidecars in LR.  At the time I felt that this was unwise, and the problems we have seen with ratings in CR2 files demonstrates why this approach is dangerous - there are problems that occur if you have different metadata embedded in the file to what's in the sidecar.  Having to sequence things carefully, as described by Mario, is a pain.  But I think the alternative is worse.

[Mario]

Unless a camera vendor comes up with funny ideas (e.g. embedding XMP data in files, always writing a rating=0, etc.) things usually work pretty smooth. IMatch allows you to even overcome the funny embedded XMP in CR2 by switching the "favor XMP sidecar" option on for CR2 files.

I've written a blog entry at:

http://www.photools.com/3172/adobe-lightroom-imatch-5/

(for better exposure and search engine ranking) and I will link to that from a FAQ. I need more incoming links to photools.com or else Google will push my web site down in the search results...today you often don't write for your users/readers, but for the search engines...

[sinus]

When you have LR and IMatch running at the same time and you make changes to metadata in IMatch and you write it back, LR usually picks up the changes automatically. Sometimes this does not happen then you need to manually Synchronize the folder.

I have never seen it happen automatically.  I have always had to manually sync.

I suggest the above post be copied to a FAQ.

My recollection is that at one stage Markus was thinking of working with embedded XMP in IMatch and sidecars in LR.  At the time I felt that this was unwise, and the problems we have seen with ratings in CR2 files demonstrates why this approach is dangerous - there are problems that occur if you have different metadata embedded in the file to what's in the sidecar.  Having to sequence things carefully, as described by Mario, is a pain.  But I think the alternative is worse.

I agree with Ferdinand, very interesting informations, though Ferdinand gave also a lot of very good tips.

Ferdinand, about embedded XMP in IMatch, you mean, embedd XMP into the file, like this was possible in IM3 with xmp (experimental). I thought, embedding XMP into the file is not more possible?

[cytochrome]

...
Ferdinand, about embedded XMP in IMatch, you mean, embedd XMP into the file, like this was possible in IM3 with xmp (experimental). I thought, embedding XMP into the file is not more possible?

All my NEF (and RW2 also) have embedded XMP. But this is a different workflow, I would not mix it  with sidecars.

In the first LR versions (LR1 and 2) if I remember well the metadata were read correctly from the NEF but writtent to XMP sidecars.

Francis

[Ferdinand]

Ferdinand, about embedded XMP in IMatch, you mean, embedd XMP into the file, like this was possible in IM3 with xmp (experimental). I thought, embedding XMP into the file is not more possible?

It's possible (metadata2 preferences), but not recommended.

[cytochrome]

Just out of curiosity: am I really the only one to use IMatch with a sidecar-less workflow?

Francis

[Erik]

Just out of curiosity: am I really the only one to use IMatch with a sidecar-less workflow?

Francis

I don't use sidecars, but my camera only shoots DNG, so it doesn't make sense to use sidecars. LR wouldn't like sidecars for those files and wouldn't save to them.

[Ferdinand]

Just out of curiosity: am I really the only one to use IMatch with a sidecar-less workflow?

If you shoot RAW (not DNG), then IMHO it only makes sense if your RAW converter uses embedded XMP, and that's not common.

[Lord_Helmchen]

FYI: A "few releases ago" Canon enabled in DPP (their free raw converter) embedded XMP. Also the camera now write embedded XMP (that was the reason I had to set "Favor XMP sidecar file" in CR2 settings.

[cytochrome]

...
If you shoot RAW (not DNG), then IMHO it only makes sense if your RAW converter uses embedded XMP, and that's not common.

I don't see your point Ferdinand.

I use mostly ASP2, DxO9 when I need Prime, and sometimes VNX2. VNX2 embeds XMP and can read them, ASP and DxO have their own sidecars that nobody else can use (same as the private part of Adobe XMP-sidecars).

I embed XMP because I can use the raw container as a container for all I care in terms of metadata. It is self-sufficient and can reconstitute a minimal DB anytime, anyplace.
VNX2 and Dx0 transcribe IMatch's hierarchical keywords to the JPG, ASP dosn't but since my RAWs are all versioned, once IMatch is open all JPG get the same set of metadata.

I controlled extensively that my RAWs with embedded XMP are still legible my the converters I use (they may not be read by the Adobe programs, I did not test).

Francis

[sinus]

...
If you shoot RAW (not DNG), then IMHO it only makes sense if your RAW converter uses embedded XMP, and that's not common.

I don't see your point Ferdinand.

I use mostly ASP2, DxO9 when I need Prime, and sometimes VNX2. VNX2 embeds XMP and can read them, ASP and DxO have their own sidecars that nobody else can use (same as the private part of Adobe XMP-sidecars).

I embed XMP because I can use the raw container as a container for all I care in terms of metadata. It is self-sufficient and can reconstitute a minimal DB anytime, anyplace.
VNX2 and Dx0 transcribe IMatch's hierarchical keywords to the JPG, ASP dosn't but since my RAWs are all versioned, once IMatch is open all JPG get the same set of metadata.

I controlled extensively that my RAWs with embedded XMP are still legible my the converters I use (they may not be read by the Adobe programs, I did not test).

Francis

So I have lerned from here and from other forums, I hope, my points are correct:

1) IMatch can embedd xmp into RAWs (NEFs in my case)
2) IMatch can create/read/write xmp-sidecars
3) IMatch can hold xmp in its database

4) Lightroom can NOT embedd xmp into RAWs (NEFs in my case)
5) LR can create/read/write xmp-sidecars
6) LR can hold xmp in its database

So, I can choose, what I want:
Work with embedded xmp into RAWs
a) no Problem in IM
b) I have not that load of xmp-files, no xmp-files will be created
c) LR can read this embedded xmp (?), but writes its xmp (like cropping) into its database
d) IM cannot read the xmp from LR, from example cropping
e) I must have a good look, that the db from LR is intact (including backup), because a lost would be lost all xmp in the DB of LR
f) If I change xmp, in IM or LR, all writings will go slower, because the whole RAW must be written (?)
g) Backup RAWs will go slow, because always the whole RAW (because changed) must be backuped
h) If I send a RAW to a friend/client/artist, every change means, sending the whole RAW again
i) Here I change my RAW, what CAN be dangerous
...

Work with xmp-sidecars
a) no Problem in IM
b) I have a load of xmp-files, for each RAW one, does a bit "clustering" my folders
c) LR can read this sidecars, and write
d) IM can read the xmp from LR and display cropping
e) If the DB of LR will be lost, nothing happens, because I have the RAW and its xmp, where all is written
f) If I change xmp, in IM or LR, all writings will go very quick, because only a small file must be written
g) Changing xmp of a RAW, means, not change the RAW, only the xmp-file, hence for a Backup only the xmp-file must be written, and this is very quickly
h) If I send a RAW to a friend/client/artist, every change means, I can send only the smallo xmp-file over the line
i) The RAW will not be touched, it is in a way safer than if I change the RAW
...

If I think about all, I think, normally the xmp-sidecar-line has more advantages, specially if we work with Adobe - Products (like I do). And if we do so, we can expect no problems.

But, Francis, I mean, if your approach works for you, then I would not change it, then you have everything under your control and this is fine.

[Erik]

...
If you shoot RAW (not DNG), then IMHO it only makes sense if your RAW converter uses embedded XMP, and that's not common.

I don't see your point Ferdinand.

I use mostly ASP2, DxO9 when I need Prime, and sometimes VNX2. VNX2 embeds XMP and can read them, ASP and DxO have their own sidecars that nobody else can use (same as the private part of Adobe XMP-sidecars).

I embed XMP because I can use the raw container as a container for all I care in terms of metadata. It is self-sufficient and can reconstitute a minimal DB anytime, anyplace.
VNX2 and Dx0 transcribe IMatch's hierarchical keywords to the JPG, ASP dosn't but since my RAWs are all versioned, once IMatch is open all JPG get the same set of metadata.

I controlled extensively that my RAWs with embedded XMP are still legible my the converters I use (they may not be read by the Adobe programs, I did not test).

Francis

I think it's really that there are so many RAW developers out there, that certainly what works depends on exactly what the software does.  A lot of software seems to read the XMP embedded in RAW files, but LR for instance will not write XMP to those RAW files (at least from what I recall).  Of course, as you point out, many RAW processors don't write XMP at all using proprietary files to store develop settings.  I think it really comes down to how you use XMP data outside of IM.  For instance, if you don't ever want to save XMP using LR, then it doesn't matter too much where the XMP data is stored.  If all you need is for the XMP data to potentially be transferred to a derived Tiff, Jpeg, etc, then you still may not care that much.

The important thing is to be aware of what might not work and what does work, and ultimately to use what works best for you. 

[Jingo]

Just out of curiosity: am I really the only one to use IMatch with a sidecar-less workflow?

Francis

Hi Francis - I don't use sidecar files either and never have with the various DAM's I've worked with.. though I do believe they have some big advantages over embedding within the file:
  a) only updating the XMP sidecar file prevents possible corruption within the image file itself
  b) using sidecar files means you only need to backup tiny text like files which are much faster than backing up larger image files

Personally, I've only used sidecars for my RAW files and all JPG/DNG/TIFF files have embedded metadata... but there really is no right or wrong about it!

Enjoy!!

[cytochrome]

.......
But, Francis, I mean, if your approach works for you, then I would not change it, then you have everything under your control and this is fine.

Markus,

My initial remark was just to say that xmp could be embedded in the NEF (and some other RAW files). It was not to convince you to change your workflow, to the contrary, I think it is a bad idea for those using LR or Photoshop to convert. Embedding has also a clear disadvantage in terms of speed, it takes a lot of raw opening and closing, I had to buy a new computer my old Dell was out of breath all the time...

But it is interesting to discuss different alternatives and how they can be handled with IM.

Francis

[cytochrome]

Jingo,

I think JPG/DNG/Tiff don't need sidecars anyway, it is well admitted that it is safe and practical to store xmp in them. For JPG it is even clearly asking for trouble to have external XMP since most converters also write metadata per default into the JPG.

I agree with xmp being much faster to handle and save. Storing XMP in RAW has a price, in terms of speed and also it limits the type of converter you can use. With LR/Photoshop I would not do it.

Francis

[sinus]

But it is interesting to discuss different alternatives and how they can be handled with IM.

Francis

Exactly right.

I tried just now a bit with xmp-sidecars in LR and IM.
Hmmmm, I think, as people like you and specially Ferdinand pointed out, we should have then a workflow, what works fine and then not change this.

As in my intial posting, there is of course a third way (instead sidecars or embedding), like I wrote:

I could simply hold the xmp-data in the db of IMatch.
If I use LR only for developing, I could hold these datas in the db of LR.

All editing data would be in LR. I must simply be careful, that this db will be backuped (but this I valid for IMatch also).
So LR must know nothing about Labels, categories, keywords and so on.
And IMatch knows nothing about editing includig cropping.

But usually, I think, if we editing a RAW, then we create a jpg and this jpg could be handled in IM as proxy, so we had all advantages.

For this scenario, I do not know, if IMatch can hold such a huge amount of data in the db, has this an impact on the speed and so on. If not really, then this would be also an alternative, quite interesting.

In fact, I did this partially in IM3.

What do you think about this approach?

[cytochrome]

Markus,

The part in your "third way"  I would be cautious of is "I could simply hold the xmp-data in the db of IMatch. "  I know nothing about the internals of IM but if you want to see your metadata (like for example Location) you will have to set up a metadata panel with tags to show up. At times IM reads back metadata from the file (image or xmp). If the tags are empty you'll erase your data.

Or? not sure, maybe Mario can comment on such a schema.

I think that with IM + LR the best is to go along Ferdinand workflow. It makes sense and also it seems IM is built along these lines.

Francis

[Ferdinand]

I don't see your point Ferdinand.

A lot has been posted in this thread today and I don't have the time now to go through all of it.

I guess I was thinking of LR mainly.  If you use embedded metadata in IMatch, then I think LR will put a copy in the XMP sidecar along with its edit parameters.  So you will have two copies.  If you then edit the metadata in Imatch these two copies will be out of sync.  We have seen the problems this causes.

If you use ASP and its own sidecars then I guess you are right - you won't have a problem.  Unless you tell ASP to write out standard XMP files, in which case you end up with the same problem as with LR.  I think PN willl be the same as LR as it uses standard XMP.

Programs like DxO and ASP should be ok as long as they don't write out standard XMP files.

[sinus]

Hey,
thanks a lot, all of you.

I have read yesterday some forums about xmp, and I have read your comments here and elsewhere in this IM forum.
Once I must come to a conclusion. 

I decided, to use the system with xmp-sidecars (between IM and LR). Sidecars are new for me, but I think, if I use LR and IM properly, I should not have big problems.
At the moment I use LR 4.4, I must look, what are the correct preferences in LR (import Metadata and so on), and I must read, if it is wise to update LR.

So, I am on the way to convert my db from 3.6 into IM5 (still 7 hours to go) and I will see, how all works.

[cytochrome]

I don't see your point Ferdinand.

...
I guess I was thinking of LR mainly. ...

Ferdinand, I think we all agree that LR users are far better of with XMP sidecars than embedding their XMP into the RAW. Since I don't use LR or other Adobe software I have a choice. ASP and DxO integrate well with a "in-file metadata" workflow. Of course as you note one should refrain from writing XMP sidecars from these programs (DxO probably can't right away). but why would one?

Francis

[Ferdinand]

but why would one?

To exchange ratings and labels

[jch2103]

Ferdinand, I think we all agree that LR users are far better of with XMP sidecars than embedding their XMP into the RAW. Since I don't use LR or other Adobe software I have a choice. ASP and DxO integrate well with a "in-file metadata" workflow. Of course as you note one should refrain from writing XMP sidecars from these programs (DxO probably can't right away). but why would one?

I can't speak about ASP, but DxO also integrates with IMatch and an XMP sidecar workflow. (As you know, DxO stores its 'instructions' in sidecar .dop text files, not in the XMP sidecar files, which DxO doesn't touch.) IMatch makes all this transparent with its handling of buddy files.

[cytochrome]

but why would one?

To exchange ratings and labels

OK, makes sense, I didn't think about that because I don't use them. If I would it would be at the time of ingestion. My ingester (PhotoMechanic) would write them into the NEF and IMatch would read them. No sidecar involved.

Francis