Workflow: store Metadata in the database only?

[sinus]

Hi,

How handling Metadata, we have a very good helpfile, even with some Example Configurations (like Adobe, Nikon and so on).

I think, quite a lot of people does use the Adobe-scenario, like Mario wrote in the help-file.
I guess, also a lot of people does shoot photos in RAW (nef in my case) and use Lightroom (in my case version 4.4) ONLY for RAW-processing.

And in this scenario I use IMatch 5 as the ONLY place for writing Metadata except some basics in the Camera.

Hence I use only 3 softwares for my photos:
Lightroom as a plain RAW-processor
Photoshop as a kind of "fine-tuning" some pictures
IMatch 5 for a real great DAM and the only place for edition Metadata

Well, this brings me to the problem (like Ferdinand in another thread pointed great out https://www.photools.com/community/index.php?topic=276.msg3399#msg3399 ), where should I store my Metadata?

Metadata in my case are:
XMP
IPTC (what are generated from the xmp-entries)
EXIF
Ratings and Labels
Development-settings of RAWS (Cropping, Colour, Contrast and so on)
GEO-stuff (what I do not use yet, but will change)

Well, for me one thing is easy:
all standard formats like jpg, tiff, psd, DNG and so on, I let the Metadata embedd in the file itself

The problem is the RAW.
Here I have a question:

Why not simply do ONLY use the database of Lightroom and from IMatch to store these stuff?
WITHOUT embedd in the RAW (what seems to be normally not possible) and WITHOUT store these settings into a sidecare.

This would mean:
Lightroom:
I use only the database of LR to store all "Develop-settings". I do NOT use sidecars. I can NOT see any Metadata, what I wrote in IMatch, but since I usw LR only as a RAW-converter it is not necessary.

IMatch:
I use for ALL of my Metadata only the intern database of IM5, I do not use sidecars.

This means: I had only RAWS, what would be untouched.

But in LR I could use full advantage of the develop-settings, each time, when I edit a RAW, all the settings are there, because they are stored inside of LR.

In IMatch I could NOT see some cropping. But I would have all the Metadata-stuff there.

Usually I make (like many other people) from a RAW a version in a standard -format. Hence I would end in IM5, thanks to the version-possibilities, with a RAW and with a jpg, neatly nearby.
For me personally, all problems would be solved, no DNG-stuff (what has negative and positive parts), no fiddling with xmp-sidecars, nothing.

This leads finally to some questions and point to think about:

1) I know, that LR does hold the Metadata (in this scenario for me only the develop-settings) in the intern database. The question is, what about, if I have 300'000 images in there? Is LR good enough to handle this or must I create a new catalog each year (what would be ok).

2) The same for IMatch, only that I guess, the Metadata of, say, 300'000 images, is very huge. What about the speed and the db-size of IM5?

3) Since the Metadata would be only in IMatch inside, I would have to make regular backups of the db (the same of course for LR)

4) If I gives images away, I must and would of course create jpg or tiffs and EMBEDD the xmp-stuff from the IM5-db into these files, but I am sure, this can be made on some way (native or by script).

Because my head is in a bad manner and some butterflies flies around in my brain, because this metadata-stuff is so complicated, that I am not sure, if I made some very wrong thoughts. 

Would be great, to hear some of  your ideas.
Thanks a lot!

Markus

[cytochrome]

Hello Sinus,

These are quasi existential questions we have all to solve

I use NEF and my solution is very simple but you cannot use it because of Adobe and the believing in the virginity of the holy raw...

Why not consider the JPG ("with a RAW and with a jpg, neatly nearby") as a sidecar and repository for your metadata? You can even have a special very small JPG, with minimal image but full metadata.

When LR converts the NEF to JPG it writes certainly the full XMP/IPTC to the JPG. This includes ratings, for categories you would have to use keywords.

And in IMatch whenever you modify the metadata for the NEF, the file relation will transcribe it to the JPG...

But I think most people using Adobe will end up with XMP sidecars, it is after all the purpose of the thing.

Francis

[sinus]

Hello Sinus,

These are quasi existential questions we have all to solve

True, true!

I use NEF and my solution is very simple but you cannot use it because of Adobe and the believing in the virginity of the holy raw...

Why not consider the JPG ("with a RAW and with a jpg, neatly nearby") as a sidecar and repository for your metadata? You can even have a special very small JPG, with minimal image but full metadata.

Yep, this is also a good idea. But it depends on my question 2, IF IMatch is able to hold a lot of images in the internal db, then I could fill the raw also with metadata (internal of IM). I think, if IM5  can not handle such a amount of data intern, then I will go the sidecar-route.
If I shoot 100 RAWs, I do never develop for each RAW a jpg, but want describe these RAWs also with Metadata. And simply to create a jpg for a "placeholder" for these RAWs, well, I have to think about it.

When LR converts the NEF to JPG it writes certainly the full XMP/IPTC to the JPG. This includes ratings, for categories you would have to use keywords.

And in IMatch whenever you modify the metadata for the NEF, the file relation will transcribe it to the JPG...

Hm, that is true. But if LR does convert the NEF to a jpg, than (almost) all metadata comes from IMatch, and this will be ok, I think. If LR does embed these metadata (and LR does) into the image, then it is ok.

Your second sentece, about the versioning rules, I am not sure yet, but maybe I could say IM5, to avoid to do this. But on the other hand, because it is IMatch, I would allow it and it would be ok 

But I think most people using Adobe will end up with XMP sidecars, it is after all the purpose of the thing.

Francis

Yep, I think too, this is true! Most people goes tho the xmp or DNG - route. But before I decide finally, I want to check all possibilites.

Francis, thanks a lot for your thoughts and answers!

[Ferdinand]

My workflow at the moment is similar to Markus', in that I shoot NEF and I process in LR5 (since I lost confidence in AfterShot Pro).  I have thought long and hard about this.  I don't have all the answers and I can't answer all Markus' questions.  I don't have enough experience with LR to comment on its performance and whether multiple LR catalogs are desirable.  LR is slower than Bibble5 and ASP.

My current workflow is as follows:

   
• I enter all metadata, other than rating and label, before LR sees the image.  I enter nothing in LR, except perhaps rating and label.
• I have Imatch 3.6 configured to use sidecars for XMP, so XMP should be in the sidecar and IPTC in the NEF and both XMP and IPTC will be embedded in JPG, TIFF.  The same in IMatch 5.
• I have IMatch 3.6 in synchronous mode so changes are written immediately, and background processing off in IMatch 5.  You don't have to have these settings if you want faster performance, but you must then remember to flush pending changes before you ingest into LR.
• I ingest into LR and all the metadata is there, since LR will read it from the XMP.
• I have LR configured "Edit | Catalog Settings | Automatically Write Changes Into XMP" to be on.  I think that this is important, so that you don't lose anything.
• I do often rate them and set labels in LR, which is why it is important to always write the settings to XMP.  If you don't you may find that both LR and IMatch have conflicting edits to the XMP queued.  Also, if you want to refresh the metadata in LR you will need to reread the XMP sidecar, and so your LR edits must be stored in the sidecar or you will lose them.
• I export converted images into subfolders, and never import versions into LR.
• LR will ensure that the standard metadata is embedded in the JPG or TIFF.
• I then ingest the versions into Imatch, and if I have done any ratings or labels in LR I refresh XMP for the NEFs.  In Imatch 5 you probably won't need to refresh, unless you've turned off automatic updating.
• I then run my synchronisation script, although in IMatch 5 I just need to trigger propagation.  You may or may not need this - it depends on other aspects of your workflow
• I manage all metadata in IMatch.

I still haven't done a full production run in IMatch5 - there are a couple of longstanding bugs that need to be fixed, so this is preliminary, but I think it will work. 

The other possible addition is to make the NEFs read-only, which will prevent IPTC from being embedded in the NEF.  This is what I do for ARW, as most programs don't support embedded IPTC in ARW.  The advantage that this has is that it is much quicker to write to the sidecars only than to also write to the RAW.  But I have so much embedded IPTC in NEFs that I'm not sure I see the point of changing now.

Hope this helps.

[cytochrome]

....
Hm, that is true. But if LR does convert the NEF to a jpg, than (almost) all metadata comes from IMatch, and this will be ok, I think. If LR does embed these metadata (and LR does) into the image, then it is ok.
......

Sinus/Markus,

I don't think LR will write anything to the NEF (apart maybe GPS data?, I don't use LR). Adobe can read from the NEF (I suppose it has to be set in the LR settings) but will only write to XMP sidecars and the output files like JPG or TIFF.

Ferdinand,

Concerning the RW2, the viewers I use, Irfan, FPV, xnviewMP.. all can read the metadata written in the file. And of course ASP, Lightzone too. So i am surprised at the ARW, they are certainly more common than RW2.

Francis

[sinus]

The other possible addition is to make the NEFs read-only, which will prevent IPTC from being embedded in the NEF. 
Hope this helps.

Ferdinand, thanks a lot, at the moment (I have no time) only some question, what is really unclear to me, maybe you know it for sure (or someone else):

When the RAW is NOT write-protected, does IM5 writes some IPTC INSIDE the raw? I thougt, wether LR nor IM5 does write something in the RAW.

In IM3.6 Mario explained something about "experimental" mode for embed XMP-data in the RAW-file! What about that? Is this gone in IM5?

[Ferdinand]

When the RAW is NOT write-protected, does IM5 writes some IPTC INSIDE the raw? I thougt, wether LR nor IM5 does write something in the RAW.

In IM3.6 Mario explained something about "experimental" mode for embed XMP-data in the RAW-file! What about that? Is this gone in IM5?

This depends on your settings.  In the Metadata2 panel in preferences there is an XMP export block and the first item is XMP write mode.  The default is "Use existing XMP sidecar file else use Mode 1", and Mode 1 is "Write to all supported formats". 

What this means is the default is to write XMP to the sidecar file if it already exists, otherwise it will write it to the NEF, since ExifTool considers NEF a supported format.

If you don't like this you can change it.  What I use is the fifth option "Always write to XMP sidecar files".  (Note that this option does not apply to standard formats like JPG & TIFF.  The next option in that list controls whether XMP is embedded in standard formats, and the default is yes.)

In the default install, any XMP that you write will also be written to the corresponding IPTC field, if one exists.  IPTC is always embedded in the file, so in the default install you will get IPTC written to the NEF.

This is harder to turn off, if you don't like it.  The easiest way is to make your NEFs read-only, then nothing at all gets written to the NEF.  (The other way to stop IPTC being written to the NEF is to turn off MWG compatibility, but this isn't a good idea, and is more complex.)

[sinus]

Hi Ferdinand

Thanks really a lot for your answers. This stuff drives me crazy 
I think, I have to do some tests, when I have enought time (after all we must earn some money  )

a) I will test, does and what writes LR and IM5 relly into a nef.
b) I will test, if I use sidecars, are these xml-information still inside IM5 (in the database) or not, because if yes, then I will do anyway a test
c) about the speed of IM5, if I have a lot of infomation in a lot of images

Maybe I will do some more tests, simply I cannot find some really good and clear answers on this (or I do not find them).
You wrote, Ferdinand, you use sidecars. How long do you this and would you say, this is the best for you to do or could you think about another good choice?
The DNG-stuff, honestly, I am not trust really and somehow, I find it too clumpsy.

Finally I guess, I will go the same route like you.

[Ferdinand]

You will find that LR does not write to the NEF.  Almost no RAW converter does, except Capture and Capture NX.

So LR will write to and read from the XMP sidecar. 

So if you want to be sure that LR can read what IMatch writes there and IMatch reads what LR writes there, you need to be sure that IMatch uses sidecars for NEF / RAW.

So I recommend the option "Always write to XMP sidecar files".

But after LR writes to the sidecar, IMatch will need to reread it.

And when IMatch writes to the XMP, LR will need to reread it.  ***But, this is dangerous if don't have LR configured so that "Edit | Catalog Settings | Automatically Write Changes Into XMP" is on.  Otherwise you risk losing your LR edits.  Read my first post for more detail***

[sinus]

This is (for me) a very good, first test with Build 110:

If I choose in the Metadata-preferences MWG-compatibility (default) and change simply to MODE1, then IMatch writes a lot of infos into the RAW (nef) without sidecars!

I tried it with

change dots, pins
change labels, ratings
change (edit) Caption writer
change city, country
change even GPS

ALL this IM5 writes into the NEF.
Because if I open such a Nef in Photoshop, all information, written in IM5, are there! Without xmp-sidecar! GREAT!

Then, if I instruct Photoshop (Raw) to write sidecars, then PS will do so and IM5 will read them (because I see this in the displaying from the cropping). GREAT!

And if I alter now, when the xmp-sidecar from PS is here, again some stuff, then IM5 will also NOT write into the xmp, but in the RAW (nef).

Fazit:
This means, IM5 can writes only into the RAW (NEF), but read the xmp-sidecar.
Photoshop (I am sure, LR also) can read the information, written by IMatch, and writes a sidecar (filled only with develeppmente-stuff).

Means, the XMP-sidecar will be written only by Photoshop, the information will be written by IM5 only into the RAW!

And it seems, that they both does not mix some stuff.
If my further test will be so correct (for me), I will got this route.

PS:
This is also a great advantages, because if I shoot 500 images in RAW, usually I will only edit in Photoshop, say 100.  Hence I must have only for 100 a xmp-sidecar, the others can be stay as they are (simply a nef). Hence I know, all nefs with a sidecar has been edited in PS, the other are usually not that good.

[Ferdinand]

It will be interesting to see what Mario says, but my opinion is that this is a risky approach.  What this means is that you have two copies of XMP - one in the NEF put there by IMatch 5 and one in the sidecar put there by Adobe s/w.  How will you keep them synchronised?  IMatch V5 won't and nor will PS/LR.  Well, not easily, and perhaps not at all.

It might work if you *never* enter metadata in Adobe s/w.  But I'm not sure how you would get PS/LR to see any changes that you might make in IMatch without losing your edit settings.

[Mario]

General rule: If you work with Adobe software, do as Adobe does.
The default settings in IMatch 5 (always write to XMP sidecar file, embed XMP in standard formats) is exactly what you need to a seamless integration with Adobe software.

The only reason for embedding XMP in NEF files I see is when you have to use Nikon Capture. I dropped Capture a while ago and used ExifTool to extract the XMP data from the NEF into sidecar files.

[sinus]

It will be interesting to see what Mario says, but my opinion is that this is a risky approach.  What this means is that you have two copies of XMP - one in the NEF put there by IMatch 5 and one in the sidecar put there by Adobe s/w.  How will you keep them synchronised?  IMatch V5 won't and nor will PS/LR.  Well, not easily, and perhaps not at all.

It might work if you *never* enter metadata in Adobe s/w.  But I'm not sure how you would get PS/LR to see any changes that you might make in IMatch without losing your edit settings.

Ferdinand,
I want NOT synchronise them. 

I do ALL xmp-stuff in IM5 (including rating), nothing in PS/LR except the developpement-stuff, what PS/LR writes into the sidecar.
IM5 does read at least the cropping-infos from the sidecar and display the image correct cropped.

I will do further tests, but it seems to work very good.

[sinus]

General rule: If you work with Adobe software, do as Adobe does.
The default settings in IMatch 5 (always write to XMP sidecar file, embed XMP in standard formats) is exactly what you need to a seamless integration with Adobe software.

The only reason for embedding XMP in NEF files I see is when you have to use Nikon Capture. I dropped Capture a while ago and used ExifTool to extract the XMP data from the NEF into sidecar files.

Thanks Mario,
I work ONLY with IM5, where I do ALL xmp-stuff, ratings, labeling, iptc and so on. And my tests (so far) shows, that IM5 does neatly embedd these infos into the nef.

In PS/LR I do ONLY the developpement, for these I let PS/LR write a xmp-sidecar, where all the settings are stored.
Hence PS/LR does NOT (it seems so) embedd any Infos into the NEF, because I choose in the preferences to use a sidecar.

IM5 reads at least the cropping - infos from the sidecar and displays the images-croppings correct! GREAT!

Why I want not, that IM5 writes simply the stuff into a xmp-sidecar, as you suggests?
Well, simply, because I do not like sidecars.

I think, a lot of photographers does work like me: I take, say, 500 images. 50 I delete, 150 I develop in PS. The rest of 300 images I do enter some xmp-stuff like City, Caption, Headline and so on.
BUT I do NOT develop them, hence they stay usually simply in my db. Hence they must not have a sidecar.

If you count this up, say with 100x, then I have 50'000 nefs, and for 30'000 I would simply not use sidecars. Why then write these?

If I have this workflow (DAM in IM5 including xmp-stuff, LR/PS ONLY for edition images), then it would be best, from my point of view, if IMatch 5 would do this:

1) read the xmp-sidecar from PS/LR and use the settings, what are of use for IM5, like cropping
2) writes the xmp-stuff, entered in IM5, ONLY into the intern DB of IM5, not in the sidecar, not in embedd them into the RAW

But maybe this could be a speed-problem, because the intern DB of IM5 would grow too large???

[Mario]

2) writes the xmp-stuff, entered in IM5, ONLY into the intern DB of IM5, not in the sidecar, not in embedd them into the RAW

If you don't enable automatic write-back, IMatch writes back data to the NEF, XMP sidecar or other files only when you manually write-back files.
This does not require additional database space.

[Richard]

But maybe this could be a speed-problem, because the intern DB of IM5 would grow too large???

Hi Markus,

Whether you write data back to files or not is not going to change the amount of data stored with your database. You are not moving the data, you would be copying it to image files. But if you do write data back to files, I am sure that will make them grow.

[Ferdinand]

I want NOT synchronise them. 
I do ALL xmp-stuff in IM5 (including rating), nothing in PS/LR except the developpement-stuff, what PS/LR writes into the sidecar.
IM5 does read at least the cropping-infos from the sidecar and display the image correct cropped.

If you are disciplined and only operate as you suggest, then it might work. 

But if you change metadata in IMatch after you have edited the image in PS/LR, then I think it will be difficult to get PS/LR to reread this information without losing your edits. 

The ability to share ratings and labels between the two programs is useful, but you won't be able to share any changes you make to rating / label once you've edited the images.

[sinus]

But maybe this could be a speed-problem, because the intern DB of IM5 would grow too large???

Hi Markus,

Whether you write data back to files or not is not going to change the amount of data stored with your database. You are not moving the data, you would be copying it to image files. But if you do write data back to files, I am sure that will make them grow.

Thanks, Mario and Richard

Sorry to be so "stubburn" but I want it know only exactly (besides I will do some more tests).

So, say, I have a NEF and a JPG.
IM5 reads all Metadata and holds it in its database.
If I change or add some Metadata, IMatch holds it in the db.

If I do a write-back (manual or automatic), IMatch writes thes data to the file (embedding or as sidecar, depending on the settings). But this writing is in fact a copy, all Metadata is also in the db. This means in such cases, Metadata is in the file AND in the db.

Hence, it does not matter, where I store Metadata, the db of IMatch will be the same!

Is this correct?

[sinus]

I want NOT synchronise them. 
I do ALL xmp-stuff in IM5 (including rating), nothing in PS/LR except the developpement-stuff, what PS/LR writes into the sidecar.
IM5 does read at least the cropping-infos from the sidecar and display the image correct cropped.

If you are disciplined and only operate as you suggest, then it might work. 

But if you change metadata in IMatch after you have edited the image in PS/LR, then I think it will be difficult to get PS/LR to reread this information without losing your edits. 

The ability to share ratings and labels between the two programs is useful, but you won't be able to share any changes you make to rating / label once you've edited the images.

Ferdinand,thanks, your are right, it could be dangerous.
I will do some tests, very carefully, and see, what I will do.
If it works really without problems, I will do it.

If not, then I will do like you, work with sidecars.

[Richard]

Hence, it does not matter, where I store Metadata, the db of IMatch will be the same!

That is how I understand it.

[cytochrome]

Ah Sinus, I hope I have not contaminated you...

I do as you would like, I write all the metadata in the NEF (or the RW2 for my Panasonic files) and JPGs (via raw decoder or IM propagation) and all the real cataloging in IM, only categories for now, attributes and so on will come when Marion has settled for a definitive database structure.

I feel this is safe, what is in the NEF is out of IM, so any change in IM usage or philosophy will not affect it at all. And what is in the IM DB, I hope at least with categories, to be able to transfer it also into the image file as keywords, if it becomes needed. All the programs I use can retrieve this information.

This said, your situation is very different because you use Adobe products. I would be cautious. Mario told me in a recent post that he simply ditched all Nikon programs and lives happy with Adobe and sidecars. And I fear that IMatch is going to more and more reflect this choice: being optimized for sidecars instead of in image file metadata.

Some years I made an opposite choice ago: I ditched LR, am happy with ASP and often use ViewNx2 which over the years has become excellent albeit slow. And I rarely use PS for JPG editing. My JPGs are not in the same folder than the NEF/RW2, so no interference.
As Ferdinand says, you will have to test and be disciplined because Nikon + Adobe - sidecars = trouble  if ever you let LR write to xmp.

Francis

[Ferdinand]

If not, then I will do like you, work with sidecars.

You still need to be disciplined if you decide to use sidecars.  You need to be sure that both IMatch and LR write their settings to sidecars and that only one program at a time does so.  My other post has details.

[sinus]

Thanks to all for your answers.
Mario pointed out several times in the past, that metata is complicated. And indeed, it is.

I have to do some tests, reading your posts, think about it and make a decision. I think, it is quite important, because is is not very good, if we change our "metadata-workflow" often.

Thanks and have a good time.

[cytochrome]

Sinus, you are swimming in troubled waters.

From what I have seen (I tested a lot of writing into the NEF): the only safe way is to set Metadata2 as "Ignore sidecar". With this setting, whatever LR/PS write into the sidecar will be ignored: will not go into IM DB and not into the NEF. But you will not see the Adobe edits like cropping.

If you have old NEF with metadata only in sidecars you can temporarily check "Favor sidecars", import the metadata and immediately check again Ignore sidecars. The "Merge sidecar and embedded.." is treacherous: if something has been modified in LR/PS it will overwrite your metadata in the NEF. In case of doubt, ExifTollGUI will show what is in the NEF and what in the xmp.

The whole is complicated by the fact that metadata can be imported in the DB but not written to the files (NEF or sidecars). This is what the "Protect unwritten metadata" is for. It protects but if you forget it and make new changes before write-back they will be lost... It took me quite some time to understand why I was loosing edits to metadata. The Help tells it clearly, but since it is not so obvious one can overlook it easily.

If, as you plan, you use LR only for conversion it can work but if you do edits in IM while working in LR, the xmp sidecar will contain your old metadata and erase any new work done in the meanwhile in IM.

Good luck

Francis