How to force metadata back-write to raw??

[cytochrome]

Here my problem (again..) :

- I like basic metadata in the raw, the rest in IMatch
- with NEF this shines, simple and no trouble since years
- with Panasonic RW2 it was impossible since last year so metadata are in xmp (some are lost ) or in jpg
- since last year PhotoMechanic 5 allows writting to the RW2, and now IM5 also
- so all my recent Pana photos have metadata in the RW2, simple and no trouble since.

- For my old Pana photos, I would like to get the metadata either from xmp or jpg into the corresponding raw. Just for consistency, and also because I ma stubborn...

- I setup a reverse versioning scheme (jpg -->  RW2) it works, but not always (don't understand why) and it is terribly slow. And it confuses the Relations array *

I tried the following (when I still have the xmp):

- set Metadata2 as MWG ON, xmp write mode = 1, Protect unwritten metadata NO (I want to enforce the copy) and xmp sidecar FAVORED
- CTL ALT s  : everything is nicely copied and shows in the RW2 metadata panels
- set Metadata2 as xmp sidecar IGNORED
- if I simply wait a bit nothing happens (I expected an automatic write back since in Background processing I have ticked the immediate write back (and also the file relation refresh)
- if I do CTL ALT s my metadata is simply lost from the metadata panel

So question 1 : where is the metadata I can read in the panel after importing from the xmp? the sidecar should be ignored and it is not (alas) in the RW2.

The only way to get my metadata block into the RW2 is to copy it via the icon in metadata panel from the jpg and paste it on the RW2!! Then it works fine, the whole block appears (ExifToolGui) in the IPTC and XMP of the raw.

This is cumbersome, so question 2: how to get simply the metadata once it shows in the RW2 meta panel into the RW2. There must be a setting.... or not?

*A propos the Relation rules: when one sets up an inverse relation rule nothing is simple (of course I untick the direct rule): sometimes all files (RW2 and JPG) are seen as masters AND derivates (blue and orange bar together), and sometimes, it disappears, NO master and NO target despite numerous "Refresh". Is there a way to completely reset the system? I made a database verification and compacting, seems to have reset, but this takes time.

Francis

[RalfC]

The only way to get my metadata block into the RW2 is to copy it via the icon in metadata panel from the jpg and paste it on the RW2!! Then it works fine, the whole block appears (ExifToolGui) in the IPTC and XMP of the raw.

This is cumbersome, so question 2: how to get simply the metadata once it shows in the RW2 meta panel into the RW2. There must be a setting.... or not?

There might be another possibility without IMatch. I used a batch file when I had a similar problem: Calling exiftool for every jpg-image in a directory and copying the tags to the right place:

Code Select Expand


echo off

for %%X in (*.jpg) do (
  For %%Y in (nef tif jpg psd dng) do (

    rem Uncomment next line to keep track of files where the target does not exist
    If not exist ..\%%~nX.%%Y echo ..\%%~nX.%%Y >> additional.txt

    rem Uncomment one of the next lines if you want to see the currently processed file
    rem echo %%X %%Y
    rem If exist "..\%%~nX.%%Y" echo "C:\Program Files\exiftool\exiftool" -overwrite_original -addtagsfromfile "%%X" "..\%%~nX.%%Y"
   
    rem Do the processing (path to Exiftool should be adjusted!)
    If exist "..\%%~nX.%%Y" "C:\Program Files\exiftool\exiftool" -overwrite_original -m -addtagsfromfile "%%X" "..\%%~nX.%%Y"
  )
)

Requirement is that the filenames (without extension) are identical.
Oh, and my Jpgs where in a subdirectory from the destination files.

Let me know if you have questions.

Regards,
Ralf

[Mario]

This is indeed a very specific problem and outside the scope of IMatch. Metadata clean-up on this scale is best done with ExifTool, and before pulling the files into the DAM of your choice.

*A propos the Relation rules: when one sets up an inverse relation rule nothing is simple (of course I untick the direct rule): sometimes all files (RW2 and JPG) are seen as masters AND derivates (blue and orange bar together), and sometimes, it disappears, NO master and NO target despite numerous "Refresh". Is there a way to completely reset the system? I made a database verification and compacting, seems to have reset, but this takes time.

(Better: Open a new topic for questions like this. Mixing too many information in one post makes it hard to sort everything out)

Relations can be simple. And relations can be complex. And if you work with not really supported concepts as reverse versioning, you can shot yourself in your own foot pretty easily. You can reset your entire relation setup by just disabling all rules and then to a Relation Refresh for your entire database: Select the database node and do a <F4>,<R>.

[cytochrome]

..
There might be another possibility without IMatch. I used a batch file when I had a similar problem:
..
Requirement is that the filenames (without extension) are identical.
Oh, and my Jpgs where in a subdirectory from the destination files.

Let me know if you have questions.

Regards,
Ralf

Hello Ralf,

Looks good. My source is also in a sud-folder to the target folder, bu my case is a bit more complicated since the source file is in the form P10001234_xxx.jpg or P10001234-xxx.jpg and the target is P10001234.rw2. I remember having seen Regex talk on the Exiftool site, so I suppose it can be done, stripping [-_] from the source file name? I am very regex illiterate...

Francis

[cytochrome]

This is indeed a very specific problem and outside the scope of IMatch. Metadata clean-up on this scale is best done with ExifTool, and before pulling the files into the DAM of your choice.
..

Mario, my post was too long and wordy... simply it is not a metadata cleanup, there are no metadata in these rw2.
They are in xmp sidecars, I import them into IMatch from the xmp, they show up in the metadata panel.

Question: can IMatch write this data (as it shows in the metadata panel) into the raw file?

I know it can, because it works, but not all the time.

Francis

Thanks for the relation cleanup tip. I had deselected everything but not the reverse rule..

[Mario]

Question: can IMatch write this data (as it shows in the metadata panel) into the raw file?

If ExifTool support it, yes.
But you don't want to write to the RAW, you want to copy the existing XMP contents to the RAW. That's different.

When IMatch writes changes to files, it only writes diffs.
IMatch keeps track of which data was changed, and then instructs ExifTool to update exactly these changed tags. Finally IMatch runs the ARGS files provided by Phil to propagate metadata changes from XMP to IPTC and EXIF, and vice-versa.

What you want is to copy/merge an existing XMP file completely into the RAW. IMatch can do this when you do an explicit copy (which you did), but not with any of the regular write-back commands. This also depends on the write mode you use etc. under Edit > Preferences > Metadata 2.

This is why I said, best to the merge with ExifTool once, before adding these files into IMatch.

[cytochrome]

Thanks for the clarification, the difference was not evident, since copy/paste worked I thought it was only my setting in metadata that blocked it (I tried several possible combinations, including MWG On/Off).

I don't know how PM does it (apparently not via Exiftool cause it is very fast) but it writes the complete PM metadata panel to the empty rw2...). These files open fine in ASP and in IMatch.

I will find another way, it concerns only pre-2013 images and the Pana is not my main camera.

Francis

[Mario]

I don't know how PM does it (apparently not via Exiftool cause it is very fast) but it writes the complete PM metadata panel to the empty rw2...). These files open fine in ASP and in IMatch.

ExifTool is very fast too. How long does it take to write-back XMP data in IMatch on your machine? I just checked a NEF file and it takes IMatch 0.3 seconds to update rating, label, some keywords, time stamps, IPTC/EXIF propagation included.

but it writes the complete PM metadata panel to the empty rw2...).

Yes, that's not the problem. IMatch/ExifTool can also create a new XMP record in your RAW file. It's just your "migrate existing XMP data in a sidecar file" which causes the extra work. When you set IMatch to write XMP data to RAW files, and the file has no XMP data, IMatch embeds a fresh XMP record in the file. It sets up a basic MWG compliant XMP record, imports EXIF and IPTC data which may be contained in the RAW file into that XMP record as well, and finally takes care that all the IPTC/EXIF digest data, metadata timestamps and whatnot is up-to-date so the fresh XMP record is compatible with the MWG rules.

Did you try that? The resulting XMP record will be compatible with all applications which support embedded XMP data in your specific RAW format (many preference sidecar files of course).

But when I understand your initial post correctly, this is not what you need to do with your old files. They already have XMP data, but in a sidecar file. And you want IMatch to take that XMP data and copy it in the RAW file? That cannot be done with the standard "write-back changes after I've made changes" method. That's what I'm trying to explain here.

I would either add a new feature (Import existing XMP sidecar file into RAW and delete sidecar file if not used by other files) or you run an ExifTool command processor script which copies the data from the XMP sidecar file into the RAW.

For example, I think something like

-tagsfromfile
{File.PathS}{File.NameS}.xmp
{File.FullNameS}

would be a good start. Make sure you set the Run for each file in selection option as well. This command line copies the XMP data from the XMP file into the image file. Other commands can be added, e.g. to import existing EXIF data in the RAW into the XMP record etc.

[cytochrome]

Thank you Mario for all the information. I was not aware of the complexity although now I find it logical to create the XMP record first before populating it with whatever is available.

What I don't understand is this:

It sets up a basic MWG compliant XMP record, imports EXIF and IPTC data which may be contained in the RAW file into that XMP record as well, and finally takes care that all the IPTC/EXIF digest data, metadata timestamps and whatnot is up-to-date so the fresh XMP record is compatible with the MWG rules.

Did you try that? The resulting XMP record will be compatible with all applications which support embedded XMP data in your specific RAW format (many preference sidecar files of course).

What could I try? First setup an empty record? How? and then it would accept to write back my xmp sidecar to this in-raw record?

But when I understand your initial post correctly, this is not what you need to do with your old files.

Yes this is the situation, the raw is empty (apart from Exif and Panasonic specific stuff like lens correction etc).

I will study the exiftool scripts and try to set something up along the lines you suggest. Will take me some time, my programming is rusty..

Francis

[RalfC]

Looks good. My source is also in a sud-folder to the target folder, bu my case is a bit more complicated since the source file is in the form P10001234_xxx.jpg or P10001234-xxx.jpg and the target is P10001234.rw2. I remember having seen Regex talk on the Exiftool site, so I suppose it can be done, stripping [-_] from the source file name? I am very regex illiterate...

Hello Francis,

my skills for regex are not really existing either.

I suppose that the XMP-file has the same name as the Raw, i.e. P10001234.xmp and P10001234.rw2? The you could modify the batch script to use the XMP-File instead of the jpg...

(Line 3 in my batch contains the extention from where to copy and line 4 contains the extensions to which to copy. Filepaths should be adjustable by replacing "..\")

Regards,
Ralf

PS: as usually, test with some copies first...

[Ferdinand]

I will study the exiftool scripts and try to set something up along the lines you suggest. Will take me some time, my programming is rusty..

That cats to keywords script that I wrote has ExifTool code that can create XMP, along with a lot of options that you don't need which will make understanding it tricky.  Also, that script processes files through ExifTool in batches of up to 50 images, to make things more complex.  If I had the time I could write a script to do this based on that script, but life is just a little too busy at present.

[Mario]

What I don't understand is this:

Metadata is really complicated. Sad thing. We cannot change that. And it gets worse when users have tried different approaches over time (which is often the case), and then decide to settle to some approach. This usually requires some manual processing of files, e.g. with the help of ExifTool. If you look at the ExifTool user forum, you'll find plenty of users who had these problems and needed a procedure to standardize their metadata.

I'll try to explain your specific problem in more detail with respect to IMatch 5:

You have a file.RAW and a file.XMP.

When you add this file to your IMatch 5 database, IMatch either pulls the XMP data from the RAW, the sidecar file, or merges both XMP records, depending on the settings you use under Edit > Preferences > Metadata for XMP import.

If you have set IMatch to read XMP data from the XMP file (or merge), all the XMP data (and IPTC/EXIF) will be visible in IMatch, in the metadata panel.

When you update an XMP tag (e.g. set a Rating) and then let IMatch write-back metadata, IMatch either updates the XMP data in the RAW or in the XMP file, depending on the options you have set under Edit > Preferences > Metadata for XMP export.

During write-back, IMatch checks which tags have been updated (in this example, rating). IMatch then tells ExifTool to update this tag, plus some required secondary tags like metadata update timestamps which always need to be written.

Since your RAW file has no XMP record when IMatch updates it, ExifTool write produce a empty valid XMP record, and then write the rating information and the timestamps into this record. This is the important bit: IMatch will not copy the data in the XMP sidecar file into the RAW in this step. Or write all metadata tags contained in the database. IMatch always works with the existing metadata in the file it writes to, and updates it. And IMatch only adds/updates/deletes tags which need to be modified. This ensures that the existing metadata remains as unchanged as possible, which is important especially for maker notes and such.

If you read metadata from the XMP sidecar file, but write XMP to the RAW file, the XMP data in the RAW will not contain all the XMP data contained in the sidecar file. The XMP in the RAW file contains only the minimum XMP data produced by ExifTool when it sets up a fresh record, plus the changes IMatch has written (the rating).

I hope this makes it more clear why the Copy XMP from sidecar file into the RAW file has to be done once, and in an extra step. It cannot be done by the standard IMatch update metadata process.

[cytochrome]

...
I suppose that the XMP-file has the same name as the Raw, i.e. P10001234.xmp and P10001234.rw2? The you could modify the batch script to use the XMP-File instead of the jpg...

(Line 3 in my batch contains the extention from where to copy and line 4 contains the extensions to which to copy. Filepaths should be adjustable by replacing "..\")

Regards,
Ralf

PS: as usually, test with some copies first...

Hello Ralf,

Thanks for the help, but I am stuck...

- in my test folder I have files like P1003011.RW2 and P1003011.XMP

- I modified your script like
for %%X in (*.xmp) do (
  For %%Y in (nef rw2 tif jpg psd dng) do (
.....

copied the bat file in the RW2 folder, gave a path to exiftool (C:\windows\system32\exiftool.exe)

Well the Additional.txt file contains:
..\P1003011.RW2.nef
..\P1003011.RW2.rw2
..\P1003011.RW2.tif
..\P1003011.RW2.jpg
..\P1003011.RW2.psd
..\P1003011.RW2.dng
..\P1003011.nef
..\P1003011.rw2
..\P1003011.tif
..\P1003011.jpg
..\P1003011.psd
..\P1003011.dng
..\P1003012.RW2.nef
..\...........

EDIT: Well there is no need to plague you further, I can do it very simply folder by folder, with a simple command line like:

myDIR exiftool -tagsfromfile %d%f.XMP  -ext RW2  myDIR

Works well and fast... And I can refine it to copy only IPTC etc.. So thank you everybody.

@Mario: thanks alot for the time I cost you, in fact I am quite ashamed. It is because when I first used IM5 I was enthused by the file relations and thought that would be perfect to write back despite different naming format in my raw and jpgs, and do it from within IMatch. I supposed it to be simple, it is not...  I just read again your explanation, and yes it is exactly what I see: using write-back only certain fields were updated, while copy/paste copies the whole set. Ithought I could defeat this by playing with the different settings in Metadata2 panel, but no.

As you note it is a one time thing, since some month I write the XMP directly to the RW2, so no problem, I will do it "by hand". If I would understand the IMatch Exiftool interface I would try it from there, but let it be.

@Ferdinand: thanks also for your interest and help. In fact I was alluding to Exiftool scripts (like the one Ralph setup) and not your IMatch scripts. It would be overkill to try to modify them.

Voila, lets say the subject is closed... et encore merci à tous

Francis

PS I finally got it to work in the ExifTool Control Panel, very handy...