IM5 collecting embedded metadata from raw even though I am using sidecar files

[KimAbel]

Hello

I have many raw files with embedded metadata and this is causing me a lot of headace! In IM5 I have chosen to use sidecar files for raw files (IM5 is set to default in metadata2 settings) and this has always been my preferred setting, but by mistake I have at one time set IM5 to use embedded metadata and IM5 therefore wrote metadata into the raws. I quickly changed this setting when I discovered it, but to late to switch back to a backup of my files.

When I edit metadata in these files with embedded metadata and I am doing a metadata writeback IM5 is reading back the embedded metadata in stead of writing my changes into the sidecar file. So all my edits are lost. In order to get it to work I have to delete xmp and Iptc from the raw and then do my edits.

Is this the correct way of reading metadata when I have chosen to store metadata in sidecar files?

Will embedded metadata always be preferred if it is conflicts between embedded and sidecar?

Is it impossible to have IM5 to ignore embedded metadata in raws, or will this cause trouble?

I am trying to do a delete Iptc and xmp from all my raws, but this is very time consuming in IM5 ExifTool command processor. I have found ExifToolGUI much faster at this, but GUI wont let me do this in many folders at the same time. I am waiting for a confirmation of my new user account at ExifTool forum to ask for advice in how to do this operation in the command line version of ExifTool (delete embedded xmp and Iptc from cr2 files in one folder with many subfolders), but perhaps someone here knows how the proper command should be in order to do this?

Regards
Kim Abel

[Mario]

Embedded metadata is considered more important and will overrule XMP data from sidecar files. We are only talking about XMP data here, because that's the only metadata for which this problem can show up.

Having competing XMP records for a file (one embedded and one in the sidecar file) is calling for trouble. Just don't.
You can use the ExifTool Command Processor to strip embedded XMP data from your files if you only want to use sidecar files.

-overwrite_original_in_place
-xmp:all=
{Files}

[KimAbel]

Thanks Mario

We are only talking about XMP data here, because that's the only metadata for which this problem can show up.

In many of my files the embedded xmp record are deleted, but Iptc not. These files also have this problem. The embedded Iptc is favoured. So I have to delete Iptc to.

You can use the ExifTool Command Processor to strip embedded XMP data from your files if you only want to use sidecar files.

I dont know why, but the ExifTool Command Processor in IM5 is way much slower than ExifToolGUI in stripping xmp and Iptc. Thats why I am trying to find a method to delete xmp and Iptc from cr2 files which are in several subfolders with the standard exiftool. Attached is my settings in ExifTool Command Processor in IM5.

Kim Abel

[attachment deleted by admin]

[Mario]

I cannot comment on performance without a log file. But IMatch sends the commands "as-is" to ExifTool. There should be no speed difference - except that IMatch of courser reloads the images afterwards.

If your files have IPTC Core (legacy IPTC) data as well as IPTC Core in XMP, you of course have again two sets of data because by default (MWG compliance) IMatch imports legacy IPTC into XMP. Legacy IPTC can only exist in the image file, which is also true for EXIF and GPS data. All this data is imported into XMP by IMatch. But IMatch will synch back IPTC, EXIF and GPS into the original image file when it contains that data so after the first import/write-back this data will be in synch  with the XMP. You only need so make sure that you have only one XMP record and that this has been created from the actual IPTC/EXIF/GPS data in the image.

[KimAbel]

Heres a log file from a session where I deleted IPTC from 58 files in one folder with ExifTool Command Processor in IM5. I took about 2 minutes and 20 seconds to finish.

In ExifToolGUI it took about 20 seconds to delete Iptc from 74 files.

Not the same files, but files from the same camera two days apart.

Log file is attached.

Kim Abel

[attachment deleted by admin]

[Mario]

Does ExifTool GUI run the MWG-compliance scripts and migrate data between IPTC, EXIF, GPS to produce a compliant MWG record?
This costs extra time and is performed by IMatch of course.

Plus, since you change files on disk and probably metadata which impacts the visuals, IMatch has to rescan the files as well. And reload the changed metadata...

[KimAbel]

My time examples was from the time I started the ExifTool Command Processor and until the Output window showed how many files were updated. The rescan times was not included in this and this happens after I have closed the window. Is the MWG-compliance script running at the same time as ExifTool does it job, or after I have closed the ExifTool Command Processor window?

Perhaps some of the time differences has to do with the choice "Run once" or "Run for each file in selection"? I earlier tried both without noticing any big time differences, but when I try now I can see that "Run once" is quite a bit faster. It works when I select a few files, but I tried with 2175 files and soon got this error message:

Error: Temporary file already exists: F:/Bilder/BILDEA~1/2005/0413SI~1/KA2D8C~1.CR2_exiftool_tmp
Error: Temporary file already exists: F:/Bilder/BILDEA~1/2005/0413SI~1/KA5858~1.CR2_exiftool_tmp

I cant find those files in that folder so I can not delete them!

Im not sure if that choice will be good when I have selected from 1000-5000 cr2 files.

Kim Abel

[Mario]

These files are created by ExifTool. Maybe it got confused or hit a limit when you processed 2500 files in one argument file.

For all files at once is quite a lot faster.

[KimAbel]

I tried the "Run once" with a little over 260 files, but also got the error message. It seems like some of the photos are stripped from Iptc, but it can not finish all of them! So both methods with the ExifTool Command Processor is no good in my case!

Jörg (joel23) was very kind and wrote a lilttle script that stripped both xmp and Iptc from the selected files. This works very well and it is now chewing through my files. Thanks a lot for the script Jörg :-)

If your files have IPTC Core (legacy IPTC) data as well as IPTC Core in XMP, you of course have again two sets of data because by default (MWG compliance) IMatch imports legacy IPTC into XMP. Legacy IPTC can only exist in the image file, which is also true for EXIF and GPS data. All this data is imported into XMP by IMatch. But IMatch will synch back IPTC, EXIF and GPS into the original image file when it contains that data so after the first import/write-back this data will be in synch  with the XMP. You only need so make sure that you have only one XMP record and that this has been created from the actual IPTC/EXIF/GPS data in the image.

My metadata has always been edited in IM3 and IM5. My database was recreated in IM5 in version 124. This problem appeared after I did a plain rescan of all my folders a few days ago in version 150 i think (or 148). Then many photos appeared with the need for a metadata writeback. Many of these files would not do a successful metadata writeback until I stripped them from embedded xmp and Iptc. Now, when I have been trying to edit location fields and description fields in my files it apparently works since the metadata writeback goes as planned, but when I do a reload of metadata on these files the "old" metadata comes back. I can see that the sidecar (xmp file) has different values from the embedded (cr2 file) in several of the fields (often location tags and description)(my Metadata2 settings are default and I have cr2 as master files in versioning).

Composite\Country
Composite\State
Composite\City
Composite\Location
Xmp Dublin Core\Description

Why and how I am not sure, but it seems that my sidecar Iptc has not synced successfully into my embedded Iptc since they are not the same. All I have understood is that this problem disappears after I have deleted the embedded Iptc and xmp. So that is what I am doing now with Jörg's script. Im not sure if this is interesting for you to look into, since I dont have any log file from the time of my rescan of all the folders. I have sent Jörg a cr2, jpg and xmp which have this problem and he is looking into it, but Im not sure if the files will tell us anything.

Regards
Kim Abel

[Mario]

ut it can not finish all of them! So both methods with the ExifTool Command Processor is no good in my case!

In that case ExifTool should log some important info into the Output panel and the IMatch log file.
I have processed thousands of NEF files, moving from embedded XMP/IPTC to sidecar files after abandoning Nikon Capture for good and I know that this module (and ExifTool) can handle hundreds of files usually...

[KimAbel]

Jörg's script has now finished deleting xmp and Iptc from all my cr2 files, and my initial problem with IM5 dismissing my edits and reverting to the "old" embedden metadata is mostly gone.

There is one problem that still exists.

I think I have narrowed it down to the tag "Copyright" in the Tag selector.

By the way: This tag has the name "XMP Dublin Core\Copyright" when you are making your own metadata panel and "xmp::dc\Rights" when you are making your own metadata template! Very confusing that these dont have the same name even though you select the same tag in the Tag selector.

Anyhow, I am trying to set "Kim Abel" in this field. I do the metadata writeback and it all looks fine. IM5 is saving the edits and my metadata panel shows the right value. But when I do a forced rescan of the file (because the thumbnail shows the wrong orientation) the field goes empty. So IM5 is colleting information somewhere else and dismisses my edits. I can see in ExifToolGUI that there exists a tag in the cr2's exif with the name ----IFD0---- Copyright. This tag is empty. My guess is that IM5 is collecting this info from my cr2 file and dismisses my "Kim Abel". Is there any way I can correct this, or should I not use this field? IM5 is also not getting the correct orientation for some of my cr2's thumbnail. Portraits are shown as landscape, and its the same files with the copyright trouble that has this wrong orientation. I can send an example cr2 file for testing if needed.

As mentioned before my Metadata2 settings are the default with raws using sidecars. I have in IM3 used Iptc Core. I have cr2 as master and jpg and tif as versions. Metadata is set to propagate.

Attatced is what Exif output reports, and Imatch log file.

Kim Abel

[attachment deleted by admin]

[Mario]

This tag is empty. My guess is that IM5 is collecting this info from my cr2 file and dismisses my "Kim Abel".

This field is mapped to the corresponding XMP field based on the Metadata Working guidelines. If your XMP and EXIF data is out-of-synch, either let IMatch sync the data from XMP to EXIF or remove that EXIF field from your files.

[joel23]

Sorry, I'm late with the following up.

Does ExifTool GUI run the MWG-compliance scripts and migrate data between IPTC, EXIF, GPS to produce a compliant MWG record?
This costs extra time and is performed by IMatch of course.

Do you mean the arg files when saying "MWG-compliance scripts"? Nope, they are not used when they are not explicit are applied in ExifToolGUI's "command line".  There is no need to do so when deleting XMP and/or IPTC, because nothing have to kept in sync (at least not IPTC <-> XMP), no digest needs to be written.

Btw: just using these arg files (in general) does not ensure MWG-compliance, as I told before in other posts. As today I really wonder why - as it seems - you still believe those (IMHO example) files ensure MWG compliance. Import the MWG test files and check what happens, once with arg files (more or less a mess for dates and keywords) and without them: still not fully compliant, but much better.

Anyway: here IMatch does the same as ExifToolGui does: with the default settings for RAW/CR2 (which are: using sidecars, allow create EXIF/IPTC/GPS=no, write EXIF=no, write IPTC=no) no arg files are launched.  Means XMP <-> EXIF and XMP <-> IPTC are not synced and by this we are not MWG compliant, even when MWG is set to YES. But at least now, when IPTC was purged, EXIF and XMP have to be kept in sync, for being MWG compliant.

I am not sure, but this might be because MGW strict mode is disabled in the config file (imatch_et.config): $Image::ExifTool::MWG::strict = 0;  Shouldn't this toggle when a user switches MWG compliance to YES or NO?
Will do some test with this config file soon, but so far its seems as it is just a 'dummy call': if the file is there or not, set to 0 or 1, it changes nothing in IMatch's behavior when loading the MWG test files.

Anyway, as Kim told, he accidentally has used embedded XMP and later switched to the default settings for CR2. He also explained that he had edited Composite/Location later in the Default Metadata Panel. What happens when we do that is that we get different IPTC and XMP data.  And as it seems IPTC is preferred when the data is reloaded.
MWG compliance guide says: if both exist and digest matches or does not exist: prefer XMP... It does not say where from, the file itself or the sidecar :-/ but since you prefer XMP IMatch should use the data in the sidecar, not what is to find in the file. IMHO.

@Kim: funny that you mentioned Copyright (I started writing until here two days ago): even when you now have purged IPTC for all files, you should consider to enable "write EXIF". Otherwise it might happen that you soon have the same problem here, between EXIF <-> XMP: editing Copyright or GPS data (XMP) will not be synced to EXIF when it is disabled. Aside that it is not MWG compliant, as said.

about syncing Copyright tags:

This tag is empty. My guess is that IM5 is collecting this info from my cr2 file and dismisses my "Kim Abel".

This field is mapped to the corresponding XMP field based on the Metadata Working guidelines. If your XMP and EXIF data is out-of-synch, either let IMatch sync the data from XMP to EXIF or remove that EXIF field from your files.

Of course they will get out of sync very soon with the default settings for RAW. Not only EXIF but also all other related tags, like XMP TIFF\Copyright will not be synced with the default settings. See the attachments after XMP DC Copyright and XMP GPS was edited.

I wonder this one is closed, because I would call that a bug - with the addition "under certain circumstances". I believe users should at least get informed that the default RAW settings breaks the sync and MWG compliance.Sorry Just realized now, we are in General Discussion.

[attachment deleted by admin]

[KimAbel]

Thanks again Jörg for your in dept discussion of this. This is clearly something that a ordinary user like me have trouble understanding 

MWG compliance guide says: if both exist and digest matches or does not exist: prefer XMP... It does not say where from, the file itself or the sidecar :-/ but since you prefer XMP IMatch should use the data in the sidecar, not what is to find in the file. IMHO.

I cant tell whats right in regard to MWG, but for me the logic is clear. Its the sidecar I am editing, and therefore it should be the sidecar that determines what the field should be. As it is now it only makes thing difficult to understand since I dont get a clue of whats happening in IM5. It has given me quite a bit of "headache" to figure out what IM5 is doing in this case. Perhaps an error message in IM5 should be displayed when there exists two sets of metadata? Perhaps this also results in other problems that could be linked (wrong orientation in cr2 thumbnails and a lot of files with a blue updating icon which "hangs". They wont update and I have to do a forced update on them).

Edit: Found ot that the orientaton problem was due to using jpg as visual proxy and xmp was set to propagate from cr2 to raw with orientation. Used xmp without orientation instead, but I had to delete orientation in vertical jpg to get it right. Much to think about to get it all right

This field is mapped to the corresponding XMP field based on the Metadata Working guidelines. If your XMP and EXIF data is out-of-synch, either let IMatch sync the data from XMP to EXIF or remove that EXIF field from your files.

How many more fields do I have to remove from my cr2 in order to not get into trouble with the default setting in IM5? I have already deleted xmp and Iptc, but as I have understood it from Mario's and Jörg's reply there's several other fields that needs to be updated in embedded exif (my fields in my metadata template is those shown in the attachment plus labels, ratings and gps (from Lightroom)). As I have understood it from Jörg's reply no one who edits gps and copyright fields will get MWG correct data.

Is there any big speed differences in IM5 between the two following choices in regard to saving metadata?

1 - Open up exif writing in cr2
2 - Disable exif writing in cr2 and just edit sidecar (but in my example this is'nt possible if I want MWG correct metadata?). Smaller filesize and possibly quicker to save.

If I choose to delete some of the exif fields in my cr2's then what is the correct command for ExifTool in order to delete them?

Why is the Copyright tag named different in the metadata panel and the metadata template?

Sorry for my many questions, but this is real important to me in order to make my database work correctly. Hope this also is useful to other users since I guess this is something all users with old legacy Iptc will encounter :-)

Kim Abel

[attachment deleted by admin]

[joel23]

[...]
Sorry for my many questions, but this is real important to me in order to make my database work correctly. Hope this also is useful to other users since I guess this is something all users with old legacy Iptc will encounter :-)

Kim Abel

Kim,
don't forget we are in still in Beta-test and things might not work as they should yet. With V152 another glitch seems to raise when RAW was edited - you might consider waiting until IMatch was released for developing a strategy how to cope with metadata.

I have a different approach: I write everything to RAW, I always did and never would delete EXIF from my RAW. My believing is that using sidecars probably produces more mess than doing good. But that's me. Even when I would work with sidecars, I would make sure to keep XMP in sync with the EXIF and IPTC in the files. But I am to narrow-minded when it comes to this, to give good advices.
I accept the impact it has when writing metadata to RAW or syncing them = a tad longer processing time and larger backups, 'cause the RAW files are touched. But I count that as peanuts. More time consuming is writing metadata to real large files like PSD (several hundreds MBs here)

When I pointed to MWG compliance than as said because we are still in Beta-test and I believe programers should know where the glitches and bugs are, in order to fix them. Especially because IMatch claims to be MWG compliant by default (which IMHO is not the case, still trying to convince Mario, respectively to get his attention regarding this).
But of course its up to you to be MWG compliant or not.

See what the helpfile tells regarding the IPTC/EXIF sync:

When you prevent IMatch from updating existing IPTC/EXIF data in files, the XMP metadata record and the IPTC/EXIF records may end up containing different data. And this may cause IMatch to replace newer data in XMP with older data imported from IPTC/EXIF on import or re-scan operations. Usually you disable writing IPTC/EXIF only if the image has no IPTC/EXIF record, and when you have good reasons to do so.

But its disabled by default - guess here just was forgotten, that this are the default setting for RAW files;) and as said: users might not be aware about the negative impacts this has, when they are using the defaults settings:

Me, preventing what? I don't prevent anything - I just used the default settings

[cytochrome]

...
I have a different approach: I write everything to RAW, I always did and never would delete EXIF from my RAW. My believing is that using sidecars probably produces more mess than doing good. But that's me. Even when I would work with sidecars, I would make sure to keep XMP in sync with the EXIF and IPTC in the files. But I am to narrow-minded when it comes to this, to give good advices. ..........

Same here, everything into raw, ever since I catalogue my pictures. I have used the trial and error method to find the settings that suit my needs and now IM5 is very stable concerning metadata handling in my NEF and RW2. I don't use Adobe programs but AftershotPro and DxO for raw conversion and they fit well in this workflow. DxO has no catalog and that of ASP can be ignored completely (and for many reasons).

I use PM5 for ingestion, renaming, sorting and first pass metadata writing, this goes directly into the NEF. With Panasonic raw it is a bit more complicated, PM5 does not write to the RW2, so produces XMP instead. In IM5 these XMP are copied to RW2 with the ECP, it is a one click thing, fast and reliable. Then I take the XMP out of the way to prevent XMP/RAW synchrony problems.

Mario no longer uses Nikon software and has switched to Adobe so IM is "naturally" tuned for an XMP workflow. But an "everything into raw" works fine, IM is flexible and can be used in more than one way, which is one of its forte.  I don't proselyte for the raw way since it requires some testing, but I am glad to see that I am not alone

Francis

[joel23]

Hi Francis,
what is PM5?

[cytochrome]

Hello Joerg,

It is Photomechanic 5. It is a very capable and flexible ingester, but pricey!! I got it on a "special sale" with reduced price. It seems mostly used by sport and other event photographers who have to add iptc quickly on the field before sending the photos to press.

It has an excellent interface for xmp/iptc entry, a bit like the iptc editor in IM3 but easier and is iptc/xmp while the xmp editor in IM3 was horrific (to my taste). And it writes xmp/iptc to NEF, CR2 very fast (they don't use Exiftool, but I never had a glitch).

Francis

[joel23]

Hi Francis,

ah okay. Heard of it.
I just wondered why you go this extra steps, since IM5 writes metadata to RW2 and its metadata handling is far better than it was with IM3.

[cytochrome]

Well, I bought PM5 before I could lay a hand on IM5. At that time PM5 could write directly to the RW2, and IM3 could not. Then some PM users complained that Adobe (LR or ACR) could no longer open these Raw files. So PM ditched this possibility, there are many times more Adobe users than of all other converters together...

And PM writes metadata to 100 NEF in 1-2 seconds. For a long time I had a slow computer, writing to the raw was very slow with IM5, with a lot of blocking and white screens. This is solved now because Mario solved many bugs and I bought a fast up to date machine. But still I find the metadata entry form of PM more convenient than that of IM. IM is more flexible, can be adapted, coloried etc, but for fast and efficient metadata entry I prefer PM. And the EPC is very powerful so copying metadata from XMP to RW2 is a snap (on a fast computer).

Works for me, in fact workflow is mostly a personal thing, the best is the one you are familiar with and confident in. This said, with my present experience of IM5 I would probably never have hunted for an alternate metadata sampler, but now I am hooked. 

By the way PM prepares (since years) a cataloger. They have discovered that it is more difficult than they thought, so no they no longer communicate on it, simply it is in the works. I hope IM5 comes out first, the market is not that big and if it is as good as the ingester it will be some competition for iM.

Francis

PS I just copied metadata from XMP to 84 RW2 files: the ECP script took 9 seconds to run, than 30 seconds more to update everything in the DB and down 2 levels of JPGs. Really good and no hassle at all.