WriteBack Not Working as Expected

[richard49]

I'm sorry to bother you again; I have looked through all of the questions regarding writeback and the pen icon but still can't solve my problem.

I'm using iMatch 2018 on a Window 10 computer.

I manage to edit the keyword in one image and then the next image won't do the same.

I have several hundred images that have the pen icon stating that the have unwritten metadata.

When I adjust the keywording for the image and then click on the pen icon the image takes the adjusted keyword and then almost instantly reverts to the original keywords it had in the first place.

I've tried selecting Shift>CTRL.F5 and told it to Force Update but still no luck.

I've been two day attempting to correct the keywording in iMatch and am still no nearer my goal; it's so frustrating as I like iMatch but am finding it just so complicated.

[Mario]

Select one of your files and then open the ExifTool Command Processor via <F9>,<E>
Select the "All Keywords" preset and run it. Show us the results.
Note: If you work with RAW files, also show use the contents of the XMP file by temporarily changing {Files} into {XMPFiles} in the "All Keywords" preset.

In general: It usually helps in such metadata problem cases to upload a sample file somewhere. If you work with RAW files, ZIP the RAW and the XMP file and upload it.
Also state which software and workflow you have used on the files, if any. There are applications out there which are prone to produce corrupted metadata or data without properly synchronized keywords. You notice this only when you change your workflow or switch to other applications or send your files to clients.

The typical cause for this kind of problem are

a) Existing keywords in the file which are out-of-sync and thus cause IMatch to revert back after the reload following the write-back

b) The user preventing IMatch from writing legacy IPTC data but the file has legacy IPTC data (Do you use non-standard settings under Edit > Preferences > Metadata 2 > Configure File Formats)?`

c) RAW files (in your case NEF probbaly) which have out-of-sync metadata embedded in the RAW file and the sidecar file.

a) is the most likely case, especially if legacy IPTC keywords are stored in the file. Here it helps to delete the out-of-sync IPTC keywords in the ECP and let IMatch re-create them cleanly during write-back.

[richard49]

Mario,
Within the Presets in the ExifTool Command there is no option I can find for "All Keywords"?

Regards,
Richard

[Jingo]

Hi Richard - it should be the first one in the list right above "Create XMP Files"... perhaps you need to scroll up in the dropdown?

[richard49]

Jingo,
There is no scroll bar to scroll with, that's it...

[Jingo]

Jingo,
There is no scroll bar to scroll with, that's it...

Strange.. these are stored in the preferences I believe so perhaps something is blocking them.  In any event - you should be able to just paste this in and run it:

-G1
-IPTC:Keywords
-XMP:Subject
-XMP:HierarchicalSubject
-charset
filename=UTF8
{Files}

[sinus]

Jingo, you are ahead of me.
I wanted also write something, but you were quicker.

Just to say, I have also not this Keyword-entry in the ECP, but with your help I could do it.

Richard, you should be able now to do it with Jingo's lines.

[richard49]

Jingo and Sinus,
I'm new to this application and have no idea where to paste this text and then what to do after I've pasted it?

Regards,
Richard

[Jingo]

No worries - it is easy

So, in the ECP, in the Arguments section, just delete the lines that appear and paste in the ones provided.  Then, with the images you want selected, just hit the Run Button... the results should appear in the Output box for the keywords... Enjoy!


NOTE: You can type a new name into the Presets box such as "All Keywords" and then hit the SAVE button to create a new preset.  That way, the preset will be there going forward.  Enjoy!!

[richard49]

The results of the input you gave is...

[IPTC]          Keywords                        : North Weald, XVII

The Keywords shown in the iMatch Keywords section show...

North Weald; Transport|Aircraft; Supermarine Spitfire Mk| XVII;

That's odd, surely?

PS. I've confirmed that the keywords shown using the ExifTool Command Processor are correct by using another application (Exif Pilot).

It still doesn't (as yet) tell me what to do to solve my problem of being able to delete certain keywords from this image and then them re-appearing a second , or so, later?

Bear in mind that there are loads of other images with exactly the same problem.

Please note that I very much appreciate everyone helping me battle my way through an application that is new to me and somewhat... not logical. 

I'm still evaluating iMatch and want to bear with it as it's, so far, the best DAM application I've used.

Regards,
Richard

[Jingo]

The Keywords shown in the Exiftool processor are showing keywords that exist INSIDE the file (or sidecar file if RAW).... the keywords that show in the keyword panel in IM might still only be in the database... that is a significant difference for most users.  It seems like you only have IPTC keywords in the file which are a very old format and perhaps that is the problem with your writeback.  I would suggest deleting the IPTC keywords (you can use the Delete all metadata preset or delete Legacy IPTC preset in the ECP) and trying to write the catalog info back to the files again.   Good luck!

[Mario]

If you have used {Files} in your ECP the keywords you see are the keywords embedded in the NEF file.
If you use {XMPFiles} instead you also see the keywords in the XMP sidecar file.

It seems your NEF file has an embedded legacy IPTC record and that the keywords in this record do not match the keywords in the XMP file.
This should not be a problem because IMatch will write the keywords you see in the Keyword Panel when you write back, and synchronize them between legacy IPTC, XMP flat keywords and XMP hierarchical keywords.

Since your NEF files have embedded legacy IPTC data, IMatch must write metadata to the NEF file and the XMP sidecar file. This is the default for NEF files and usually not a problem. NEF is a common format and safe to write. There are many old NEF files with partial IPTC records around or sometimes even with embedded partial XMP records. Nikon software and cameras did a lot of weird things over the years.

Just to be sure: Open Edit > Preferences > Metadata 2: Configure File Formats. In the list at the top select the NEF format. The default settings are as follows:



As I said, things would be much easier if you can just send a NEF and XMP file and the keywords you see in IMatch in the Keyword Panel.
I need to see if ExifTool reports problems writing to the NEF file, which data is written etc.

You can upload the files and info to your cloud space and post a link, or send the files to support email address.

[richard49]

Mario,
I have forwarded onto Support the email requested with the files I mentioned.

I have also checked the Configure File Formats and your example is the same as mine.

Regards,
Richard

[Mario]

Richard,

I have received your email. But you did include the wrong XMP file. The image is named _DSC4371.NEF so the XMP should be _DSC4371.XMP, not _DSC4369.xmp

The NEF file has legacy IPTC metadata, XMP metadata (!) and a lot of proprietary Nikon and Capture metadata.
The metadata was created by Capture 2.4 which unfortunately insisted on embedded XMP inside the NEF RAW file, despite the accepted standard to keep XMP data for RAW files in sidecar files.
IMatch can handle that, but it's a real nuisance if you work with software who does not know about XMP embedded in the NEF and produces a totally independent XMP file with probably very different XMP data. I can tell more when you send me the correct XMP file.

I made a first test and added the keywords "Mario" and "Test|IMatch" to the file.
No problems. IMatch updated the embedded IPTC record in the NEF with the new keywords and created a clean XMP file with the same keywords in the XMP flat and hierarchical sections.
Existing IPTC and XMP data from the file was properly merged.

[richard49]

Mario,
My aplogies for getting the files mixed up, it always happens when you're in a rush.

You say that my files have IPTC Metadata from Capture 2.4 and by using iMatch you were able to correctt his, how?

I'm sending you, via the support email you sent, a new RAW file together with the correct sidecar file.

Regards,
Richard

[Mario]

You say that my files have IPTC Metadata from Capture 2.4 and by using iMatch you were able to correctt his, how?

There was nothing to correct. IMatch reads and updates the legacy IPTC data in your NEF file, and uses the XMP data of the partial XMP record embedded in the file.
But by default it will only write XMP to sidecar files for NEF, to be compatible with all applications out there (except the old Capture product).
Since the XMP record in your NEF does not contain keywords there is no problem.

But removing it via the ExifTool Command Processor would be a good idea, because dealing with two XMP records for the same file is problematic and may cause trouble.
I have long ago removed XMP and legacy IPTC embedded by Capture in my old NEF files and only keep the EXIF in the NEF (important) and the XMP data in the sidecar file. This works best for modern workflows and cooperates with all RAW processors an image editors out there.

[richard49]

Mario,
You say that "removing it via the ExifTool Command Processor would be a good idea" with regard to the (I think) the XMP data.

a) That the XMP data in the .nef file or the sidecar file needs to be removed?

b) How exactly would I do this as I have no idea?

I'm still having the problem that within iMatch I remove keywords and they then appear back almost instantly.

Sorry, I'm just getting very frustrated and am about to give up (sadly) on iMatch as it's not obvious in its use as all I wanted was a simple application that would catalogue my images without all of this bother.

On saying that I really do appreciate you'rs (and everyone else) in taking time to help, it restore my faith in human kind. 

Regards,
Richard

[Mario]

The latest file and sidecar file you have sent has even worse metadata.

I used ExifTool to pull the data from the NEF and the XMP file.
The XMP file was aparrently created by Nikon Capture and probably later updated by Adobe Lr 3.4.


The NEF file contains not one but two (!) legacy IPTC records with the following keywords:

[IPTC]          Keywords                        : North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII
[IPTC2]         Keywords                        : North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII


The XMP file has the following flat and hierarchical keywords:


[XMP-dc]        Subject                         : North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII

[XMP-lr]        Hierarchical Subject            : North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, Transport|Aircraft|Supermarine Seafire Mk XVII, Transport|Aircraft, Supermarine Seafire Mk| XVII


To make the hierarchical keywords easier to see (and the mess they are in) I have flattened this out:

North Weald
Aircraft
Supermarine Seafire Mk XVII
Transport
Supermarine Seafire Mk
XVII
North Weald
Aircraft
Supermarine Seafire Mk XVII
Transport
Supermarine Seafire Mk
XVII
North Weald
Aircraft
Supermarine Seafire Mk XVII
Transport
Supermarine Seafire Mk
XVII
North Weald
Aircraft
Supermarine Seafire Mk XVII
Transport
Supermarine Seafire Mk
XVII
North Weald
Aircraft
Supermarine Seafire Mk XVII
Transport
Supermarine Seafire Mk
XVII
Transport|Aircraft|Supermarine Seafire Mk XVII
Transport|Aircraft
Supermarine Seafire Mk| XVII


You see the problem? The hierarchical keywords are a mess, many keywords are repeated numerous times, with or without hierarchy.
This is what's in your file. IMatch does not show the duplicate keywords because they are filtered.

What I now did was just to click on the Pen icon in the Keyword Panel (in front of the input field where you can enter new keywords).
This marks all keywords as "updated" and writes what you see in the keyword panel (without the dupes) into the database.
Now I click on the write-back pen for the image in the file window.
IMatch writes out XMP data (to the XMP sidecar file) and also synchronizes the legacy IPTC inside the NEF file.

As expected the keywords in the NEF are the same as before, but the many duplicate keywords in XMP are removed and now look like this:

[XMP-dc]        Subject                         : North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, Supermarine Seafire Mk| XVII, Transport|Aircraft, Transport|Aircraft|Supermarine Seafire Mk XVII
[XMP-lr]        Hierarchical Subject            : North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, Transport|Aircraft|Supermarine Seafire Mk XVII, Transport|Aircraft, Supermarine Seafire Mk| XVII

I used the IMatch default settings.

Now I added a two new keywords: "Mario" and "Test|IMatch" in the keyword panel and then wrote back the file again.

The legacy IPTC record in the NEF file now shows the new keywords:

[IPTC]          Keywords                        : North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, Supermarine Seafire Mk| XVII, Transport|Aircraft, Transport|Aircraft|Supermarine Seafire Mk XVII, Mario, Test|IMatch
[IPTC2]         Keywords                        : North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, Supermarine Seafire Mk| XVII, Transport|Aircraft, Transport|Aircraft|Supermarine Seafire Mk XVII, Mario, Test|IMatch

and the XMP metadata in the sidecar file is also updated and correct:

[XMP-dc]        Subject                         : North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, Supermarine Seafire Mk| XVII, Transport|Aircraft, Transport|Aircraft|Supermarine Seafire Mk XVII, Mario, Test|IMatch
[XMP-lr]        Hierarchical Subject            : North Weald, Aircraft, Supermarine Seafire Mk XVII, Transport, Supermarine Seafire Mk, XVII, Transport|Aircraft|Supermarine Seafire Mk XVII, Transport|Aircraft, Supermarine Seafire Mk| XVII, Mario, Test|IMatch

So, IMatch / ExifTool fixed the metadata mess in your files superbly. Removed the duplicate and out-of-sync keywords and creates three sets of keywords in IPTC and XMP which are identical.

There is really no need for you to be upset.
IMatch is just the first application which handles the metadata mess in your files properly and shows what the problem is.
Many users with NEF files created by Nikon Capture and then processed with other applications like Lr discover this kind of problems when they first use IMatch or another pro-grade DAM.

IMatch not only knows about NEF files with embedded XMP and legacy IPTC data, but also handles it properly. IMatch also sticks to all metadata standards, applies the Metadata Working Group rules and properly synchronizes metadata between the XMP sidecar file and the NEF file. Even updates the legacy IPTC data so it stays in sync with the XMP metadata.

To make your NEF files clean I would suggest to remove the partial XMP record (which does have keywords, for example) with the "Remove XMP metadata" preset in the ExifTool Command Processor.
Make sure yoi have backups of your NEF files. Also, unless you need legacy IPTC metadata (which is discontinued for 15 years now) you also may consider to strip it from your NEF files (there is a preset in the ECP for that as well). This leaves you with a NEF file and the XMP data only in the sidecar file.

BEFORE you do that make sure IMatch has imported the files, and then write them back once to produce a high-quality XMP record in the sidecar file (a composite from the legacy IPTC, EXIF and XMP). Then strip the legacy IPTC data and partial XMP from the NEF if you like.

Discovering metadata problems is typical for users who have worked on their files with multiple applications over the years. Especially with Capture or older Photoshop/Lr versions. All software had issues and the problems only show up when you switch to another software or service. Or to an application which really cares for clean metadata like IMatch.

[Jingo]

Sorry, I'm just getting very frustrated and am about to give up (sadly) on iMatch as it's not obvious in its use as all I wanted was a simple application that would catalogue my images without all of this bother.

Don't give up Richard... this is just an unfortunate side effect of how messed up other software can make your files.  Once you set things up correct - you will be set going forward.  For many many years, I used to use Thumbsplus to manage my photos... back then, it was easy because there was only EXIF and IPTC... you read EXIF (camera settings, etc) and you wrote to IPTC (captions, description, keywords, ratings, etc).  Along came XMP which advanced the metadata ability.. but opened a can of worms as well.  Now, your older photos store in IPTC but newer images might store in IPTC and XMP.  And.. there are TONS of XMP namespaces.... and each software handles thing differently now....

[jch2103]

Discovering metadata problems is typical for users who have worked on their files with multiple applications over the years. Especially with Capture or older Photoshop/Lr versions. All software had issues and the problems only show up when you switch to another software or service. Or to an application which really cares for clean metadata like IMatch.

I can endorse what Mario and Jingo have said. You only find out how 'dirty' your metadata are when you use a tool like IMatch that can show you what problems been lurking, hidden, in your files.

[sinus]

Discovering metadata problems is typical for users who have worked on their files with multiple applications over the years. Especially with Capture or older Photoshop/Lr versions. All software had issues and the problems only show up when you switch to another software or service. Or to an application which really cares for clean metadata like IMatch.

I can endorse what Mario and Jingo have said. You only find out how 'dirty' your metadata are when you use a tool like IMatch that can show you what problems been lurking, hidden, in your files.

Yep, agree.
And you will be very satisfied and relaxed when you know that your files are clean again.

[richard49]

Mario,
Where you state...

What I now did was just to click on the Pen icon in the Keyword Panel (in front of the input field where you can enter new keywords).
This marks all keywords as "updated" and writes what you see in the keyword panel (without the dupes) into the database.
Now I click on the write-back pen for the image in the file window.
IMatch writes out XMP data (to the XMP sidecar file) and also synchronizes the legacy IPTC inside the NEF file.

Under Edit>Preferences>Metadata Write-Back I have (as requested earlier) the I have Write-Back changes to metadata immediately unticked.

I take it that this is still the case otherwise my images are corrected and then re-appear again?

Regards,
Richard

[Mario]

Immediate Write-back is off by default. It slows down IMatch and may interfere with the workflow. This option exists only for users who need it for specific reasons.

I take it that this is still the case otherwise my images are corrected and then re-appear again?

I don't understand.
1. Use one file and mark the keywords as "updated" by clicking the pen and then committing the changes with the green hook button.



2 .Keep the Exiftool output panel open (View > Panels > Output) so we have a protocol of what ExifTool is writing to which file.

3. Click on the pen in the file window thumbnail panel. This writes back the metadata into the file, synchronizing all 3 keyword tags.



This is the workflow I used successfully with your file. I could afterwards add/remove keyword and nothing 're-appeared'. I assume you mean that keywords you have deleted or added vanish?

If this does not work on your PC:

- the file you tried has a different metadata problem than the sample file you've provided. From experience, the problem is always having different keywords in different keyword tags. As we have seen in your sample image.

- IMatch / ExifTool is unable to update the metadata in the NEF or XMP file - but then the log in the output panel will show detailed info about the problem encountered.

Copy the text to the clipboard, paste in Windows Notepad and save it to a file. Attach that file in your reply.

[richard49]

Mario,
Everything is working as expected now.

My mistake was that when I clicked on the green tick on the keywords panel I was getting a warning about write-back so I only clicked on the yellow pencil on the image itself.

Once I had realised that I should click on the green tick for keywords and ignore the warning and then click on the pencil everything worked.

Of course I had to remove some of the keywords which was quite easy.

My utmost appreciation for your (and everyone's) help with this, you've been very patient with me for which I'm grateful.

Regards,
Richard

[Mario]

Once I had realised that I should click on the green tick for keywords and ignore the warning and then click on the pencil everything worked.

That would have been the information that the data you have just added/edited exists only in the database until you write back to the file.