Cannot get rid of "Pending Metadata Write-back"

[fischmir]

Hallo,

I have two pics I can not get rid off in the collection "Pending Metadata Write-back". I already updated these files multiple times including "Force Update" etc.  But two pictures are always there. These files are not offline. If I click on "Information" everything is fine. App-Panel "info & Activity" is not showing any error.

Any idea out there?

Regards,
Christian

[jch2103]

Do these files show the yellow pencil? If so, do a mouse-over on the pencil and see what IMatch thinks still needs to be written.

[JohnZeman]

What happens when you select those images and press CTRL+SHIFT+F5 and choose Reload Metadata?

[Mario]

Check the tooltip of the yellow pen icon. It will tell us why IMatch thinks it needs to write the files.
Please also give us info about the file format you are using, and if you use non-standard metadata settings.

The behavior you describe is usually caused by a mismatch between EXIF/IPTC and XMP which IMatch cannot correct because the metadata options chosen by the user don't allow this. For example, if your file has legacy IPTC metadata but you don't allow IMatch to synchronize XMP and IPTC, a mismatch between data in IPTC and XMP may occur after a write-back, causing IMatch to flag the file as pending...

[fischmir]

Hmm...it seems to be another problem. My database is damaged and cannot be repaired. I restored a backup and checked that database. That db is ok.

This (new, restored) db shows again 355 pending metadata write-backs. After updating all files, the collection was showing 80 files still to be updated. But this number increased step-by-step to 355 again.

The yellow pen is yellow and shows the following tags to write:

IPTC::ApplicationRecord\Keyowrds
XMP::dc\Subject

I checked my preferences, but I don't know if there is "setting-mismatch". I not aware of any change I did here. What do I need to check?

__
If I reload Metadata, IM tells me, that some metadata is protected (which is fine in my case). After the update, there are still 34 files in this collection. 
__


Regards,
Christian

[Mario]

Keywords (IPTC keywords, XMP subject) is the typical case for what I mentioned above.

IMatch has hierarchical keywords and tries to write them to the file. This creates a mismatch between the keywords in the database and the keywords in the file on re-import. Since you did not answer my questions, I cannot tell more. If you use RAW, make sure you allow IMatch to update existing IPTC/EXIF data in the RAW file Make sure your keyword import settings (Edit > Preferences > Metadata) and your thesaurus don't create 'new' keywords on every import. Unless you give us more info to work with, everything else would be guesswork.

[fischmir]

Please also give us info about the file format you are using, and if you use non-standard metadata settings.

I am using jpg and jpeg and I am NOT aware of using any non-standard metadata settings ("Metadata Working Group Compliance" is set to "Yes").

If I check these 355 files I should mention, that they are basically from three different folders. Two of three folders contain images which were scanned. One folders contains pictures taken with a digitale camera.

I attached a logfile. Check at this line:
01.03 09:22:17+ 6661 [0D6C] 00  I>             # Loglevel changed.


[attachment deleted by admin]

[Mario]

JPEG. Important info. JPEG files are by default set to allow update or existing IPTC/EXIF and to contain XMP data. This avoids many of the typical pitfalls caused by RAW files.

If you have image files which reliable produce this problem, please attach one to your reply. Or send it to me, with a link back to this topic or the ID 3989.

[fischmir]

Hi Mario,

I have new information.

I have 355 pending files. If I "Reload Masterdata", this number decreases to 34. Directly after this, I do a "Database Diagnostic" and "Metabase" shows erros. There is one warning in the Log File:

Checking Metabase:
      Warning: 321 files marked as pending for write-back which have no data to write-back. Fixed.

Cool. I start the test again and Metabase is NOT showing any warning. But my collection "Pending Metadata Write-back" is now showing 355 files again.  The next "Database Diagnostic" runs without warnings. This behaviour is reproducable every time I "Reload Masterdata" (34 files, warnings in "Database Diagnostic" and fixed, 355 files again).

The file I send you, is disappaers after "Reload Masterdata", but is shown again after "Database Diagnostic". I will now send you one file, which remains even after I reloaded masterdata.

Hope this helps.

Regards,
Christian

[Mario]

I have tested the two sample files.

The first problem showed up when I tried to ZIP the two files into an archive using Windows' built-in ZIP (Save to compressed folder). Windows informed me that one of the files has a file name which is invalid for a ZIP file and then refused to pack the files.

The file name is

remains after reload masterdata 2003-06-01__12-00-00__Košíre__16.jog

IMatch handles this, though. No problem.

I opened the Output Panel (<F9>,<O>) to see details about what IMatch is writing (keywords) and what ExifTool responds. ExifToo. reports

Warning: Some character(s) could not be encoded in Latin - F:/Download/COMMUN~1/3989/REMAIN~1.JPG
Warning: Error converting value for ExifIFD:SubSecTimeOriginal (ValueConvInv) - F:\Download\COMMUN~1\3989\REMAIN~1.JPG
Warning: Some character(s) could not be encoded in Latin - F:/Download/COMMUN~1/3989/REMAIN~1.JPG

Again, this is for the file name above.

The keywords you are attempting to write cannot be mapped into the Latin character set and hence ExifTool refuses to update the legacy IPTC data in the file. This is the reason why the file always remains in the write-back state. The keywords you want to write are:

CZE; Czech Republic; Hlavní mesto Praha; Hlavní město Praha; Košíre; Košíře; Staré Mesto; Staré Město; geo:lat=50|08615672; geo:lon=14|41348314; geotagged

This cannot be converted into legacy IPTC.

Solutions:

1. Remove the legacy IPTC from the file and use only XMP or
2. Fix the keyword to use only ASCII/ANSI characters or
3. Tell IMatch to use a different character set when writing IPTC data (Edit > Preferences > Metadata). MAKE ABSOLUTELY SHURE you understand the implications of this and especially the risk of mixing different character sets in your files.

[joel23]

I had the same problem with IPTC and the same keywords "Košíře, Staré Mesto" (for those who are interested: one is a district in Prague and the other the name for its Old Town) and with other keywords when using a wide character set, eg. Cambodian locations/keywords.
I solved this by first setting the "IPTC:CodedCharacterSet tag" to UTF8 before ingesting the files and before adding locations and keywords in IMatch. I let Geosetter do the job.

Solution #4
If you want to stick to IPTC, close IMatch, delete IPTC with ExifTool or its GUI and create a new IPTC record by only setting IPTC:CodedCharacterSet to UTF8 - again using ExifTool or its GUI.
Or chase them through Geosetter (for Geosetter's settings see the 1st attachment) which nicely set this tag as well.

Once this tag was set, IMatch behaves nice regarding keywords (and also IPTC [Sub-]Location and City) and writes IPTC keywords and locations in UTF8 See the 2nd attachment.
No need to use a different character set (3. on Mario's list)

Reminder:
Set "IPTC:CodedCharacterSet" to UTF8 when IMatch is the creator of the IPTC record = when "allow to create IPTC/EXIF/GPS" is set to Yes.
And, when IPTC already exist, convert it to UTF8 as Geosetter does and MWG demands it. IMHO.

[attachment deleted by admin]

[Mario]

Changing the IPTC encoding tag can only be done after all IPTC data was removed! ExifTool uses the IPTC encoding found in the file when reading/writing data if not overridden in Edit > Preferences > Metadata. If the file already has IPTC data and the character set is changed, a mix of different character sets is created, which can make the IPTC data unreadable. More details on this in the help for the E > P > Metadata dialog.

[ColinIM]

I've suffered for the first time (yesterday) with this symptom of recurring writebacks on 3 JPG files.

To cut to the main point of my note here - after trying to use IMatch (of course) and ExifToolGUI and GeoSetter to correct, or tidy, or to delete (only) the problem metadata (Keywords) which I thought were causing the IMatch writeback loop, I was able to fix each of the files in turn with one mouse-click using the program called Picture Information Extractor (PIE) from Picmeta.

The PIE program has a menu option labelled Metadata | Repair Metadata ...  and although it's a "black box" procedure (the PIE program doesn't reveal what it 'fixes' in the metadata), once I had used this step on each file, the ExifTool/IMatch metadata-writeback loop was cured. The PIE 'fix' appeared to keep ALL of the (significant) metadata intact in these files.  Any changes were invisible to me, and I kept IMatch open, monitoring the files, throughout the seemingly-instant PIE 'fix'.

Although - to repeat my 'black-box' point from the previous paragraph - I don't know exactly what PIE did to fix my files, as soon as IMatch had responded to the changed "modified time" on the repaired file and had re-ingested it, the most visible result was that I was able to get IMatch to erase the Keywords which were being repeatedly RE-inserted into the {File.MD.XMP::Lightroom\hierarchicalSubject\HierarchicalSubject\0} fields after my attempts to do a writeback.

I'm trying to keep this brief, but if they'd be any use to you Mario I can upload some logs and a sample image.  I've kept copies of the related IMatch logs and the ExifTool Output panel during attempts to do writebacks in IMatch. (I see nothing in the logs that gave me any clues!).  I've also kept copies of the "faulty" images (before and after their fix by PIE), and I also took command-line ExifTool extracts of the metadata in two of these files before and after the PIE fix. (Although again, the metadata extracts show me nothing suspicious except to reveal multiple entries of some EXIF/XMP fields in the files - which I note is VERY common among my images, and is usually benign.)

To conclude: I had done nothing to 'provoke' the apparent changes on these 3 troublesome images. They are among a few dozen files with similar editing histories (last edited in 2008) and - as far as I can tell - none of them contains any non-UK-English characters, but (like nearly all of my files) they do still contain some legacy IPTC data.

Colin P.

Note that there's a free version of the PIE program but the free version only reads metadata. It won't write back any changes. The paid-for version costs USD39.

[joel23]

Changing the IPTC encoding tag can only be done after all IPTC data was removed!

Yes, this is what I had told:

Solution #4
If you want to stick to IPTC, close IMatch, delete IPTC with ExifTool or its GUI and create a new IPTC record by only setting IPTC:CodedCharacterSet to UTF8 - again using ExifTool or its GUI.

After this was done IMatch/Exiftool nicely imports all the other XMP and EXIF data to IPTC even when a wide character set is used.

[Mario]

Probably the program only does what ExifTool does when you use one of the repair modes: read the metadata, delete it, write it back. You can do that with ExifTool or the ECP in IMatch.

Check out, for example:

http://www.sno.phy.queensu.ca/~phil/exiftool/faq.html

or

https://www.google.de/?gfe_rd=cr&ei=FR22VOjQNoum8wf8xYCADw&gws_rd=ssl#q=exiftool+repair

The procedure required of course depends on what is to repair. In the case for this thread, the problem was that the user wanted to write metadata that could not safely be written because of a mismatch of the character set of the existing IPTC metadata (ANSI/Latin) and the data to be written (which was in a different character set that could not be mapped to ANSI/Latin).

To repair this problem, ExifTool has to read the IPTC data in the current character set, delete the IPTC data in the file, and then write the IPTC data back in UTF-8 encoding. From then on, all data can be safely written. (IMatch 3 has produced UTF-8 IPTC data only since 2008).

To convert data in a file from one encoding to another, I successfully used this in the past:

exiftool -tagsfromfile @ -iptc:all -codedcharacterset=utf8 a.jpg

from the ExifTool FAQ page.

All these character set issues only affect legacy IPTC data, which was never really designed for multi-language or global use. But legacy IPTC was replaced 10 years ago by XMP and the web sites and application vendors should just get their backsides up and cope with it.

[jch2103]

Check out, for example:

http://www.sno.phy.queensu.ca/~phil/exiftool/faq.html

See especially FAQ #20: http://www.sno.phy.queensu.ca/~phil/exiftool/faq.html#Q20

[joel23]

And see especially FAQ #10 http://www.sno.phy.queensu.ca/~phil/exiftool/faq.html#Q10

Note that unless CodedCharacterSet is UTF‑8, applications have no reliable way to determine the IPTC character encoding. For this reason, it is recommended that CodedCharacterSet be set to "UTF8" when creating new IPTC.

Jazz IPTC is not dead, it just smells funny.
Original © 1974 by Frank Zappa

[fischmir]

After this discussion, my problem is almost sorted out.

There are two files left in the category "Pending Metadata Write-back". If I check ExifTool OUtput it tells me:

__
-overwrite_original_in_place
-m
-use
MWG
-charset
ExifTool={PTETCHARSET}
-ex
-tagsfromfile
P:\iMatch\OJ8D3S~K\29H4W4~D
-@
C:\Program Files (x86)\photools.com\IMatch5\arg_files\exif2xmp.args
--Exif:rating
-@
C:\Program Files (x86)\photools.com\IMatch5\arg_files\iptc2xmp.args
-@
C:\Program Files (x86)\photools.com\IMatch5\arg_files\gps2xmp.args
-sep


-XMP-dc:Subject=
-XMP-lr:HierarchicalSubject=DEU
-XMP-lr:HierarchicalSubject=Dutum
-XMP-lr:HierarchicalSubject=Germany
-XMP-lr:HierarchicalSubject=Nordrhein-Westfalen
-XMP-lr:HierarchicalSubject=Rheine
-XMP-lr:HierarchicalSubject=geo:lat=52|27590556
-XMP-lr:HierarchicalSubject=geo:lon=7|42613889
-XMP-lr:HierarchicalSubject=geotagged
-XMP-dc:Subject=DEU
-XMP-dc:Subject=Dutum
-XMP-dc:Subject=geotagged
-XMP-dc:Subject=Germany
-XMP-dc:Subject=Nordrhein-Westfalen
-XMP-dc:Subject=Rheine
-XMP-dc:Subject=geo:lat=52
-XMP-dc:Subject=27590556
-XMP-dc:Subject=geo:lon=7
-XMP-dc:Subject=42613889
-XMP:CreatorTool=photools.com IMatch 5.2.0.18 (Windows)

-xmp:DocumentID=xmp.did:df8f788a-212d-442d-a629-e66d5268d4cd

-xmp:OriginalDocumentID=xmp.did:df8f788a-212d-442d-a629-e66d5268d4cd

-xmp:InstanceID=xmp.iid:9a3f4ab4-d391-4b11-8ddc-abe35ecd93de

-XMP:MetadataDate=now
-XMP:ModifyDate=now

-execute
-execute9999



----- Runtime: 0,2 s.
No file specified
__

Any idea how to solve this?

Regards,
Christian

[Aubrey]

I'm coming late to the discussion, but my 2 cents worth...

I recently had files which were continuously appearing as unwritten after metadata write back.
I ended up doing a a forced reload of the highlighted files using crtl shift F5.
After this all problems disappeared.

Aubrey

[joel23]

After this discussion, my problem is almost sorted out.

There are two files left in the category "Pending Metadata Write-back". If I check ExifTool OUtput it tells me:

__
-overwrite_original_in_place
-m
-use
MWG
-charset
ExifTool={PTETCHARSET}
-ex
-tagsfromfile
P:\iMatch\OJ8D3S~K\29H4W4~D

I miss the filename extension here!?

[Mario]

P:\iMatch\OJ8D3S~K\29H4W4~D

This file name looks strange, even for a shortened 8.3 file name.
What is the name of the original file?

ExifTool reports

No file specified

which means that it cannot find a file or the parameters sent by IMatch are incorrect (highly unlikely because these are the same for all files).

[fischmir]

I agree. My files are stored on a NAS Synology. Could this be reason? I forced the update and renamed this file and back again to its old name. Now, it's gone. Original filename is: 2014-03-26__12-00-00__Rheine.jpeg (scanned image).

Now, there is only one file left. I'll try to fix it myself...

[Mario]

NAS boxes are usually not a problem, as long as the installed SAMBA version supports Windows file names correctly. I use NAS storage myself.

[Menace]

exiftool -tagsfromfile @ -iptc:all -codedcharacterset=utf8 a.jpg

Sorry, I'm not sure, how to write it in ECP. If I write: 

-tagsfromfile @
-iptc:all
-codedcharacterset=utf8
{Files}

ECP wrote: Ignored superfluous tag names or invalid options: -tagsfromfile  ...

[Mario]

What are you trying to achieve?

Do you want to forcefully convert the existing legacy IPTC data to UTF8? Why?
Make sure you read the ExifTool FAQ where you got this from and that you understand the consequences.

Your arguments are wrong and cannot work that way.
{Files} is wrong when used with tagsfromfile. You need to run your ECP this for each file in the selection individually, and use the name of the file, not the {Files} variable.

I suggest you use the following arguments on some copy of your files, then test the result in all your applications:

-tagsfromfile
@
-iptc:all
-codedcharacterset=utf8
{File.FullName}


Make sure you set the "run for each file in selection" option in the ECP as well.

[Menace]

First, thank you a lot for your help.

I try to "repair" my metatags (see: https://www.photools.com/community/index.php?topic=4168.0 ). Maybe it would be better to rewrite all metatags?

[joel23]

First, thank you a lot for your help.

I try to "repair" my metatags (see: https://www.photools.com/community/index.php?topic=4168.0 ). Maybe it would be better to rewrite all metatags?

Yes, converting might get you into more problems.
Delete all IPTC and after that add the "-IPTC:CodedCharacterSet=utf8" tag only in ECP. The rest (syncing IPTC with XMP) IMHO can be done with a force update.

[Menace]

Hello joel23,

thank you for your help. Sorry for my stupid questions:

1. How can I delete all IPTC (for example, when I have hundred or thousands files)?
2. The commandline in ECP would:

-IPTC:CodedCharacterSet=utf8
{File.FullName}

3. Would it be a solution to set in Edit Menu > Preferences > Metadata: IPTC: read and write from default to UTF-8?

[joel23]

Hello joel23,

thank you for your help. Sorry for my stupid questions:

1. How can I delete all IPTC (for example, when I have hundred or thousands files)?
2. The commandline in ECP would:

-IPTC:CodedCharacterSet=utf8
{File.FullName}

Haven't much time, so a quick answer (seems deleting and adding in one run does not work).
deleting:
-overwrite_original_in_place
-m
-IPTC=
{Files}

adding CodeCharacterSet only:
-overwrite_original_in_place
-m
-IPTC:CodedCharacterSet=utf8
{Files}

3. Would it be a solution to set in Edit Menu > Preferences > Metadata: IPTC: read and write from default to UTF-8?

No, just leave it to "default".

EDIT
here you are using {Files}

[Menace]

Thank you a lot. It works perfect, but is not a solution for my problem. Still pending Metadata Write-back.

[joel23]

Thank you a lot. It works perfect, but is not a solution for my problem. Still pending Metadata Write-back.

If this problem is the same like in this thread then it's the xmp-xap tag.
I downloaded the images from your dropbox and have the same prob, even after removing IPTC.
IMatch/ExifTool complains:
Warning: Undefined XMP namespace: xap - M:/_Bearbeiten/work/triptych/Original.jpg

Try to remove the tag - I can't tell how on the fly; I get

Warning: Tag 'XMP' does not support alternate languages
Nothing to do.

when trying to do. Maybe someone else can help on this.

[Menace]

I fear, my "old" images are messed up completely. If I remove "write hierarchical keywords" I can write the files and there are no pending. If I write hierarchical keywords, it is pending. Maybe my hierarchy is to long (vor example: Systematik|Stamm    Gliederfüßer (Arthropoda)|Klasse Insekten|Neuflügler (Neoptera)|Eumetabola|Holometabola|Ord Hautflügler (Hymenoptera)|UOrd Taillenwespen (Apocrita)|ÜF Vespoidea|Fam Faltenwespen (Vespidae)|UF Feldwespen (Polistinae)|Gattung Polistes|Franzoesische Feldwespe (Polistes dominula) ).

Edit: Anyway: Thank you a lot for your help.

[Mario]

Geez. Just slow down a bit.
You are posting way to many messages (in multiple threads) and information for any other user to keep up...
We need to check your files first, see what they contain, what the problem is. Don't damage your files by fiddling with the metadata in several applications. This will do no good.

[Menace]

Ok. But I fear, that my images have not just one problem, but a lot. I keep calm until you have the time to look in the files.

[Mario]

See your other thread and my reply.