Write back problem

[Mees Dekker]

I'm puzzled by the following. 

The dashboard says: 68 files to write-back. I do that and no files are shown any more to write-back. This is expected of course. All seems fine.

However: when I then run a database diagnosis, a warning is returned that says that 68 files are marked for writeback, but have no data to write back. Why are there 68 files marked when the dashboard says there are none to write back? The diagnosis also says that this problem is fixed.

But when I now check the dashboard, it states that here are once again 68 files to write-back. 

What could this behaviour? Both diagnosis log and debug log are attached.

[Mario]

Do these files show a pen in the File Window?
If so, when you point the mouse cursor at the pen, are tag names displayed?

The diagnosis is message can appear when, for some reason, IMatch did not reset the "to write" flags associated with each tag value. I have seen this very rarely, mostly when I shut-down IMatch in the developer tools. I have not figured out a specific workflow or steps to produce this particular warning.

However, when the diagnosis encounters this, it resets the flag in the database.
Maybe the Dashboard has not refreshed? This is a rather particular situation and pressing <F5> once after clicking into the Dashboard might fix this.

[Mees Dekker]

Yes, these files show a pen. See attached screenshot for the tags involved. (I'm using the Dutch translation of IMatch)

Alas: pressing F5 in the dashboard does not change anything: the same 68 files are shown as pending write-back.

[Mario]

If you see tags to write-back, IMatch is right. These tags need to be written.
What happens if you write-back again?
Do you use automatic reverse geocoding?

[Mees Dekker]

When I write back, all goes according to my very first entry. Pencil disappears, dashboard shows no files to write back, but after a database diagnosis, all 68 are again marked for pending write back. 

I do not use automatisch reverse geocoding, that is always performed/initiated manually. Google is used as Geocoding service. 

As far as I can recall, this never happend before the last update to 2023.3.2

[Mario]

The warning is misleading I think.
This is an updated test which checks two-ways now:
a) for files marked as pending (in the file record) but no metadata tags marked as modified.
b) for files not marked as pending (in the file record) but with metadata tags marked as modified.
The diagnosis does (I have fixed that already) does not use different warnings for the a/b case.

My guess is that, for some reason, there are tags marked as modified (the tags you see in the pen icon tool tip) but the file itself is not marked as pending. This is why the pen comes back after the diagnosis.
So that's OK, just the warning text is wrong.

The standard scenario for a file to become pending again after write-back is when ExifTool failed to write the data or when the re-read of the metadata after the write-back produced "new" data because something did not sync correctly.

Usually this is caused by legacy and out-of-sync keywords. I don't recall having seen that with GPS coordinates.

If this affects the same files all the time, please send me a sample file (with a link back to this topic).

Writing GPS and location data is really complex, since there is EXIF GPS, XMP GPS, location data in photoshop XMP and EXIF XMP and all that needs to be synchronized during write-back, copying coordinates and location info into a large number of places.

As outlined in the release notes, IMatch 2023.3.2 introduces some changes and now does "more" mapping to improve location data synchronization. Maybe this somehow collides with the existing data in your files? Not sure.

What is your GPS and location workflow?
Do you use the outdated composite tags or the new IPTC Location Metadata Panel layouts?
Do the files show data in the Location Created and Location Shown structured tags?

[Mees Dekker]

My GPS workflow is as follows:

In the mappanel I only mark the location where the picture is taken. That location is applied to all files and after that all files are reverse geocoded (through the metadata panel) and written back

I guess I'm still using the composite tags. Location created and location shown do indeed have data.

A couple of sample files will be send to you.

[dcb]

I'm seeing the same behaviour, except for me it's across 7.5K files that were updated when I made a change to some keywords. It is only the GPS related tags that are showing a problem, and the problem only occurs after I run a database diagnosis, but it shows repeatedly no matter how many times metadata is written back.

A force rescan of all files has them load with no metadata issues identified. Only once database diagnosis is run does the problem show.

I don't use reverse geocoding.

This is the GPS data prior to the writeback.

Code Select Expand

[GPS]           GPS Version ID                  : 2.2.0.0
[GPS]           GPS Latitude Ref                : South
[GPS]           GPS Latitude                    : 37 deg 46' 39.39
[GPS]           GPS Longitude Ref               : East
[GPS]           GPS Longitude                   : 144 deg 55' 39.02
[GPS]           GPS Altitude Ref                : Above Sea Level
[GPS]           GPS Altitude                    : 42 m
[GPS]           GPS Time Stamp                  : 14:40:41
[GPS]           GPS Map Datum                   : WGS-84
[GPS]           GPS Date Stamp                  : 2014:12:14
[XMP-exif]      GPS Altitude                    : 42 m
[XMP-exif]      GPS Altitude Ref                : Above Sea Level
[XMP-exif]      GPS Latitude                    : 37 deg 46' 39.39 S
[XMP-exif]      GPS Longitude                   : 144 deg 55' 39.02 E
[XMP-exif]      GPS Map Datum                   : WGS-84
[XMP-exif]      GPS Date/Time                   : 2014:12:14 14:40:41Z
[XMP-exif]      GPS Version ID                  : 2.2.0.0
[Composite]     GPS Altitude                    : 42 m Above Sea Level
[Composite]     GPS Date/Time                   : 2014:12:14 14:40:41Z
[Composite]     GPS Latitude                    : 37 deg 46' 39.39 S
[Composite]     GPS Longitude                   : 144 deg 55' 39.02 E
[Composite]     GPS Latitude Ref                : South
[Composite]     GPS Longitude Ref               : East
[Composite]     GPS Position                    : 37 deg 46' 39.39 S, 144 deg 55' 39.02 E
Then, after reapplying GPS from the map and writing back metadata. iptcExt fields are added, as are destination fields. File is not showing the writeback flag at this point. I see that GPS datestamp has not updated to now which is not what I'd expect.

Code Select Expand

[GPS]           GPS Version ID                  : 2.2.0.0
[GPS]           GPS Latitude Ref                : South
[GPS]           GPS Latitude                    : 37 deg 46' 39.39
[GPS]           GPS Longitude Ref               : East
[GPS]           GPS Longitude                   : 144 deg 55' 39.02
[GPS]           GPS Altitude Ref                : Above Sea Level
[GPS]           GPS Altitude                    : 42 m
[GPS]           GPS Time Stamp                  : 14:40:41
[GPS]           GPS Map Datum                   : WGS-84
[GPS]           GPS Dest Latitude Ref           : South
[GPS]           GPS Dest Latitude               : 37 deg 46' 39.39
[GPS]           GPS Dest Longitude Ref          : East
[GPS]           GPS Dest Longitude              : 144 deg 55' 39.02
[GPS]           GPS Date Stamp                  : 2014:12:14
[XMP-iptcExt]   Location Created GPS Altitude   : 42 m
[XMP-iptcExt]   Location Created GPS Latitude   : 37 deg 46' 39.39 S
[XMP-iptcExt]   Location Created GPS Longitude  : 144 deg 55' 39.02 E
[XMP-iptcExt]   Location Shown GPS Latitude     : 37 deg 46' 39.39 S
[XMP-iptcExt]   Location Shown GPS Longitude    : 144 deg 55' 39.02 E
[XMP-exif]      GPS Altitude                    : 42 m
[XMP-exif]      GPS Altitude Ref                : Above Sea Level
[XMP-exif]      GPS Dest Latitude               : 37 deg 46' 39.39 S
[XMP-exif]      GPS Dest Longitude              : 144 deg 55' 39.02 E
[XMP-exif]      GPS Latitude                    : 37 deg 46' 39.39 S
[XMP-exif]      GPS Longitude                   : 144 deg 55' 39.02 E
[XMP-exif]      GPS Map Datum                   : WGS-84
[XMP-exif]      GPS Date/Time                   : 2014:12:14 14:40:41Z
[XMP-exif]      GPS Version ID                  : 2.2.0.0
[Composite]     GPS Altitude                    : 42 m Above Sea Level
[Composite]     GPS Date/Time                   : 2014:12:14 14:40:41Z
[Composite]     GPS Dest Latitude               : 37 deg 46' 39.39 S
[Composite]     GPS Dest Longitude              : 144 deg 55' 39.02 E
[Composite]     GPS Latitude                    : 37 deg 46' 39.39 S
[Composite]     GPS Longitude                   : 144 deg 55' 39.02 E
[Composite]     GPS Dest Latitude Ref           : South
[Composite]     GPS Dest Longitude Ref          : East
[Composite]     GPS Latitude Ref                : South
[Composite]     GPS Longitude Ref               : East
[Composite]     GPS Position                    : 37 deg 46' 39.39 S, 144 deg 55' 39.02 E
And after a database diagnosis once again showing writeback. No change in values from what were just recorded as written by IMatch.

Code Select Expand

[GPS]           GPS Version ID                  : 2.2.0.0
[GPS]           GPS Latitude Ref                : South
[GPS]           GPS Latitude                    : 37 deg 46' 39.39
[GPS]           GPS Longitude Ref               : East
[GPS]           GPS Longitude                   : 144 deg 55' 39.02
[GPS]           GPS Altitude Ref                : Above Sea Level
[GPS]           GPS Altitude                    : 42 m
[GPS]           GPS Time Stamp                  : 14:40:41
[GPS]           GPS Map Datum                   : WGS-84
[GPS]           GPS Dest Latitude Ref           : South
[GPS]           GPS Dest Latitude               : 37 deg 46' 39.39
[GPS]           GPS Dest Longitude Ref          : East
[GPS]           GPS Dest Longitude              : 144 deg 55' 39.02
[GPS]           GPS Date Stamp                  : 2014:12:14
[XMP-iptcExt]   Location Created GPS Altitude   : 42 m
[XMP-iptcExt]   Location Created GPS Latitude   : 37 deg 46' 39.39 S
[XMP-iptcExt]   Location Created GPS Longitude  : 144 deg 55' 39.02 E
[XMP-iptcExt]   Location Shown GPS Latitude     : 37 deg 46' 39.39 S
[XMP-iptcExt]   Location Shown GPS Longitude    : 144 deg 55' 39.02 E
[XMP-exif]      GPS Altitude                    : 42 m
[XMP-exif]      GPS Altitude Ref                : Above Sea Level
[XMP-exif]      GPS Dest Latitude               : 37 deg 46' 39.39 S
[XMP-exif]      GPS Dest Longitude              : 144 deg 55' 39.02 E
[XMP-exif]      GPS Latitude                    : 37 deg 46' 39.39 S
[XMP-exif]      GPS Longitude                   : 144 deg 55' 39.02 E
[XMP-exif]      GPS Map Datum                   : WGS-84
[XMP-exif]      GPS Date/Time                   : 2014:12:14 14:40:41Z
[XMP-exif]      GPS Version ID                  : 2.2.0.0
[Composite]     GPS Altitude                    : 42 m Above Sea Level
[Composite]     GPS Date/Time                   : 2014:12:14 14:40:41Z
[Composite]     GPS Dest Latitude               : 37 deg 46' 39.39 S
[Composite]     GPS Dest Longitude              : 144 deg 55' 39.02 E
[Composite]     GPS Latitude                    : 37 deg 46' 39.39 S
[Composite]     GPS Longitude                   : 144 deg 55' 39.02 E
[Composite]     GPS Dest Latitude Ref           : South
[Composite]     GPS Dest Longitude Ref          : East
[Composite]     GPS Latitude Ref                : South
[Composite]     GPS Longitude Ref               : East
[Composite]     GPS Position                    : 37 deg 46' 39.39 S, 144 deg 55' 39.02 E


I hope this helps.

[Mario]

I have identified the source for the original issue of this thread.
When comparing the internal state (files marked as pending for write-back) with information about pending files retrieved directly from the database, a new flag was not considered properly, causing a discrepancy between the number of files in the internal state and the number of files actually having tags marked as modified.

[Mees Dekker]

And can you fix it for the next release?

[Mario]

And can you fix it for the next release?

I have fixed this already (see release notes).

But I've spent some more hours on this because the one image (the one with the elder gentleman) produces a duplicate entry for IPTCExt::LocationCreated, consisting of two identical sets of lat/lon/alt.

I first thought this was an issue in IMatch, and tried to track it down.
But then I recognized that the XML output ExifTool delivers to IMatch actually contains the duplicate tag values for XMP-iptcExt:LocationCreatedGPSAltitude, XMP-iptcExt:LocationCreatedGPSLongitude and XMP-iptcExt:LocationCreatedGPSLatitude.

I have no idea where this comes from. The other file you've sent does not produce duplicate entries for location created.

I will post a question in the ExifTool user forum about this. Maybe Phil can tell me what the problem is.
Do I have your permission you provide Phil with a copy of the image?

[Mees Dekker]

Of course you have this permission. Anything that helps IMatch improve.

In the mean time I just found out that the GPS tags also come back for writing when I change the filename.

Workflow: I imported a set of CR-2 files into IMatch. Then geotagged them through the mappanel and wrote back the new metadata. They all had no data to write back (no pencil) after that. Then I renamed them all by using the renamer. Now not only the "preserved filename" tag was marked for writing, but also all the GPS related tags.

It is therefore not only the database diagnosis as a cause. Problaby/possibly related to the Exiftool problem too?

[Mario]

Workflow: I imported a set of CR-2 files into IMatch. Then geotagged them through the mappanel and wrote back the new metadata. They all had no data to write back (no pencil) after that. Then I renamed them all by using the renamer. Now not only the "preserved filename" tag was marked for writing, but also all the GPS related tags.

This has been fixed for the upcoming release.
When mapping GPS/location data (new) between native GPS, EXIF GPS and XMP, IMatch under some conditions did not recognize that the data was already mapped and mapped it again. This caused the re-occurring write-backs.

I will look into at the strange duplicated created coordinates after shipping the next release.
This seems to be something unique to this particular file and hence I prioritized it low.

[Ger]

Hi Mario,
It seems I have the same problem in my database, with about 15k images returning to pending metadata writeback (after diagnoses). I confirm this issue did not exist with version 2023.2.

(I also see the wrong values for GPS longitude and GPS altitude (this thread).

It would be great if you release the new version soon.

Thanks in advance

ger

[Mario]

Over the weekend and today's holiday I've finally figured out why the image produced a duplicate entry for location created.
It was not ExifTools fault but mine. Stupid me!
A change in functionality to deal with unwanted tags in original files caused a side effect that may have caused accidental duplication of some tags with the multiValue option set. Like, for example, IPTCExt::LocationCreated.

Note: This problem only happened here in my lab, with the development version of IMatch.
The modified code was not shipped to customers. I only noticed with Mees' files and some files in my test suite.

[Mario]

(I also see the wrong values for GPS longitude and GPS altitude (this thread).

This thread is about incorrectly formatted formatted values for Altitude. Which only shows when a user decides to display the formatted Altitude or uses the corresponding variable. IMatch does not use this value for anything.

And it's only about the Altitude, not longitude.
Please open a bug report when you think the longitude shown by IMatch is wrong. Also provide a test file and your GPS workflow.