IMatch failed to import GPS Data for some newly indexed files

[Tveloso]

I had another instance of the issue discussed in this topic:
 
https://www.photools.com/community/index.php/topic,13478.0.html

That topic has been archived, so I could no longer post in it, but the same issue described there has happened again - after running a Database Diagnostics, several instances of this warning were generated:

Code Select Expand

Warning: File [<OIDvalue>] with GPS flag but no GPS lat/lon. Fixed.
But the files in question actually did all contain GPS Data, and IMatch simply had not imported it during indexing (or it was somehow cleared before the diagnostics run).

Reloading Metadata for these Files (Shift+Ctrl+F5, Reload Metadata) then successfully imported the missing GPS data.

This time I do have the log from the IMatch Session where the files were originally indexed, but unfortunately I forgot to enable Debug Logging before indexing.  There were about 400 files indexed in two separate operations of about 200 each, and only 18 files were discovered to have the "missing GPS data" condition during the diagnostics that was run shortly afterwards, so this seems to be something that happens infrequently.

I'll send the following logs to the support email:

• The log from the IMatch Session where the files were originally indexed (with Normal Logging)
• The Diagnostics Log containing the 18 Warnings
• The log from the IMatch Session where Metadata for the 18 files was successfully loaded (with Debug Logging)

[Mario]

Also send some of the files where IMatch / ExifTool failed to extract the GPS data.

[Tveloso]

Sample Files have been sent.

But given that that they later received their GPS data with a "Reload Metadata" request, they're probably not very different from the files that did get GPS at the initial indexing.
__PRESENT

[Mario]

I have received the sample files, thanks.
I've tried a few things to somehow coerce the behavior you have experienced, but so far could now.
Even creating hundreds of copies of your images, mixing in about 100 of GPS test files I have in my library and then indexing that folder did not reveal any issues.

Maybe we're lucky and you have IMatch running with debug logging and this provides more info about the problem...

In many cases when something does not work when a folder is added to IMatch and works when a "force update" is used later when the system is at rest, the most plausible explanation is that there was some sort of overload that caused a WIC codec or ExifTool to fail. Or maybe a virus checker was triggered.

But any of these issues would leave traces in the log file, e.g. an ExifTool error or IMatch logging that a file could not be processed because the WIC system returned an error.

[Tveloso]

Thank you Mario.

I expect to have a small batch of files to index tomorrow night, and will be sure to turn on Debug Logging before doing it...and will run a diagnostics right afterwards.  Hopefully it will happen again, and there will be something valuable in the log.

I'll report back tomorrow night.

[Tveloso]

Tonight I added a small batch of about 50 files to IMatch (with Debug Logging on), and was glad to find that there were 11 instances of the "GPS Flag Warning" in a Diagnostics run immediately afterwards:

Code Select Expand

Warning: File [<OIDvalue>] with GPS flag but no GPS lat/lon. Fixed.
I was able to re-load Metadata on all 11 Files, and the missing GPS coordinates were then successfully loaded.  I'll send the logs to the support email.

[Mario]

Is this reproducible?
I mean when you copy the 50 files in Windows Explorer into a new folder and then index that folder in IMatch, does this issue repeat? That would be good news.

If you can repro it, try it again with 10 of the files. And if this produces the issue, upload the files to your cloud space and send me a link. If I can reproduce it here, I can fix it.

[Tveloso]

Mario, I was able to reproduce this using your suggestion!

There were again 11 files with the issue (although, a slight variation in the specific files involved).

I have sent all the files that were indexed for this test to the support email address.

[Mario]

Thanks for sending the files.
Unfortunately, I was so far unable to reproduce the problem.

I've indexed the images and videos. All 51 files showed "with GPS data" in IMatch. Running the diagnosis revealed no problems.
I then changing the indexing settings to run a metadata template that applies a location based on GPS data found in the images and re-indexed the images again - with the same perfect result.

I will run some additional tests when there is a free time slot. As of now, this is a norepro and thus I cannot do anything about it.

Since something similar was never reported AFAIK, this might be something that happens only on your PC.
But you can reproduce it, which is good.

Please try this for your next test: Reduce the number of parallel indexing threads to 2 (see Process Control (Advanced Setting) and then force re-import the files again. Do you sill see the same problems about files being imported without GPS data?

[Tveloso]

I changed the threads from zero to two, for both File Import and Metadata Import, but got the same results (11 Files with the "GPS flag but no GPS lat/lon" warning at the next Database Diagnostics).  I tried it a couple of times to see if the file count would change, but it was 11 both times.

I then installed IMatch 2023.4, and repeated the test there, with again 11 files having the warning.

Mario, this is not that big an issue for me.  I put together a small App that can open a list of OIDs in a result window, so I copy the column of OIDs from the Diagnostics Log into my App, and can easily rescan all the files (using the Relaod Metadata option), which so far has always successfully loaded the GPS coordinates.

And since this may be something happening on my PC only, it's probably not worth your time.

I'll report back if I stumble on some commonality that might suggest a cause.

[Mario]

OK. I've did some additional tests, but it always works without problems.
Even duplicating your samples 100 times and processing them all into a new database caused no problem.
I have no idea what could cause this behavior on your computer, sorry.

[Tveloso]

Mario, this may be related to Bug Fix #02557, which you have made for Release .15.2 (although this is really the "opposite problem" from that - where the file has the GPS Flag, but no lat/lon), so perhaps that fix will address this "opposite issue" as well?

But I just wanted to mention a variation on this issue that I noticed today.  I actually haven't had the:

Code Select Expand

Warning: File [<OIDvalue>] with GPS flag but no GPS lat/lon. Fixed.
...issue in a while.  But today, I found a file that had neither GPS coords, nor a GPS Flag Icon, but did have a GPS TimeStamp showing in the MD Panel:

   

The Pen Icon's ToolTip indicates that GPS Data is among what's pending to write back:

   

I'm quite sure that I didn't change those values though.  All I've done with this file so far is Face Recognition. 

As with the other posts here, the ECP shows that the file does have a complete set of GPS data, so it does seem that IMatch has somehow removed the coordinates (but left the TimeStamp).  Unlike in those other instances though, the Diagnostics didn't complain about this file (since no lat/lon, and an absent GPS flag, represent the in-synch condition).

As before, a rescan (with the reload metadata option) fixed the issue.

[Mario]

Under which condition has IMatch removed the GPS data?
I have no idea how this can happen. Any hint appreciated. I cannot diagnose things I cannot reproduce.

What's the workflow? Versioning? Propagation? MD Templates? AutoFill? MapPanel? Locations?...
IMatch modifies the timestamp only when setting GPS coordinates for a file, which is a decisive operation caused by e.g. the Map Panel and Locations.

[Tveloso]

What's the workflow? Versioning? Propagation? MD Templates? AutoFill? MapPanel? Locations?...

The only thing that would come into play here I think, is the MD Template that runs when indexing new files:

   

It includes the option to assign IMatch Locations, but has the Assign GPS coordinates of Location unchecked (could that be a factor?).

It also sets the Preserved FileName Tag.  Although very much a tangent, just for completeness, here's a description of why that's being done (which should probably be ignored):
Like many users here I think, I "work on" my files locally, and then when everything is done, the files are written back, and then moved to archive storage.  A Renamer Preset called Store Photos handles that work (both renaming, and then moving the files to a NAS), but in the course of the rename operation, if Preserved FileName did not already have a value, it would be updated at that time, and the files would once again become pending for writeback (with the Preserved FileName tag as the only thing needing writeback).  Since the files now live on a NAS, writing them back again would be terribly slow, so that tag is set at ingest, in order to prevent that second Pending Writeback condition when the Renamer "stores them".

IMatch modifies the timestamp only when setting GPS coordinates for a file, which is a decisive operation caused by e.g. the Map Panel and Locations.

One thing I noticed about the problem file, is that following the Reload Metadata rescan (which fixes the issue), the GPS Timestamp now contains my local UTC Offset, when it didn't before (when the coords were missing):

   

It now looks like this:

   

...and clicking into the field gives:

   

That's actually an incorrect Timestamp, because the file actually contains the following:

    [GPS]               1 GPS Latitude Ref                : North
    [GPS]               2 GPS Latitude                    : ** deg **' 24.63"
    [GPS]               3 GPS Longitude Ref               : West
    [GPS]               4 GPS Longitude                   : ** deg **' 0.28"
    [GPS]               5 GPS Altitude Ref                : Above Sea Level
    [GPS]               6 GPS Altitude                    : 6.435843992 m
    [GPS]               7 GPS Time Stamp                  : 08:26:33
    [GPS]              12 GPS Speed Ref                   : km/h
    [GPS]              13 GPS Speed                       : 19.55999947
    [GPS]              16 GPS Img Direction Ref           : True North
    [GPS]              17 GPS Img Direction               : 203.9145203
    [GPS]              23 GPS Dest Bearing Ref            : True North
    [GPS]              24 GPS Dest Bearing                : 23.91452027
    [GPS]              29 GPS Date Stamp                  : 2024:08:28
    [GPS]              31 GPS Horizontal Positioning Error: 4.593283369 m
    [Composite]         - GPS Altitude                    : 6.4 m Above Sea Level
    [Composite]         - GPS Date/Time                   : 2024:08:28 08:26:33Z
    [Composite]         - GPS Latitude                    : ** deg **' 24.63" N
    [Composite]         - GPS Longitude                   : ** deg **' 0.28" W
    [Composite]         - GPS Position                    : ** deg **' 24.63" N, ** deg **' 0.28" W

...so the original version of that TimeStamp (created at ingest) was correct...(the photo actually was taken at near 4:30 AM in my timezone):

[Composite]         - Date/Time Original              : 2024:08:28 04:26:35.192-04:00
[Composite]         - GPS Date/Time                   : 2024:08:28 08:26:33Z

Last night, I indexed another small batch of files.  Prior to doing anything with them, I filtered for files lacking GPS Latitude, but none turned up.  I then did the Face Recognition work (which is the only thing I had done with the batch of files containing the "bad file").  And again no files "lost" their coords.  But one difference between the two batches of files, is that this one contained no files eligible to have an IMatch Location assigned to them, whereas the previous batch did have some (and locations were successfully assigned to those).  The file in question did not match an IMatch Location (but the file immediately prior to it - in FileName Order - did get a location assigned).

I'm sure that I'll have another batch of files that will include some that get an IMatch Location assigned, so I'll keep a close eye on this, and will report back if I have more details.

[Mario]

I have so far identified twp issues:

When IMatch sets the GPS timestamp when setting coordinates, it does so correctly in UTC but does not add the +00:00 time zone offset. This is why the timestamp displays your local time zone when you click it.
IMatch writes the time stamp in UTC, though, which is why it comes back with the 00:00 or Z afterwards.
I have resolved this for the next release.

Also, when applying a location from the Map Panel, the GPS timestamp was not set. When applying a location via the Commands menu, the time stamp was changed correctly to the current time in UTC. This also impacted the automatic assignment of locations via Metadata Templates.

Let me know when you find out what operation causes the GPS coordinates to be removed.