Diagnostics sets GPS flag for files without GPS coordinates

[thrinn]

After upgrading to IM 2023.1.10, I performe a Database Diagnostics, as recommended in the release notes. I was astonished that I got 1.700 warnings of type

Code Select Expand

  Warning: File [18441] with missing GPS flag. Fixed.I knew that I had not worked with so many files with IM 2023 (mostly because I was on vacation when 2023 came out).
Investigating further, I discovered that more or less all of these files do not have any GPS coordinates. They are now (after the diagnostics) showing the globe symbol, but they do not have GPS coordinates. The only (strange) thing is that they have some tag called "GPS Map Datum" set to "WGS-84". No idea where this tag comes from. Some of the files are out-of-camera JPGs and RW2 files, without any metadata change in IMatch. It is of course possible that the camera just chooses to put this tag into EXIF, but I had never problems with it before. The first attachment shows that all relevant GPS tags are empty.

I also checked some files with the ECP. Searching for GPS, I only find the tags in the second attachment.

So, my question is: Why does the diagnostics sets the GPS flag when there are no Lat/Long tags set? Does it look for any tag in the GPS group to decide if the file should have the flag set?

[cg]

I got about 300 of those warnings when I ran the diagnostics, and those files do indeed have the GPS flag set now. It's a pretty varied collection of files of different ages, cameras, etc.

With a quick look, it doesn't seem like there's anything particularly consistent with the GPS flags across all of the files. Some have 0" coordinates, others are empty, some have the WGS-84 map datum flag, and other various tags set in some and empty in others.

[Mario]

For convenience, I've used an existing "FileHasGPSData" routine provided by the IMatch database engine.
This engine checks if a file has a GPS record. Native GPS, as shown in your 2nd screen shot, is mapped to XMP GPS during import. The presence of the GPS record is enough for a file to having GPS. A GPS record without coordinates should be quire rare in the wild.

The "file has GPS data but flag is not set" must be fast, because the test needs to be run once for each of the 50K, 100K or 1 million files in your database. So I've used the check for an existing GPS record.

I understand you have files with GPS records, map datum, GPS version id etc. but no lat/lon coordinates.
One could create such records, I think, by deleting the lat/lon/alt but leaving the GPS record in place. Did you do this?

To handle this situation, the diagnosis would need to do a second check, after determining if a file has a GPS record it would also load the lat/lon metadata to deal with files having a GPS record but no coordinates. Which I would have assumed to be quite uncommon.
Doing this will slow down the diagnosis considerably I guess. I wonder how common this is...

[thrinn]

Additional info to reproduce the issue:
Import the attached JPG and run a diagnostics. Get a warning that missing GPS flag has been fixed.
I can of course manually delete the GPS Map Datum tag. Now, the globe icon disappears. Write back metadata. So far, so good.
But now, run the diagnostics again - and the GPS flag is set again.

Edit: Sorry, attachment size limit reached.
Edit: OK, found a file small enough.

[thrinn]

One could create such records, I think, by deleting the lat/lon/alt but leaving the GPS record in place. Did you do this?

No. As I said, this also happens with files fresh from the camera.

[Mario]

No. As I said, this also happens with files fresh from the camera.

This looks like your camera is always adding a pro-forma GPS record, but if it has no coordinates, it just leaves them out.
This would no be the first stupid thing a camera vendor has done...
...like, adding a hard-coded "Rating=None" to the incomplete XMP record written to each file. Among other things.
They really give a shit, don't they?

So, I probably need to slow down the diagnosis for all users and change it to:

a) does the file have a GPS record?
b) if yes, do a second check to see if this is not a fake and there are actually a lat/long coordinate pair?

I have to admit that I'm not trilled about this. But if it has to be done, it has to be done.

[thrinn]

So, I probably need to slow down the diagnosis for all users and change it to:

a) does the file have a GPS record?
b) if yes, do a second check to see if this is not a fake and there are actually a lat/long coordinate pair?

I have to admit that I'm not trilled about this. But if it has to be done, it has to be done.

Yes, I know what you mean. But I doubt that only my camera is producing incomplete GPS records.
I could work around this issue by somehow deleting the GPS record for files where it is not wanted (metadata template, ECP...).
But I will not be the only one running into this issue.

[axel.hennig]

Hi Thorsten,

interesting. I also had these error warnings, but due to the fact that IMatch said "fixed" I didn't further investigate.

I could reproduce that with your provided file (RXT08031.JPG).

I was able to fix it by setting up the following Metadata-Layout and deleting everything (or setting to empty) which was possible.

[axel.hennig]

By the way: I don't have the log-file anymore from my productive database when running the Database Diagnostics. But am I somehow able to find out which files have been reported with these errors?

[axel.hennig]

I made a quick test with a data-driven category to find some images with globe symbol, but without having Lat/Lon. Found some and they are from several different Cameras (not sure if the metadata was changed before importing to IMatch). The Cameras are:

Canon PowerShot S110
Canon EOS 5D Mark II
Canon EOS 5D Mark III
Canon EOS 5D Mark IV
Canon EOS 80D
Canon EOS 700D
Canon EOS 500D
NIKON D700
NIKON D5100
NIKON D200
HTC Desire HD A9191
iPhone 6

[cg]

I created a special category for the files that had the GPS flag "fixed" as they were all listed as recently modified when the db opened after the optimization.

In my case, they are files of all ages and provenance, some from scanners, some from phones but perhaps post-processed. The main commonality seems to be that the state of the metadata is messy. 2023.1.10 was the first time they had triggered a warning in the log.

In my case, as I mentioned, some of the files have coordinates of 0'0"/0'0" while others have empty fields.

If there's any data that I can output from analyzing these files (there are 298 of them), I'm happy to help.

[thrinn]

Thanks to Mario's explanation of the technical background, I understand now what happened. It is also clear to me that the issue is caused, in the end, by the hardware and software (camera, smartphone) creating the metadata.
But from a user perspective, it is still not ideal. Apparently, Sony, Canon, Nikon, HTC, Apple... all seem to create this kind of incomplete GPS record. So, I would expect to see quite a lot of "My globe icon reappears" postings here in the future. Similar to the "My metadata keeps reappearing" topics popping up again and again. Long term users of IMatch might know what to do, but for a newbie it is not easily understandable.

[Mario]

I have changed the "HasGPS" check to not look for an existing GPS record, but to look for the existence of
Latitude/Longitude or DestLatitude/DestLongitude. If at least one of the tags exists, IMatch assumes the file has GPS data.

I have modified the diagnosis to detect files flagged marked as having GPS but having no lat/lon/destlat/destlon and reset the GPS flag in this case.

Running the diagnosis in 2023.10.12 will thus solve the problem, for both cases.

[thrinn]

Just installed 2023.10.12. Running diagnostics gave lots of warnings about resetting the GPS flag the other way around, which was to be expected. Second diagnostics run without any warnings.

The result is now "correct" (from my user perspective). There was no noticable run time penalty for the disgnostics, now using a more complex new logic. But my database is small (< 30.000 files), therefore the diagnostics run time has never been an issue for me anyway.

Thank you for the quick fix!

[Mario]

Thanks for the feedback.
The runtime penalty for this new check is minimal, even for larger databases. I was able to make it fast