[2023.2.2] Capture Time sort out of order with newly imported files

[cg]

It appears as though files imported into the latest version (2023.2.2) don't sort in the correct order by Capture Time. Files in the db imported into previous versions of iMatch are fine.

All of the files were taken with an iPhone. I'm attaching screen captures of the metadata panel for two files (with and without the cursor in one of the fields).

You can see there is a timezone offset (-4) for IMG_2653.JPG, while IMG_2654.JPG does not have one. Both show the time in the file window as "3:19 PM". However it looks like IMG_2654.JPG has a different tz offset (-7) visible only when the field is selected to edit.

The files are of the same subject, but one difference I DID notice is that IMG_2654 has a face annotation field in it generated by the iPhone, while IMG_2653 does not.

Has something changed with iMatch, ExifTool, or both, with the latest version that affects how the time is loaded when the file is imported, which in turn scrambles the Capture Time sorting?

Thanks!

[Tveloso]

IMatch by default does not show the UTC Offset when it matches your local Time Zone, unless you click into the field to edit the DateTime.  But Mario recently added an option to always show that Offset.

But Mario, I'm finding that turning on the Always show time zone offset option:

   

...does not actually show it consistently for all files. 

I too have a number of iPhone Photos that don't show the offset (even though that option is turned on).  Here is an example of a file that does not show the offset:

   

...until I click into the field:

   

And another one, taken only a few hours earlier, which does show it, both when the field is not being edited:

   

...and when it is:

   

The difference appears to be the presence of XMP Dates...File1 (which does not show the offset in the Metadata Panel) includes XMP Dates (lacking an offset) among its DateTimes:

Code Select Expand

[System]            - File Modification Date/Time    : 2023:08:12 21:11:38-04:00
[System]            - File Access Date/Time          : 2023:08:15 22:13:29-04:00
[System]            - File Creation Date/Time        : 2023:08:12 21:16:22-04:00
[IFD0]            306 Modify Date                    : 2023:08:12 17:53:24
[ExifIFD]      36867 Date/Time Original              : 2023:08:12 17:53:24
[ExifIFD]      36868 Create Date                    : 2023:08:12 17:53:24
[XMP-xmp]          - Create Date                    : 2023:08:12 17:53:24
[XMP-xmp]          - Modify Date                    : 2023:08:12 17:53:24
[XMP-photoshop]    - Date Created                    : 2023:08:12 17:53:24
[ICC-header]      24 Profile Date Time              : 2022:01:01 00:00:00
[Composite]        - Create Date                    : 2023:08:12 17:53:24.435-04:00
[Composite]        - Date/Time Original              : 2023:08:12 17:53:24.435-04:00
[Composite]        - Modify Date                    : 2023:08:12 17:53:24-04:00
...while File2 (which does show the offset) does not have them:

Code Select Expand

[System]            - File Modification Date/Time    : 2023:08:12 14:40:28-04:00
[System]            - File Access Date/Time          : 2023:08:15 22:13:50-04:00
[System]            - File Creation Date/Time        : 2023:08:12 21:16:22-04:00
[IFD0]            306 Modify Date                    : 2023:08:12 14:40:28
[ExifIFD]      36867 Date/Time Original              : 2023:08:12 14:40:28
[ExifIFD]      36868 Create Date                    : 2023:08:12 14:40:28
[GPS]              29 GPS Date Stamp                  : 2023:08:12
[ICC-header]      24 Profile Date Time              : 2022:01:01 00:00:00
[Composite]        - Create Date                    : 2023:08:12 14:40:28.651-04:00
[Composite]        - Date/Time Original              : 2023:08:12 14:40:28.651-04:00
[Composite]        - Modify Date                    : 2023:08:12 14:40:28-04:00
I'm pretty sure that I haven't done any writebacks to either of these files yet.

In my case, the files were originally indexed under IMatch 2023.1.22.  I just installed 2023.2.2 but have not indexed any new files yet.

[Mario]

Please provide the images used for your report.
I need to see what ExifTool/IMatch produce from the dates in your files, how File.DateTime is produced etc.

[Tveloso]

Thank you Mario.  Sample Files have been sent.
__PRESENT

[Mario]

@Tveloso Thanks for sending the two files.

The file IMG_2653.JPG has no XMP record.
IMatch imports it and produces the XMP "date subject created" and "date created" as described in How IMatch uses Date and Time Information

The XMP::photoshop\DateCreated\DateCreated and XMP::xmp\CreateDate\CreateDate are set to

"2023:08:11 15:19:53.498-04:00"

applying the EXIF time zone offset to the two EXIF timestamps found in the file.

The IMG_2654.JPG file has an XMP record.
The XMP record was created by the XMP Core 6.0.0 toolkit and the Creator Tool is set to 16.5.1 - which is not terribly helpful. No idea.

The file holds the following timestamps in XMP:

[XMP-xmp]      Create Date       : 2023:08:11 15:19:55
[XMP-photoshop] Date Created  : 2023:08:11 15:19:55

and these will be used by ExifTool and transmitted to IMatch - which takes the existing date and time and does not override it by re-evaluating EXIF date. This is the intended behavior - XMP data is more relevant/important than native EXIF data, since it is usually created by the user or an editing software.

The device/software writing the XMP metadata should of course have used the EXIF time zone offset when producing the XMP timestamps. This is a bug in the firmware/software. Or the usual don't care, don't bother approach so many devices / applications have these days...

Since both time stamps have no time-zone offset, IMatch assumes them to be in local time or whatever has been configured for the mode (see File.DateTime Mapping Mode (User-defined Time Zone Offsets)).

However, for some reason, the offset is not explicitly added to the existing XMP data.
The File.DateTime.UTC is set to "2023:08:11 15:19:55" and the UTC offset is correctly recorded as +02:00 (my current time zone). But IMatch stores the XMP time stamps as found in the file to the database, instead of adding the +02:00 time zone offset explicitly.

Which also messes with the "Always show time zone" in the Metadata Panel. Since the XMP timestamps have no time zone, the MD panel cannot show any. And when the user clicks on the time stamp to edit, the validation routine detects the missing time zone offset and adds the local time zone offset as it should.

I assume a glitch in the special case that detects and deals with existing XMP date and time values during file ingest.
It takes existing XMP time stamps and uses them "as-is". There should be a check to see if the producing device/software messed up and produced XMP time stamps without a time zone offset.

I will add this check. And if there are existing XMP time stamps without time zone offset, search for a usable offset in the file and apply the correct mapping mode as configured by the user.

Please contact the device/software vendor and tell them that they have a bug the code that produces the XMP.
If they would not have messed up, everything would work nicely in IMatch.

[Mario]

Stupid me.

The code to detect a missing time zone offset in XMP time stamps was already in place, but due to a wrong flag, it was not used/applied.

IMatch now adds missing time zone offsets to XMP time stamps during import.

@cg

This thread was unfortunately a bit side-tracked by unrelated discussion.
You wrote something about Capture Time and wrong sort order.
I've asked for some sample files which exhibit the problem but so far have not received any from you.
I will close this bug ticket now since the main issue has been fixed.
Please open a new bug report for the Capture Time issue alone.

[cg]

Hi, Mario.

I sent sample files when I posted the bug report along with a link to this forum post. I'll re-send them in case they got lost in the shuffle.

Again, from what I can tell, it seems like iMatch is not calculating the actual capture time correctly for the purposes of the sort order like before. It may or may not have something to do with what Tveloso mentioned, but it does seem to have to do with the existence of an iphone-generated face annotation in one file and not the other.

Thanks!

[Mario]

The Capture time sort attribute uses File.DateTime.UTC, see UTC and Local Time
If the calculation of UTC based on what is found in the file is wrong, sorting will be wrong.

Did you check the File.DateTime.UTC for the files in question?
You can use these variables in VarToy:

Code Select Expand

DT: {File.DateTime}
DT: {File.DateTime.TZO}
UTC: {File.DateTime.UTC}
LT: {File.DateTime.Local}
LT: {File.DateTime.Local.TZO}
Org: {File.DateTime.Original}
Org: {File.DateTime.Original.TZO}