Inconsistent Time Zone Display?

[jch2103]

Background
I've had an issue with my raw processor properly displaying Time Zone data in ,.jpg output files. The developer has made some code changes but I'm seeing inconsistent results in my IMatch metadata windows (some images properly show TZ data but others don't - but output from the ExifTool Command processor shows both files are consistent regarding TZ display!!). 

Specifics
I'm having issues with inconsistent display of the Time zone in the tag {File.MD.XMP::exif\DateTimeOriginal\DateTimeOriginal\0} as displayed in my metadata window but no issues with display of TZ in the tag {File.MD.XMP::xmp\CreateDate\CreateDate\0}. Here's are links to two sets of relevant files (raw and jpg): 
https://1drv.ms/u/s!AtlBwiIf8wQzgZ1uubpFOfdNBbcEGg?e=vhqjZe 
and 
https://1drv.ms/u/s!AtlBwiIf8wQzgZ1t731p4O6ebr6U_g?e=04lsfH

2021-10-14 06-11-39 NIKON Z 6_DxO PL7_3.jpg displays TZ for the above tag, while the other .jpg doesn't. However, when I run the ExifTool Command Processor with 

Code Select Expand

-time:all
-a
-G0:1
-s
-charset
filename=UTF8
{Files}

there's no difference in TZ in the ExifTool data for the two files. I'm having trouble understanding why the two image files are displaying differently in IMatch given what the ECP is showing. See attached screenshots of the two .jpg files. 

I hope this is clear. Let me know if I need to explain further. 

[Mario]

I've looked at the two JPG files. The JPG files contain both EXIF and XMP metadata.

Code Select Expand

2021-10-14 05-57-22 NIKON Z 6_DxO PL7_3.jpg

[ExifIFD]      Date/Time Original              : 2021:10:14 05:57:22
[ExifIFD]      Create Date                    : 2021:10:14 05:57:22
[XMP-exif]      Date/Time Original              : 2021:10:14 05:57:22
[XMP-photoshop] Date Created                    : 2021:10:14 05:57:22
[XMP-xmp]      Create Date                    : 2021:10:14 05:57:22.26-06:00
[ExifIFD]      Offset Time                    : -07:00
[ExifIFD]      Offset Time Digitized          : -06:00


2021-10-14 06-11-39 NIKON Z 6_DxO PL7_3.jpg

[ExifIFD]      Date/Time Original              : 2021:10:14 06:11:39
[ExifIFD]      Create Date                    : 2021:10:14 06:11:39
[XMP-exif]      Date/Time Original              : 2021:10:14 06:11:39
[XMP-photoshop] Date Created                    : 2021:10:14 06:11:39
[XMP-xmp]      Create Date                    : 2021:10:14 06:11:39.19-06:00
[ExifIFD]      Offset Time                    : -07:00
[ExifIFD]      Offset Time Digitized          : -06:00
Only one of the required EXIF Time Zone Offset tags is included, which is why the XMP::photoshop\DateCreated\DateCreated ends up without a time zone offset.

As per definition,

DateTimeOriginal gets time zone from Offset Time Original, with the fallback TimeZoneOffset
CreateDate gets the time zone from Offset Time Digitized, with the fallback imeZoneOffset

Offset Time Original is missing in the file => no time zone offset can be applied to XMP::photoshop\DateCreated\DateCreated. This causes IMatch to set a time zone offset based on the mode configured by the user.

Note: The Offset Time tag provides the time zone offset for the Metadata Modified timestamp.

When I manually add/correct the missing time zone offset for the tag (Metadata Panel or TimeWiz) and write back, the missing time zone offset is added to the file:

Code Select Expand

[ExifIFD]      Offset Time                    : +01:00
[ExifIFD]      Offset Time Original           : -06:00
[ExifIFD]      Offset Time Digitized          : -06:00
This is how it should have looked from the beginning.

The NEF files include all required time zone offsets, which explains why the timestamps for the NEFs files display correctly in IMatch.

In my opinion, IMatch is working as designed and the flawed metadata in the file is the cause for this.

[jch2103]

Thank you for looking into this, especially about the missing EXIF Time Zone (Offset Time Original) tag. 

The thing that I don't understand is that although both jpg images are apparently missing the Offset Time Original tag, one of the images displays the TZ in IMatch while the other one doesn't (see prior screenshots). Is there a reason for this discrepancy?

[Mario]

I see this in the MD Panel for the two files:



I use the default setting for the time zone mapping mode (apply local time zone if there is none).

For the NEF files, I see a time zone offset of -06:00 for both files.

[sinus]

I do also not understand the discrepancy, what you have.

I have downloaded your files and tried it on my system (but only the jpgs). I have exactly the same results like Mario posted.

[Mario]

@jch2103

Which File.DateTime Mapping Mode (User-defined Time Zone Offsets) do you use?
This is the only thing I could think of that would cause the difference between what you see for these files in comparison to @sinus and me...?!

[jch2103]

Thank you both for checking. M Preferences setting is the default File.DateTime Mapping Mode 2, so results my results should be the same as yours... (except for local time zone).  

I did a Force Update on both jpg files; now they do show the same information. 



This second screenshot is after I clicked into the tag field. 


If I understand correctly, if I add the missing Offset Time Original, then both Create Date and Date Subject Created should display TZ as -06:00, which would match what's shown for the original NEFs?

[thrinn]

I also downloaded the files. As Mario and Sinus, both JPG show the same metadata information.
If you see something different, maybe it is related to the option to always display the timezone offset, even if it matches the local timezone. All three of us have the same local timezone offset (+01:00), but if your local TZ is -06:00 it might explaing the difference of what is displayed.

[Mario]

Good point.

The Metadata Analyst has been changed for the next release and now treats a missing Offset TimeOriginal and/or Offset TimeDigitized as a warning.

[jch2103]

Thanks for the insights, everyone. The Metadata Analyst changes should be helpful in my case; thanks!

[jch2103]

Another update on this (complicated) issue:

My raw processor developer (DxO) has responded to my recent input, saying
The DxO camera-lens team replied to say that the problem comes from the XMP attached to the image, which was generated by another software application.

The timezone for DateTimeOriginal is missing. It's our input data so it's normal that it's missing too after DxO process.

I ran the Metadata Analyst on a sample file (after it had been ingested into IM and an XMP file was created) and also on the JPG output file (both attached, so I haven't duplicated the date/time information).

I'm under the impression that 'XMP-exif DateTimeOriginal' doesn't necessarily include TZ information, but it appears that DxO relies on having TZ present in that tag for output. Am I correct about TZ being optional? It seems their choice of input metadata assumes TZ is present.

In my situation, I've been using the TimeWiz app to add TZ to output jpg files which addresses my problem in IM, although this is a extra processing step.

[Mario]

ExifTool produces a time zone offset automatically in the source data IMatch uses if there are time zone offsets available in EXIF. Which are optional. Sometimes both offsets are provided, sometimes only once (depends on camera/device/firmware/moon phase).

I don't have the time to scan and read hundred of rows of JSON from the MDA, sorry.
Just provide the images you have used. Then I can quickly check if they contain time zone offsets, if ExifTool is using them, if IMatch is using them etc.

See also How IMatch uses Date and Time Information where I explained all that and how difficult things are.
Besides, time zone offsets in XMP timestamps are optional. IMatch always adds them (unless they already exist in the XMP timestamp ExifTool produces from the file), based on time zone offsets found in the file or the mode configured by the user.

[jch2103]

ExifTool produces a time zone offset automatically in the source data IMatch uses if there are time zone offsets available in EXIF. Which are optional. Sometimes both offsets are provided, sometimes only once (depends on camera/device/firmware/moon phase).

I certainly understand why you didn't want to go through the JSON files in my earlier post (I think perhaps I guessed that you had better tool than I do to parse JSON files).

The DxO camera-lens team replied to say that the problem comes from the XMP attached to the image, which was generated by another software application.

The timezone for DateTimeOriginal is missing. It's our input data so it's normal that it's missing too after DxO process.

I checked an out-of-camera Nikon Z6 NEF image plus XMP sidecar after importing the image into IMatch. An ExifTool Command Processor dump of the XMP sidecar produced these tags referencing 'date':

[System]        File Modification Date/Time    : 2024:04:18 13:08:21-06:00
[System]        File Access Date/Time          : 2024:04:18 13:08:21-06:00
[System]        File Creation Date/Time        : 2024:04:18 12:59:41-06:00
[XMP-exif]      Date/Time Original              : 2024:03:29 11:15:08
[XMP-exif]      GPS Date/Time                  : 2024:03:29 02:15:05.74Z
[XMP-photoshop] Date Created                    : 2024:03:29 11:15:08.23+09:00
[XMP-xmp]      Create Date                    : 2024:03:29 11:15:08.23+09:00
[XMP-xmp]      Metadata Date                  : 2024:04:18 13:08:21-06:00
[XMP-xmp]      Modify Date                    : 2024:04:18 13:08:21-06:00

It appears that the DxO camera-lens team is correct that the [XMP-exif] Date/Time Original tag does not include a TZ. Is this expected behavior, given that the camera recorded both time zone and GPS date/time (via Snapbridge)?

I've attached:
1. ExifTool dump of the original NEF
2. ExifTool dump of XMP sidecar produced via IMatch after (only) doing a reverse geocode on the NEF

Let me know if you need more information.

[Mario]

Do you see a time zone offset in the Metadata Panel?

Is this expected behavior, given that the camera recorded both time zone and GPS date/time (via Snapbridge)?

Why would it not be. ExifTool and IMatch don't look at GPS data for setting date subject created and date created time zones. They look at the EXIF time zone offsets designed for these two time stamps.

When I read my analysis above correctly:

Only one of the required EXIF Time Zone Offset tags is included, which is why the XMP::photoshop\DateCreated\DateCreated ends up without a time zone offset.

EXIF has 3 optional time tone offsets, two of which are used for create date and date subject created. The file you have provided has only one of the offsets, which is why one of the XMP time stamp ends up without. IMatch adds the local time zone in this case.

Do you see time zone offsets for both timestamps in the MD Panel? Click into the timestamps.
What you see here will be written to the file (if the timestamp is marked as modified).

Are the ExifTool dumps from yet another set of images or the same images as before?
I think you used another set of files I don't have since the first dump (Out of camera) shows 3 time zone offsets

Code Select Expand

[ExifIFD]      Offset Time                    : +09:00
[ExifIFD]      Offset Time Original           : +09:00
[ExifIFD]      Offset Time Digitized          : +09:00
which does not match what I saw in the original file I had?

The XMP ExifTool Dump shows both time stamps (create date and date subject created) with the correct time zone offset.

[XMP-photoshop] Date Created               : 2024:03:29 11:15:08.23+09:00
[XMP-xmp]      Create Date                    : 2024:03:29 11:15:08.23+09:00

Which is expected, since the from camera file has all required time zone offsets, different that the file you have provided first! What was the problem again?

The MDA will show you when a required time zone offset is missing. If no offset is missing, you should see both time stamps in the MD panel with a time zone offset match the EXIF offsets.

[jch2103]

Apologies for not clarifying that the DSC_9523 metadata I sent above is indeed for a different image than I had previously mentioned. The reason for doing this was to try to isolate what exactly was happening while minimizing the number of metadata processing variables that might affect the results. Recording all this also helps me figure out what's going on. 

Yes, all of the 'offset' tags are populated in this particular image NEF.XMP. However, DxO's observation is that 

The DxO camera-lens team replied to say that the problem comes from the XMP attached to the image, which was generated by another software application.

The timezone for DateTimeOriginal is missing. It's our input data so it's normal that it's missing too after DxO process.

Their key point is that they use DateTimeOriginal as input data for their processing, and a missing time zone there will affect their output. Metadata Analyst for the NEF says

XMP


...
XMP Date and Time Information (embedded).
XMP-xmp CreateDate: 2024:03:29 11:15:08.23
XMP Date and Time Information (sidecar).
XMP-exif DateTimeOriginal: 2024:03:29 11:15:08
XMP-photoshop DateCreated: 2024:03:29 11:15:08.23+09:00
XMP-xmp CreateDate: 2024:03:29 11:15:08.23+09:00
XMP-xmp MetadataDate: 2024:04:24 15:25:26-06:00
XMP-xmp ModifyDate: 2024:04:24 15:25:26-06:00

So MA confirms that DateTimeOriginal doesn't have TZ information, which DxO claims to use as input processing. MA also confirms that the three Offset tags are present in the original NEF. 

[ExifIFD]      Offset Time                    : +09:00
[ExifIFD]      Offset Time Original           : +09:00
[ExifIFD]      Offset Time Digitized          : +09:00

However, MA also documents that the JPG output file produced by DxO does not include the Offset TimeOriginal tag:

EXIF

DateTimeOriginal: 2024:03:29 11:15:08
CreateDate: 2024:03:29 11:15:08
Offset TimeOriginal missing. 'Date Subject Created' may have no time zone offset.
Offset TimeDigitized: +09:00
Offset ModifyDate: -06:00

XMP
 Embedded XMP record (DxO PhotoLab 7.6) found.

XMP Date and Time Information (embedded).
  XMP-exif DateTimeOriginal: 2024:03:29 11:15:08
  XMP-photoshop DateCreated: 2024:03:29 11:15:08.23+09:00
  XMP-xmp CreateDate: 2024:03:29 11:15:08.23+09:00
  XMP-xmp MetadataDate: 2024:04:24 12:13:24-06:00
  XMP-xmp ModifyDate: 2024:04:24 12:13:24-06:00

I've written to DxO support to ask them to include the Exif Offset TimeOriginal tag in the output JPG files. We'll see what response I get. (In the meantime, my workaround of using the TimeWiz app to assign TZ to the JPG files does work, even though it's an extra hassle. That ensures that the NEF originals and the JPG output files sort together by Capture Time in IMatch.)

Thank you for your time on this issue!

[Mario]

MA = "Metadata Analyst"?

When DxO says "attached to the image", what do they mean? The XMP data embedded in the NEF or the sidecar file?
Note that some Nikon cameras write some XMP data into the NEF directly. Maybe this causes issues for them? IMatch maintains XMP data for RAW files only in sidecar files.

Why does DxO rely on XMP metadata which usually does not even exist in fresh RAW files? They should rely on the native EXIF time zone offsets when this is so important for them.

I have seen only one NEF from you, and that NEF was missing one of the required EXIF time zone offsets.
IMatch will set the time zone for the corresponding XMP tag to your local time zone, and when you don't have the option enabled to not mark time stamps produced when ingesting the file (default is to mark them as pending) or you mark the the two time stamps as modified in the Metadata Panel by clicking the pen, IMatch will write both XMP tags with time zone information into your file. Give it a try. The time zone information will be written in the same format as you can see in the MD panel when you click into one of the timestamps.
This will also cause ExifTool to copy the time zone information from the two XMP timestamps into the two native EXIF time zone offsets so everything will be filled and in-sync after the write-back.

I cannot comment on metadata in files I don't have here.
I just looked at the two text files you provided. The NEF file in this case has all required time zone offsets (at +09:00) and the metadata produced by IMatch shows these time zone offsets used for the two XMP time stamps, which means all is in order.

If these metadata dumps are from other files or my conclusion is wrong, please provide the NEF files and JPG files and sidecar files you have used or send to DxO so I can do a proper analysis.

[jch2103]

MA = "Metadata Analyst"?

When DxO says "attached to the image", what do they mean? The XMP data embedded in the NEF or the sidecar file?

Yes, MA in this case means Metadata Analyst.
DxO was explicitly referring to the XMP sidecar when they referred to 'attached to the image'. The reason they use the sidecar is to be able to incorporate metadata contained there (e.g., location descriptions, ratings that override the '0' default rating used by Nikon, keywords, etc., etc.) into output files. Otherwise the output JPG or other format files wouldn't be able to incorporate this additional information not contained in the raw file. If there's no XMP sidecar, DxO will use whatever metadata information is in the raw file. (Like IMatch, DxO needs to be able to use various user workflows.) 

To summarize (and hopefully not oversimplify):

1. DxO reads raw file and XMP sidecar metadata when processing raw files into JPG or other output, allowing metadata editing via their own metadata editor or via another editor of the user's choice, such as IMatch. They pass on metadata from the raw file and/or in the XMP sidecar file into the output file (JPG in my case). Nearly all the appropriate metadata from the raw and/or XMP is included in the JPG file, with the exception below. 

2. I have found that my raw and (DxO) JPG output files don't sort the same in IMatch using a Capture Time sort. This is due to different representations of time zones between the raw file and the output file. 

3. DxO attributes this discrepancy to a missing time zone in the DateTimeOriginal tag. (They didn't specify whether this is for the raw file or the XMP, but there's no time zone in either, which I believe is consistent with this Exif tag's definition. Therefore it doesn't seem to matter in my case which tag is used.)

4. DxO passes on to the JPG output file the Offset TimeDigitized tag but not the Offset TimeOriginal tag. You have said above that IMatch relies on having both of these tags to properly assign time zone information to other tags. I have asked DxO to also add the Offset TimeOriginal tag to metadata copied to JPG output files. I await their response.

5. I have a workaround for this discrepancy: Use the TimeWiz app to assign the raw file time zone to the output JPG file. This fixes the discrepancy, adding the Offset TimeOriginal to the JPG. IMatch is now able to properly sort the raw and JPG files. The downside is that this adds additional complication and steps to my workflow.

Here's a link to the NEF, XMP and JPG files (DSC_9523) mentioned above: Time Zone Issues.zip

[Mario]

Native EXIF data may have two time zone offsets which are linked to "Create Date" and "Date Subject Created".
IMatch finds these and uses them when it produces XMP::CreateDate and XMP::DateSubjectCreated while importing a file.
If one or both of the EXIF time zones are missing, IMatch by default assumes the local time zone, or applies the "mode" configured by the user: File.DateTime Mapping Mode (User-defined Time Zone Offsets)

The two relevant XMP time stamps will thus always have a time-zone offset in the database (which also shows in the MD panel).
When you write back the file, the XMP time stamps will be written with the time zone offset  and ExifTool will, when IMatch runs the XMP2EXIF.args file) update the native EXIF time zone offsets from these time stamps.

All this one write-back and you can see the result in the MDA (Metadata Analyst) and in the MD Panel in IMatch. Or in the ExifTool Command Processor.

Note that when you have disabled the option to mark "new" XMP timestamps produced by IMatch during ingest for write-back, the new XMP time stamps (with offset) will not be written back - since they have been marked for write-back. You can force a write-back by clicking the pen in front of the two time stamps.

If DxO does not look at the native EXIF time zone data but relies on XMP (?), this is a mandatory step to allow IMatch to fix the missing time zone offset in your file.

This applies for the first file you have sent. The file in the middle where you only posted some ExifTool dumps had both time stamps and IMatch produce XMP time stamps with time zone offsets.

The NEF file in the ZIP file has both EXIF time stamps, at +09:00.
It has some embedded XMP data (stupid Nikon again) in form of a rudimentary XMP record with many mandatory tags missing. No timestamps, for example.

IMatch produces the following (correct) time stamps from the EXIF data and offsets.



and when I write back, IMatch produces a complete, compliant and rich XMP sidecar file with time stamps containing the correct offset:

Code Select Expand

[XMP-photoshop] Date Created                  : 2024:03:29 11:15:08.23+09:00
[XMP-xmp]      Create Date                    : 2024:03:29 11:15:08.23+09:00
everything is well and as it should be. Not sure what DxO sees as a problem.
If the source file has EXIF time zones, IMatch uses them. Else it substitutes the local time zone or applies the user-specified mapping mode.

I have explained all that 3 times now and also explained it in deep detail How IMatch uses Date and Time Information

As long as you make IMatch write-back the time stamps it has produced, the XMP record will always contain XMP time stamps with time zone offsets.

Make sure you don't have enabled the option to not mark time stamps created/modified during import as pending.