Broken MP4 metadata

[rolandgifford]

Suddenly I have 4000+ MP4 files where the metadata is pending writeback and every attempt to do that fails.

I'm primarily interested in how to remove them from the pending queue. All of them have been written back successfully in the past, I haven't intentionally changed anything, but I clearly have.

My final attempt to fix this has failed so I'm now asking what I should do.

I have done the following in my last attempt:

Moved the MP4 files from their correct folder to another folder (using Windows and not IMatch)
Rescan the folder to remove them from the database
Rescan the folder I have moved them to to add them to the database
Writeback (several times)

The Keywords and GPS data that I'm expecting have been imported

The output window for writeback shows:
Warning: Month '00' out of range 1..12 in XMP-xmp:ModifyDate (PrintConvInv) - H:\Import\Ann\FZ300\AnnArgentina 2022_0917_123714.MP4

This is the same error for all files and is the error that I'm trying to clear

Hovering on the yellow pen icon for theses files shows either:

XMP::xmp\CreateDate

or

XMP::exif\DateTimeOriginal
XMP::photoshop\DateCreated
XMP::xmp\CreateDate

it toggles between these two with each attempted writeback

[Mario]

The XMP-xmp:ModifyDate tag is set by ExifTool during write-back (when it writes the XMP data).

Do you have XMP sidecar files for the MP4 files?
IMatch switched to embedding XMP into MP4 files maybe a year (?) ago.
At that time I've explained in the release notes how to import the XMP sidecar file into the MP4 and then remove it.
Did you do that?

When you run the The Metadata Analyst on one of the MP4 files, are there warnings or errors reported?

[axel.hennig]

I think I've had this problem also (quite somt time ago). At the end the problem was corrupted Metadata.

We would need much more information from you, e.g.:

• Output of Metadata-Analyst app
• What does "Output - Exiftool" (F9,O) show?
• Could you upload one problematic Video to your Cloud and past the link here?

If I'm correct, I've solved the problem by (of course applied to Videos not JPGs): https://exiftool.org/faq.html#Q20

[rolandgifford]

Do you have XMP sidecar files for the MP4 files?
IMatch switched to embedding XMP into MP4 files maybe a year (?) ago.
At that time I've explained in the release notes how to import the XMP sidecar file into the MP4 and then remove it.
Did you do that?

No sidecar files, I did import following the release notes.

Metadata Analyst for one file reports:

Metadata Analyst Results. Version 2023.5.2. 12/20/2023 12:27:35 PM
File analyzed: H:\Import\Ann\FZ300\AnnArgentina 2022_0917_115441.MP4
Errors: 0
Warnings: 6

Warning: [System] File has unwritten metadata (pending write-back).<br/>The metadata loaded from the image and the data in the database may not match.
Warning: [Detailed Validation] [minor] Month+Day out of range for EXIF:ModifyDate
Warning: [Detailed Validation] [minor] Undefined value for MakerNotes:ClearRetouchValue
Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0xc6d2 PanasonicTitle
Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0xc6d3 PanasonicTitle2
Warning: [Detailed Validation] IFD1:ThumbnailLength is zero

[rolandgifford]

If I'm correct, I've solved the problem by (of course applied to Videos not JPGs): https://exiftool.org/faq.html#Q20

I'm fairly confident that I tried that (having fixed jpgs that way) but ExifTool wouldn't fix MP4s.

Exiftool from one update attached

[Mario]

This looks pretty, normal, except

Warning: [Detailed Validation] [minor] Month+Day out of range for EXIF:ModifyDate 

When you look at the file in the Metadata Panel in the Browser layout, you should be able to see this tag.
Which value does it have?

What happens when you make the "date subject created" and "create date" tags in the Metadata Panel as modified (click the pen in front of the two tags) and then write back?

IMatch should write both timestamps and this should result in both the EXIF::ModifyDate and XMP::ModifyDate to be set.
Either ExifTool does that during write-back or IMatch does this using standard ExifTool syntax like -EXIF::ModifyDate=now

[rolandgifford]

ModifyDate shows as 0000:00:00 00:00:00.112 in the example I have used in this post.

I think that each video has a different suffix number

No change when I flag the Create Date tags to be updated and do an update

ECP File Dates and Times shows:

[File:System]   FileModifyDate                  : 2023:12:20 13:09:09+00:00
[File:System]   FileAccessDate                  : 2023:12:20 13:09:12+00:00
[File:System]   FileCreateDate                  : 2022:09:17 11:54:41+01:00
[QuickTime]     CreateDate                      : 2022:09:17 11:54:41
[QuickTime]     ModifyDate                      : 2022:09:17 11:54:41
[QuickTime:Track1] TrackCreateDate              : 2022:09:17 11:54:41
[QuickTime:Track1] TrackModifyDate              : 2022:09:17 11:54:41
[QuickTime:Track1] MediaCreateDate              : 2022:09:17 11:54:41
[QuickTime:Track1] MediaModifyDate              : 2022:09:17 11:54:41
[QuickTime:Track2] TrackCreateDate              : 2022:09:17 11:54:41
[QuickTime:Track2] TrackModifyDate              : 2022:09:17 11:54:41
[QuickTime:Track2] MediaCreateDate              : 2022:09:17 11:54:41
[QuickTime:Track2] MediaModifyDate              : 2022:09:17 11:54:41
[EXIF:IFD0]     ModifyDate                      : 0000:00:00 00:00:00
[EXIF:ExifIFD]  DateTimeOriginal                : 2022:09:17 11:54:41
[EXIF:ExifIFD]  CreateDate                      : 2022:09:17 11:54:41
[MakerNotes:Panasonic] TimeStamp                : 2022:09:17 11:54:41
[EXIF:ExifIFD]  SubSecTime                      : 112
[EXIF:ExifIFD]  SubSecTimeOriginal              : 112
[EXIF:ExifIFD]  SubSecTimeDigitized             : 112
[XMP:XMP-exif]  DateTimeOriginal                : 2022:09:17 11:54:41
[XMP:XMP-exif]  GPSDateTime                     : 2023:01:21 11:29:52
[XMP:XMP-photoshop] DateCreated                 : 2022:09:17 11:54:41.112
[XMP:XMP-xmp]   CreateDate                      : 2022:09:17 11:54:41.112+00:00
[XMP:XMP-xmp]   MetadataDate                    : 2023:12:20 13:09:09+00:00
[XMP:XMP-xmp]   ModifyDate                      : 2023:12:20 13:09:09+00:00
[Composite]     SubSecCreateDate                : 2022:09:17 11:54:41.112
[Composite]     SubSecDateTimeOriginal          : 2022:09:17 11:54:41.112
[Composite]     SubSecModifyDate                : 0000:00:00 00:00:00.112

[Mario]

These are only warnings, they will not hinder ExifTool from updating the file.
If the write-back comes "back" afterwards, there is some sort of mismatch between time stamps or something similar.

Please attach a sample MP4 file or upload to your cloud and post a link.

Do you use the default settings for File.DateTime in Edit > Preferences > Metadata?
When this worked before, do you have an idea since when this failed, or what happened before it failed?

[rolandgifford]

Please attach a sample MP4 file or upload to your cloud and post a link.

Do you use the default settings for File.DateTime in Edit > Preferences > Metadata?
When this worked before, do you have an idea since when this failed, or what happened before it failed?

I use option 5 for FileDateTime. When this option was introduced I used something else but DST was causing confusion for the way I like to work so I changed it. I have no interest in any time other than the time shown on a clock where and when I took the photo.

This failed fairly recently. I have done some recent widespread Keyword changes resulting in most of my files being out of sync. Updating metadata has worked for all photo images, most of my video files have been left in this odd out of sync state.

Link to file:

https://drive.google.com/file/d/1FroJh-UdbfbwhIkZ3u79TF70ja_alsqK/view?usp=sharing

[axel.hennig]

I was able to "fix" the problem by running (command-line) the following exiftool-command:

Code Select Expand

exiftool.exe -all= -tagsfromfile @ -all:all -unsafe -icc_profile video.mp4
But be careful: Try it with a copy of your video-file, not with the origional one. And: You will loose some metadata with this command (mainly Panasonic-specific ones).

[rolandgifford]

I was able to "fix" the problem by running (command-line) the following exiftool-command:

It worked for me too which surprises me as I was already aware of this (having used it before) and was sure that I had tried it this time as well before asking here.

Thanks

[Mario]

I've tried to download the sample file but Google tells me that is has been deleted?

[rolandgifford]

I've tried to download the sample file but Google tells me that is has been deleted?

Sorry, I assumed that you had downloaded it at about the same time as axel's reply and chose to 'recommend' his fix by not replying yourself - as you are usually very fast replying.

I have removed it as I choose not to leave things out there with downloadable links.

I have applied axel's suggestion and everything is again in sync. I have metadata going out of sync from time to time for files which have previously been in sync (Taiwan place names the last time I remember) and my main interest every time is to fix that with 'how did I do that' as an aside. I usually put it down to tighter checks as software is amended/fixed so something previously acceptable isn't any more.

I will have a broken copy of the file on backup if it would be useful for you to check and you would like me to upload it again.

Thanks as ever for your support

[rolandgifford]

Easier to restore than I expected as it was still in the Google Bin

https://drive.google.com/file/d/1FroJh-UdbfbwhIkZ3u79TF70ja_alsqK/view?usp=sharing

As in my previous reply I don't need to fix my data any more as it is fixed. Only look if it would be useful to you to run a quick safety check or whatever.

[Mario]

I've had a look at the MP4 file, thank you.
Running it in the MDA brings up a few minor warnings:

Code Select Expand

Warning: [Detailed Validation] [minor] Month+Day out of range for EXIF:ModifyDate
Warning: [Detailed Validation] [minor] Undefined value for MakerNotes:ClearRetouchValue
Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0xc6d2 PanasonicTitle
Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0xc6d3 PanasonicTitle2
Warning: [Detailed Validation] IFD1:ThumbnailLength is zero

The EXIF modify date is 0000:00:00 00:00:00, which is surely not a valid time stamp.
IMatch imports the file and sets create date / date subject created to

2022:09:17 11:54:41.112+01:00

applying my local time zone. There is no time zone offset in the EXIF record and neither a QuickTime zone tag, so IMatch applies my local time zone as per my settings.

I give the file a rating, add some keywords and a headline and write back.
ExifTool reports:

1 image files updated
Warning: Month '00' out of range 1..12 in XMP-xmp:ModifyDate (PrintConvInv)

The headline, rating and keyword I've added appear in the file. The write-back was successful.

The [IFD0] Modify Date still shows "0000:00:00 00:00:00", which is why ExifTool displays a warning.
The XMP Modify date was set correctly.
Apparently ExifTool does not update the IFD0 Modify Date.

I switch the Metadata Panel to Browser Mode and change the tags
Composite\Exif-SubSecModifyDate\SubSecModifyDate
Composite\MWG-ModifyDate\ModifyDate to
to the current date and time.

Write back again. The time stamps switch back to 0000:00:00 ... and the IFD0 modify date remains unchanged.
ExifTool reports a successful write-back and the same warning about the invalid IFD timestamp.

When I try to explicitly change IFD0::ModifyDate on the command line, ExifTool fails to do so. It updates all modify time stamps, except the IFD0. I think this is because this is a video file:

exiftool -overwrite_original_in_place -v2 -modifydate=now "E:\data\download\community\13854\AnnArgentina 2022_0917_115441.MP4"
Writing PDF:ModifyDate if tag exists
Writing MIE-Doc:ModifyDate
Writing PNG:ModifyDate
Writing PostScript:ModifyDate
Writing QuickTime:ModifyDate if tag exists
Writing XMP-xmp:ModifyDate if tag exists
Writing IFD0:ModifyDate
======== E:/data/download/community/13854/AnnArgentina 2022_0917_115441.MP4
Rewriting E:/data/download/community/13854/AnnArgentina 2022_0917_115441.MP4...
  FileType = MP4
  FileTypeExtension = MP4
  MIMEType = video/mp4
  Editing tags in: IFD0 ItemList MIE-Doc MOV Meta Movie PDF PNG PostScript QuickTime UserData XMP
  Creating tags in: IFD0 MIE-Doc MOV PNG PostScript XMP
  Rewriting Movie
  Rewriting MovieHeader
    - MovieHeader:ModifyDate = '3786264206'
    + MovieHeader:ModifyDate = '3786264240'
  Rewriting XMP
    - XMP-xmp:ModifyDate = '2023-12-24T12:03:26+01:00'
    + XMP-xmp:ModifyDate = '2023-12-24T12:04:00+01:00'
    1 image files updated

I was unable to find a command that removes or sets the invalid IFD0 timestamp in the limited time I had for this test.
Since it causes only a warning and does not interfere with adding/updating the XMP data in the file, I would ignore it.
Rewriting all data as suggested by Axel will fix this, at the cost of losing other metadata. Your choice.

[rolandgifford]

I was unable to find a command that removes or sets the invalid IFD0 timestamp in the limited time I had for this test.
Since it causes only a warning and does not interfere with adding/updating the XMP data in the file, I would ignore it.
Rewriting all data as suggested by Axel will fix this, at the cost of losing other metadata. Your choice.

Thank you for looking and a very comprehensive reply.

My problem with writing back is that IMatch says that it is Pending Writeback even though writeback succeeded. I usually make several changes then writeback all outstanding, I can't do that if I have 4000+ videos which shouldn't be in the queue.

The command from axel preserves the metadata that I'm interested in and I have used it before. I thought that I had tried it before posting here with this problem but I clearly hadn't.

[Mario]

I have figured this out. I guess

It is caused by the option "Ignore problem XMP tags in embedded XMP" under Edit > Preferences > Metadata 2.

I've enhanced the previous "Ignore embedded XMP Rating" option (which was designed to ignore the hard-coded Rating=None XMP tag several camera vendors now embed) to also include CreateDate and DateSubjectCreated after running into problem caused by Nikon cameras which

a) Embed proper EXIF time stamps and EXIF time zone
b) Embed XMP time stamps which ignore the time zone

For example, EXIF states that the timestamps where at UTC+04:00, but the XMP timestamps are "2023:12:25 12:00:00", which is wrong since the +04:00 is missing. Why? Don't know. Olny Nikon knows.

IMatch by default prioritizes XMP over native metadata. So it used the XMP timestamp instead of making up it's own (correct) time stamp with time zone offset from the EXIF and EXIF time zone.

This resulting in a bug report. After figuring out the problem, I've added CreateDate and DateSubjectCreated to the list of potentially unsafe tags.

And this is what bites us with your MP4 file!

Because, in this particular case, the time stamp and time zone offset produced by ExifTool is wrong (for some reason?). And by ignoring the DateCreated XMP time stamp in the file (when the aforementioned option is on), IMatch picks the wrong time stamp when it re-imports the file after write back.

On the first ingest, IMatch produces correct time stamps for CreateDate, DateSubjectCreated and ModifyDate. And want to write them back. When you do that, the time stamps ion the embedded XMP are correct. During re-import, IMatch ignores these timestamps, however, and favors the EXIF-based time stamp ExifTool produces.

But that time stamp has no time zone offset! And during import, IMatch adds the missing time zone, base on the local time zone and user settings. This then flags the DateCreated for write-back. And after the write-back, the whole thing starts over.
The file remains in "pending write-back" limbo all the time.

Disabling the "potential problem tags" is not a solution for most users. Since it is very likely that your camera vendor embedded a hard-coded "Rating=None" in your RAW files, which would override the rating you set in IMatch.

What I do now, is

a) When IMatch finds CreateDate and/or DateSubjectCreated in the file during ingest, it stores them in memory.
b) When the "potential problem tags" is enabled (default) it ignores these tags as before
c) BUT when it processes the CreateDate and DateSubject created by ExifTool from EXIF, it

c.1) checks if the time stamps in the embedded XMP exists
c.2) checks if these are longer (indicating a time zone)
c.3) checks if the time stamp produced by ExifTool is contained in the saved XMP time stamp
c.4) in that case, uses the time stamp from the embedded XMP record and not the time stamp produced by ExifTool

For example, the time stamps in the embedded XMP record are
A: "2023:12:25 12:00:00+04:00"
and are set to be ignored.

ExifTool produces:
B: "2023:12:25 12:00:00"
for the same file.

IMatch sees that A is longer than B and that B is contained in A
A: "2023:12:25 12:00:00+04:00"
B: "2023:12:25 12:00:00"

and uses A instead of B. Which solves the problem.

I'm still not sure why some of the timestamps in the file use a 00:00 offset explicitly, and some don't.
It's a bit of a mess and ExifTool somehow adds to it in this case.

With the code change explained below, I can now

Import the file
Write-back once
And the pen is gone.

[rolandgifford]

Because, in this particular case, the time stamp and time zone offset produced by ExifTool is wrong (for some reason?). And by ignoring the DateCreated XMP time stamp in the file (when the aforementioned option is on), IMatch picks the wrong time stamp when it re-imports the file after write back.

As is so often the case, I'm impressed by the effort you make.

My guess is that ExifTool used to update these dates correctly. All of these MP4 files have written back successfully in the past with no flag that they are out of sync after doing that.

I expect that I will leave the files as they are with their potentially reduced metadata following axel's data fix as exporting metadata again from IMatch will probably have put back what was deleted by the unsafe option.

[Mario]

Try with the next IMatch release next year. Should be fine.

[axel.hennig]

That's great news, because I've used my suggestion above to fix some (maybe 50) *.mp4 videos which had the same problem. Now I will restore these ~50 files from my backup, try if your solution works and if it does replace the "less metadata" ones (which are currently in my IM-database) with the restored ones from the backup.

Same as rolandgifford: Your support is really great.

[J.D. McDowell]

Wow, thank you!!! I was reading the release notes and wanted to see exactly what the "Enhancement" to "Ignore problematic tags" was.

I'm mainly a Nikon user and have run into this for years. As soon as you mentioned, time zone offsets, it was like a light bulb went on. I could never figure out why only some files in some folders would apparently one day open with bad time stamps. I'm now thinking it was only the ones where I had used a time zone offset in the camera.

Thank you Mario for your expert sleuthing in sorting that out. I'm super happy to see that resolved  

[Mario]

You're welcome.
This is one facet of what I call metadata mess. It's complicated and challenging.

[axel.hennig]

Initially I was also thinking that the problem I had with some *.mp4 files was fixed with IMatch 2023.6.4, but:

Steps to reproduce

• Import the sample file which can be downloaded here: https://c.gmx.net/@329885836172591305/FX0k5gs-QxG6VNJAFpjGUw
• Write-back (which is needed after import)
• Show the following Tag in a Metadate-template {File.MD.XMP::exifEX\CameraOwnerName\OwnerName\0} and write some text to it and save it
• Write-back (which is needed after a value has added in the step before)
• What can be observed: the OwnerName has been added, but an additional write-back is needed
• Write-back
• What can be observed: the OwnerName is empty again

I think after the last write-back the OwnerName (exifEX) gets overwritten by Canon:OwnerName, but I'm not sure.

I think this problem is somehow similar to the initial problem in this discussion. Can this problem also be solved somehow?

[Mario]

IMatch does not deal with these tags directly or in any special way.
Maybe try to fill CanonOwnerName and see if ExifTool maps this to ExifEX::OwnerName?

[axel.hennig]

I tried to fill Canon:OwnerName, but wasn't successful. I did run

Code Select Expand

exiftool.exe -overwrite_original -OwnerName=test vid.mp4
But the result is just:

Code Select Expand

[Canon]         OwnerName                       :
[ExifIFD]       OwnerName                       :
[XMP-exifEX]    OwnerName                       : test

And even running the following does not work:

Code Select Expand

exiftool.exe -overwrite_original -Canon:OwnerName=test vid.mp4
I've also posted this in the exiftool-Forum (because I think the exiftool-manual says that this tag is writeable): https://exiftool.org/forum/index.php?topic=15568.0

[axel.hennig]

Even if I import the above mentioned *.mp4-file containing ExifEX::OwnerName "JohnDoe" (written by exiftool outside of IMatch) it gets wiped out after first write-back within IMatch which is requested immediately after importing.

[Mario]

Probably caused by the XMP => EXIF mapping IMatch during write-back?
I don't see this tag being explicitly mentioned in the ExifTool XMP2EXIF.args file.
But there is an -EXIF:all < XMP-exifEX:all copy operation in the args file.

[axel.hennig]

I got an answer from Phil from ExifTool Forum. Can be found here: https://exiftool.org/forum/index.php?topic=15568.msg83624#msg83624

His answer is

• interesting
• working (both with exiftool and afterwards within IMatch)

[Mario]

I see. Metadata mess again

But that is not something I could do during write-back (extracting the thumbnail to a file, updating that file, re-embedding the thumbnail in the MP4. That would be way to complicated and add even more complexity to an already overly complex process.

You can write yourself a CMD batch file and run it via a favorite perhaps? If you have to use the Owner tag explicitly for some reason.

[axel.hennig]

I already did this, CMD batch and so on...

Wasn't expecting this to be implemented in IMatch (as you said too complicated), just wanted to inform the community about this (interesting) exiftool-behaviour.

I see. Metadata mess again

Not sure if this is true, because initially the Metadata in the Video and in the extracted thumbnail are in sync. It is ExifTool which does not map the one to the other causing the problem.