Propagation of data from master to version VIDEO files

[DaweP]

PROBLEM:
I need to take the data date/time and used lens from master video files and automatically transfer these data to derived version video files (created by external video editor).

CONTEXT:
Video files (*.mp4) from Nikon Z6, typically tens of seconds of up to two minutes long, are cut in DaVinci Resolve to remove unwanted parts (to reduce of HDD requirements and facilitate later search and use). No color grading, no effects, just cutting. Then the files are rendered in DaVinci Resolve as one (or more) "shorter" clips and named according to my convention: ####_VZ6_####_C01.mp4 (or _C02, _C03, etc.). Once indexed in IMatch, these automatically become VERSIONS of the original MASTER file (temporarily stored alongside, to be deleted later). These version clips have DATE and TIME stamp of their origin that reflects rendering by DaVinci Resolve. I would like to replace these data by DATE and TIME corresponding to when the original video was filmed and I want these values to be displayed in IMatch (for obvious reasons, I guess).
Unfortunately, my efforts to automatically propagate date/time from the masters to the versions was not successful. I tried several possibilities of "What to propagate" checklist, but even the ultimate "XMP, All data" does not work. Perhaps some of you could guide me how to better set the propagation for this purpose.

CURRENT (suboptimal) SOLUTION:
I use the "TimeWiz" function in IMatch to manually correct the timestamp in the clips. As this has to be done one-by-one master file and involves complex dialog and a lot of mouse-clicks, it becomes really time-consuming if I have to process more files this way.

Further, I tried to use "Metadata Mechanic" in IMatch to edit Lens data (what lens was used for filming that clip) by manually modifying the content of:
photools.com::IMatch\2900\lens\0
- this field is empty in the version clips. As I can create presets for several lens used, it works fine and is relatively fast... But when I save the metadata (yellow pencil -> metadata write-back), the new entered values are removed and the field is empty again. Probably this field is read-only and cannot be modified.
Is there any solution for this?

I know that dealing with metadata is really a mess.  But I dare to hope that somebody can help me with this. Actually, all I want is to copy content of two fields (tags) from master files to version(s) files and store these. The relation between masters and versions is easily established (thanks to my naming conventions), so this should not be that difficult. But I cannot resolve it myself.
Help/advice is highly appreciated!

David

[Mario]

Wow, that's quite a particular problem you have there...

ExifTool can write XMP data to MP4 files, and so can IMatch.
Do these MP4 files contain EXIF metadata? Do they contain XMP data written by your other software?
Which tags did you try to propagate and what was the result?
All XMP comes to mind, since it contains the XMP timestamps. If this does not work, something is problematic with the MP4 files or they contain conflicting metadata or...
Probably a time-consuming, complex and 'costly' analysis would required...
Did ExifTool report any problems in the ExifTool output panel while you were writing the files?
If all else fails, the ExifTool command line is a very powerful tool: https://exiftool.org/exiftool_pod.html

Further, I tried to use "Metadata Mechanic" in IMatch to edit Lens data (what lens was used for filming that clip) by manually modifying the content of: photools.com::IMatch\2900\lens\0

This is a synthetic tag that IMatch derives from about a dozen other lens-related tags ExifTool delivers for different image formats. The tag solely exists to offer IMatch users the "best" lens name that's available for a particular file format. You cannot change this synthetic tag, it does not exist in images or videos.

Lens data typically consists of 4 or more numeric fields inside the EXIF record. ExifTool uses internal tables and algorithms to convert these numbers into a user-readable "lens name".
Quite often camera and lens vendors reuse the same lens numbers for different lenses, which complicates things even more.
You can write lens data directly with ExifTool on the command line (also in the IMatch ECP), employing the reverse lookup magic ExifTool provides and when a) you know what you are doing and b) ExifTool can write the target format (which is always tricky for video formats).

[DaweP]

Hi Mario, thanks for the reply.
The *.mp4 files produced by DaVinci Resolve likely contain some metadata, but I cannot report more to this with confidence right now (I am out of my PC till tomorrow evening). I use default export settings so I guess at least some metadata should be there. And as this program belongs to well-established video editing tools, I hope the files should have correct metadata, as opposed to corrupted in whatever way. Perhaps this assumption is naive, though.
Second, while I tried several options "what to propagate" from master to version video file, it never led to the desired result - i.e., to change the date/time to reflect that of the master file. As it remained the same, there was no need to "save" the metadata, hence I cannot report what the ExifTool's output panel message was. The target file simply remained unaltered after the command to propagate the data. Just for a record -  I use propagation of rating, keywords and categories regularly for my photo files without any issues.
As you repeatedly emphasized in past, metadata are mess. And if I am correct, numerous tags hold values for date and time in EXIF, IPTC, XMP... What is best to choose as a data source for display with the thumbnails? I guess I sticked to IMatch's default, but perhaps some other tag would be better in my case... (for thumbnails view optimized for video files).

Concerning the lens data - OK, I understand why I cannot modify the synthetic tag. But again, if I managed to copy the "lens code" inside the respective lens tags from master video file (where it IS displayed in IMatch) to the version file (where it is empty), it should be displayed correctly, I would say. So this brings me back to the original - theoretically simple - task: To copy a handful of metadata data from master file to version file.

If possible, I would link (tomorrow evening) some sample short video files, one out of the camera, second modified in DaVinci Resolve, so that you (or whoever is interested) can analyze the contained metadata and/or to propagate date/time as well as lens code to the version.

Thank you!
David

[Mario]

Keep in mind that propagation is performed when you change the master, or when you perform a manual propagation via the menu of F4,P.
Changing the propagation option does nothing, it only affects what will be propagated in the future.

[DaweP]

Sure, I know and by "I propagate" I actually mean the command F4, P.

[jch2103]

The *.mp4 files produced by DaVinci Resolve likely contain some metadata, but I cannot report more to this with confidence right now (I am out of my PC till tomorrow evening). I use default export settings so I guess at least some metadata should be there. And as this program belongs to well-established video editing tools, I hope the files should have correct metadata, as opposed to corrupted in whatever way.

I did a quick test to confirm what I thought: Export of a Nikon Z6 .mov file from DaVinci Resolve 18 contains very little metadata, most of it having to do with Quicktime video metadata, but essentially no metadata from the original .mov/.xmp file. I haven't tried using propagation yet, so I'm interested in what you discover. What I've been doing is to copy .xmp metadata that I've added to the original clip from the camera to the Resolve output file. I don't do much video, so it's not a lot of extra work for my few files. My biggest pain point is getting date/time data from the original to the Resolve clip (this makes sense in my limited video workflow, but I can understand that someone who's doing a more complicated video production wouldn't want to do this).

[jch2103]

ps - I see that Resolve 18 (and maybe earlier versions) has its own metadata management functions https://www.blackmagicdesign.com/ca/products/davinciresolve/media

I haven't dug in, but these seem to be designed for managing video production, so really a different purpose than IM metadata management. (Another walled garden, but perhaps makes sense in this video production context?)

Also, attached is an ExifTool dump of metadata from a Resolve clip.

[Mario]

The metadata in the attached file looks like the standard Apple QuickTime format (in one of its many variants). QuickTime is write-once only.
I wonder where DaVinci Resolve stores the metadata they explain about in the link you've provided. I hope as XMP in the video file so it can be accessed by other software. IMatch will pick up XMP automatically.

[DaweP]

Here is a Dropbox link to download ZIP archive with 2 video files:
https://www.dropbox.com/s/1tpu3pj25mnao5r/2_vid_files.zip?dl=0
0843_VZ6_8893.MP4 is original file produced by Nikon Z6 (6 seconds long). Date/time is today (7th Aug 2022), 18:43 - exactly the date/time I want to be shown. It also shows that the lens in use was Nikon Z 40mm f/2. This is the MASTER file in my system.
0843_VZ6_8893_C01.mp4 is a 3 seconds long "cut" from the above, rendered in DaVinci Resolve 18. It is a VERSION. As this file was rendered today, the date is the same, but the time reflects time of its rendering (18:51). This is technically correct, but not what I would like to see. Further, it shows no data for lens used.

Detection rule of file relations is set as follows
^[0-9]*_VZ6_[0-9]*\.(MP4|MOV)$
/^_*//
^{name}(_C[0-9])*.*\.(MP4|MOV)$
...and is working well.

The goal of the game is to propagate date/time of filming as well as the lens used contained in master file's metadata to the target (version) file's metadata and visualize these in IMatch.

[jch2103]

The metadata in the attached file looks like the standard Apple QuickTime format (in one of its many variants). QuickTime is write-once only.
I wonder where DaVinci Resolve stores the metadata they explain about in the link you've provided. I hope as XMP in the video file so it can be accessed by other software. IMatch will pick up XMP automatically.

After further digging, Resolve metadata is apparently contained in a Resolve database, not in video file-specific xml. I believe documentation of the Resolve database is available, but my quick look didn't turn up a reference to Resolve 18's database specifics. Not helpful to the OP, I think.

[DaweP]

Please, if anyone from the group of experienced users would download the two files (described earlier) and finds some way how to propagate date/time and lens metadata from one to the other, it would really help me. Thank you.
https://www.dropbox.com/s/1tpu3pj25mnao5r/2_vid_files.zip?dl=0

[Mario]

I tried it and it worked instantly.

Keep in mind that you work with video file, which have a much bigger metadata mess than image files.
For some reason, you want to fiddle with that mess and prop timestamps on top of temporary video clips produced by daVinci.
I guess this is the reason why you won't get much support for this from the community. I doubt anybody ever tried that or saw a need for time-stamping temporary media clips.

Anyway, here is what my analysis shows.

1. The original video has no EXIF data (naturally) and only a small XMP with a minimum of metadata. No timestamps in XMP => your problem!

2.  When IMatch encounters an image without EXIF data, it does some clever things to figure out a suitable timestamp. I have described that in details in How IMatch uses Date and Time Information

3.  This means you end up (in most cases) with XMP timestamps in the Metadata Panel. In the case of this file, IMatch used timestamps from the Apple QuickTime tags contained in the file.

4.  I guess you have disabled the option Edit > Preferences > Metadata 2: Mark files as pending write-back so IMatch does not mark the 'produced' XMP timestamps as pending.

5.  When you propagate data (I used XMP all data as I suggested above), IMatch thus does not write the timestamps into the master and so no timestamps can be copied from the master to the version.

6. Solution: Mark the two timestamps as pending for the master (click the pen in front of the tags in the Metadata Panel).
Now write back the master. IMatch will write the two XMP timestamps created from QuickTime into the master file, and then propagation will copy that data into the versions.



7. Alternatively, select the master and all versions (master last) and then click the pen icon in front of the tags in the MD panel. This will copy the data into all files. No propagation or complex versioning needed at all.

Ps.: Lens Data

The lens data is contained in proprietary Nikon maker notes in the original file. Yes, Nikon still continues to use that crap in 2022..
You'll have to figure out how to copy these maker notes safely yourself, I won't spend time on that. Learn the ExifTool command line and ask on the ExifTool forum if this is possible at all. Knowing ExifTool, there is maybe a way, but I cannot spend so much time on one particular problem for one user, sorry. This is nothing that will help more than one user, and my time is very limited.

[DaweP]

Hi Mario,
Thank you, I appreciate your support. I will go through your steps and hopefully resolve my unique issue.
Throughout the years, I appreciated enormous flexibility of IMatch to fulfill all variable needs of a wide range of users. I am not a professional filmmaker and thus my needs are most likely different from the others. My current need for handling video files might seem strange to you, but is it really that weird? I mostly film some clips of my young daughter. From these clips, I cut/render those "moments" that are worth keeping for later use - i.e., these are then put together to make some clips worth seeing. Is it that strange to want to know date/time when it was filmed?
Again, thank you very much for your assistance. Consider this topic closed and dedicate your time to other things that need your attention, not to this minority report... 
David

[DaweP]

Just wanted to add this for possible reference of other users:
Mario was right (as always) - for some reason, my setting in...
Edit > Preferences > Metadata 2: Mark files as pending write-back was set to No.
Once I switched it to Yes, the imported video file had indeed indicated non-written data. I was then able to follow the suggested steps and synchronize the time stamp.

As expected, one single setting, overlooked among so many, caused these complications.
I hope this would be useful to the other users