How I copied metadata from my version files to the master files using ExifTool

[GrantRobertson]

Howdy. I'm pretty proud that I was finally able to figure out how to copy specific metadata from my version files to the master files using ExifTool. So, I thought I would write up a set of instructions, with complete explanations to help other users do the same thing for their particular situations.

I decided to write this whole "article" to give you a feel for what it's like to start with a problem you want to solve, and develop all the parts of that solution in such a way that they all work together most efficiently. It is more of a narrative of my thought process than a specific set of steps to accomplish a specific task.

Background:
My Samsung S7 phone's camera can create raw, .DNG, files in addition to the standard .JPG files. It will only do this for pictures I take using the "Pro" mode. This means that some of the pictures that come off of my phone are in pairs of .DNG and .JPG, while some of the pictures are just .JPG files. Unlike many other raw files, these particular .DNG files look almost identical to the .JPG files, except for the total resolution and file size. So, I wanted to be able to simply delete the .JPG files and only keep the .DNG files, to save space. (Please don't debate me as to whether that is the best idea possible. That is what I wanted to do.) Unfortunately, the .DNG files don't have the GPS data that was in the .JPG files. Therefore, I wanted to copy that GPS metadata from the .JPG files to the .DNG files before deleting the .JPG files. I did some research in the forum to see how I could achieve this. Unfortunately, the only things that were suggested were to manually copy and paste data from one file to another, one... file... at... a... time....  But, I want to do this to perhaps thousands of files, and I want to be able to do it quickly as part of my standard ingestion process.

So, here is what I did instead:

First, I created the Relation Definitions for both the Buddy File relations and the Master-Version relations. This wasn't absolutely necessary, but I wanted to make sure the files were kept together until I was ready to delete the .JPG files. I also wanted this relation to be visible to me within the File Window. But I also prefer to make my Relation Definitions as specific as possible, so they don't accidentally relate files that I don't want to be related. So, open the Relation Definitions dialog by choosing { Commands / Relations / Manage Relations... } from the main menu. Here is the Relation Definition that I used:

[ See the 2020-05-22_Relation Definition.png attachment. ]

The master files (the .DNG files) have 8 digits, an underscore, and six digits, with a .DNG extension. So that is what I specified in the Master Expression. The buddy/version files have identical names, except for a .JPG extension. So the Link Expression uses the {name} "variable" (as far as I know, this "variable" only works in this dialog), followed by the extension of the buddy files, ".JPG". Notice that both expressions use a backslash in front of the period/dot to "escape" the period/dot, because the period/dot character has special meaning within regular expressions. (To "escape" a character means to let that character "escape" from the normal processing. In this case the period/dot will not be processed by the regular expression processor. It has "escaped" from the processor.)

But, I also wanted to make sure that these relations would only work if the files were specifically from this camera. Who knows, I may end up with some other camera that creates similarly named files in the future. So, I also used the "Limit by Variables" feature. I set the Master Expression to be exactly the make and model of the camera, as it shows up in the Metadata Browser, with a space between them for better readability. To get the correct variable names, I simply right-clicked on the field in the Metadata Browser and chose "Copy as Variable." Then I just pasted that into the Version Expression field in the above dialog, putting a space between the variables as well. Here is the text of those variables, because they are too long to show the whole thing in the screen shot: {File.MD.Exif::Main\271\Make\0} {File.MD.Exif::Main\272\Model\0}. Note that I chose these fields from the Exif group within the Metadata Browser. This is because I know the camera writes to the Exif fields rather than to any XMP fields. I want this relation to work even before IMatch has had a chance to copy any metadata to any XMP fields.

I used the same Relation Definitions for both the buddy relation and the version relations. Next, I went through all of the "Versioning" settings in the "Versioning" tab and disabled all of the options other than the "Don't copy..." options, just to be safe. I have found that if you do anything that will cause IMatch to propagate data from masters to versions, that will delete the GPS metadata from the version that I am trying to get into the master.  Later, as part of my ingestion process, I will change the names of the .DNG files, and so I will need a different set of buddy/version Relation Definitions to relate the .DNG files to any subsequent versions I may create. Those versions will have metadata propagated to them as normal.


Filtering to show only the .DNG files of these relations:

Alrighty, after I had the relations set up, I needed a way to show only the .DNG files that were part of this relation. This is because, later, I will need to select all these files so I can run them through the ExifTool Command Processor (ECP).  It is easier to select only a certain set of files if those files are the only files you can see in the File Window.

The first, most obvious filter to set up is to filter based on the file extension. Make sure the File Name filter panel is visible by left-clicking on the triangle to the right of the "Configure the filter panel" gear icon in the Filter Panel's toolbar:

[ See the 2020-05-22_Gear Icon.png attachment. ]

...then make sure "File Name" is selected.

Next, find the File Name filter panel in the, um, "Filter Panel." {Note to Mario: For future reference, it is usually best to not give two different things the same name. Especially if one is a part of another. That makes it confusing for people reading the documentation.} Click the triangle along the left edge to expand the filter. In the first drop-down list, choose "Ends with." In the field to the right of that, enter ".DNG" (without the quotes, naturally). Make sure all three of the options below that are unchecked. Now, check the box to the left of the name of the filter to activate that filter. Next, make sure all the other filter panels are deactivated by removing any checks next to their names. Finally, go all the way up to the Filter Manager panel within the Filter Panel and make sure the top "Filter Panel" checkbox is checked.

But this filter is not precise enough. What if I have other .DNG files from other cameras, or saved from some software, within the scope of the File Window? I don't want those to appear. I only want to see .DNG files from this camera. For the next part of the filter, I used two of the "Value Filter" filter panels. If you cannot see two of these filter panels then use the "Configure the filter panel" gear icon to display them, just as before. For these filter panels, I used the same tags as I used to create the Relation Definitions, except not in variable form. Unfortunately, IMatch does not allow you to paste a tag name into the selection box, so you have to go find it in the tag selector. But first, you need to do two things. You need to go back to the Relation Definitions dialog and copy the two tag variables that were used in the Relation Definitions. Paste them into a text editor or something, just to have them for easy reference. You want to make sure you are selecting the exact same tags as were used in the Relation Definitions. You also need to have at least one of the desired files displayed in the File Window. You will see why in just a second.

Now, expand one of the Value Filter filter panels by clicking on its triangle, just as before:

[ See the 2020-05-22_Make Filter.png attachment. ]

Select the first of the top three options (1). Because we are only going to choose one value from the bottom list, this option will work just fine. Now you have to select the correct tag to filter for. Do not use the drop-down list. It may be tempting, but you cannot be sure that the tag names that are in the list represent the exact tags that you are looking for. Instead, open the standard Tag Selector by clicking on the "Three dot" button to the right of that (2). In the tag selector (not shown), go to the "Search all Tags" tab. In the "Group name contains:" field, enter the major group from the first tag name you set aside previously. (You can only use one word from the long list of family and group names in the full tag name.) In my case that was "Exif." In the "Tag name contains:" field, enter the name of the tag. In my case, that was "make." This will filter the long list above these fields to only show tags that contain those words. Now you have to scroll through and find the exact tag that you used before. In my case, it was "Exif::Main\271\Make."  You will notice that the "File.MD." part is missing from the beginning and the "\0" part is missing from the end. I can only assume that the Tag Selector does not display those parts. Double-click the matching tag name to copy it down to the "Selected Tags:" box, then click "OK." You will see that just the tag name (without the group names) is shown in that drop-down box to the left. The one you didn't click on previously.

Now, you will see a list of all the values for this tag for all the images within the scope of the File Window. The desired value will only show up if one of the desired files is within the scope of the File Window. Select the one that matches the camera (3). In my case, it is "samsung." Finish up by checking the checkbox next to the name of this filter panel, to activate it.

And... do the exact same thing for the "model" tag and the other Value Filter filter panel. Choose only the model name that matches the desired camera.


But wait, there's even more! What if I have already copied metadata to some of my .DNG files from this camera and already deleted their buddy .JPG files? Now, the way I set up the ExifTool command in the ECP (ExifTool Command Processor), ExifTool will skip over .DNG files that don't have matching .JPG files (yes, I'll get to that part eventually), however, things will go faster if ExifTool doesn't have to bother to look for those missing .JPG files. Remember, there could be thousands. For this part of the filter, I made use of the fact that I had already set up version relations for these files. If there is no .JPG file to go with the .DNG file, then the .DNG file is not considered to be a Master. So I used the "File Properties" filter panel to show me only the files that are master files. Again, if that filter panel is not showing, use the gear icon to display it.

In the File Properties filter panel, check only the "Hide masters" option and the "Invert filter" option. This will cause the filter to only show files that are masters. And, of course, enable that filter panel by checking the box next to its name.

Remember, all these options from all these filter panels are put together into a Boolean equation with ANDs between each of them. So, only files that match all of these criteria will be displayed. In this case, it will be only the .DNG files from this camera that still have version files which haven't been deleted.

After going to all this work, you don't want to have to recreate that filter from scratch every time. So just save that filter by clicking the "Save the active filter into an existing or new filter" button:

[ See the 2020-05-22_Save Filter.png attachment. ]

...and enter a new name. I called mine, "Samsung S7 .DNG Masters." Now, as part of my ingest process from this camera, I will go to the ingest folder (I use a folder named "Samsung S7"), Enable this filter, use Ctrl-A to select all the displayed files, and then run the ECP preset which I am sure you are dying for me to tell you about.

... Continued in the first comment ...

[GrantRobertson]

The ExifTool Command Processor preset:

Open the ECP (ExifTool Command Processor) window { Main Menu : Tools / ExifTool Command Processor... }.  The first thing you want to do is save/create a new preset. This way, you won't accidentally overwrite an existing preset. Simply select the text of the existing preset name in the "Presets" field at the top of the dialog, and type your new name, then click the "Save" button. This will save a copy of whatever was in the "Arguments:" field under that new name. I named mine, "Fix S7 DNG." You don't need to bother including the word "metadata" in your preset name. After all, this is the ExifTool Command Processor. What are you gonna be fixing other than metadata?

Now, select all the text in the "Arguments:" field and delete it, then replace it with the following:

Code Select Expand


# im-warn

# Fix Samsung S7 DNG
# Copies GPS data from .JPG buddy file so the .JPG can be deleted.

# Make sure only .DNG files are selected by using the
# "Samsung S7 .DNG Masters" filter in the Filter Panel.

-overwrite_original_in_place
-charset
filename=UTF8

-tagsFromFile
%d%f.JPG
-GPSLatitudeRef
-GPSLatitude
-GPSLongitudeRef
-GPSLongitude
-GPSAltitude

{Files}

(Don't forget to scroll down and copy all of the code.)


Then make sure the "Run once" option, at the bottom left, is chosen. Even though you can run this command on multiple files at once, ExifTool can process them all in one go because of the features of the -tagsFromFile option. This makes the command run relatively fast, compared to restarting ExifTool once for each selected file. Notice that the last line in the command is the "{Files}" variable that IMatch provides. That will cause IMatch to pass the full pathnames of all the files that were selected when the ECP was run. You will see how this comes in handy later.

Naturally, you will not want your ExifTool command to do the exact same thing as mine, so you will need to edit it some. Here are some tips:
   1. The lines that begin with # are comment lines. You can only put a comment on a line by itself. You cannot stick a comment on the end of a line.
      ○ Notice how I included reminders and instructions to myself, telling me what the command is supposed to do and what to do before I run the command.
   2. If the first line contains the "# im-warn" comment, that tells IMatch to put up a dialog box making sure you really want to modify your metadata. If you are brave, you can leave this out.
   3. Because this command will modify the file(s), you need to include the "-overwrite_original_in_place" option or ExifTool will make a copy of the file and modify that, which will confuse IMatch.
   4. Every separate part of the command needs to be on a separate line, even if it really goes with the line above it. For instance, "-charset filename=UTF8" is actually one option, with its argument. But you still need to put the option on one line and the argument on the next.
   5. The "-charset filename=UTF8" option ensures that this command will work even if the filenames include UTF8 characters.
   6. The "-tagsFromFile" and "%d%f.JPG" lines tell ExifTool to copy tag data from the specified file. But how does this work?
      ○ Notice that I used the "{Files}" variable at the end of the command. This passes the full pathnames of all of the selected files to ExifTool when the command is run.
      ○ ExifTool has a handy feature as part of the -tagsFromFile option that allows it to parse out those pathnames into the directory, filename, and extension parts. As ExifTool processes each file in turn, it will take its directory name, filename, and extension and use them to allow you to specify slightly different filenames from which it should copy that metadata. The "%d" is replaced with the directory of the file, the "%f" is replaced with the filename without the extension, and "%e" indicates the extension of the original file but without the period. So, when I specified "%d%f.JPG" what I did was tell ExifTool to grab the metadata from an identically named file in the same folder, except with a .JPG extension instead of a .DNG extension.
      ○ This allows ExifTool to create its own batch to process all the files in one go.
      ○ Now, it is also possible to use variables provided by IMatch to specify both the main file to be operated on as well as the file used by the -tagsFromFile option. However, then you could not use the "{Files} variable to pass all the files at once, and you would have to select the "Run for each file in selection" option in the bottom left of the ECP, which would be slower. So, if at all possible, make use of the batching feature of the -tagsFromFile option rather than the batching features  provided by IMatch.

Now, how did I figure out what tags to copy from my .JPG files to my .DNG files? I used the "List Metadata" preset to list all the metadata in one of my .DNG files and the same for its matching .JPG file. I copied the results of each into a separate document in my text editor. I use Notepad++. Then I used the "Compare" plugin in Notepad++ to show me all the lines that were different. Both Notepad++ and the Compare plugin are free. When I ran the compare, I left the option to "Show moved lines" enabled. This is because each file may have identical metadata in identical tags, but those tags may be stored in a different order in different file formats. So they show up in a different order in the metadata list. Then I just looked for lines in the .JPG that were not in the .DNG. I ignored lines for things like modify date, and things that were likely to be specific to the .JPG file format. As it turns out, the only metadata that was not in the .DNG files was the GPS data. Just enter each of the tag names, with a dash as a prefix, on a separate line after the -tagsFromFile option. Only tags listed after the -tagsFromFile option will be copied from the other file.

So, I just listed each of the GPS metadata tag names, as they appeared in the metadata listing. I did not use the specific group names. This allowed ExifTool to choose which was the prefered groups in which to place this metadata. Remember, the goal was not to make the .DNG files identical to the .JPG. Only to make it contain the same information. I trust ExifTool to know where is the best place to store it.
      
After you run the command, ExifTool will tell you how many files it updated. However, Imatch won't know about these changes until you close the ECP and Imatch updates the files, which it does automatically, after you close the window. If you had a lot of files selected, it may take a while for imatch to update all the files. When it is done, all the files will still be selected. This comes in handy for the next step.


Deleting only the .JPG files that are versions of the selected .DNG files:

While all the files are still selected, left-click the Versions Indicator on any one of the files. This will pop up the Versions context menu. (If you right-clicked, you will get a different context menu.) Choose "Show all versions..." Now click somewhere in the Results Window to make that window active. (Note to Mario: I think this is yet another lost focus bug. The window should be active by default.) Then press Ctrl-a to select all the files. (Ctrl-a won't work unless you have first made the window active.) Notice that only the .JPG files are selected. Perhaps this was part of the plan, but in a "Show all versions" results window, you cannot select the master files. Well, now you can delete all the selected .JPG files and not worry about deleting any that aren't really a version of a .DNG file.


Summary:
Once you have this all set up, all you will have to do is go to the folder that contains the files you just copied from your camera, turn on the appropriate filter, select all the files that are now visible, run the ECP preset, choose "Show all versions," and then select all and delete those versions from the results window. Then just close the, now empty, Results Window. You should be able to do the whole thing in less than a minute. Just don't forget to turn off your filter when you are finished. Otherwise it will look as if you deleted all the files in your folder.

I have tested this procedure with the Relation Definitions enabled. It works perfectly. This is because ExifTool modifies the master files independently from IMatch, and then IMatch updates itself only after the files are modified.

[Mario]

Thanks for sharing 

[sinus]

Thanks for sharing 

+1
Thanks!

[thrinn]

Thanks for putting in so much effort to writing this up! While I do not have a requirement at the moment to use the ECP in a similar way as you, I immediately bookmarked your post for future reference. Because I can easily imagine that I will run into similar problems when I will deal with smartphone images. Your very detailed explanation of not only what you did, but also why you did it, will surely help!

[Jingo]

Thanks for sharing 

Seriously... many thx.. this must have taken a good chunk of your day!

[GrantRobertson]

Thanks for sharing 

Seriously... many thx.. this must have taken a good chunk of your day!

At least a week, off and on, to figure out. Then two full days to implement and write up. I proofread and edit a LOT.

I eventually plan to start making YouTube videos.

[jch2103]

Well done! Very complete description that should benefit others here.

[ubacher]

I have written a script to fix this lack of GPS data in Samsung dng files.
I run it at the end of a renaming script which I use.

PS: I have since switched to using the Lightroom camera app for taking pictures. It only
produces .dng files and these do have gps data.
Samsung dng files only show COLOR and MONOCHROME profiles in the ACR while the Lightroom produces dng files which do show a range of Adobe Profiles to select from.
I have never able to figure out what metadata causes this different behaviour inside ACR.