How does background relation refresh and propagation works?

[cytochrome]

With V110, lang Neutral, Metadata 2 : no sidecars, write mode 1, Background : no immediate write back, relations refresh and propagation ON, a small NEF folder without metadata and a JPG subfolder:

On the NEF folder:

- select all, enter metadata (Author, Description,Headline, Location, Rating)
- commit ==> some activity, the pencil appears in all NEF
- click one pencil ==> activity ==> metadata is written in the NEF, xmp and iptc (lookup with ExiftoolGUI)
- nothing happens in the JPG sub folder (the relations are setup and tested since some weeks)
- manually refresh relations for all NEF ==> some HD activity, the relation icons show up for NEF and JPG but the metadata is not written to jPG
- wait 3 hours with PC idle, no competing program activity ==> nothing happens
- manually propagate with F4,P the metadata panel for the JPG the xmp and iptc fields are populated (with Author in triplicate), the metadata is written to the corresponding xmp and iptc fields in the JPG, including the triplicate Author (this is not always the case, sometimes it is triplicate only in the IM DB)

If NOW I make a change in one of the NEF metadata followed by Ctrl s AND a click on the pencil the metadata is immediately propagated WITHOUT having to refresh relations or activate F4,P

I don't get the rationale of all this.

Francis

[Mario]

- manually refresh relations for all NEF ==> some HD activity, the relation icons show up for NEF and JPG but the metadata is not written to jPG

Refreshing relations does not cause metadata to be written. It just applies your current set of relation rules to the files in the database.
Data is only written to versions when you make changes to the master and write them.
Or when you use the "Propagate to Version" command.

If NOW I make a change in one of the NEF metadata followed by Ctrl s AND a click on the pencil the metadata is immediately propagated WITHOUT having to refresh relations or activate F4,P

Logical. You have now your relation rules in place and they are carried out.

[cytochrome]

This means there is no difference whether "Automatically propagate data from master to versions" is ON or OFF ?

Francis

[Mario]

When you set this setting to off, no data is propagated automatically.  It should always stay ON.

In your example you changed relation rules and updated the database so IMatch established links between masters and versions.  This process will not trigger write-back or metadata propagation. It just creates links between masters and versions.

When you make changes to master files and trigger a write-back, IMatch checks for versions and then carries out the propagation rules you have defined in your relation setup.

If you make changes to your relations and you want to propagate data from the master to the versions immediately, use <F4>,<P>. This starts a write-back of the master and all the versions - to make sure that the data in these files on disk is current. It then reloads the metadata into the database to ensure that the database is up-to-date. And then it runs the propagation rules to copy data from the master to the versions on disk.

[cytochrome]

OK, thank you. It makes sense, simply I forget often that what the metadata panel shows may not be in the image files.

So, Automatically propagate data from master to versions happens in the database, this is mirrored in the image files only after F4,P.

Francis

This is why I had asked for an icon or letter to let us know if the metadata panel reflects the state of the database, of the image files, or both.

[Mario]

So, Automatically propagate data from master to versions happens in the database, this is mirrored in the image files only after F4,P.

Propagation happens at write-back time. IMatch delegates copying and merging tags and files between metadata standards to ExifTool, because only ExifTool knows all the details, and in many cases data propagation from master to versions can only be done after the data in the master has been written to disk.

When you update tag X in the user interface, this tag may be copied into three or more other tags when IMatch updates the file on disk, depending on whatever MWG rules ExifTool has implemented. The display of the metadata panel shall be considered with caution until the data has been written to the file - if you show tags from multiple metadata formats.

For example, if you show XMP description and IPTC Caption. IMatch only allows you to edit the XMP caption because what you fill in there will be copied into the IPTC Caption at write-back time. Until then, the IPTC Caption will be empty or show old content.

The same is true for versions. Until the write-back has been completed, they will not show correct data. When you tell IMatch to copy XMP data from the master to versions, IMatch does this as follows:

1. When the write-back runs, IMatch first flushes all changes from the database into the master file. It does only write out updated or deleted tags in order to keep the metadata in the master file as complete as possible.

2. When IMatch sends instructions to update tag "A", ExifTool may in addition update tags "B","C","Y", "Z", e.g. to apply MWG rules, to update digest data, timestamps etc.

3. If the master file is complete, IMatch performs the write-back to all version files.


4. Then IMatch looks at which tags need to be copied from the master to its versions (e.g. all XMP tags). IMatch produces instructions for ExifTool from these rules and runs them. This ensures that ExifTool stays in charge of all writing and copying, which guarantees optimal data quality and conformance.

5. Finally, IMatch re-imports the current metadata from all written-to files into the database. This step ensures that IMatch shows the correct data for all tags which have been changed during the write-back - which are usually many more tags than IMatch has intentionally written.

At this time, the metadata panel shows exactly the data that is in the files.

(There is also a section on this in the help, with more details). Search for How IMatch writes Metadata.


Note 1: IMatch does not attempt to perform metadata migration between standards itself. The rules are complex and I would have to rebuild (and maintain!) the same rules ExifTool has implemented. Impossible.

Note 2: IMatch makes exception to the above for certain tags: Rating, Label and Keywords because these tags are so frequently used and it would cause to much confusion otherwise.

Remember: Data is correct after the write-back has completed.

[Ferdinand]

Propagation happens at write-back time.

Surely this doesn't apply to everything, e.g. pins and bookmarks?  Even ratings and labels are propagated prior to write-back, aren't they?  There must be a line between what needs writeback for propagation and what doesn't?

[Mario]

Pins and Bookmarks are an internal IMatch concept. They are not stored in the metadata of your files and thus don't require ExifTool to be propagated.

See my explanation above again. I mentioned that some frequently used tags (rating, label, and keywords) are migrated by IMatch in the database to avoid confusion. But that's it.

If you look again at the Relation dialog box again, notice the * behind most of the metadata attributes, and the meaning of the * below that box:



[attachment deleted by admin]

[cytochrome]

Thanks a lot Mario for the detailed explanation. It should figure in the Help  either in the Metadata Write-back or the metadata Storage chapters. It is a more complex mechanism than I thought, particularly the final reimport from file(s) into the DB. Is it then that the "Merge xmp from sidecar and raw" is done?

Of course most people wont care, but those who watch the Metadata panel during versioning and write-back and see sudden changes of certain fields will have a chance to understand what happens and why. I was puzzled more than once.

A propos Ferdinand's question, I think the "* indicates on-write" attibutes" is misleading. I had interpreted it as "within say All Metadata" only those that Exiftool will/can write will be updated. In fact it should read "will be 'really' propagated only at write-back.

I wonder how many users are clear on the fact that DB and files are often out of synchrony and that it depends on  various settings and actions.

Francis

PS As an after thought : is a graphic schematic of what happens to the metadata in DB, sidecar and image file from import of the image to a stable state (everything at its definitive place) possible?

[Mario]

Is it then that the "Merge xmp from sidecar and raw" is done?

This is always done when metadata is read, whether when a file is added to the database, updated after a change on disk or after write-back.
ExifTool has a very complex logic which fields are copied where between standards, and this can change in every ExifTool version or when Adobe or the MWG come up with new ideas.

You will only see a mismatch in the metadata panel when you a) use versioning and b) don't let IMatch write changes immediately. Since writing to files, especially with large files, is a time-consuming process, IMatch gives you the choice. If you work with versioning and tend to forget that files marked as "pending write-back" may not reflect the actual metadata in the metadata panel yet, enable the background write-back under Edit > Preferences > Background processing.