Metadata Write-Back - No Files

[StanRohrer]

I try to Write-Back Metadata for All Pending Files and iMatch thinks there are no files to write back.

- Opened iMatch.
- Copied folder to a junk (test) folder location.
- Showed junk folder file data to be written back.
- Attempted Write-Back of all Pending Files. [No files with pending data].

It seems the folder copy is adding metadata to the files related to document identifications. But the Write-Back command is not seeing the files as having data to write back. I think I can repeat this at will using source folders and files originally added to my database last October. Imatch has had a major (paid) update since then as I recall. Since then I see this problem on occasion. If I select a file, or a group of files, the write-back will work by clicking the pencil icon.

[Mario]

How did you copy files to junk folders? In IMatch? Externally?
Which write-back command did you use?
What does the pending write-back collection show?
When you point the mouse cursor at a pen, which tags are listed for write-back?
Does the database diagnosis find any problem?

[StanRohrer]

- How did you copy files to junk folders? In IMatch? Externally?
In IMatch. Right Click on the source folder (in Media & Folders) and select Clipboard / Copy. Right click on my Junk folder and select Clipboard / Paste. Click on the newly minted folder and view the files.

- Which write-back command did you use?
Menu / Commands / Meta-Data Writeback / For All Pending Files.

- What does the pending write-back collection show?
Pending Metadata Write-back (0).

- When you point the mouse cursor at a pen, which tags are listed for write-back?
Image attached to the first message. A Debug level Log is attached to the first message for that days session.

- Does the database diagnosis find any problem?
The Diagnosis Completed Successfully.

Working in IMatch 2021.14.4.

[Mario]

Please use quotes to make your post better readable (toolbar button).

When the collection tells no files to write-back and the write-back command tells "Nothing to do", it seems correct. No files to write-back.
When you still see pen icons in the File Window in this situation, point the mouse cursor at a pen in a thumbnail panel so we can see which tags are marked as write-back.

[StanRohrer]

Please use quotes to make your post better readable (toolbar button).

When the collection tells no files to write-back and the write-back command tells "Nothing to do", it seems correct. No files to write-back.
When you still see pen icons in the File Window in this situation, point the mouse cursor at a pen in a thumbnail panel so we can see which tags are marked as write-back.

Attached is the image (from the first post) that shows the 5 items of pending writeback data.

[Mario]

Interesting.
When you close and repoen the database, does the Pending Write-back collection update to show these files?
Does the database diagnosis find anything?

[StanRohrer]

Closing and reopening IMatch does not seem to change the status. Collection "Pending Meta-data- Write-back" still indicates zero.

As noted above, Diagnostics pass (Complete Successfully). When I close IMatch I almost always run a Backup. The Backup checkbox for Database Diagnostics is always checked. So my diagnostics are run almost daily and some days multiple times. No errors.

[StanRohrer]

Copy and Paste of a file was observed again today. A .PSD file Clipboard Copied and Pasted and Write-back All Pending worked as expected. Doing a Clipboard Copy and Paste of a .JPG file resulted in data to write (per the pencil icon) but the Write-Back All Pending showed no files available to write-back. Clicking the pencil on the file indeed completed a write-back and cleared the pencil. Again, the Write-Back status was not properly conveyed through the iMatch system. I think it perhaps notable that the PSD file works as expected and the JPG file did not.

iMatch 221.16.4 x64

[Mario]

Is this repeatable on your system?
I wonder why the PSD works and the JPG does not? Internally, it's all the same.
Do you use versioning and propagation between PSD and JPG?
Is there another file with the same name as the PSD/JPG in the same folder, using an XMP sidecar file?
Did IMatch log any warnings or errors in the log file?

[StanRohrer]

Is this repeatable on your system?...
I wonder why the PSD works and the JPG does not? Internally, it's all the same. .....
Mostly yes. My attempt just now had problems with both the JPG and the PSD. Unlike earlier in the day.

Do you use versioning and propagation between PSD and JPG? ...
No. I tend to work on a PSD file and then save the JPG from Photoshop as a new file. iMatch discovers it as a new file.

Is there another file with the same name as the PSD/JPG in the same folder, using an XMP sidecar file? ...
There are indeed 2 files the same file name but using extensions of PSD and JPG. There are no XMP sidecar files in the source or destination folders.

Did IMatch log any warnings or errors in the log file? ...
Debug Log file attached. Open IM. Set debug. Copy/Paste 2 files. Attempt Metadata Write-Back All Pending (reports none pending even though pencil icons are visible).

[Mario]

I've tried to reproduce this as follows:

1. Add a folder to a database. It contains two files.
2. After import, no data needs to be written back (the files were already processed by IMatch).
3. Create a new folder.
4. Copy one of the files into the new folder.

This adds the Metadata Creator Tool to both files (source and target, because it was not set yet) and it also adds the XMP document tracking (derived from etc.). OK.
The "Pending Writeback" collection updates and shows two files. OK.
I use Commands menu > Writeback > For all pending files  and it asks me if I want to write back 2 files. OK.
I use Commands menu > Writeback > For all selected (in the folder with the copied file) and it writes back the selected file. OK.
Pending Metadata Writeback Collection updates and now shows only 1 file. OK.

1. I delete the new folder and remove the original folder from the database.
2. I add the folder again. Two files. No pending write-back. OK.
3. I copy the entire folder with Ctrl+C, Ctrl+V in the Media & Folders View.
4. The two files in the new folder show the pen icon and the XMP document... tags added. OK.
5. Pending Metadata Writeback Collection shows 4 files (the source files have got their Creator Tool added, and the copied files all the XMP document tags). OK.
6. Commands > Metadata Writeback > All Files shows 4 files to write-back. OK.

Seems to work just fine. I've tried the same when you first posted your report and it worked then as well.

I must do something different or in a different order. Or it just fails with your specific database, for some reason?
Can you also reproduce this when you create a new database and add a few folders?

The uploaded log file is clean, no errors or warnings or unusual entries.

[StanRohrer]

I did your above steps 1-4. The "source" files did not get a pending writeback pencil. The one destination file did get a pencil. Commands > Write Back > All Pending shows no files to write back (I expect to be 1)

Collections > Pending Writeback = 0 (I expect to be 1).

It's been many years (20?) since I created a new iMatch database. I'll report in next message.

[StanRohrer]

New database. Imported test folder with 2 files. Created subfolder. Clipboard > Copy 1 file from top folder to Clipboard > Paste into second folder. Pencil appeared on single file in second folder. Commands > Writeback > All shows nothing to write.

Commands > Writeback >Selected did indeed work and removed the pencil.

[Mario]

I cannot reproduce this using your steps, sorry. Just tried again.

[StanRohrer]

Mario, are there preferences/settings that are common to both my working database and the new created database that I should play with? Maybe something that would provide indirect clues?

[StanRohrer]

Using folders outside of my normal photo library and the new small/empty database, I did some work with a larger group of files to see if I could see anything of interest. My test file set included files directly from camera, files edited and/or created by Photoshop. Files created by some other effects programs.

In Windows file manager I grabbed 340 files from a working photo folder and pasted them to a new test folder (to be the source folder). Imported into the fresh IM database from the new test folder. All looked well and as expected in iMatch.

In Windows file managerI created another test folder (to be the destination folder). In iMatch, imported this empty folder. All as expected.

In iMatch I went to the source folder and copied all 340 files and pasted to the destination folder.

Curiously some (not all) of the source files in the _source_ folder now got a pencil icon. The mouseover indicated the files needed to writeback: XMP::xmp\CreatorTool. This may or may not be correct operation, I don't know enough about XMP operations – but I find it curious that the IM Copy/Paste routines change data in the _source_ files. Is it correct to be modifying the source files with XMP data?

In the destination folder I reviewed files and found many with pencil icons. All indicated files had 5 XMP DerivedFrom type fields as displayed in previous messages. Some of these also had the XMP::xmp\CreatorTool field to write back.

My quick analysis indicates JPG's straight from camera (Canon 7Dm2) get the XMP CreatorTool info added during the copy. Files created by Photoshop (PSD and JPG) do not get CreatorTool added. Files created by an outside paint program get the CreatorTool (and some other XMP fields).

All that being said, XMP::xmp\CreatorTool seems significant in this problem. Files with CreatorTool needing updated will register in the Write-Back Collection and in Write-Back All. Pencil icon files not including CreatorTool are missed by the Write-Back Collection and in Write-Back All.

After a Write-Back All, the destination files still with pencil icons appear to be Photoshop output created files (PSD and JPG) which have XMP data to write back but did NOT include CreatorTool. These do not show in Write-Back Collection nor Write-Back All. I have to manually go find the pencil icon files and manually force write-back to clear the pencil.

I may need to go re-run this scenario and check my notes. I've run out of time for this session to see if I can duplicate the scenario.

[sinus]

In the preferences for "Indexing" there is nothing special?
In my case I have there a Metadata Template to apply, what changes the metadata and gives a pencil.

But this is only one needle in a haystack from 1000 

[Mario]

The CreatorTool is the "name" of the first tool creating a resource.
IMatch sets it when it creates metadata for a file and the tag is not yet set. Photoshop e.g. sets it.
IMatch updates the tag when it refers to and older IMatch version when it modifies metadata.

When you copy files inside IMatch, IMatch creates the links required by XMP (document id, instance id, etc.) and during this step, when the metadata in the original file has some missing XMP document tags, they are created and then used to produce the links for the copy. This is normal. When you just import a file and don't copy it, missing XMP document tags are created during the first write-back.
A lot of things are going on behind the curtain with XMP metadata 

I have checked the code that takes care of this, and also looked at the code that is run to copy metadata when you copy a file.
There is an optimization which may cause the particular issue you are experiencing. The code tries to avoid marking files as pending write-back when not needed.
I need to analyze this further, but maybe it does not always recognize that a file needs to be written back when XMP document management tags are set or updated, or maybe the file is flagged as pending, but the file is not added to the pending collection.

[StanRohrer]

I can put a half dozen image source files in a dropbox for you, Mario, if it would be helpful.

[Mario]

Thanks. But I have already resolved this issue for the next release, as far as I can tell.

[StanRohrer]

I think the IM Diagnostics need to be able to detect and fix pencil identified files from this problem and be sure they are included in Collections > Pending Writeback. I have no idea how many pencils are laying around my database that don't get updated by Commands > Metadata Writeback > All Files.

[Mario]

Such a check is already in place. I shall schedule some time later to analyze this further.

[StanRohrer]

Seems fixed in IM v2021.17.2. Thanks Mario!