Raw and JPEG Keywords out of sync

[Stevef48]

After updating to iMatch 2021 42k of my photos appeared in the No Keywords category. I'm still battling with 2021, so I haven't been able to work with it, but now I have 49k photos without keywords.
I've tried rescanning the files, but that made no difference.
Is it possible to read .xmp files for these files? Should I reconvert the iMatch 2020 database, which has 0 files without keywords?
Does anyone have a solution?
Thanks in advance,
Steve

[Mario]

Nothing related to that has changed between 2020 ans 2021.

IMatch does not remove keywords. The only way to remove keywords from a database is to strip the keywords from the file in some other app, resulting in IMatch reloading the file and removing the keywords from the database (unless metadata protection is on).

By the XMP standard, JPEG files MUST embedded XMP, not XMP in sidecar files.
Also by the XMP standard, if XMP files belong to all files in the same folder with the same name.

If you have a JPEG file with an XMP sidecar file (very unusual, unless you keep the origin RAW in the same folder or you have used something like Mylio) IMatch merges the metadata of the JPEG and the XMP sidecar file into one master XMP record maintained by the database. By convention, the XMP in the JPEG (if any) takes precedence.
So, if you somehow managed to have an XMP record embedded in the JPEG and also a sidecar file, your results may be unexpected.
During Write-back of the JPEG, IMatch only updates the embedded XMP data, according to the XMP standard.

See Metadata for Beginners and Metadata Write-back


1. Do you have changed any of the default IMatch settings for metadata under Edit > Preferences > Metadata / Metadata 2
2. Did you process these files with Mylio?
3. What do you mean by "struggle with 2021"? From the past four weeks I can tell you that this was a super-easy rollout with virtually no problems reported at all.

4. Run the Metadata Analyst on one of the JPEG files, use the GREEN BUTTON at the top to copy the outcome into the clipboard and paste the result into your reply.


Did we not discuss similar obscure issues you are having in another thread?
What are you doing to cause all these problems? Which applications do you use?
Reading and writing metadata in IMatch is usually something that just works out of the box and does require user interaction only when the metadata in the files is damaged or incomplete due to other applications used previously.

[Stevef48]

I always keep RAW in the same folder as their jpegs.
As I said in my post I did not change preferences, but they have changed. I have now checked Buddy File and Version options for the RAW formats I used, but that made no difference.

1. Do you have changed any of the default IMatch settings for metadata under Edit > Preferences > Metadata / Metadata 2 only because they had previously been changed. ie 2021 preferences looked different to those I had in 2020. I changed them to read xmp, when I discovered all the files without keywords, but it didn't help as there are no hierarchical subject in the xmp files I examined. I've attached screenshots of Metadata and Metadata2 Preferences.
2. Did you process these files with Mylio? I haven't heard of Mylio.
3. What do you mean by "struggle with 2021"? From the past four weeks I can tell you that this was a super-easy rollout with virtually no problems reported at all. iMatch 2021 has been busy reading and adding/updating data since I installed it. It constantly cycled through reading metadata and adding/updating files for many hours. You said that it should only have to writeback metadata once, or twice. Why does it keep reading metadata that it presumably just wrote?

4. Run the Metadata Analyst on one of the JPEG files, use the GREEN BUTTON at the top to copy the outcome into the clipboard and paste the result into your reply. I attached reports to the original post. See attached zip file. Only the JPEG has the correct keywords.

IMatch does not remove keywords. The only way to remove keywords from a database is to strip the keywords from the file in some other app, resulting in IMatch reloading the file and removing the keywords from the database (unless metadata protection is on). 
See Metadata for Beginners and Metadata Write-back


Did we not discuss similar obscure issues you are having in another thread? Yes.
What are you doing to cause all these problems? Which applications do you use? Nothing that writes to images,
since ceasing to use 2020 and installing 2021. Reading and writing metadata in IMatch is usually something that just works out of the box and does require user interaction only when the metadata in the files is damaged or incomplete due to other applications used previously.

I hear what you say, but the problems still exist and I need to fix them. How can I make iMatch copy keywords from JPEG to RAW for the RAW files that have associated JPEGs.
I made JPEGs and RAW files buddies, and wrote metadata back to the 49k files, that didn't help either.
Steve

[Mario]

Why do you need to do this?
RAW and JPEG files have separate metadata. IMatch does not by default keep them in sync.
If you want to copy keywords from RAW to JPG you need to define the RAW as a master and the JPEG as versions. And propagate keywords and other metadata.
See File Relations: Versioning

When you then change (!) keywords and other metadata for the RAW and writer-back, IMatch applies the changes also to the JPEG.
Or you trigger a manual propagation as described in the help topic.

I'm still struggling to understand which problems you try to solve.
In your other post you started with "Search does not work" and then you went to "IMatch is rescanning all my files", which IMatch does not do unless you have changed all your files.

Now you write

After updating to iMatch 2021 42k of my photos appeared in the No Keywords category.

which cannot happen. IMatch does not remove metadata or keywords from your files unless you perform actions which cause this (as I have described above).
Since IMatch is re-ingesting files (wiping keywords) the most plausible cause is that some other application has modified all the files and removed keywords, forcing IMatch to reload all files.

[Stevef48]

My cameras generate Jpeg and RAW at the same time, when the images are imported they go into the same folder. That has worked up to now.
I think that the problem is that something wiped the data from the xmp files.
iMatch 2021 spent many hours, when it first started, reading metadata from those corrupt files. I think this gave me the illusion that I had changed the search option to Filenames, but in the brief periods between Reading Metadata and Adding/Updating files I probably kept starting searches with search all metadata as the option.
My virus checker exacerbated the problem by checking the database files everytime they changed, thus putting extra pressure on the PC.

I think that what I need is a feature in iMatch similar to the one in Lightroom. If the metadata has been changed externally files are added to a 'Metadata Conflict' collection. This allow me to decide which files already have correct metadata in the database. I can then write metadata to those files and read it only from files where they contain the right info.
You rightly said in one of your posts there should only be one truth, I want that to be the iMatch database. It is difficult to achieve this when iMatch decides that it wants to overwrite its data because that it doesn't match in files.

The problem with having the RAW as the master is every RAW in my database is missing its keywords, so I think I better remove the connections, otherwise I will have even more data to reinput.

[Mario]

I think that what I need is a feature in iMatch similar to the one in Lightroom. If the metadata has been changed externally files are added to a 'Metadata Conflict' collection.

Enable Metadata Protection in IMatch if this is a problem you run into. Metadata Settings

And when "something" wipes keywords from XMP files on your system, I would be very worried about this. This might be a workflow issue.
The XMP files are for the RAW, JPEG uses embedded XMP data. If you keep the XMP the RAW and the JPEG in the same folder, the XMP is for both, where embedded metadata has a higher priority.

In general, it is not wise to manipulate metadata in multiple applications, because Adobe did not design XMP for this purpose.
If you modify metadata both in IMatch and Lightroom, you need to control your workflow carefully, especially that and when you write-back XMP data.
See Using Adobe Lightroom® and IMatch together for some additional background.

I use IMatch together with Lightroom, Photoshop, Affinity Designer, Affinity Photo, Blender and many other software. I have never lost any metadata or keywords.
I set my RAW files as the master, and have usually several PSD and JPEG files as buddies/versions, plus a TIFF for long-term archival safety.

[Stevef48]

I had enabled metadata protection.
Only IMatch modifies metadata, I only use Lightroom to develop images. The metadata conflict collection allows me to import keywords into LR.

I thought about my problem and did the following:-
A) Restored all XMPs from backup, after checking that the backups contained hierarchical keywords. I didn't check all of them, just the one for the first raw file without keywords.
B) Rescanned 49k files without keywords - no change to file window display.
C) Unchecked 'Protect XMP'. I didn't do this at first, because the files have no, relevant, XMP data.
D) repeated step B, still no change.
E) Write back metadata to all pending files. There were only 1804?

The file window still hasn't changed. The Info & Activity monitor shows no activity, but IMatch has stopped responding.

What am I doing wrong? Would using 'Import from XMP' work? I know it's supposed to be for videos but will it do what I want and import keywords to RAW from XMP files?

[Mario]

IMatch will automatically pick up keywords from your JPG and RAW and XMP files. This usually just works out of the box, for many years.

Did you run a simple test in the Metadata Analyst as I asked for yesterday in this or your other thread? use the GREEN BUTTON to post the results. Run it once for a RAW and a JPEG.
You can also pack one of your RAW files, the JPEG and the XMP in a ZIP and upload it somewhere so we can have a look.
If the files have keywords and they show up in a test database after adding them, we know it must be a setting or something on your system, and not a problem in the metadata of the files.

[Stevef48]

I ran metadata analyst and attached the results to my post yesterday at 18:51.

No matter what I do, I still have 49k images without keywords. I tried running the file command to read metadata from XMP, on the files without metadata. Bizarrely I selected 48,329 images, but now 48,956 do not have keywords. I did interrupt the process after an hour, it said that it had processed all 48,329 in one place and 8k further down the display.

I think I will copy xmps again, to replace any that were deleted by the import from xmp command. Make sure xmp protection is on, then start again with the IMatch 2020 database

[Mario]

You did not attach the results produced by the GREEN BUTTON, but the raw JSON data. I don't have time to wade through that, sorry.
Provide a sample file as required. Don't make any experiments, which are probably the cause of the problems you have. Let us have a look at one of RAW/JPG/XMP.

[Stevef48]

Sorry.
I did what I said and everything looks better. There are still some odd things going on, but it's a lot better than yesterday.
There are still items displaying as without keywords, which appear to have keywords. I will do what you asked and press the Green Button on one of them
I did a rescan of everything and let the PC for a couple of hours. When I returned the scan had bearly started and the estimated time was 98 hours. I checked that real-time virus scanning was off and Search Indexer paused and the metadata read process sped up dramatically. N.B. I only checked those, I didn't need to change them.
I initiated Metadata Writeback; 827 items needed to be written. After 826 nothing happened for several minutes, so I looked at the Log.
There are several million lines:
Attempt to reload Image/ExifTool/Casio.pm aborted.
Compilation failed in require at (eval 53909) line 2.
Can't find table Image::ExifTool::Casio::Type2
Also many referring to Sony in place of Casio. I do not and have never owned a Casio camera. I think the offending file is SN856938.jpg, because this is the first line searching backwards from the end of the file (before the red lines above):-
Warning: IPTCDigest is not current. XMP may be out of sync - D:\Photos\2008\2008-03-March\SN856938.jpg
Running Metadata Analyst gives:-
Metadata Analyst Results. Version 2021.8.10. 9/5/2021 5:25:18 PM
File analyzed: D:\Photos\2008\2008-03-March\SN856938.jpg
Errors: 1
Warnings: 4

Warning: [System] File has unwritten metadata (pending write-back).<br/>The metadata loaded from the image and the data in the database may not match.
Warning: [XMP] Embedded XMP record (photools.com IMatch 21.8.0.10 (Windows)) and XMP sidecar file (1.28) found.
Warning: [XMP] [ExifIFD]:DateTimeOriginal and [XMP-exif]:DateTimeOriginal (sidecar) mismatch.
Warning: [XMP] [ExifIFD]:DateTimeOriginal and [XMP-photoshop]:DateCreated (sidecar) mismatch.
Error: [Keywords] Different XMP keywords in embedded XMP record and sidecar file.

I will delete the XMP file, as this is JPEG.

I closed iMatch at this point, then repaired the database (Successfully) and am now running diagnosis.
Steve

[Mario]

Looks like something has damaged the ExifTool installation. Weird. Did not see that ever before.
Are you sure your computer is working properly?

Delete all files in the TEMP created by ExifTool on your system. ExifTool will then create them new. See https://www.photools.com/3705/reclaim-disk-space-removing-exiftool-temporary-files/