Missing ratings and labels in converted database

[nchadwick]

Hi, I see a similar issue has been discussed but I've not managed to resolve this from the information in that one.  I have converted a database created in IMatch 3.6 using the converter, ensuring that I ran diagnostics first and processed pending XMP updates.  When loading the converted database into v5 only a very small number of labels have come over to the new database.  For example, an image with yellow rating in v3.6 has the line "<xap:Label>Yellow</xap:Label>" in the corresponding xmp file but in v5 the same image has no label.  Other metadata such as camera specific details and GPS has come over fine.  Is there any way to get the labels back? Any help would be appreciated.

[Mario]

Please don't ask questions in the Tutorials forum. This forum is for giving answers and for tutorials only.

Always post questions in "General Discussion". I will move your post over there.

Does the file you have used for this example have an embedded XMP record? If so, the embedded XMP by default overrides the XMP data in the separate sidecar file. You can check this in the ExifTool Command Processor. Select the file, press <F9>,<E>. In the ECP select the "List Metadata" preset.

See also this FAQ

[nchadwick]

There is embedded XMP data in the files that are not showing their labels in the converted database, although the embedded XMP does not include "Label".  I assume then that this overrides the sidecar file and sets no label.  Is there anyway to force using the sidecar file XMP record or update the embedded metadata with that in the sidecar file, or is this not the way to go?  Further testing has shown only raw files actually keep their labels correctly - both NEF and RW2 files are ok but jpgs are not.  There are no other filetypes in the database I have used.

[Mario]

JPEG use embedded metadata by default and by the XMP and MWG standards. If your JPEG files have embedded XMP data with rating, IMatch will pick them up.
Your other question is answered in the FAQ (see the link I included in my reply above).

[Ferdinand]

Is there anyway to force using the sidecar file XMP record or update the embedded metadata with that in the sidecar file, or is this not the way to go?  Further testing has shown only raw files actually keep their labels correctly - both NEF and RW2 files are ok but jpgs are not.  There are no other filetypes in the database I have used.

This reads to me like you have sidecars for your JPGs - is this right?  The default workflow is to embed metadata in JPGs and not to use sidecars.  If you search the forum you'll find instances of people who use JPGs with sidecars and who have had difficulties.  As noted in the FAQ Mario linked to, you can set your Metadata2 preferences to do this for any file format, but it's not recommended for JPGs and other so-called standard file formats.  You would be better to use the Exiftool Command Processor to migrate the metadata from the sidecars to the JPGs.

This also reads like you might have both embedded metadata in JPGs and also sidecars.  If so, this is a recipe for trouble and I'd recommend to use the Exiftool Command Processor to consolidate the metadata into the JPGs, but you may need to use some advanced settings if you need to copy some metadata from the sidecar while preserving some existing embedded.

If course if you don't have sidecars for your JPGs then you can ignore all of this.

[nchadwick]

Thanks for the advice.  I didn't set out to have sidecars for the jpgs,they were created by IMatch 3.6 when I wrote the unsaved metadata before converting the database, I may have had some setting wrong to cause the metadata to be saved to new sidecars rather than embedding it in the jpgs themselves.  Ideally I would like to write the data in the sidecars to the jpgs then delete the sidecars.  I'll read-up on the Exiftool Command Processor and try to do this.

[Ferdinand]

I have a lot of sympathy for people in your situation, because I was once in it myself. 

When IMatch 3.6 first started using XMP we all had sidecars for all file formats.  I think that must have been the default setting at that time.  Over many years I changed my XMP settings several times as my understanding changed.  It was only during the alpha and early beta testing that I realised what was going to be needed for IMatch 5, and so I did a series of exercises to clean this all up - making sure that RAW files had no embedded XMP and only sidecars and the opposite for JPG and other standard file formats.  And removing redundant sidecars in JPG-only folders.

This was discussed quite a bit on the beta testers forum but perhaps it should have been aired more on the 3.6 forum, once the public beta was out.

Actually my brother is in your situation.  I bought him his copy of both V3.6 and V5.1 and so I have a responsibility to get it to work and help him with the migration.  Unfortunately he lives a 12-hour drive away, so I have to take control of his computer remotely.  His situation is worse, because I think he has sidecars for older JPGs but not for newer JPGs, and so it's messy to know which files need the clean-up.

I can provide more commentary as the exercise progresses if you like, but I'm going to try to do the clean-up in V3.6, whereas you have already migrated.  I'll probably adapt a couple of V3.6 scripts that I used for my own clean-up.

[nchadwick]

Thanks for that, I think it's starting to make sense to me.  Fortunately I'm still using 3.6 day to day and I've only converted a copy, although to test it properly I will have to embed metadata into "live" jpgs.  Would be interesting to hear about progress with your brother's set-up.

[Ferdinand]

Well, the main thing you need to do in v3.6 is make sure that your XMP settings are consistent with V5 and then trigger an XMP writeback.  If you haven't used ratings of labels this is easy.  You turn on asynchronous mode, give all files a rating or label, then remove it, turn synchronous mode on and write out pending changes.

If you're like me, then you have used ratings and labels.  So what I did was create a set of temporary categories to store the labels in before you start this process.  After you've set and removed the temporary label, then you use the temporary categories to reset the real labels before you turn synchronous mode back on and write the pending changes.

This works, but it will write to all sidecars for RAW files and to all standard files like JPG & TIFF.  This can take quite some time and increase the size of your incremental backups quite a lot. 

What I'm trying to do is identify only those files of my brothers that need this.  Not sure how best to do that, but I have written a script that will find files with redundant XMP sidecars, i.e. situations where there is file.xmp in a folder but no file.raw.  There's a good chance that these files will need the above treatment, so it's possible that I can just focus on those.  Alternatively we just trigger an XMP write for all files.  The script can also remove the redundant sidecars once I use the above procedure. 

There is one other thing you may need to do, depending on what your XMP settings have been in 3.6 and what file formats you have used.  If you have let IMatch write XMP to RAW files then ideally you should strip it out.  This may be easiest to do in V5 using the ExifTool command processor, although I have several  scripts that can also do this (using ExifTool).

[Ferdinand]

I'm not sure if the OP is still interested, but I thought I'd document this anyway, while it is still fresh in my mind.

My brother had two problems.  First, IMatch 3.6 was configured to use sidecars for all files.  (I don't know whether I did this at one point many years ago when I thought it was a good idea, or perhaps it simply was the default when he first installed IMatch 3.6.)

Second, he had asynchronous mode on.  (I guess I must have done this for him at some stage, since old files had sidecars but newer files had pending write-backs, but I don't recall when and why.)  This actually wasn't that much of a problem, as it saved him from too many redundant sidecars and all we needed to do was trigger the write-backs.

So the first thing we did was run a script of mine that identified "standard" files with XMP sidecars and bookmarked them.  (The script can also optionally get rid of these sidecars, but we haven't done that yet.  I have migrated this script to IMatch 5, and we will get rid of them shortly using this script.)

With asynchronous mode still on, I set a label for these files and removed it.  So they were added to the writeback queue and embedded XMP would be written once the write-back was triggered.  (See my comments in my previous post about how I have dealt with this if ratings have been used for some files.)

At this stage most of the files in his DB were in the writeback queue, and I couldn't be sure what would happen to the ones that were already in the queue.  They were initially queued for a write-back to sidecars, and it wasn't clear to me whether turning this option off in preferences would change existing pending writebacks.  Although there is a list of these files, I couldn't quickly think of a way to select them in order to set and unset a label and so be confident of the nature of the write-back.  Given that most of the DB was already in the write-back queue, it seemed easiest to set and unset a label for all files in the DB.  So we did.

We could have made the process simpler by just doing this in the first place.  But doing so writes to all standard files and to all sidecars for other file types, and this can be slow and it will also increase the size of the incremental backup.

So now every file in the DB had a pending write-back.  We turned synchronous mode back on and triggered the pending write-backs.  This took a while.

Then we converted the DB to IMatch 5 and all metadata was there.  He is now enjoying the delights of IMatch 5.  Was a long wait for his 2009 birthday present, but he is happy and impressed.

[Mario]

Then we converted the DB to IMatch 5 and all metadata was there.  He is now enjoying the delights of IMatch 5.  Was a long wait for his 2009 birthday present, but he is happy and impressed.