Rating (5 Stars) and Labels showing in Imatch 3 but not IM5 after export/import

[kiwilink]

Now that 5.106 has come out I was able to export the property database as XML Schema and then open IM5 and go to Preferences and "Edit Attributes" and import IMATCH3Property Schema and all of my Title and description data now show up.  However, I noticed I don't see my ratings (4 stars) and Labels (colors) appearing. 

It says in the help menu that this data is Metadata XMP data but I'm not sure how I get it exported from IM3 to IM5 so it shows up.

Thanks!

Michael

[DigPeter]

I think that IM3.6 intially holds such data in the database.  "Process pending XMP updates" has to be implemented to put the data in the meatadata if the file.  If this is done, it should show in IM5.

[kiwilink]

Digpeter:

Thanks for the response.  Is this a command or a script that I must run in 3.6?

Mike

[thrinn]

Have a look at the Tools menu:
Tools -> XMP -> Process Pending XMP Updates.

[kiwilink]

Thanks THrinn

Oh boy, I have 36,4233 pending updates.  I have never used this command so I'm running it right now.

I'm not sure why there are so many.  What exactly does this do that would involve 36,000 files?

[Richard]

Hi Mike,

I don't believe that the XMP updates are necessarily one per image file. Thus your 36,4233 pending updates might be for a much smaller number of image files.

[kiwilink]

Richard

Thanks for the reply.  I just looked and that is how many actual images I have in my 3.6 database.  When I went into tools and then chose "Process pending updates" it came up with that number.

There was a box that said "Allow to create sidecar files or embed XMP data".  I did not check this box and then hit "Start" and the process ran for about 10 minutes but I don't think it updated anything.  Was I supposed to check this box?  Did I hurt anything or change any of my images when I ran this "Process"?

I then copied a file from 3.6 into Imatch 5 that had rating info.  I then looked and it was no different than when I started. In otherwords, no rating info or label info.

Thanks !

Mike

[Richard]

Hi Mike,

I believe that "Allow to create sidecar files or embed XMP data" was important. Without checking that box the changes probably didn't get written to the image files. The fact that it only took ten minutes for that many changes tends to indicate to me that the pending changes were not written. Just removed.

[Richard]

Hi Mike,

In IMatch 3.6 I had 4,242 updates pending. With "Allow to create sidecar files or embed XMP data" checked it took 21 minutes on a six year old computer. Part of the time difference is due to slower computer speed but 36,4233 files in ten minutes can not all be attributed to a faster computer.

[kiwilink]

Richard:

I guess I better wait for Mario to weigh in before I do anything further.  I backed up my 3.6 database before I ran that "Process Pending Updates".  If you are correct in that I lost all of those changes because I didn't check the box I'm hoping I can go back to yesterdays database backup and restore.

However, if it wiped out all of my individual images XMP data then I'm scared.

I noticed now that in 3.6 every directory in my database is asking now for a rescan (see attachment).

Mike

[attachment deleted by admin]

[Richard]

I totally agree that you better wait for Mario to weigh in. From reading the 3.6 Help I am not sure what happened for you. Since you were smart and have a backup, you should be fine.

[Ferdinand]

I don't pretend to understand in detail what happened here, but I will make some general points.

I suspect that I lot of people don't understand what IMatch 3.6 is doing with XMP and where it's storing it.  Personally I find V3.6 XMP preferences hard to understand. Even when you do, there's the question of where you *want* IMatch 3.6 to store XMP in order to smooth the migration to IMatch 5.

I wrote this post to highlight some of these issues:
https://www.photools.com/community/index.php?topic=276.0

My advice is to work out first how you want XMP handled in IMatch 5 and then have IMatch 3.6 match this.

[Mario]

Writing XMP data to image files or sidecar files is slow. Like most other software, IMatch 3 hence keeps changes made to XMP data in the database - which is 1000 times faster. When the user needs to  flush out changed XMP data to his image files, he runs a Tools > Process Pending XMP updates. This command checks for files with unwritten changes to XMP data and then writes these changes to the image file or the sidecar file.

This behavior can be controlled via Options > Preferences > XMP.

If all the files in your database showed up with pending XMP data, this is most likely caused by the following situation:

You used the default settings in IMatch which caused IMatch to create an XMP record for files without one, and then importing the EXIF/IPTC data in the image file into this XMP record. This is the standard behavior and it is best for most use cases. This produces a new XMP record for every file on import. If a file already has an XMP record, IMatch imports it and merges in data extracted from existing EXIF and IPTC data in the original image file.

Since write-back of XMP data is off by default (for performance reasons, see help for details), IMatch maintains this XMP data only in the database until you explicitly flush it to the image file / sidecar file. Unless you do that, the XMP data will only be visible in IMatch 3, but not in other applications, including IMatch 5, LR or whatever you use.

[DigPeter]

Mike,  if you check "Allow to create sidecar files or embed XMP data", you will create sidecar files.  If you do not want these, leave it alone;  the script will work by writing the infromation direct to to the image files.  The process can take a long time.

[Richard]

Mike,  if you check "Allow to create sidecar files or embed XMP data", you will create sidecar files.

That is not true if the image files allow the data to be embedded. My files are TIFF and JPEG and no sidecar files were created because I checked the box.

The process can take a long time.

That is my experience and thus I doubted that Mike's 36,4233 images were changed in just ten minutes.

[Ferdinand]

It would aid understanding if people could indicate when they're talking about V3.6 and when about V5.  I suspect that there's a mix in the the last few posts.

[kiwilink]

Thanks Ferdinand, Richard, Digpeter, and Mario for the comments.  I will assume that I didn't do any damage to my 3.6 images although I am not sure.  Since I only use IMATCH and Photoshop on my Images I am fine with letting IMATCH manage the XMP data.  Mario, I looked at my directories after I ran that "process Pending Update" and now it appears that it created a XMP file for every Image in my database (see attachment).  Those were not there before I ran that routine.  What should I do? Can I delete all of these XMP files since I only want IMATCH to manage the data for me?  WHAT WOULD BE THE EASIEST WAY FOR ME TO DELETE ALL OF THESE SINCE I HAVE HUNDREDS OF DIRECTORIES TO GO TO AND DELETE THEM?  Should I also restore to a previous version of my database?  Please advise.

Now, my original issue was that I wanted to test IMATCH5 and try and understand what I should do to get the "Ratings" and the "Label" information into IMATCH5 from my IMATCH3.6.  I have a few thousand images where I used the 1 to 5 "Star" ratings and I also chose different colors for some of my images.  If you can tell me what steps I should do so I can do this that would be great.  Maybe when you write the migration piece this will happen automatically but if not I need to figure out how to do this so I don't have to redo it in IMATCH5. 

Thanks!

Michael



[attachment deleted by admin]

[Richard]

I can't say for sure that this will work but it is worth trying. In Windows select the top folder where you store your image files. Perform a search for .xmp. In the results window you should be able to select large batches and delete them.

[kiwilink]

Richard:

Thanks.  That worked!  I think I will wait for Mario to respond before I delete them all and also my other question above about what I should do to get the ratings and label info into Imatch5.  Thanks for the great tip!

Mike

[kiwilink]

Mario:

I went ahead and deleted all 36,423 .xmp files.  Then I used True Image and restored my Imatch 3.6 database (see attachment 1) back to last Thursday.  Then I opened Imatch 3.6.  Look at attachment 2.  IMATCH 3.6 now wants me to re-sync every folder.   What should I do?

Mike

[attachment deleted by admin]

[DigPeter]

Talking about IM3.6 'Process pending XMP'!

Mike,  if you check "Allow to create sidecar files or embed XMP data", you will create sidecar files.

That is not true if the image files allow the data to be embedded. My files are TIFF and JPEG and no sidecar files were created because I checked the box.

I stand corrected - sorry if that mislead Mike.  I confess I have never checked the box, so was going only what it says 'on the box'. So the result with jpeg would seem not to be affected by that check-box.

[DigPeter]

Now, my original issue was that I wanted to test IMATCH5 and try and understand what I should do to get the "Ratings" and the "Label" information into IMATCH5 from my IMATCH3.6.  I have a few thousand images where I used the 1 to 5 "Star" ratings and I also chose different colors for some of my images.  If you can tell me what steps I should do so I can do this that would be great.  Maybe when you write the migration piece this will happen automatically but if not I need to figure out how to do this so I don't have to redo it in IMATCH5. 

I went ahead and deleted all 36,423 .xmp files.  Then I used True Image and restored my Imatch 3.6 database (see attachment 1) back to last Thursday.  Then I opened Imatch 3.6.  Look at attachment 2.  IMATCH 3.6 now wants me to re-sync every folder.   What should I do?

To re-sync, right-click on a folder, select 'one-click rescan' and it then happens.  With 36,000 images, you would have to select the top folder and the re-sync will apply to all your images, but will probably take a long time.  I agree, await Mario's guidance for that.

What I would do meanwhile is to experiment with getting your labels etc into IM5:

- In IM3.6 copy a smallish folder to another location.  If a resync notice appears, click 'yes', then do the copy.
- Perform 'Process pending XMP' on the new folder.
- Create a new IM5 database with the new folder.
- Then check whether the labels etc are available in IM5 - I think that they should be.

[kiwilink]

DigPeter

It only partially worked.

Thanks for the suggestion.  I tried this just now.  I created a test database in 3.6.  Chose purple on one image and set 4 stars on one image.  I then ran "Process Pending XMP".  I checked the box that says "Allow to create sidecar or embed existing XMP records.  It will not create new XMP records".  See attachment 1 where I did this.  I was surprised to see that even though it says it will not create XMP records it did create one XMP record for every image.  Then I ran a rescan.  I created a new database in Imatch5 and the added theses new files.  What I found out is that it did show the "Star" ratings but the label ("purple") did not show up.  So it appears that ratings worked but label did not.  See attachment 2.

Thanks!

P.S.  Mario:  When you read this please see my earlier posts with questions about whether or not I did something to my database in 3.6.  I rescanned 3.6 and all looks O.K. but I want to make sure I didn't do any damage I'm unaware of.  Thanks!

[attachment deleted by admin]

[DigPeter]

DigPeter

It only partially worked.

My labels do transfer OK, so I do not know why they do not for you.  Atleast you are getting some useful information from this test.  I await Mario with interest.

[kiwilink]

DipPeter:

Thanks for the response.  I just made all of them different labels (colors) and then opened up the XMP editor to see if they are there in Imatch 3.6.  Since I see it in IMATCH3.6 it , the label problem I am seeing must be in Imatch5 (see attachment).

Mike

[attachment deleted by admin]

[DigPeter]

Since I see it in IMATCH3.6 it , the label problem I am seeing must be in Imatch5 (see attachment).

Mike - I have just done a little test. 

In IM3.6, apply yellow label to an image.
'Process pending XMP' - check that yellow is in XMP - OK.
Copy the image to a test folder for IM5.  There it is complete with yellow label.

So it is working for me.  It might be something to do with your IM5 settings, but I would not know which - sorry.

Peter

[kiwilink]

DigPeter:

Yes, it must be in my settings because I've tried it several different times with no success.

Thanks for trying!

Michael

[Mario]

When you look at the metadata of this file in the Metadata Panel in IMatch 5 (Browser layout), can you see the label tag? You can just search for label in the panel to find it. It is very unlikely that IMatch imports all metadata but not the label.

I see in your screen shot that you use a custom label with the name Purple.  This custom name is not associated with a color by default in IMatch 5. To associate the label name Purple with a color (and to see it in the file window) go to Edit > Preferences > Metadata and add Purple as a new label and associate it with a color of your liking.

XMP labels are just text. If and which color is associated in an application depends on the application. Different applications use different label names, and different colors. IMatch by default ships with label colors matching the common RAW processors.

[kiwilink]

Mario:

Thanks for the reply.  I will check this when I get home.  In the meantime can you look at my above post dated August 04, 2013, 12:38:36 pm.

1.  When running "Process Pending XMP" under 3.6 why did IMATCH create one XMP record for every image REGARDLESS of whether or not I checked the box that says:

"Allow to create sidecar or embed existing XMP records.  It will not create new XMP records"?

I'm asking because it didn't matter whether or not I checked the box because it created an XMP record either way.  Is this a bug because I thought it would just embed the data into the jpg files.

2.  After I did this 'Process Pending XMP" and it created 36,423 XMP records, I deleted all of them.  But my question is did I do anything to my images (jpg files) when I ran this routine??  Was it O.K. for me to delete these 36,423 XMP records?

3.  Maybe under 3.6 I did not understand what labels were used for.  I used the ratings to rate my pictures by picture quality (1-5 stars) and I used the labels (colors) for my picture favorites (red = best, blue = 2nd best, green = 3rd best, etc.).  I just used the colors that IMATCH provided me in 3.6 so I'm not sure why I would have to add purple as a new label in Imatch 5.  I just assumed that ratings and colors I assigned to images in 3.6 would show up in IMATCH5.  I now understand that this data is not embedded with the image but stored in the IMATCH database.  I am confused as to why I have to run that "Process Pending XMP" routine to write the data to an external XMP file or the image itself when all I really want to do is migrate this database data in IMATCH 3.6 to IMATCH5 if it is possible.

Sorry for the long winded response but I'm confused here and I want to understand just why that routine in 3.6 is creating all of those XMP files and just what I should do to get my ratings and colors to show up in Imatch5 like they do in IMATCH 3.6.

Thanks Mario for your patience!

Michael

[BenAW]

when all I really want to do is migrate this database data in IMATCH 3.6 to IMATCH5 if it is possible.

Mario has told sometime ago that the migration from the 3.6 dbase to the 5 dbase will be one of the last things that will be added due to the many changes in the dbase structure.

Workaround for now if you really want those rating and stars in IM5 already would be to create a normal category with subcats for every rating and star you have in IM3. Select for each rating and star ALL assigned images and assign them to the appropriate category. Export your IM3 category structure and import it in IM5. No you can do the reverse process and have your rating and stars in IM5.

[Mario]

I'm asking because it didn't matter whether or not I checked the box because it created an XMP record either way.  Is this a bug because I thought it would just embed the data into the jpg files.

I don't know. This feature exists for years and I don't recall any bug reports about it. Can you add this as a bug report in the bug report forum for IMatch 3? I will then check for it when I make the next bug fix round for IMatch 3 (if there ever will be one).

If you configure IMatch to always write XMP sidecar files, even for formats for which XMP is usually embedded (JPEG), IMatch will produce XMP files for all files in your collection which already have XMP data. This seems to be the case for your system. It will not produce fresh XMP data in this step.

2.  After I did this 'Process Pending XMP" and it created 36,423 XMP records, I deleted all of them.  But my question is did I do anything to my images (jpg files) when I ran this routine??  Was it O.K. for me to delete these 36,423 XMP records?

IMatch either writes XMP data into the image file or into a sidecar file. Which option do you use for XMP in IMatch 3? You can ask IMatch to always use XMP sidecar files, even for formats like JPEG which allow for embedded XMP data (or require embedded XMP data).

Was it O.K. for me to delete these 36,423 XMP records?

If you have deleted XMP records created by IMatch 3, the modified XMP data will still be in the database. If you have modified XMP data in IMatch. If these records where just the result of flushing out the base XMP record IMatch 3 creates when it ingests an image without an XMP record, you should be fine. Adobe products etc. will not use XMP sidecar files for JPEG files but always embed XMP data in JPEG files.

I just assumed that ratings and colors I assigned to images in 3.6 would show up in IMATCH5.

They do. If the data is embedded in the XMP record. IMatch 5 cannot look into IMatch 3 databases. The way IMatch 3 stores XMP in the database is not compatible with how ExifTool processes XMP and how IMatch 5 manages it. XMP data for your files must be written to the file (or sidecar) in order to be picked up by IMatch 5 or other XMP aware applications.

[kiwilink]

Mario:

Thanks for the outstanding reply!  I think I'm much closer to undrstanding how this works.

I have attached my settings.   These should be the defaults because I don't remember ever changing them.


This 3rd option says

"Always store XMP data in external sidecar files"

My Imatch database doesn't have any "sidecar" files that I have ever seen  (unless these files are hidden somewhere).  They ARE NOT in my directories where all of my images are stored.

As I mentioned earlier, when I run the "Process Pending XMP updates" 36,000 XMP files are generated. 

Question 1 - Do you see a setting that I need to change?

Question 2 - When I run Process Pending XMP updates should I check the box that says "Allow to create sidecar or embed existing XMP records.  It will not create new XMP records"?

Thanks Mario!

P.S.  maybe I should follow BENAW's suggestion (Thanks Ben).  I just want to understand the process.

[attachment deleted by admin]

[Richard]

To add XMP labels and ratings categories to IMatch 3.6 you can import them. From the Database Menu select Import and Export then Import Category Definitions. From there you need to end up with "7 XMP Rating and Label.imcs". On my computer it is here:
C:\Documents and Settings\All Users\Application Data\photools.com\IMatch\Presets\7 XMP Rating and Label.imcs

[kiwilink]

I got it to work but I'm not sure I understand why.

Attachment 1 shows the shows the two different XMP preferences I used in IMATCH 3.6 (showing the one that worked and the one that did not)

Attachment 2 shows the result in IMATCH5.

Mario, if you can explain that would be great.

Thanks!

Michael

[attachment deleted by admin]

[Mario]

The "non-working" settings force IMatch to produce sidecar XMP files also for file formats which by default use embedded XMP metadata (stored in the image).
If you configure IMatch 3 that way, you'll end up with XMP sidecar files e.g. for JPEG files. This is non-standard.

If you have that, you must configure IMatch 5 in similar ways (Edit > Preferences > Metadata 2) so it looks for and uses XMP files for JPEG files.

By default, IMatch expects embedded XMP data for standard formats like JPEG, TIFF, PSD, DNG, ... and ignores sidecar files. This is in accordance with the XMP specification, the Metadata Working Group specification and the ways standard applications like Photoshop treat XMP data for these file formats.

By letting IMatch 3 embed XMP data in JPEG files (your "working" settings) you used the standard way to do things, and IMatch 5 immediately found and picked up the XMP data.