Raw+Jpeg ranking/stacking

[wachdam]

Hello,

New user testing imatch v5 beta 0130  .

My workflow is:

1)  ingesting raw+jpeg images with photomechanic v469 and inputting IPTC/XMP metadata with structured keywords

2) rating raw+ jpeg  images with Breezebrowser pro

3) indexing raw + jpeg  images with Imatch v5

The @keywords creates well  hierarchical categories based on the structured keywords inputted with PhotoMechanic .

When indexing the images, I see the rating on the jpeg but not on the raw images (Canon 5DMKIII .cr2)
When doing an auto-stack, the CR2 appear on top of stack with no rating

Do I miss something that the raw images do not display the rating made with Breezebrowser pro ?

Best , Pierre W

[Mario]

Where does BB write the rating to?

By default, IMatch expects XMP sidecar files for all RAW formats - in accordance with the XMP standard, the Metadata Working Group recommendations and the 'industry standard' set by Adobe products.

Your CR2 file should have a accompanying .XMP sidecar file. I this the case?
This sidecar file (you can open it in Windows Notepad and search) should contain the rating and label written by BB.
If this is the case, IMatch can pick up the metadata.

If BB writes XMP data into the CR2 file, this is non-standard. In this case you'll need to tell IMatch that it should look for embedded XMP data for .CR2 files under Edit > Preferences > Metadata 2. Click on Configure File Formats... and in the list select the .CR2 file extension. Change the options as needed and close the dialog. Select the CR2 files in a File Window and press <Shift>+<Ctrl>+<F5>. In the dialog, choose Reload Metadata to load the metadata again using your new rules.

Note: Mixing this many applications in adding/editing metadata may cause trouble. XMP is a complex thing, and especially if you derive from the standards set by XMP and MWG/Adobe you'll may end up in trouble. I explain all that in detail in the IMatch help. Just press <F1> while you are in Edit > Preferences > Metadata 2 to open the corresponding help topic.

[wachdam]

Hello Mario,

Thank You for your feedback and support .

Breezebrowser writes well to the .xmp that was created during ingest by Photo Mechanic .

I append the jpeg and the xmp to my post ( the cr2 is 64 mb so that it does not fit here as attachment)

for example

  <rdf:Description rdf:about="" xmlns:xmp="http://ns.adobe.com/xap/1.0/">
    <xmp:Rating>5</xmp:Rating>
  </rdf:Description>

 

The rating was created  by Breezebrowser

When indexing in Imatch, the jpeg shows the rating but not the cr2

I tried to change the Protect rating and label from default No to Yes in IMATCH preferences/metadata2 but it makes no difference, the rating shows only on the jpeg .

I wonder if I miss some parameter about raw+jpeg coupling or if it is another problem

Best Pierre

Best Pierre

[attachment deleted by admin]

[wachdam]

Hi Mario,

Forgot to write to thank you for the extensive documentation that you have made available with Imatch .

I took a couple of hours yesterday to print and go through the 350 pages help that you have put up prior to create this post  .

Sincere thanks for the awesome job that you have put up on this .

Best Pierre

[wachdam]

hello Mario,

I have tested to rank the images within Photo Mechanic and leave Breeze Browser out of the equation , to obtain the same result:

when indexing the file in Imatch v5 0130, only the jpeg file shows the rating .

I append an xmp created by Photo Mechanic (v469)

Best Pierre

[attachment deleted by admin]

[Mario]

The JPEG file has an embedded XMP record with a rating of 2.
The XMP file also has a rating record with the value 2.

I tried to reproduce the behavior you are seeing by these steps:

Adding the JPEG file to an IMatch 5 database correctly shows the rating in the file window. OK.

I then added a plain .CR2 file (without embedded XMP) data to the same folder as the JPEG and the XMP.
Renamed this .CR2 file to match the name of the XMP file and rescanned the folder in IMatch.
IMatch pulls the XMP data from the XMP file and uses it for the CR2. The CR2 file shows up with a rating of 2. OK
I used the IMatch 5 default values for Metadata 2 and File Formats.

My guess is that the CR2 file also contains XMP data and that this overrides the XMP data from the sidecar file.
Please try this:

Select the CR2 in a file window.
Open the ExifTool Command Processor via Tools > ExifTool Command Processor.
Type these arguments into the left edit window (Copy/paste from below):

Code Select Expand

-rating
-xmp:all
-G1
{Files}


This shows us the XMP data (if any) contained in the CR2 file.

[wachdam]

Hello Mario ,

Herafter teh result of the exiftool command as you proposed :

raw file (no rating in IMATCH)

-use
MWG
-rating
-xmp:all
-G1

D:\Z_TEST~1\TEST03\131221~1\677_53\A_SELECT\A_SELE~1\131221~1.CR2


-execute9999
[MWG]           Rating                          : 0

jpeg buddy file (rating 1 in IMATCH)

-execute9999
[MWG]           Rating                          : 1
[XMP-x]         XMP Toolkit                     : XMP Core 4.1.1
[XMP-iptcCore]  Creator Work Email              : email: pierrew@happygraphy.com
[XMP-xmpRights] Marked                          : True
[XMP-xmpRights] Web Statement                   : http://www.happygraphy.com
[XMP-lr]        Hierarchical Subject            : type|private, event|wedding
[XMP-photoshop] Date Created                    : 2013:12:21 18:21:01+01:00
[XMP-dc]        Rights                          : copyright Pierre Wachholder
[XMP-photomech] Prefs                           : Tagged:1, ColorClass:0, Rating:1, FrameNum:003982
[XMP-photomech] PM Version                      : PM4


I double chacked Photo Mechanic preferences and made sure to select the "don't update iptc/iptcxm4P even if it exists" field in IPTC/XMP preferences .
The other point is that files updated with both Photo Mechanic and Breeze Browser experience the same issue .

Do you see any error from PM in the exif data sent above ?

Best Pierre

[Mario]

The CR2 file has a rating of 0 (unusual, but allowed). This overwrites the rating pulled in from the XMP because IMatch merges metadata when ingesting. It first pulls in the XMP and then this rating (2) is wiped out by the 0-rating in the CR2 file.

[wachdam]

Hi Mario,

thanks again for your feedback .

I have forwarded your feedback to Photo Mechanics support .


I have tried a workflow with Breezesys Downloader pro and Breezebrowser pro and obtain teh same result when ingesting the files in IMATCH :

the .cr2 has a rating of 0 and the jpeg has a rating of 5 .

This makes me wonder if I would not be better of to either use only the jpegs in my database or try to declare  the raw  as buddy files from the jpegs .

Do you see potential issues with this ?

Best Pierre

[wachdam]

Hello Mario,

As I have an Adobe PS CC cloud subscription which includes LR CC; I also tried to import the ranked CR2+jpeg images from PM and BB Pro into an empty LR catalog .

In both cases, the LR catalog displays  the raw with the ranking .

Whatever my personal feelings about the situation, my understanding  is that all "small" players adjust their software to the way Adobe create/modify its "standards" .

I have no plan to move to LR, I personnally think that a specialized piece of sw for each part of the workflow will make a better job than a "one fits it all" software such as LR, but adjusting to LR may be for all these specialized sw, including IMATCH, the way to interact nicely together .

If I cant make rankings work OK, I thought about creating a keyword which would mimic the ranking and allow me to work with Imatch @keywords and have the indexed  images going automatically to these categories  , which is exactly what I was looking for and work OK .

My 2 cents,

Best Pierre

[Mario]

The problem can only be because your files contain competing XMP data in the XMP sidecar file and embedded in the CR2 file.
The easiest method would be to strip the outdated XMP record in the CR2 file with ExifTool. This can be done by

exiftool.exe -xmp= <Name of CR2 file>

I cannot reproduce this behavior here with any of the CR2 files in my collection. Please send me one of your CR2 files or upload it somewhere.
If you have no web space, contact me via email (see link below) for details about how to upload files to my FTP server.

IMatch currently merges XMP data when it imports files and the image file has both a sidecar and an embedded XMP record. It is hard to decide which of the competing XMP records to favor, the embedded or the sidecar file? IMatch favors the sidecar file because my assumption is that since most applications write sidecar files for RAW files (including the Adobe products you mention). So the sidecar rating is supposed to override the rating embedded in your CR2 file. I will need to look at your files and the ExifTool XMP output to see why this does not work in this case.

[wachdam]

Hello Mario,

Many thanks for your feedback .

I am preparing and sending to your support email a zip containing the original raw and jpeg (from a canon 5DMKIII) as well as copies of the same files after ranking, both with photo mechanic and breeze sys .

My understanding is that the raw files were totally untouched by PM/Breeze Sys ( PM has a lots of options to embed and/or create xmp side cars to accomodate different scenarios, BreezeSys is xmp side car only ) .

Maybe an panel in the db preferences to allow the user to go for one or another option would be easier than having you to decide and solve all potential scenarios.
I read your posts in the user guide that the current situation is a mess so maybe the user being able to select options that will address his workflow would be easier ?

My 2 cents again.

Also, many thanks forbringing to life Imatch5 which I value as an excellent dam .

All the best to you and the people you care for for 2014;

Pierre W

[Mario]

My understanding is that the raw files were totally untouched by PM/Breeze Sys ( PM has a lots of options to embed and/or create xmp side cars to accomodate different scenarios, BreezeSys is xmp side car only ) .

The ExifTool output shows that the file contains XMP data, at least a partial set of XMP. I cannot say which application put it there.

Maybe an panel in the db preferences to allow the user to go for one or another option would be easier than having you to decide and solve all potential scenarios.
I read your posts in the user guide that the current situation is a mess so maybe the user being able to select options that will address his workflow would be easier ?

I'm quite sure that IMatch 5 has more options in this area than will ever bee needed. You can even customize internal/external XMP data by file extension. See the IMatch help for details on how to achieve this.

[wachdam]

Hello Mario ,

Thanks again


Just revisited Metadata2 preferences/ configure file formats for .CR2 raw format

Does XMP sidecar file / Force XMP sidecar file option make that only data from sidecar is read and that whatever data is embedded in the raw cr2 is ignored ?

Some other options mention write iptc and write exif but I do not see any Read IPTC or Read Exif option to set to No in order to avoid that Imatch tries and read the embedded data .

ps To reiterate my last post, Imatch is great , I hope that my numerous posts are not seen as critics, they are not meant to ; I am still learning and the last days have been busy with absorbing  the (splendid) online documentation/help manuals  and posts on the forum

Best ,

Pierre

[wachdam]

Hi again,

Just tried to change the .CR2 option to force xmp ; emptied db, closed it and restarted it, the reimported the raw (without jpeg)   with rating and Imatch does not display the rating yet .

Herafter the xmp data as seeing with edit+ ; last lines of the file mention rating 1 as seen when using PM or LR

<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.1.1">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
            xmlns:photomechanic="http://ns.camerabits.com/photomechanic/1.0/">
         <photomechanic:HasCrop>False</photomechanic:HasCrop>
         <photomechanic:Prefs>0:0:1:-00001</photomechanic:Prefs>
         <photomechanic:PMVersion>PM4</photomechanic:PMVersion>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:Iptc4xmpCore="http://iptc.org/std/Iptc4xmpCore/1.0/xmlns/">
         <Iptc4xmpCore:CreatorContactInfo rdf:parseType="Resource">
            <Iptc4xmpCore:CiEmailWork>email: pierrew@happygraphy.com</Iptc4xmpCore:CiEmailWork>
         </Iptc4xmpCore:CreatorContactInfo>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:xapRights="http://ns.adobe.com/xap/1.0/rights/">
         <xapRights:Marked>True</xapRights:Marked>
         <xapRights:WebStatement>http://www.happygraphy.com</xapRights:WebStatement>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:lr="http://ns.adobe.com/lightroom/1.0/">
         <lr:hierarchicalSubject>
            <rdf:Bag>
               <rdf:li>type|private</rdf:li>
               <rdf:li>people|guylaine_pietjan</rdf:li>
               <rdf:li>location|chateau_rocq</rdf:li>
               <rdf:li>event|wedding</rdf:li>
               <rdf:li>community|happygraphy</rdf:li>
            </rdf:Bag>
         </lr:hierarchicalSubject>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">
         <photoshop:DateCreated>2013-12-21T18:20:33+01:00</photoshop:DateCreated>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:dc="http://purl.org/dc/elements/1.1/">
         <dc:description>
            <rdf:Alt>
               <rdf:li xml:lang="x-default">wedding Guylaine Pietjan </rdf:li>
            </rdf:Alt>
         </dc:description>
         <dc:rights>
            <rdf:Alt>
               <rdf:li xml:lang="x-default">copyright Pierre Wachholder</rdf:li>
            </rdf:Alt>
         </dc:rights>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:xap="http://ns.adobe.com/xap/1.0/">
         <xap:Rating>1</xap:Rating>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>

Best Pierre

[wachdam]

Hello Mario,

I have just sent to you an email with an ftp link and some explanation on the zip content .

Best,

Pierre

[wachdam]

Hello Mario,

This email to wish you well for 2014 and enquire whether you received the files that I have sent .

Also, I received feedback from Photo Mechanic support  to which I have written and sent files  to enquire whether data was inputted into the raw images .

The answer was that the raw files were not modified at all and that Canon has indeed set rating=0 in the raw file .

As mentioned previously, I have changed the db preferences for .cr2 file to force xmp without solving the issue which seems to indicate that Imatch stills reads the embedded .cr2 data .

Best Pierre

Here's PM feedback in long

The RAW files in each of the five folders is absolutely identical.  PM 4.6.9 has not written to the CR2 file in any way.  I used "shasum" to generate checksums for the files and they're the same.  One byte out of place and the sums would differ.

The XMP sidecar file has been created in folder two and has an Adobe namespace rating of zero and a Photo Mechanic prefs string with the value "0:0:0:-00001"
The XMP sidecar file in the fifth folder has an Adobe namespace rating of 5 and a Photo Mechanic prefs string with the value "0:0:5:-00001"

To interpret the Photo Mechanic prefs string the field values are as follows:

Tag:Color Class index:Rating:Frame number

So both the Adobe rating and the PM rating in the sidecar file are both 5.

Now, Canon has been for a while embedding a small amount of XMP data in CR2 files generated in the camera and your original image does indeed have a rating embedded and it is zero.  But Photo Mechanic didn't put it there.  Your camera did.  Photo Mechanic did not update the value (it wouldn't anyway even if you told PM to embed XMP since PM would now embed all XMP fields and the embedded XMP would be far too small to update in place so a new XMP block would be added at the end of the file) so it still exists.

The IMatch folks need to interpret the XMP metadata in the sidecar file with priority over the XMP metadata contained within the CR2 file.  If IMatch did this, it would see the rating in the XMP sidecar file and would ignore the zero rating found in the CR2 file.

[Richard]

The IMatch folks need to interpret the XMP metadata in the sidecar file with priority over the XMP metadata contained within the CR2 file.

I don't think that IMatch should "interpret" and give any data priority over other data. IMO, IMatch should simply report to the user that embedded data is in conflict with sidecar data.

[wachdam]

Hi Richard,

Imatch preferences exist to let the user give preferences to one or the other workflow and accomodate for the current  "mess"/lack of standardization  in Exif/IPTC/XMP as mentioned by Mario in the Imatch documentation.

My understanding in this case  is that the  0 rating in the cr2 file originates from Canon at cr2 creation in the camera .

If the preferences would allow full priority to xmp and not reading the embedded cr2 data, the rating would display correctly .

As mentioned in a previous post, I tested the same images with two ingest/rating  programs in full xmp workflow (Photo Mechanic and BreezeSys Downloaderpro/BreezeBrowser pro )  and the import them to Lightroom CC and  LR displays the ratings OK for the raw in both cases.

This points me thinking that, although Imatch preferences for .cr2 files have been set to Force XMP, the embedded rating data in the CR2; which originates from Canon, has been read by Imatch .

Best Pierre

[Richard]

Hi Pierre,

Just because a user selects side cars as the place to write XMP data, I don't think that IMatch or any other application should simply ignore what Cannon embedded in the file. I would prefer that an application let me know that there is a conflict and help me fix the problem. Ignoring a problem will not eliminate it. It will remain a potential problem forever.

[wachdam]

Hi Richard,

I suppose we are trying to say the same thing :

- in case of conflicting info between embedded and side car the app should :

- warn the user about conflicting data

- let the user fix the conflict ; for example by setting the priorities in Imatch preferences, for instance xmp has priority over xmp embedded and not display warning messages if the user choses to (like PS colour spaces warnings which the user can chose to have not displayed )

In the current case , Canon has chosen to add a rating data value by default to the cr2 , which causes the issue .

By using force xmp in Imatch  prefs, I was assuming that the embedded data  is being discarded vs the xmp data

This is not the case right now if i understand correctly, as Imatch ignores the xmp rating for the cr2 because of the conflicting info .

Only Mario can tell if the Force xmp was meant also for read operations or only for write, and in that case, if a feature request would be the way to go further .

Best Pierre

[Mario]

I looked at the CR2 file with the rating problem you sent (thanks) in the 07_BREEZESYS_BBPRO_RANKING folder.

The sidecar file has a rating of 5, the XMP metadata record embedded in the CR2 has a rating of 0 ( = no rating).

In the Default XMP mode (configurable per format under Edit > Preferences > Metadata: File Formats) IMatch merges embedded XMP metadata (and XMP data generated by ExifTool) with the XMP metadata in sidecar files. ExifTool usually produces a much richer and more precise XMP record (especially from EXIF data and maker notes) than other software and RAW processors. By combining the XMP data generated by ExifTool with the XMP sidecar file, IMatch gets the best possible XMP data.

This process also merges XMP data that may be embedded in the image into the resulting composite XMP record IMatch uses. The XMP data embedded in the file / produced by ExifTool gets priority over the XMP metadata in the side car file. If the XMP data embedded in the image contains different values for XMP than the sidecar file, the embedded data wins. This is why IMatch shows no rating for the CR2 file in your example.

If the user configures "Favor XMP" or "Force XMP" for a file format (Edit > Preferences > Metadata 2: File Formats), IMatch should ignore embedded XMP data when a sidecar file exists. This was not working in all cases. I have fixed that for the upcoming 5.0.134 version.

If this version becomes available, you need to change the settings for .CR2 files to Favor or Force XMP in order to work around the fact that your CR2 files have two competing XMP records. If you set these option, IMatch uses only the XMP data in the sidecar file.

A cleaner solution would be to use ExifTool to strip the XMP data from the CR2 file, keeping only the sidecar file. It's always a potential source for problems when you have files with two competing XMP records. Only one of them is updated, so they are out-of-synch anyway.

[wachdam]

Hello Mario ;

Many thanks for your feedback on this .

I will install the new version when available and test it with the 3 generations of Canon 5D dslr (classic,mkII,mkIII) and feedback you .

As Photo Mechanics support mentioned, it is in this case Canon that embeds the 0 rating in the 5D MKIII raw file .

I am using a 100% xmp sidecar workflow and do not want to edit the raws and take chances on their integrity , sp that I welcome the fix in Imatch .

Best ; and again thank you for your excellent software, documentation and support .

Pierre W