Versioning and image orientation

[DigPeter]

I have been trying versioning for the first time in a small database of 52 raw (RW2) images and 52 jpegs derived from these in Lightroom.  These include some portait images, which were corrrectly displayed vertically.

The Relations Definition linking RW2 to jpg files works and metadata written in the raw files successfully was successfully applied to the jpegs.  So far OK.  But after the background processing finished, the portrait images are displayed horizontally - they have been rotated by 90 degrees.

The log file is attached.

Is there a reason for this and how can I prevent it happening?



[attachment deleted by admin]

[Mario]

LR and other Adobe software stores all kinds of useful and less useful data in XMP. Copying the entire XMP record to another file can cause strange results.
"Versioning" was never a concept of XMP data, and copying the entire XMP record from a RAW to a JPEG file can cause such issues.

Do the RAW/JPEG files show the crop/transformation icon? If so, the XMP record written by Adobe contains a crop/rotation record in the XMP:crs namespace.
Copying this record from the RAW to the JPEG can result in a wrongly cropped or rotated image because the crop record belongs to the RAW and not the JPEG. The JPEG is most likely stored already cropped and rotated.

Which data do you propagate in your relation rule? All XMP?
Maybe copying propagating "XMP without crop" or "XMP without LR/CRS data" is the better option for your workflow? The first disables the crop record in the target file but keeps it. The second option strips proprietary XMP data produced by LR and other Adobe software before storing the data in the target file.

Also, If this is crop/transform related, which crop settings do you use under Edit > Preferences > Metadata 2?

I know this is all complicated, but Adobe did not design XMP data for versioning or for copying it between files. They designed XMP mainly with their products in mind, and since then added proprietary stuff to it. For example the crop record. IMatch, when enabled, uses the crop record to crop and rotate files. Which is useful. But can lead to problems when you copy crop records between different files.

Another cause for this can be that you copy the entire EXIF metadata record, which includes an orientation tag. Copying the orientation tag from the RAW to the JPEG may cause the JPEG to be orientated wrong as well. Basically it's often a combination between EXIF orientation, how the resulting file is physically rotated and the XMP crop record.

When you propagate EXIF data, try "EXIF without orientation".

[joel23]

I have been trying versioning for the first time in a small database of 52 raw (RW2) images and 52 jpegs derived from these in Lightroom.  These include some portait images, which were corrrectly displayed vertically.

The Relations Definition linking RW2 to jpg files works and metadata written in the raw files successfully was successfully applied to the jpegs.  So far OK.  But after the background processing finished, the portrait images are displayed horizontally - they have been rotated by 90 degrees.

The log file is attached.

Is there a reason for this and how can I prevent it happening?

Try using "EXIF Data wthout Orientation"

[DigPeter]

Try using "EXIF Data wthout Orientation"

Thanks - experimentation is clearly required.

[DigPeter]

Mario, thank you for this very full reply.

Do the RAW/JPEG files show the crop/transformation icon? If so, the XMP record written by Adobe contains a crop/rotation record in the XMP:crs namespace.
Copying this record from the RAW to the JPEG can result in a wrongly cropped or rotated image because the crop record belongs to the RAW and not the JPEG. The JPEG is most likely stored already cropped and rotated.

There is no XMP:crs namespace tag in any of the XMP files.  Attached are XMP files for both raw and jpeg images from before and after versioning.

Which data do you propagate in your relation rule? All XMP?
Maybe copying propagating "XMP without crop" or "XMP without LR/CRS data" is the better option for your workflow? The first disables the crop record in the target file but keeps it. The second option strips proprietary XMP data produced by LR and other Adobe software before storing the data in the target file.

Also, If this is crop/transform related, which crop settings do you use under Edit > Preferences > Metadata 2?

I do not fully understand the versioning dialog, so I checked only "all metadata.  I will have to experiment with different settings.  Perhaps "XMP without lr/crs data" should also be checked.

When you propagate EXIF data, try "EXIF without orientation".

I will try this.

[attachment deleted by admin]

[cytochrome]

I have the same problem with orientation of RW2 when I shoot with the Panasonic 20mm/1.7 pancake.

With this lens the orientation is not recorded, apparently it has no orientation sensor (an no stabilisation..) but is excellent anyway.

Francis

[Mario]

I can't see the EXIF data of the JPEG, but the post version JPEG XMP lists an EXIF orientation different from "none". So I guess that the RAW has an EXIF orientation different from none, and copying that over to the JPEG causes the JPEG to be displayed wrong.

LR writes the JPEG with the pixel data rotated correctly, and when you unconditionally replace the EXIF record in the JPEG by your version setup, you copy the orientation of the RAW, which, when applied to the JPEG, causes a wrong orientation.

1. Don't copy metadata unless you know exactly what is caused by that - copying EXIF records between different file formats can cause great problems!
2. If you have to copy the EXIF record for whatever reason, don't copy the orientation if it causes problems.
3. Read the corresponding sections in the IMatch help which give explanations of every setting and option, and examples on when to use what.

Versioning and metadata propagation in IMatch 5 is very flexible and powerful. But the user still has to decide if and which metadata he copies. And this unfortunately requires a deeper understanding of metadata. EXIF data (or the EXIF data in the XMP record) is a potential cause of problems. Because EXIF contains data directly "tied" to the original image, starting at the EXIF orientation tag to maker notes, lens and white balance information and more. Copying this data between files, and especially between RAW and other file formats, is usually not a good idea. It may be needed in some cases (that's why IMatch allows it) but the side effects have to be understood.

When you produce JPEG files in LR, LR already copies a lot of "save" metadata into the JPEG. There is usually not need to copy metadata also in IMatch from the RAW to the JPEG, especially not EXIF data.

[DigPeter]

I have the same problem with orientation of RW2 when I shoot with the Panasonic 20mm/1.7 pancake.

With this lens the orientation is not recorded, apparently it has no orientation sensor (an no stabilisation..) but is excellent anyway.

Francis

An impressive seeming lens.

[DigPeter]

I can't see the EXIF data of the JPEG, but the post version JPEG XMP lists an EXIF orientation different from "none". So I guess that the RAW has an EXIF orientation different from none, and copying that over to the JPEG causes the JPEG to be displayed wrong.

LR writes the JPEG with the pixel data rotated correctly, and when you unconditionally replace the EXIF record in the JPEG by your version setup, you copy the orientation of the RAW, which, when applied to the JPEG, causes a wrong orientation.

1. Don't copy metadata unless you know exactly what is caused by that - copying EXIF records between different file formats can cause great problems!
2. If you have to copy the EXIF record for whatever reason, don't copy the orientation if it causes problems.
3. Read the corresponding sections in the IMatch help which give explanations of every setting and option, and examples on when to use what.

Versioning and metadata propagation in IMatch 5 is very flexible and powerful. But the user still has to decide if and which metadata he copies. And this unfortunately requires a deeper understanding of metadata. EXIF data (or the EXIF data in the XMP record) is a potential cause of problems. Because EXIF contains data directly "tied" to the original image, starting at the EXIF orientation tag to maker notes, lens and white balance information and more. Copying this data between files, and especially between RAW and other file formats, is usually not a good idea. It may be needed in some cases (that's why IMatch allows it) but the side effects have to be understood.

When you produce JPEG files in LR, LR already copies a lot of "save" metadata into the JPEG. There is usually not need to copy metadata also in IMatch from the RAW to the JPEG, especially not EXIF data.

Thanks - I am now much clearer.   I am experimenting with workflow:
- Download raw files from camera, applying standard info, like photographer, and applying gps info if this available in a gps track.
- Convert raw in LR to jpg, so jpg will have same metadata as the raw, including exif, plus a load of redundant editing XMP info.  (Why does LR not offer the opportunity to drop this and just leave the exif and any other XMP/IPTC info?)
- in IM with versioning, write Title, Description, keywords through the metadata and keyword panels simultaneously to both the raw and jpg files.

Mario can you please confirm that I need only check "XMP without LR/CRS data", in addition to the default settings?  Or should I also check exif without orientation", even if the jpg files have exif?

[Mario]

Mario can you please confirm that I need only check "XMP without LR/CRS data", in addition to the default settings?  Or should I also check exif without orientation", even if the jpg files have exif?

If you want to copy XMP data from the master to the version you must check at least one of the XMP options. Copying the LR/CRS data is usually not a good idea because all the proprietary data in the CRS namespace is only meaningful to Adobe applications and valid only for the RAW file but not the JPEG.

If you don't want to copy EXIF data, don't check any of the EXIF entries in the "What to propagate" list. Problem is then that when you apply the MWG rules, the XMP data (XMP:exif namespace and tags) copied to the JPEG file will no longer match the EXIF data in the JPEG file.

[DigPeter]

Mario can you please confirm that I need only check "XMP without LR/CRS data", in addition to the default settings?  Or should I also check exif without orientation", even if the jpg files have exif?

If you don't want to copy EXIF data, don't check any of the EXIF entries in the "What to propagate" list. Problem is then that when you apply the MWG rules, the XMP data (XMP:exif namespace and tags) copied to the JPEG file will no longer match the EXIF data in the JPEG file.

Thanks, Mario.

[ubacher]

Encountered the same problem with badly oriented jpg's after conversion in PS.
I have All Metadata checked and also Maker Notes and Nikon Capture Data.

( And I just realized that NIKON CAPTURE refers probably to the Nikon program of that name and not the process of capturing data which should
anyway be in the EXIF. )

Mario: what settings would you recommend for someone working with RAW and jpg and the occasional PSD files? I am using Keywords though.
Just Classic IPTC Data? Is there any need to copy XMP in such a case?


The help file does not explain the various settings nor give recommended defaults - I am also wondering about the exact meaning of * indicates on-write attributes

[Mario]

The help file does not explain the various settings nor give recommended defaults - I am also wondering about the exact meaning of * indicates on-write attributes

I have added some more text to the corresponding help section based on this thread, including my orientation and crop examples from above.

The problem is that users want "versioning" and copy metadata around between their files without having a real understanding of what all the metadata really means and what copying EXIF data from a RAW to a JPEG file can cause. And of course all this depends on the applications you work with, what exactly you consider a version, what you plan to do with a version, which clients and customers you have, which web services, on-line galleries you use etc.

I have written lots of this over the last years, and the Internet is full of tips and workflow recommendations. Often people only encounter that their metadata is a mess when they switch from one application (workflow) to another application. Or when they first publish their files or need to deliver to clients which have strict requirements considering metadata and formats...

Classic IPTC is safe, but deprecated. Still, works pretty well.

Copying EXIF is potentially harmful, and not only because of the maker notes which become often invalid if copied between files (although ExifTool does a marvelous job here). The orientation field is one example. But there are also white balance, and even lens can cause problems if you open the version in an application which does "lens correction" - but on the wrong data.

XMP is safe, except the XMP:exif namespace and all the proprietary namespace Adobe and other software companies have invented. Especially Adobe is flooding XMP data with all kinds of proprietary data. I have seen XMP records with dozens of megabytes (!), larger than the actual image. Just because a user did check a wrong option and from then on LR/CS recorded every pushed pixel in the XMP "history".

This is all really a moving target. Every week a user sends me another weird file or weird workflow idea he/she wants to implement in IMatch using versioning...

I personally, and most pro's I know implement a very strict but simple workflow.
Copying EXIF data between files is usually a no-no.
Classic IPTC and XMP a must.
Modifying metadata only in *one* application. XMP never stood up to it's universal cross-application and platform promise. Unless you are really careful.
Besides, there is not one XMP - in fact there are many generations of XMP data in use in image archives (pro or amateur). Each and every of these XMP generations had its own flaws, and Adobe adds stuff to XMP whenever they need something new for one of their applications.

Metadata is a moving target and I can only try to cope and to support as many different useful workflows as possible.
My general tip is to KISS.

[DigPeter]

@ Mario  I have set up a new database, with versioning exif without orientation.  This fixes the problem of horizontal, portrait images.

Your last post indicates that versioning is not a safe practice in the hands of the technically illiterate, like myself.  My new database creates keywords in versions which are different to those in the master files.  See topic https://www.photools.com/community/index.php?topic=1089.0

[BenAW]

@ MarioYour last post indicates that versioning is not a safe practice in the hands of the technically illiterate, like myself.

This was one of the discussion points in the early Alpha testing.
I don't like the present Versioning setup, because it gives the FALSE impression that propagation of (meta)data has anything to do with Versioning. It has not.
The default for propagation should be NO propagation.

If you assign categories, keywords and whatever only to the Master file, you can simply locate that file.
Through Versioning you have now also found all Versions, just do "Show all Versions". This works fine, even if the version has no categories, keywords etc.

If you have a NEED to propagate anything at all, make very sure you understand what you are doing.

[Ferdinand]

I don't like the present Versioning setup, because it gives the FALSE impression that propagation of (meta)data has anything to do with Versioning. It has not.

For some people it's a part of versioning, but not for others.

But let's not refight these battles again.  I have no problem with the default being nothing being propagated if versioning is enabled, so that the user has to choose what is propagated.

[DigPeter]

If you have a NEED to propagate anything at all, make very sure you understand what you are doing.

Thanks for the advice Ben - I think I am going off versioning -  see my recent topic https://www.photools.com/community/index.php?topic=1089.0

[BenAW]

Thanks for the advice Ben - I think I am going off versioning -  see my recent topic https://www.photools.com/community/index.php?topic=1089.0

That would be a shame, because Versioning is very usefull.
Just setup some versioning using only the Exif timestamp (see screenshot)


The Label "Detection"should actually be called "Versioning", and Versioning should be called "Propagation".
Just make sure all checkboxes under "What to propagate" are empty.

[attachment deleted by admin]

[Mario]

This is called detection because on this tab you tell IMatch how to find (detect) versions. I think the name is good.
The Versioning tab combines several settings and options related to versions, from categories to collections to colors to metadata propagation.
Detection is used for buddy and version relations, the Version tab is only available for relations of type version.

Of course versioning can be used without copying metadata around. Or just copying rating, label, collections and categories. Which are all safe.

[DigPeter]

@ MarioYour last post indicates that versioning is not a safe practice in the hands of the technically illiterate, like myself.

This was one of the discussion points in the early Alpha testing.

I have searched in the old Alpha test forum, but cannot find these discussions, which I missed.  Would it be possible to  let me have a link?

[BenAW]

I have searched in the old Alpha test forum, but cannot find these discussions, which I missed.  Would it be possible to  let me have a link?

Try these:
http://ptforum.photoolsweb.com/ubbthreads.php?ubb=showflat&Number=63097&page=1
or
http://ptforum.photoolsweb.com/ubbthreads.php?ubb=showflat&Number=63097&page=1

I really urge you to try Versioning without propagating anything. Once you are comfortable using the Versioning, you can always decide to propagate something if you  see a reason to do so.

[BenAW]

The Versioning tab combines several settings and options related to versions, from categories to collections to colors to metadata propagation.

Problem appearently was that it gave the impression that propagation is required to get versioning to work. I belief that as default even some items are checked for propagation.
I suggest to at least rename the Versioning tab into Propagation, and have nothing checked for propagation as default.

[DigPeter]

Ben - thanks  for those links.

[Mario]

A new version relation does not propagate anything by default.

[BenAW]

A new version relation does not propagate anything by default.

I believe that the relations shipped with IM5 have also some items checked for propagation.
Since users who start with versioning likely will first edit an existing  relation they "inherit" the checked propagation items as well.

[Ferdinand]

I support the idea that the shipped versioning definitions have no propagation enabled.

I think the biggest potential risk in propagating is the one that Peter found, propagating orientation along with other EXIF.  I can't imagine when you'd want to propagate orientation.  My V3.6 synchronisation script had an option "Reset EXIF orientation to Top/Left", which was enabled by default. 

I wonder if V5 needs an EXIF propagation option that includes orientation.

[cytochrome]

Most people correct orientation, if needed, in the raw converter, so both raw and jpg/tiff appear correctly oriented.

In the RW2 propagation rules I had to checked Exif data without orientation because one of my Pana lenses (20mm/1.7) has no orientation sensor. Since ASP does not write to the RW2, the flag was not raised (it was set only  in the xxx.rw2.xmp buddy) and IM5 showed the portrait images as landscape. So I turned them 90° and discovered later that the corresponding JPGs were now lying on the flank because of orientation propagation. I had to turn them back...

I agree that IM could do without propagating Exif orientation.

Francis

[Mario]

I believe that the relations shipped with IM5 have also some items checked for propagation.

I ship an example NEF versioning rule, which is disabled by default. And it only propagates safe data: collections, categories and rating/label.

[Mario]

I wonder if V5 needs an EXIF propagation option that includes orientation.

Your choice. Either propagate all EXIF data, or just use the "EXIF without orienation" sub-set.

[Ferdinand]

I know that IMatch is the DAM for adults, but even adults get tripped up sometimes.  This is one of a couple of things that I expect will trip people up often (the other being the fact that the saved version of the active workspace changes as the active workspace is adjusted).

[Mario]

...the fact that the saved version of the active workspace changes as the active workspace is adjusted).

Doesn't. IMatch only changes the currently loaded workspace.
If you save your current workspace to "Blah" it will be saved and retained in that state.
You can then continue making changes to whatever workspace you have currently loaded without ever touching Blah.

But when you load "Blah" you make it the active workspace. And the active workspace is what you work with and which is used by IMatch to record your changes.

[Mario]

I know that IMatch is the DAM for adults, but even adults get tripped up sometimes.  This is one of a couple of things that I expect will trip people up often.

All new relations don't propagate anything. Safe.
If a user starts enabling metadata propagation, all I can do is to assume that he reads the help, at least. Or posts a question in the community. Or follows one of the tutorials. If a user randomly clicks on options and modifies his files, he can and will cause problems. But that's true for all non-trivial application. A bit of common sense never fails.