Writing to Exif in raw files

[cytochrome]

There is a number of threads in the forum (last example here: https://www.photools.com/community/index.php?topic=3461.0 ) where fellow IM users don't understand why the desired action (here Rotation of a raw image, but it concerns also gps, date etc..) is not effected using the default IM configuration => MGW compliance and No writing to Exif.

Why this is so has been explained several time by Mario but as long as you are not actually affected by the problem you don't grasp the salt of it (was my case at least).

So I suggest that :

- either Write to Exif is enabled by default for all raw formats that handle it (that is all I tested) or

- that the Help states very clearly for example in this section "Change or Correct the EXIF orientation metadata in the image" and again here "EXIF Orientation" that it will not work unless the user allows writing to Exif.

At least this is my experience...

Francis

[Mario]

I've added a test and message to the EXIF date/time already for the next build.
For EXIF orientation, I have not planned to do the same. Most users never fiddle with that. That's a feature of the past, where cameras often set the orientation wrong.  Today this happens mostly with mobile phone images, but these are JPEG anyway.

If this becomes more of as burden, there is always the Feature Request board. Requests in a normal post like this will be forgotten in a week. For help update requests, please us the official method (feedback link at the bottom of each help page).

[cytochrome]

It is a thing of the past but can still be quite annoying:

- when a raw is shot vertical the camera sets the orientation tag, for example 90° CW
- the derived JPG (by ASP in my case) is rendered vertical with no orientation tag, OK
- if one forgets to check "Exif data without orientation" in the versioning rules the tag is propagated as soon as one propagates something else like keywords
- as a result the JPG is stuck with a i.e. 90° CW and since virtual rotation is broken (at least in my iM installation) this is all that is

Francis

[Ferdinand]

- either Write to Exif is enabled by default for all raw formats that handle it (that is all I tested) or

I understand the problem you have had, and if the only EXIF that would get written was date, time, orientation, gps (perhaps) than I would not be concerned.  But the trouble is that when you enable this then certain other EXIF fields will get populated when you input XMP fields like description (caption), copyright and perhaps others, and this would cause me trouble.  I have found that they can be hard to subsequently clear, although of course if you know what you're doing then the right ECP settings will do it.

So I would suggest that this is not the default.  Perhaps a warning instead.

[cytochrome]

Yes a warning is probably enough.... if people read it and understand what is going on. All the push-pull of metadata to the file than back  and forth again to keep everything in synchrony is not so intuitive.

And speaking of push-pull (the term is used in biochemistry for electron shuttling but is appropriate I think) all is not clear or working. As an example:

- take a vertically shot NEF the body sets orientation to appear as "Rotate 90 CW" in both {File.MD.XMP::tiff\Orientation\Orientation\0} and {File.MD.Exif::Main\274\Orientation\0} (my metadata tab shows both so I can follow the state of things)

- convert with ASP and let metadata be propagated (excluding Exif and XMP orientation). The NEF is "NEF" the ASP JPG is as ASP

- convert with DxO and propagate, you get DxO

The upper orientation tag is XMP::tiff\Orientation and the lower is Exif::\Orientation

I had expected that all the intense  push pull activity would synchronize at least rotation....

Not sure what will happen to the DxO JPG after some more writing and updating in IM

Francis

First Nef, then ASP jpg then DxO jgp



[attachment deleted by admin]

[Ferdinand]

I haven't been following this discussion in detail, but it looks to me like DxO is outputting conflicting and partly wrong metadata. 

When you output a JPG from a RAW converter it should be in the correct orientation, and the orientation tags either removed or set to normal.  Since you're not propagating orientation, IMatch won't change these.  The fact that the DxO image has inconsistent values, and the XMP tag is incorrect, suggests to me that it's DxO that's doing this. 

Does ExifToolGUI show the same thing immediately after DxO has output the image and before IMatch has ingested it?

[lnh]

I've added a test and message to the EXIF date/time already for the next build.
For EXIF orientation, I have not planned to do the same. Most users never fiddle with that. That's a feature of the past, where cameras often set the orientation wrong.  Today this happens mostly with mobile phone images, but these are JPEG anyway.

If this becomes more of as burden, there is always the Feature Request board. Requests in a normal post like this will be forgotten in a week. For help update requests, please us the official method (feedback link at the bottom of each help page).

As one of the people who has struggled with EXIF at times, a warning or something in the help docs would be helpful. I did consult the docs first before thinking something was wrong. Orientation sensors in cameras are understandably poor in nadir shots, and that is where the problem is most apparent with modern cameras. One place I'd use it is for rotating input images going into panoramic software like PTGui which can use raw files but they must be in the same orientation and haven't seen an ability to rotate them within PTGui. The issue with smart-cameras/smartphones may become more common in the future as the next version of Android ("L" release) will expose a RAW image format. Probably wouldn't use RAW from smartphone myself, but it will be out there soon. Off to the feature request section...

[cytochrome]

I haven't been following this discussion in detail, but it looks to me like DxO is outputting conflicting and partly wrong metadata. 

When you output a JPG from a RAW converter it should be in the correct orientation, and the orientation tags either removed or set to normal.  Since you're not propagating orientation, IMatch won't change these.  The fact that the DxO image has inconsistent values, and the XMP tag is incorrect, suggests to me that it's DxO that's doing this. 

Does ExifToolGUI show the same thing immediately after DxO has output the image and before IMatch has ingested it?

Yes it is DxO that is scrambling the orientation tags and IM just reproduce it. I thought that it would/should copy from Exif (correct) to xmp?? and so get Horizontal for both.

Metadata treatment in DxO is very poor and in addition it strips a lot of the Exif data. ASP is smarter, it converts a vertical raw to a vertical jpg and does not set any orientation tag.

Francis

[attachment deleted by admin]

[Ferdinand]

When you output a JPG from a RAW converter it should be in the correct orientation, and the orientation tags either removed or set to normal.  Since you're not propagating orientation, IMatch won't change these.  The fact that the DxO image has inconsistent values, and the XMP tag is incorrect, suggests to me that it's DxO that's doing this. 

Yes it is DxO that is scrambling the orientation tags and IM just reproduce it. I thought that it would/should copy from Exif (correct) to xmp?? and so get Horizontal for both.

Metadata treatment in DxO is very poor and in addition it strips a lot of the Exif data. ASP is smarter, it converts a vertical raw to a vertical jpg and does not set any orientation tag.

One of the reasons I like to propagate EXIF is because it cleans up what the raw converter outputs.  But one tag you *can't* clean up this way is orientation.  By definition all RAW files are really in landscape mode, and if you rotated the camera then the image is virtually rotated.  As I said, the output JPG (or TIFF) should be correctly oriented and no orientation tag set.  If DxO really is outputting incorrect metadata then I'd be complaining to them.

It's the XMP tag that's incorrect, isn't it?  You could probably use a metadata template to reset it, or the ECP. 

I assume also that you're using the latest version of IMatch and propagated recently.  There was an issue with IMatch propagating the XMP orientation field when it propagated XMP, but this was fixed several versions ago.  There's now a "don't copy xmp orientation" option that should be checked.

[cytochrome]

I propagate Exif in case I lose the raw (could happen with all my experimenting). So I still have the camera settings in the jpg, although frankly I rarely look back.

I am not sure I understand you, and the reverse

The raw in question was taken vertical, the camera writes "Rotate 90 CW" to both Exif and xmp tags. ASP converts to JPG also in portrait orientation and clears both tags (in the jpg). This is fine.

DxO writes Horizontal to the Exif tag and Rotate 90 CW to the xmp tag (of the jpg of course), this is messy. Perhaps I'll complain, they are known to be responsive but clearly they gave little thoughts to metadata. Mario could teach them a crash course...

Caution!! Just did another test with a NEF from the same body (D7000) but taken last year:

- after DxO the jpg has Exif Rotate 90 CW and xmp is blank. When the jpg gets ingested by Imatch the result is fine: both tags are Horizontal, which is OK.

Conclusion: when the xmp orientation is already set Imatch fails to reset it.

The question is why this difference? the only I can see is that I changed the menu setting in the Nikon, there is an Autorotate item in there. I'll have to experiment. Its getting complicated

Francis

[Mario]

Conclusion: when the xmp orientation is already set Imatch fails to reset it.

What? I did no longer follow this thread, so please can you summarize?
None of the WIC codecs or image libraries looks in XMP to determine if and how an image should be rotated. Only in the EXIF record.
When IMatch imports a file, ExifTool maps the EXIF orientation to the corresponding XMP tag.
Where does IMatch fail to reset anything?

[cytochrome]

Hello Mario,

It is all in reply #4 in this thread with captures of my metadata panel.  It shows the orientation tags from a NEF (first), from a JPG converted with ASP (second) and with DxO (third).

In these captures the upper orientation tag is XMP::tiff\Orientation and the lower is Exif::\Orientation

It is really complicated because depending on the Nikon body settings for "Autorotation" DxO writes nothing to the xmp tag or something different from what it writes to the Exif tag. ANd in this case Imatch copies it..

I write to the image files (no sidecar), MGW compliance ON, and Exif writing allowed.

Francis

[Mario]

I still don't understand, sorry.
Please file a bug report with the details and I will look into this later.
I don't have all the software you use, so please also provide sample files, a list of your metadata settings, what exactly you consider as a problem caused by IMatch etc.

[cytochrome]

Mario, I'll do this if I can't solve it myself.

Presently I can blank both Exif and xmp Orientation tags from the matadata tb, this works and takes not much more time than doing a virtual rotation.

But definitely something changed in the way IM handles orientation and it is DB linked: I have old jepg folders (imported during beta testing) with a mixture of vertical images:

- most have "rotate 90 CW" but are nevertheless shown vertical. I suppose I made a virtual rotation (CTRL-4,5,6) at that time and this rotation sticks since that time.
- some other JPGs (same folder) also with "rotate 90 CW"  are shown horizontal. Now CTRL-4,5,6 works only for the session. After colse/open they are back to horizontal. As I said I can easily blank both Orientation tags so no big deal.

Francis

PS I see in Bug Reports that this problem is solved and corrected in 5.2.8  merci beaucoup...

[Ferdinand]

I wasn't going to reply as I didn't want to make this thread any more incomprehensible, but sadly it seems it's too late for that.

I have read your posts and I think I understand the problem.  I mentioned previous versions of IMatch because at one point IMatch propagation of XMP produced similar symptoms to what you're experiencing.  This issue was deal with in 5.0.158.  https://www.photools.com/community/index.php?topic=2283.0 

The fact that you're getting this issue in older folders makes me suspicious that it might be a legacy of the earlier behaviour

My other point was that you could edit the metadata to fix what I think your problems is, but it seems you already discovered that.  Personally I'd use a metadata template to do this.

[cytochrome]

Ferdinand, I confess I had overlooked this threat, because 1) the problem seemed rare at that time, 2) I guess virtual rotation still functioned (IM was version 5.0...), and 3) we have different options, no sidecar on my side for example. This last one should make no difference but it seems it does.

And at that time I was not aware that orientation was governed by two different tags and had never really looked into the arg files and how the supposed synchronism mechanism was setup.

When I read all this today it makes a lot more sense. It would have saved me a lot of testing if I had understood the implications.

Seems Mario has found a cure for the virtual rotation being reset at close/open of the DB. But still, depending on settings (there are many and interfering) discrepancies between Exif, xmp and IPTC may occur. Mostly they can be ignored simply by displaying only the "good" tag, but the bad ones are hidden in the background and may creep to the surface when some particular data update occurs.

Sometime ago Mario, in a rare outburst of anger, had written something like "I hate, hate, hate metadata!!" (can't find it back right now). He even alluded at stripping IM of some of the metadata settings so everybody follows the same "simple" route... I hope he forgot about it, would void IM of a good part of its power. And charm...

Francis

[Ferdinand]

Note that I edited my previous post because some issues with formatting of the URL meant that the last few sentences were hidden.

[cytochrome]

..............

The fact that you're getting this issue in older folders makes me suspicious that it might be a legacy of the earlier behaviour

My other point was that you could edit the metadata to fix what I think your problems is, but it seems you already discovered that.  Personally I'd use a metadata template to do this.

Not so sure about the legacy. I think it was my fault, I had not checked NO propagation of xmp and  Exif rotation in my version rules. So some JPGs were mal-oriented but they were few, I corrected with CTRL-4,6, and that was it. But recently I copied keywords to some files in these old folders and portrait JPGs were reverted to landscape, and could not be corrected by virtual rotation because of the close/open DB thing. 

Lets see the state of things after Mario introduces 5.2.8

And yes, I setup a metadata template that clears xmp and exif rotation. This works OK, no reversion after DB close/open

Francis