Metadata-Analyst

[sinus]

This seems to be a cool App!
Gives you a good feeling if no errors are detected. 

I wonder, a lot of my image has always the same errors, see my attachement.
I have also attached the json-file (as txt), just in case.

Is this emported to change, what the App reported?

Thanks for any advise.

[Mario]

This such tells you that the XMP record your camera has embedded in the image is incomplete and does not follow the XMP standard and neither the Metadata Working Group recommendations.
Probable Nikon has infested your files with a rudimentary XMP record containing only one field: "rating = none".

Which makes no sense and even requires a special option in IMatch to "ignore rating in embedded XMP record".
Because XMP embedded in files overrides XMP in sidecar files by default. Embedded metadata is considered more important for obvious reasons.

You should make sure that you have only one source of truth for the metadata in your files.
Having two XMP records for the same file (embedded and in sidecar) is calling for trouble.

For RAW files, the industry standard is to use XMP in a sidecar file.
Not adding a half-assed XMP record consisting only of "rating = none" into the RAW file.
The XMP record should be complete, standard-conformant and contain a copy of the EXIF data in the corresponding namespace.

Feel free to complain to Nikon. I'm sure they will listen and provide excellent support.

[jch2103]

A familiar problem. My Nikons also add an XMP Rating to the NEF by default. Before Mario added the option in IMatch to "ignore rating in embedded XMP record", I routinely ran the ECP preset 'Delete XMP Metadata' for all newly added photos in IM.

Because the Metadata Analyst reports conflicts between these embedded and sidecar XMP records, I'm thinking I should go back to my prior practice of scrubbing all XMP records from my NEF files. (Unless Mario amends Metadata Analyst to ignore these 'standard' conflicts, i.e., the Rating and any implied conflicts between embedded and sidecar XMP records when there are no other actual embedded XMP records. Should this be a feature request?)

[Mario]

The MDA is mainly for my use, and I know how to deal with this.
I consider informing users about duplicated XMP data is a good thing. This particular problem can cause issues along the workflow, depending on which applications are used (and in which order).

[jch2103]

This is why I inquired about creating ECP presets as Favorites in a previous post (which I can't locate at the moment). Perhaps an app could do this also, to help integrate embedded XMP cleanup into standard workflow (given that I don't expect Nikon, Canon, etc. to do anything to prevent this problem).

[Mario]

Running ECP scripts which actually wipe out data in the original image file should not be to easy to run, IMHO.

If you use this often, you can use a Creating and Configuring to run ExifTool directly.
This takes only a few minutes and will do exactly what you want and how you want it done.

For example, something like

"c:\program files\photools.com\imatch6\exiftool.exe" -overwrite_original_in_place -xmp= "{File.FullName}"

or similar.

[sinus]

Thanks for your answers,  helps me a lot. 

Interesting.
I will think about this and look at this deeper. Also the GPS-stuff.

Such a tool like Metadata-Analyst gives me a good feeling, if all is ok.

I guess, this is human. If you build something, say a house (my father did) and it is not your profession, it gives you normally a good feeling, if finally an expert checks all thing (electricity, static and so on) and give you a feedback "all is ok".
If he says, the cellar should have thicker walls to avoid water from outside, then usually you would do it and finally, if all is ok, you are happy (with yourself).

MA can give maybe a bit an equal feeling, if your files are ok.
Therefore I think, it is a really cool app.

Mario, this is only a simply question, if this would make sense:
(comes simply just now in my mind). 
I believe, MA checks files, sidecars and database.

Quite a lot of users in the past has asked, the DB would be slow.
Would it not make sense to add in the MA a "speed-test"?
Only roughly of course. For example search the DB for "a" or something like this and make a relation to the size of the DB and then the MA would give some rouhly advise like:

The search-speed is OK.
The search-speed is a bit slow, make a diagnosis and compact.
The search-speed is that slow, there must be something wrong or the computer is very old.

This would avoid maybe some questions here, hmmmm, but, if I think, then would comes also other questions, why is my DB that slow. 
Maybe not a very clever idea. 

But once written, I let it here.

[Mario]

The Metadata Analyst analyzes metadata in files.
I see no need to add speed tests for the IMatch database to this tool.


IMatch logs several performance measures to the log file.
If a user complains about performance problems, this usually helps to solve the problem or at least pin the source of the speed issue.

[sinus]

The Metadata Analyst analyzes metadata in files.
I see no need to add speed tests for the IMatch database to this tool.


IMatch logs several performance measures to the log file.
If a user complains about performance problems, this usually helps to solve the problem or at least pin the source of the speed issue.

I see.
Thanks for your answer. Makes sense.

[sinus]

This such tells you that the XMP record your camera has embedded in the image is incomplete and does not follow the XMP standard and neither the Metadata Working Group recommendations.
Probable Nikon has infested your files with a rudimentary XMP record containing only one field: "rating = none".

Which makes no sense and even requires a special option in IMatch to "ignore rating in embedded XMP record".
Because XMP embedded in files overrides XMP in sidecar files by default. Embedded metadata is considered more important for obvious reasons.

You should make sure that you have only one source of truth for the metadata in your files.
Having two XMP records for the same file (embedded and in sidecar) is calling for trouble.

For RAW files, the industry standard is to use XMP in a sidecar file.
Not adding a half-assed XMP record consisting only of "rating = none" into the RAW file.
The XMP record should be complete, standard-conformant and contain a copy of the EXIF data in the corresponding namespace.

Feel free to complain to Nikon. I'm sure they will listen and provide excellent support.

Sorry to come back here again.
Reading your advise above, I did make sure, that I am with the "Metadata Working Group" conform.
I even clicked in the Metadata 2 on "default" and doublechecked, that MWG was on.
There is also choosen "ignore rating of embedded XMP record".

Means, my setup is MWG-conform and IMatch-Defaults. No other things are done during the first import into IMatch.

With theses system I let import some files import from a small Sony and a Nikon D750.
Then after the import I let run Metadata-Analyst.

Sony: perfect: no warnings, all is fine.
Gives a very good feeling! 

Nikon nef: GPS-warnings (normal, because no GPS)
XMP-warnings
Detailed Validation warning

Nikon jpg: GPS-warnings (normal, because no GPS)
XMP-warnings

This is irritating, because I did like you wrote, MWG and nothing else.
The D750 delivers only a nef and (if wished) a jpg.

BTW: the nef delivered the last warning "detailed Validation", the jpg did this not, this is the only difference between NEF and jpg (from the same camera)

Now I see 2 options for me:

1) ignore the Metadata-Analyst (what you have written, that is a tool specialy for you and problems with users)
2) You know, that I have oversee an important thing to choose, additionally to the mwg-defaults.

What do you think, ignore or do you have still an idea?

edit: I could not resist and have attached the positive result (no Problems were detected) from the sony, feels really very good 

[Mario]

Use 1)

Your NEF has an embedded XMP record. Which is non-standard and creates many problems. IMatch does not update it, it stores XMP to the sidecar file.
Ask Nikon why they produce a non-standard XMP record, sometimes only consisting of rating=none but no other required field.

[sinus]

Use 1)

Your NEF has an embedded XMP record. Which is non-standard and creates many problems. IMatch does not update it, it stores XMP to the sidecar file.
Ask Nikon why they produce a non-standard XMP record, sometimes only consisting of rating=none but no other required field.

Thanks, Mario
I will do so and I will not more use Metadata Analyst, gives me only a bad feeling  with nefs .
Sony made my day. 

Thanks! 

[Mario]

Your NEF files are OK. No worries.

[sinus]

Your NEF files are OK. No worries.

Now you made my day too! 

Thanks!!!   

[jmsantos]

This thread is very similar to a query I made some time ago, just over a year.:
https://www.photools.com/community/index.php?topic=8951.0

I took up the topic in this thread:
https://www.photools.com/community/index.php?topic=9039.0

And more recently in this other:
https://www.photools.com/community/index.php?topic=9856.0

They may be very particular cases, but for me the important thing is that the reconciliation of metadata is not well resolved when there is embedded data that does not appear in the XMP sidecar, and it is not that there are discrepancies between the two sources. To my surprise, as it says here
https://www.photools.com/community/index.php?topic=9039.msg63551#msg63551
the solution is to go through Lightroom.

[Mario]

Or just remove the partial embedded XMP record in the NEF to have only one source of truth.
The rich XMP record created by ExifTool & IMatch is usually much better than anything other software produces.

[jmsantos]

Or just remove the partial embedded XMP record in the NEF to have only one source of truth.
The rich XMP record created by ExifTool & IMatch is usually much better than anything other software produces.

I cannot do that without missing important information. I explain the problem again in a simplified way:
In an Institutional Archive there are photographs in RAW format (NEF, RAF, ...) that contain embedded geolocation metadata; and also XMP sidecar files that only have author and copyright data. With default settings (MWG compliant), IMatch reads the data from the XMP but not the embedded data; with Favor XMP sidecar, IMatch reads the embedded but not the XMP sidecar data. The only solution I find to combine both metadata sources is to go through LR.

[Mario]

Favor XMP sidecar will favor the sidecar over the embedded.
Send me one of these files so I can have a look. Embedded XMP in RAW files is really rare. Which camera did produce these files?
The most Nikons I know embed rating=none in XMP, but not much else. Old Capture versions embedded more metadata in XMP. You can also make ExifTool copy the XMP from the image into the sidecar, as a sort of one-time conversion, before wiping the XMP from the RAW.

[jch2103]

Send me one of these files so I can have a look. Embedded XMP in RAW files is really rare. Which camera did produce these files?
The most Nikons I know embed rating=none in XMP, but not much else.

Nikon's most recent cameras (including my Z6) certainly include more XMP data. Adobe products read these tags for processing. I suspect we'll only see more of this in the future. Below is straight out of the camera (via ECP searching for 'XMP'):

[XMP-xmp]       Creator Tool                    : NIKON Z 6 Ver.03.00
[XMP-xmp]       Rating                          : 0
[XMP-crd]       Exposure 2012                   : 0.00
[XMP-crd]       Highlights 2012                 : 0
[XMP-crd]       Shadows 2012                    : 0
[XMP-crd]       Luminance Smoothing             : 13
[XMP-crd]       Luminance Noise Reduction Detail: 75
[XMP-crd]       Luminance Noise Reduction Contrast: 0
[XMP-crd]       Color Noise Reduction           : 7
[XMP-crd]       Color Noise Reduction Detail    : 50
[XMP-crd]       Color Noise Reduction Smoothness: 50
[XMP-crd]       Sharpen Radius                  : 2.00
[XMP-crd]       Sharpen Detail                  : 25
[XMP-crd]       Sharpen Edge Masking            : 0
[XMP-crd]       Contrast 2012                   : 5
[XMP-crd]       Camera Profile                  : Camera Standard

Link to sample image: https://1drv.ms/u/s!AtlBwiIf8wQz9FOAkH111HdK54bh?e=X6xGeR
At this point, I've been including an extra workflow step of deleting XMP tags from the NEF to avoid all potential conflicts w/ sidecar XMP data, but that seems inefficient.

[Mario]

IMatch will also read this data.

The [XMP-crd] namespace means Adobe Camera Raw Defaults so this XMP data is usually written by Adobe ACR during import.
But ACR always writes the data into the XMP file, not in the RAW file (as far as I know).

IMatch imports all the data from your RAW file, including the Adobe Camera Raw settings, your copyright , Nikon maker notes etc. IMatch also maps the EXIF data into XMP, as per XMP standard and MWG.

By the XMP standard, XMP data for RAW files should go into the sidecar file. So when you write back, IMatch will write a external XMP file.
Now we have two separate sets of XMP data for one image.

- When I open the NEF (with the XMP created by IMatch) in Adobe Lightroom, I see the metadata (Copyright, Title, Keywords) I have added in IMatch. Not the data from the embedded XMP record.
- When I change metadata in Adobe Lightroom, it modifies the XMP file, not the XMP data embedded in the NEF file.
- The modified data is imported and shown by IMatch automatically.

- When I open the file in Photoshop via ACR, I also see the Metadata from the XMP file, not from the embedded XMP record.
- When I open the file in Affinity Photo, I see the Metadata from the XMP sidecar file.

I do the same with the original file (no XMP sidecar file).

Lr shows me the contents of the XMP record embedded in the image (as does IMatch).
When I change metadata, Lr writes it to a new XMP file. It does not modify the XMP data in the image, it does not even sync the copyright notice, which should be mapped to EXIF...!

Affinity Photo shows me (a bit) of the XMP data embedded in the file, but cannot modify it. Metadata handling is definitely not one of the strengths of Affinity Photo, still after all these years.

It looks like all applications behave the same, they favor the metadata in the XMP sidecar file, when it exits.