Tag Manager

[sinus]

Hi Mario

Sorry, if I write stuff, what is already in the help.
I wanted only to be sure, if I have understand the new Tag Manager correctly.

1 The new feature is only important, what tags from a file will be imported into the database!
The file itself with all the not imported tags will not be touched.
Is this correct, what I write?

2 If IMatch do write-back metadatas, to a xmp-sidecar or into the file itself, the not imported tags in the sidecar or in the file, they will be not wiped out of these? They are still in the sidecar or file!?
Is this also correct?

1 Because all (not imported), unnecessary tags remains in the sidecar or files, I can import them later, if I want. For this I must change the Tag Manager and import the Metadata again.
Is this also correct?

So, if you say, yep, this all is correct, then I think, I have understood the Tag Manager, if not, well, then I must reread again.
Thanks for the good work, I think, this is a good idea to reduce the amount of data.

[Mario]

yes, yes and yes.

[sinus]

yes, yes and yes.

Ah! THANKS!
I already checked some of the data, what can be wiped out of the database:

phew, a LOT of fields, what I never will use and even a lot of fields and values, what I do not understand anyway!

I think, this was a good idea of you! 

[sinus]

Hmmm, maybe I do not understand this or something is wrong.

In the attachment you can see:

Two identical raws, nefs. Both have no xmp-sidecars (also checked with ExifTool).
Both have the same Metadata inside, if I check this with ExifTool.
Means, both have some Nikon-Stuff, like in the attachement.

The left image (with the yellow OK) was imported BEFORE the new release with the new task Manager.
So this looks normal for me, this image has Nikon-Infos in the Metadata Panel, as you can see in the metadata-Panel, for example the Nikon AFInfo2.

If I want get rid of them, I think, I should do a Database Diagnostics, like it is written in the help-file.

So far so good.
But now to the image on the right side (with the pink no).
This image has been imported now (from the backup), and all Nikon-values are gone.
So I thought, fine, if I want see some of them, I have to change the Tag Manager and rescan it.

I opend the Tag Manager, see attachement 2, And did nothing than enabled the "Nikon AFInfo2", where the fields on the right side also are enabled, for example the field "AF Area Mode". I have spottet this field in both images, you can see it also in the metadata Panel in Attachement 1 (value "Single Area").

After clicking in the Tag Manager the box appears, like in the helpfile.

Now begins my problem:
First IF I enable groups or fields, this is for me INCLUDED, not excluded. Means, a enabled field with a check is for me included. Maybe I have not understand this.

But it does not really matter, because if I click on OK and then do a rescan of this (right) image (normal rescan, Metadata rescan or force does not matter), I see never the Nikon fields in the Metadata Panel.

Means, I can NOT see the changed Nikon-fields (in the Tag Manager) somewhere, only in the Exif-Tools.

I did NOT make now a Diagnostics, from my understanding, this would not change something, the diagnostics would get rid of the not more used fields, but not add news.

So I have completely misunderstood the whole thing with this Task Manager or something is wrong.
SORRY for not understand this.




[attachment deleted by admin]

[cytochrome]

Hello Markus,

Not sure I understand your ptoblem, but I just compared a NEF imported with 5.2.8 and now a fresh NEF tagged with Photomechanic and then imported into IM 5.2.10 with deault tag manager settings

In metadata browser mode the complete "Nikon" block of MakerNotes is gone. But in the NEF it is still there. So Im does wipe the info from the Raw file, if this was your concern.

Francis

[sinus]

Hello Markus,

Not sure I understand your ptoblem, but I just compared a NEF imported with 5.2.8 and now a fresh NEF tagged with Photomechanic and then imported into IM 5.2.10 with deault tag manager settings

In metadata browser mode the complete "Nikon" block of MakerNotes is gone. But in the NEF it is still there. So Im does wipe the info from the Raw file, if this was your concern.

Francis

Thanks, Francis,

So, that if fine, your results make sense for me.
The problem, what I have:

If you now want have some (or the entire block) of the Nikon fields to see in IMatch, I am not more able to do so.

Like in your case: Say, you want see not the Nikon block again in IMatch. How can you do that?
I understood the helpfile so, that I simply can open the Tag Manager and enable some fields from Nikon (or groups) and close the task Manager.
And finally I must rescan this image to see now the Nikon-fields in IMatch.

But this does not work here ... or I have understood all this stuff wrong.

[Ferdinand]

When I first saw the new tag manager I thought what a good idea - we don't want unnecessary tags from Adobe and others clogging the database.

However having had a quick look today (a little busy), I have my doubts.  It's certainly true that there's a lot of information imported into the database that I've never really wanted, but equally there is some info that I do want imported.  For example, Nikon has the habit of putting important and useful information in the maker notes.  The defaults seem to import very little.

So a bit like Markus, I've going to have to go through and examine which fields I want and enable them.  This is going to be tedious, and I can't export these settings to back them up or to try alternatives, can I?

Hmmmmm

[Mario]

There are about 40 or so groups alone for Nikon tags. Which to enable depends on your camera, firmware etc. The ExifTool documentation does not document this in detail.

When I enable the groups for the D300, D700, D3, D4 images I have, the Nikon maker notes and other proprietary tags are imported again. Same as before.

Did you only enable a tag and not the group perhaps?

[sinus]

There are about 40 or so groups alone for Nikon tags. Which to enable depends on your camera, firmware etc. The ExifTool documentation does not document this in detail.

When I enable the groups for the D300, D700, D3, D4 images I have, the Nikon maker notes and other proprietary tags are imported again. Same as before.

Did you only enable a tag and not the group perhaps?

Hi Mario
I tried first maybe 15 different tags from different Nikon-groups, did not work.
Then I tried simply one group, the group "Nikon AFInfo2", because I have seen with ExifTool, that fields from this group are present in the nef.

You can see this in the attachement below.
But a rescan does not bring any of the Nikon-fields.

[attachment deleted by admin]

[cytochrome]

MArkus,

I have the same: I cannot select some tags like Nikon NikonAFinfo2 to show again. Or than I don't understand how it works.

So we are two. At least...

Francis

[sinus]

MArkus,

I have the same: I cannot select some tags like Nikon NikonAFinfo2 to show again. Or than I don't understand how it works.

So we are two. At least...

Francis

Thanks, Francis ... good to know! 

Because sometimes I think really, maybe it is simply me ... well, and sometimes it is 

[Mario]

Are you sure you are using the right group?
Please enable all Nikon groups (check each one individually, don't use multiple selection because this does not work yet).
Then check if your data shows up.

I think that the problem is located in the way IMatch skips tag groups right when calling ExifTool. Maybe two groups fall into the same command line group name and thus blend each other out. I did not try such complicated things yet. Enabling all Nikon groups will work around that.

I just did that and it works like a charm with all Nikon files I've tested.

I need to check the code which strips the tags in the command line and see if something rules out some other Nikon groups.

[sinus]

Are you sure you are using the right group?
Please enable all Nikon groups (check each one individually, don't use multiple selection because this does not work yet).
Then check if your data shows up.

I just did that and it works like a charm with all Nikon files I've tested.

Thanks, Mario,
I will do it, just now ... takes some time

[sinus]

Hi Mario

First I opened the Tag Manager.
Then I selected "Nikon AFInfo2". On the right side automatically all fields has been selected.

I close the Tag Manager.

I am sure, that the image has a lot of Nikon-fields (nef). First I checked with ExifTool, and second, this is a copy (from backup) from a file.
And the same file, imported some days ago, shows all Nikon-fields.

-------
I open again the Tag Manager.
This time I select all groups of Nikon (with click on each single group on the left side).

Totally 31 Nikon groups, so with the previous selected groups, there are now 32! Nikon groups selected.
I close the Tag Manager, click OK in the box with the info.

Select the image again and do a rescan, first only Metadata, then a force update.

No change. All values in the Metadata Panel (Browser, Default, my own...) has not more fields, like before.
----------

What can I do more?
I did NOT let run the diagnostic. I guess, this has nothing to do with this "problem".

Should I do a rescan of the nef (the duplicate), what has all the Nikon-info, because I did import this file some days ago?
I do something completely wrong, or it works really not.
At least Francis could confirm the same behaviour.

[sinus]

I tripple-checked it 

In the attachement you see the only Nikon-group, what is present.

All other groups are not in the Browser of the Metadata-Panel.
But they are in the file, for sure. The other (identical) image, exported some days ago, has a lot of other groups and fields, like

Nikon
Nikon AFInfo2
Nikon AFTune
Nikon ColorBalanceUnknown
Nikon FileInfo
...

Hope, this helps you.

[attachment deleted by admin]

[cytochrome]

I did as Mario said: check all Nikon groups. When I do so I find a lot (maybe all - did not count yet-) of the Nikon tags in a fresh NEF freshly imported.

But when I close IM and open again, many (all? -did not count-) Nikon groups are unchecked again. Is this normal?

Markus: in my metadata browser display these Nikon tags start after IPTCEnvelopeRecord...

Francis

[RalfC]

I think that the problem is located in the way IMatch skips tag groups right when calling ExifTool. Maybe two groups fall into the same command line group name and thus blend each other out. I did not try such complicated things yet. Enabling all Nikon groups will work around that.

Having also this problem that Markus described, I enabled all Tags which start with "Nikon" including NikonCapture and NikonCustom and it worked.

Disabling the tag groups "Nikon CaptureOffset" and "Nikon CaptureOutput" caused that not all Nikon tag groups (incl the "Nikon" tag group) where visible anymore. Thus, it seems that there is some cross-effect between the tag groups.

Hope this helps in finding the problem.

Regards,
Ralf

[sinus]

Thanks, Ralf, I think, everything helps Mario here.

[ubacher]

When I click the DEFAULTS button in the tag manager it sets ALL the checkmarks.
(It did not do this when I first checked it out!)

What do I do now?

[Mario]

Having also this problem that Markus described, I enabled all Tags which start with "Nikon" including NikonCapture and NikonCustom and it worked.

Yes, that's what I said. You need to enable all Nikon* groups. I was apparently to restrictive when generating the excludes for the command line at runtime.

[sinus]

Having also this problem that Markus described, I enabled all Tags which start with "Nikon" including NikonCapture and NikonCustom and it worked.

Yes, that's what I said. You need to enable all Nikon* groups. I was apparently to restrictive when generating the excludes for the command line at runtime.

I enabled for sure all Nikon groups and it worked not (I tried it several time).

[Ferdinand]

I was apparently to restrictive when generating the excludes for the command line at runtime.

Is this something that can be rectified?

[J.D. McDowell]

I'm glad I'm not the only one with concerns here. I too shared Markus' concerns about what happens to the files, until I read the help file a few times.

I had a quick look at the fields in Tag Manager and quickly found Nikon fields that I use regularly set for culling from the database (Nikon World Time is a prime example.) As there are a lot of fields I simply selected (turned on) everything with Nikon in the name.

When I closed iMatch last night and ran my backup the database diagnostics DID NOT ask if I wanted to remove database fields (I left some Canon fields in the default settings.)

When I opened iMatch again now, all the additional Nikon fields are deselected again in the Tag Manager and all the Nikon fields in the database are now gone!!

Clearly this is an alarming bug! I've now lost the data, will need to rescan (which will take hours) and not run any database diagnostics till this is fixed.

Also, as an aside, when Nikon releases the D820 or whatever the new camera will be, how will I be able to import or configure those fields if they are not imported by default?

I understand the concept behind this addition, but clearly as of right now it's such a problem for me that I think I may uninstall, go back to 5.2.8 and restore from my last working backup.

Update: I can confirm that I did get (World Time anyway) back after reselecting the tags in Tag Manager and rescanning 1 file. I guess I'll start the entire database rescanning and head out to dinner. After reselecting all the Nikon groups I did get an odd confirmation message (see attached)

[attachment deleted by admin]

[Winfried]

I probably have the same issue with Pentax *.dng and *.pef in 5.2.10

{File.MD.Pentax::LensData\9\FocalLength\0} normally shows up the "basic" FocalLength (not reflecting the effec of the converter).
I am not able to get this field shown up or I didn't find the right settings so far.
If I use the new Pentax 1.4 converter the field {File.MD.Exif::Main\37386\FocalLength\0} shows the corrected FocalLength.
If these two values are different I know, I had used the converter.

Winfried

[Mario]

I was apparently to restrictive when generating the excludes for the command line at runtime.

Is this something that can be rectified?

Already fixed. The problem was caused by IMatch trying to skip superfluous data on the lowest level, in this case by instructing ExifTool to skip data, e.g. from Nikon already when processing the file. IMatch can skip groups and tags later, but getting rid of them already in ExifTool reduces the amount of data to transfer, the amount of data ExifTool has to process.

There problem is that there are, e.g. for Nikon, 50 'level 3' groups like Nikon AFInfo or Nikon AfInfo2, but on the command line, ExifTool can only skip level 2 groups, namely Nikon, NikonRAW and NikonCustom. IMatch thus has to check into which of the level 2 groups the level 3 groups fall and then to decide whether or not to Skip one, two or all Nikon groups. And this depends on the optional tag groups and tags included by the user.

The 5.2.10 sometimes excluded too many level 2 groups, so even if a level 3 group was enabled, or one or more tags, these never reached IMatch.

This affects only users who for some reason need to include one or more maker note groups or tags. And the work-around is to include all Nikon or Canon or whatever groups. This prevents IMatch from skipping too much and restores the behavior we had before the 5.2.10 - all maker notes will be imported.

I have that fixed already for the 5.2.12 build. IMatch now checks correctly if a included tag forces inclusion of a group, and if groups included by the user force inclusion of a level 2 group.

I've also fixed the multiple selection glitch and the 'Default' button which sometimes not reset the exclude state for groups which are by default excluded by the IMatch but have later been included by the user.

All this only affects users who try to include optional tags. But the work-around is easy so not too much pressure. I will release an update in a couple of days.

[Mario]

I probably have the same issue with Pentax *.dng and *.pef in 5.2.10

{File.MD.Pentax::LensData\9\FocalLength\0} normally shows up the "basic" FocalLength (not reflecting the effec of the converter).
I am not able to get this field shown up or I didn't find the right settings so far.
If I use the new Pentax 1.4 converter the field {File.MD.Exif::Main\37386\FocalLength\0} shows the corrected FocalLength.
If these two values are different I know, I had used the converter.

Winfried

Make sure you include all groups with names starting with Pentax (check each one individually). This will bring back in all Pentax maker notes in your files. See also my comment above.

[Ferdinand]

The workaround for me was to downgrade to 5.2.8.  I have reservations about this and won't upgrade until I understand this fully, by installing and testing 5.2.12 on the laptop.  I rely on some of this information and I'm in the middle of a big production run and can't afford to experiment right now. 

My preliminary view is that the defaults too far.  My plan is to re-enable everything and only selectively disable things progressively.  That's my advice to others who have any interest in metadata beyond basic XMP, IPTC.  For example, see my comments in another thread on ICC colour profiles.  The embedded profile is the only 100% reliable piece of metadata for this information, sadly.

[Ferdinand]

To me at least it seems more efficient to go in the opposite direction: enable everything and drop superfluous fields over time.

I have come to the same view.

[Mario]

Do whatever pleases you. Did you also restore an old backup your database? The 5.2.10 introduced some changes there as well. Did you see no warning about the database having been created by a newer version?

The defaults are based on my experience and analysis. The general goal was to keep as much useless data out of the database. Asking users to start with all tags and then remove more and more tags is not the right way to do this. Much to work-intensive and error-prone for most users. Keep it simple is key here.

Of course can not know that sinus is keen on the AFInfo2 Nikon maker note. Or that for your files none of the standard color profile tags works and you need to see the name of the color profile you have embedded in your file. I think I remember that you do a lot of grayscale with custom profiles...

But that's exactly why I created the Tag Manager dialog. And why I asked users in the release notes to check which tags are now excluded by default and if that meets their requirements.

The unfortunate glitch (bugger) with the too restrictive group level handling has been fixed, and then you can include/exclude any tags you want and in any way you see fit for your personal workflow. The majority of IMatch users will benefit from the new handling without ever doing as much as looking at the Tag Manager.

[Carlo Didier]

For me it's a great feature and works fine. As I don't need more than some basic Exif information (shutter speed, aperture, etc) and GPS info in the database, I was able to reduce the size of the database from 4.6GB to 3.5GB.
Great!

[Ferdinand]

Did you also restore an old backup your database?

I was cautious and hadn't opened by production database in 5.2.10, so only had to restore backups of my test databases and the .pts file.

Asking users to start with all tags and then remove more and more tags is not the right way to do this. Much to work-intensive and error-prone for most users.

But nor is the alternative - enable all and remove through trial and error.

I think I remember that you do a lot of grayscale with custom profiles...

You have a very good memory.  But I have avoided those files to keep my analysis simple.  I concentrated on sRGB and AdobeRGB.  I will send you a sample file.

[sinus]

And the work-around is to include all Nikon or Canon or whatever groups. This prevents IMatch from skipping too much and restores the behavior we had before the 5.2.10 - all maker notes will be imported.

Here is to say, that we must include ALL Nikon (or Canon) groups, even if we have no images with values from such groups. This means, I had to uncheck "Show only groups and tags with data" - otherwise it works not.

Only with really all groups checked (also these - see the attachement - with 0 values in it), it works, do get the fields back, with a rescan.

So I use now this version with the tag Manager, but I will not do a diagnostics, before the new .12 comes out. Because the diagnostics will wipe out unnecessary fields, but I want have better control about this.

I am glad to hear, that Mario did fix already something.

[attachment deleted by admin]

[Nytewulf]

The defaults are based on my experience and analysis. The general goal was to keep as much useless data out of the database. Asking users to start with all tags and then remove more and more tags is not the right way to do this. Much to work-intensive and error-prone for most users. Keep it simple is key here.

The Tag Manager is a welcome feature.  My suggestion about starting with everything and dropping tags over time was directed at Ferdinand, who seemed to see a problem similar to one I was looking at.  It's very easy to identify the tags I don't need; however, what could be useful or invaluable is far less clear and changes as I did deeper into the metadata available.

The problem I was trying to address wasn't with the Tag Manager -- it's with how to use it most effectively given the complexity of metadata and the way I use it in IMatch.

John