NEF files with XMP sidecars

[TomS]

I've imported a few raw files that I had tagged using other software before importing them into IMatch so the folder I indexed had xmp files accompanying each of the raw files. Those xmp files included a variety of metadata including keywords (hierarchical and flat), creator information, copyright information, and location information. None of that metadata shows up in the metadata panel when I select any of those files. If I add a keyword to a file using IMatch, the xmp file for that raw file gets updated and now all of the metadata in the file does appear in the metadata panel. Is there some way to get the metadata that is in the xmp files to show up without forcing IMatch to update some of the metadata first?

[Mario]

IMatch automatically imports XMP data from sidecar files. I work with NEF files myself every day.

But IMatch is also able to import XMP data embedded in the NEF file (e.g., by Nikon Capture or other Nikon software).
IMatch by default merges the embedded XMP data and the sidecar XMP data, giving the embedded data precedence.

This way all users who worked with Nikon software which insisted on embedding XMP data instead of using sidecar files have a smooth workflow and transition.

Please first check if your NEF file contains embedded XMP data (e.g. in the Tools menu > ExifTool Command Processor command). Use the "list metadata" preset and check if the NEF file contains XMP data.

You can control how IMatch deals with embedded XMP data (and sidecar files) on a per file format basis under Edit > Preferences > Metadata: File Formats. Open the dialog and then press <F1> for details.

It would also help if you can upload a sample NEF and XMP file somewhere, or send them to me per email (see my contact info in my signature below). If you send me an email, please include a link to this topic. I handle between 30 and 50 emails per day, and this helps me.

[TomS]

Mario,

I have emailed a xmp file to you just now. The nef file does not have any xmp data in it according to exiftool. It may be worth mentioning that the xmp file would have been created by Photo Ninja, my raw processor, because I processed this batch of raw files before I imported them into Photo Supreme. During import and after import I would have used Photo Supreme exclusively to add metadata to the file. I have my reasons for doing it in this order, which I won't go into here unless it becomes relevant.

Exiftool seems to have no problem reading the xmp file when I use the "List Data in XMP files" preset.

Let me know if you need a copy of the nef file also. It's a large file so I didn't want to upload or email it. I could easily upload it to OneDrive or somewhere else if you want.

Thanks for taking time to look at this.

[TomS]

I am noticing something else that is peculiar. In my jpegs and in the one nef file that I used IMatch to write metadata to, some of the metadata values in the metadata panel show a bunch of markup along with the actual data. I am uploading a screen shot to illustrate. Is this supposed to look like this?

[attachment deleted by admin]

[Mario]

This is correct. IMatch displays structured XMP data (e.g. regions) that way. You only see this data when you switch the Metadata Panel in Browser mode. This data is not designed for humans.

You did send me only the XMP file. I need the NEF file as well, the XMP alone is of no use to me.

[Mario]

I've added the NEF file to a database. All XMP data was imported. The Metadata Panel shows all data that is in the file. Although most of the standard XMP tags are not filled in the XMP file (e.g., title, description, credit) so these show up as empty (correct).  Location data is filled and shows up. Also marked and some other tags.

There is some metadata in a proprietary PN namespace (I assume this is Photo Ninja) and some 'labels' in the also proprietary ID Imager namespace (XMP-ics). This data is not displayed by the Default Metadata Panel layout in IMatch - this layout displays frequently used XMP, EXIF and IPTC tags only. The label concept used by IDI is not part of XMP, it's not related to XMP labels.

You can see all XMP data in your file in the Browser layout. If you want to include IDI proprietary XMP data in the Default Metadata Panel layout, just add the tags you want.

Note: IMatch by default imports only a few thousand common metadata tags. The proprietary IDI namespace is not imported by default.
You can tell IMatch to import which metadata to import in the Tag Manager.

Go to Edit > Preferences > Metadata 2 and click on Tag Manager. In the Tag Manager, put a check in the box for the XMP-ics namspace, which contains the IDI metadata. I don't know if IDI or PS also store metadata in other XMP namespaces.

Then rescan the metadata for your files using <Shift>+<Ctrl>+<F5>, reload metadata. IMatch then imports and stores the IDI metadata and you can see and use it in the Browser layout or add it to a custom Metadata Panel layout so you can see it alongside other information you are interested in.


For the proprietary PN data there is no simple way. ExifTool does not know about this namespace, at least it is not declared as a namespace and thus IMatch cannot import the data. ExifTool can dump the data on the command line because it makes up the namespace from the XMP data on-the-fly. But that does not work in IMatch, which needs more info and relies on the supported namespaces from the official list. Since ExifTool does not know about XMP-pn, IMatch cannot import the data.

If you want to import, use or display PN metadata in IMatch, Phil must add it first to ExifTool.
Contact Phil from ExifTool about this, and ask him to add support for the Photo Ninja proprietary XMP data to ExifTool. Supply a sample XMP file and all the info you can find or ask the Photo Ninja company to supply information to Phil.

[TomS]

No xmp data is displayed in my metadata panel at all. I must have something set incorrectly. The weird thing is that I took another nef file from that same set of files and wrote some metadata to it from within IMatch and all of the xmp data shows up for that file. For the file I sent to you, when I look at the metadata panel with the "browser" option set, I have headers for "Composite", "Exif", "GPS", and "IMatch Metadata". Nothing shows up below that. In the file that I wrote to using IMatch, when I scroll past the IMatch Metadata I have headers for "XMP Dublin Core", "XMP IPTC Extension", XMP Media Management", "XMP Microsoft", "XMP Photoshop", "XMP TIFF", "XMP aux", XMP exif", XMP exifEX", "XMP plus", etc...

Even if I switch from Browser to Default, all of the tags displayed in default are empty. Even the date fields. There are keywords in my xmp file that don't show up anywhere either.

I wouldn't expect or want IMatch to manage the ICS or PN proprietary metadata. That's not why I posted this issue. I usually don't bother with title or description tags unless I'm uploading the images to a sharing site. Different sharing sites handle these tags differently so I usually don't populate them until I know which site I'm uploading them to.

Please let me know what could cause you to see the xmp data when I don't. I'm copying a screen shot of the default metadata panel to confirm what I've posted.

[attachment deleted by admin]

[Ferdinand]

You know, the same thing happened to me a few days ago.  Imported some RAF files, and no metadata was shown in the metadata panel, even though there clearly was in the XMP sidecar.  I had used Geosetter to input some metadata, and in a departure from my usual workfkow, the files were still open in Geosetter.  So I closed Geosetter, removed the folder from IMatch, reingested it, and all was well.  First time that I can recall this happening.

[Mario]

Is the file open in another application and thus ExifTool cannot access the data?

Please repeat your test and attach the IMatch log file. See "Log file" in the IMatch help index for details. Look for lines starting with W> or E> which indicates Warnings or Errors.

What happens when you select the file and then press <Shift>+<Ctrl>+<F5> and then choose "Reload Metadata"? Does this change anything? Your file worked immediately here, and this is how it should be. No things to configure or anything. NEF files with sidecars work out-of-the box.

If you can't see any metadata at all, for none of your files. ExifTool may have a problem. The log file should then give us more information.

[TomS]

I removed the offending folder from my database, closed IMatch, reopened IMatch, and indexed the same folder again. The results are the same. I looked at the log file and it seems that there is some sort of error occurring when IMatch attempts to rename files in a temp folder? The log file is attached.

Could this be a permissions issue?

[attachment deleted by admin]

[Mario]

ETWARN: Error renaming temporary file to C:/Users/Tom/AppData/Local/Temp/imt7BEB575F-3A41-4DBD-BEB8-5B7350BA0FB5.xmp

These messages come from ExifTool. Apparently ExifTool is not allowed to rename files in your TEMP folder. This is very uncommon, because the TEMP folder is designed to be used by applications to store temporary files.

1. Close IMatch and other applications. Delete all files in the temp folder and then reboot. Some files cannot be deleted because they are in use or you need to be an Admin to delete them. That's OK. All files IMatch and ExifTool create there will be created under your "Tom" account and can be deleted just fine.

2. Make sure that you user account has full control (read, write, modify) to the TEMP folder and all folders below that. You can check and change that in Windows Explorer. Right-click on TEMP, then properties > Security.

3. Make sure your virus checker or similar security software is not blocking software from creating/renaming files in the TEMP folder. Or add an Exception for the application exiftool.exe in the IMatch folder: "C:\Program Files (x86)\photools.com\IMatch5\exiftool.exe"

[TomS]

I will have to try that later. I need to go to work. A quick look at my temp folder shows a whole bunch of files created by ExifTool so if it can create those files why would it not be able to rename them? Properties of my temp folder also indicate that everyone should have read, write, and modify permissions. I'll get back to you after I've had a chance to follow your instructions.

Thanks!

[Mario]

I have no clue, sorry. I don't recall a similar case.
Maybe something got 'stuck' or something. By the sound of it it's either a locking problem (some other application has the files locked (virus checker?) or it's some permission problem.

[TomS]

I've tried everything I can think to try but nothing worked. I thought I had it working but that was because I looked at one of the jpegs in the folder by accident.

I tried everything you suggested. I even uninstalled IMatch, deleted all app data including the test database that I was using and I reinstalled and started again. Still nothing worked. I still can't see the xmp data associated with my nef files.

The strange thing is that as soon as I write some metadata to one of the files, then all of my xmp data for that file becomes visible in the metadata panel.

Another strange thing I'm noticing is that now all of the jpg files in that folder have the yellow pencil icon on the thumbnail. When I hover over them, it indicates that there is XMP::dc/Subject data to write but all of the keywords are already written as far as I can tell. When I click on the pencil, the only data that I've been able to observe being added is the gps version number tag.

[Mario]

OK, let's see.

It worked out of the box here with a standard IMatch 5.4.18 installation on a clean system using your files. Metadata from your NEF/XMP showed up, was editable and I could write-back. I tried this on a W10 system and a W7 system, just to be sure.

On your system, you see no metadata for the files, but no errors or warnings are in the log file.
Writing metadata to a file (which causes a rescan/reload of the file) brings the file data into the database.

- You are using the default settings as shipped? No custom metadata or per-file format settings?
- You tried different folders?
- Maybe copied some files to another folder or disk and indexed the files from there?
- Temporarily disabled your virus checker and all background applications?

This is indeed a very peculiar case. I don't recall anything similar over the past two years since the release of IMatch 5. This must be something very specific to your system. Do you perhaps have a another computer where you can try IMatch 5?

[TomS]

OK, let's see.

It worked out of the box here with a standard IMatch 5.4.18 installation on a clean system using your files. Metadata from your NEF/XMP showed up, was editable and I could write-back. I tried this on a W10 system and a W7 system, just to be sure.

I'm running Windows 8.1.

On your system, you see no metadata for the files, but no errors or warnings are in the log file.

I'm still getting the Exiftool warnings about not being able to rename temporary files.

Writing metadata to a file (which causes a rescan/reload of the file) brings the file data into the database.

That is correct.

- You are using the default settings as shipped? No custom metadata or per-file format settings?

That is correct.

- You tried different folders?
- Maybe copied some files to another folder or disk and indexed the files from there?

Yes, I've tried two different folders that I created specifically for testing IMatch on an internal hard drive and I tried indexing a folder that had been there before. It was on the same hard drive. My system has an SSD on which my OS and applications are running and one internal hard drive on which my pictures are stored. I believe the IMatch database I created is on the regular hard drive as well. I would put it on the SSD if I was going to start using it permanently.

- Temporarily disabled your virus checker and all background applications?

I only  have Windows Defender running and I did disable it for these tests.

This is indeed a very peculiar case. I don't recall anything similar over the past two years since the release of IMatch 5. This must be something very specific to your system. Do you perhaps have a another computer where you can try IMatch 5?

One other strange thing is happening. The Load Progress Overlay Dialog doesn't show up any more. This had happened to my first installation also. After I reinstalled IMatch last night it started showing up again. This morning it doesn't any longer. I checked the application settings and it is set to "yes". I even tried setting it to no and then setting it back to yes again to see if I could get it to start working again but I can't. I have never clicked the "dismiss" button while it was displayed.

[Mario]

The load overlay shows up only when there are more than, I think, 50 files to process. Or 100.

The source of your problems is that ExifTool cannot create/rename temporary files. This makes it impossible to import metadata or write back. Or both.
Something on your machine is actively preventing the ExifTool process from operating properly. Fix that and the problems will be gone.

Maybe google for

exiftool renaming +temporary file

and see what comes up. Usually its a virus checker or file system privilege issue. Or another app, tool, service, Windows Explorer plug-in or whatever that locks the file.

You can use the free Process Monitor from sysinternals (Microsoft) to see what's going on:

https://technet.microsoft.com/en-us/sysinternals/processmonitor

The process you want to monitor is exiftool.exe and you should check for file system events which fail. Maybe this gives you more info about which other application or system setting on your machine is interfering. I can do not more from here, sorry.

[TomS]

I'll try a few more things but I don't know how much more time I'm willing to spend trying to solve this issue. It doesn't make any sense to me why any program would lock temporary files. Perhaps it has something to do with Windows File History. I know that I have that running but it shouldn't be backing up my temp folder. Now that I'm thinking of that, I did run into some issues with another program because of Windows File History. I need to make sure it's not trying to keep track of the IMatch database(s) also. I'll check it tonight and maybe try using the sysinternals program you recommend.

I will probably be upgrading to Windows 10 at some point. Perhaps I should just wait until after I do that and try IMatch again. I'm considering doing a clean install of Windows 10 after I've upgraded. I tend to trial a lot of different software and I'm sure I have remnants of many different programs hidden away all over my system not to mention a bloated Windows Registry.

One other thought I've had is to come up with a workaround. I'm thinking of applying a simple metadata template when I index folders that contain nef files that have xmp files already created. Perhaps if the metadata template just writes some trivial data into a metadata tag during indexing that will force IMatch to import the xmp data. I'm not a fan of using workarounds but it would only be a one time thing until I get my existing files indexed. I'll test that tonight if I can't come up with any other solutions.

The load overlay shows up only when there are more than, I think, 50 files to process. Or 100.

That probably explains it. I'm using a few different test folders and they have varying amounts of files in them.

Thanks for your help.

[Mario]

I will probably be upgrading to Windows 10 at some point. Perhaps I should just wait until after I do that and try IMatch again.

IMatch 5 supports Windows 7 (with SP2), Windows 8 and 8.1 and Windows 10. I test on all these systems, and the user base actively uses all these variants.

If you upgrade to Windows 10 (using the GetWindows10 auto update), Windows converts your existing W8 system to W10, keeping most of the mess you describe. Only when you install Windows 10 fresh you can get rid of whatever mess you have on your computer and start fresh and clean.

The error you see has been reported very occasionally on the ExifTool forum (google) and at least for what I read so far, ways always caused by messed up file system permissions or third party software locking files or preventing ExifTool from accessing the files.

[TomS]

If you upgrade to Windows 10 (using the GetWindows10 auto update), Windows converts your existing W8 system to W10, keeping most of the mess you describe. Only when you install Windows 10 fresh you can get rid of whatever mess you have on your computer and start fresh and clean.

I've read that once you use the GetWindows10 auto update and make sure that Windows 10 has been activated, you can then Reset your system and specify that you want to do a clean install by removing everything from the drive and reinstalling Windows 10. I will have to reinstall any software that I want to use after doing the reset but it's probably not a bad thing to do every couple of years.

[TomS]

Still no luck getting this to work. I downloaded the processmonitor but have no idea what I'm looking for in it. I'm about to give up. I would really like to try out some of IMatch's features but if I can't get my existing metadata into the catalog I don't want to spend any more time experimenting. Maybe I'll try again but right now I'm exhausted by it all.

[sinus]

The strange thing is that as soon as I write some metadata to one of the files, then all of my xmp data for that file becomes visible in the metadata panel.

So at least you could add something in a field and all would be fine, isn't it?!
That would be a solution for me.

I have over 200.000 images, mostly jpgs and nefs (D80, D100, D7000, D750....) and have never had such a problem. I have once taken over the old metadata from my previously used DAM "Portfolio" and all went well and fine.

I have no clue, what happen or happens not with your files. All what I can say from my knowledge, that IMatch does work with Metadata and gives you possibilities with them, so far I have not seen it by another DAM.
That is why I trust IMatch more than other DAMs.

I use only Lightroom/Photoshop to develep and edititing my images, but do not add there Metadatas.

[TomS]

I took a break from IMatch for a while and made some upgrades to my PC. I added a new hard drive and updated to Windows 10. I didn't do a clean install, just the free upgrade. I can see the metadata from the XMP files when I index folders now. I'm going to start doing some more testing with IMatch. I have one issue since upgrading that I would like to resolve but I will start a new topic to discuss that.