Imatch reading and writing metadata

[Aubrey]

It appears that IMatch is reading the metadata from my NEF files and then writing it back to them again.

does this make sense?

In anycase it is giving me great problems with my Backblaze backup. It means that backblaze thinks that all the files are new and wants to upload the updated files 100s Gb. While in fact I have made no changes to files.

I have a local backup of all my files (untouched by IMatch5. Is there a way to replace my current files with my backup, and then have Imatch only read the metadata.  Then Backblaze will not try to backup my library of processed files.

I also want to have the metadata incorporated into the file, to do this I have set:
Metadata 2 Allow to create XMP files: NO
I have also gone to "Configure File formats" and set NEF "XMP sidecar file" to Embed XMP in file

Thanks,
Aubrey.

[jch2103]

Just an observation: If you used XMP sidecars, the amount of data requiring a backup if you do make metadata changes will be tiny compared to making changes in the NEF files. This isn't an answer to your question, or a criticism of your work flow.

[Aubrey]

Thank you for your suggestion.
I did think of this while writing the email, however, I'm not keen to change my workflow if there is a better solution!

Thanks,
Aubrey.

[Mario]

IMatch does not write-back metadata just for fun.
Please verify:

1. You have automatic background write-back enabled under Edit > Preferences > Background Processing

2. The combination of the existing metadata in your files, and the options you have set under Metadata and Metadata 2 causes IMatch to produce "new" metadata when it imports your files (e.g. filling in XMP from IPTC, EXIF and GPS data).

If these conditions are met, IMatch will write-back data to your files because it

a) Has unwritten metadata (caused by the import) and
b) You have enabled the automatic write-back.

[cytochrome]

....
however, I'm not keen to change my workflow if there is a better solution!
...
Thanks,
Aubrey.

I have the same workflow as you: I write metadata to the image files. I see several advantages to that but the consequence is frequent read/write to the image file and this entails also that all files touched by IM are written again when I do incremental backups (I use SyncbackSE, the problem is the same).

I don't use labels and ratings (they are also written to the file) but still, and despite Mario telling us that "IMatch does not write-back metadata just for fun." I have sometime an impression that files from other folders (folders that I have not touched) are dragged along and updated maybe not for fun but for reasons I cannot decrypt.

Imatch is really geared to xmp sidecars and the "Adobe workflow", but it is flexible enough to allow for no-sidecar workflows, at a price.

Now that I have a faster machine I don't really care, and I won't give up my present workflow only to avoid the backup problem. I don't think there is a simple solution. Maybe refrain from automatic or frequent write back, keeping the whole metadata in the database for long periods, but this has other drawbacks.

Francis

[Aubrey]

Mario,
Thank you for your response, and also Francis for his comments.

In File format Metadata Options I have set the following for NEF files:
Write IPTC  NO
Write EXIF NO
Allow create IPTC/EXIF/GPS NO
(In fact everything NO!)
Except
XMP sidecar file Embed XMP in file

furthermore under Meta Templates I have removed all tags

I would have thought that with these settings nothing would get written back.

Is it possible to ask Imatch to clear its pending metadata for writeback?
Is it possible to determine what metadata is being written back to files?

One further thought, if I go to using sidecar files, is it possible to eventually have the information embedded into the NEF file?

Aubrey.

[Mario]

I don't use labels and ratings (they are also written to the file) but still, and despite Mario telling us that "IMatch does not write-back metadata just for fun." I have sometime an impression that files from other folders (folders that I have not touched) are dragged along and updated maybe not for fun but for reasons I cannot decrypt.

IMatch writes to files when you

a) Write-back metadata
b) Use the Paste Attributes command (which works directly on the files)
c) Use commands like Modify Exif date and time

I know that some software, e.g. LR, PS, Nikon Capture have the habit to "touch" files in folders you work with, may it be updating some proprietary data in the XMP sidecar file (like "last viewed") or something. Can this be the case on your system?

[Mario]

I would have thought that with these settings nothing would get written back.

No. These settings just mean that IMatch does not write IPTC or EXIF data. But it still has to update XMP data.

Is it possible to ask Imatch to clear its pending metadata for writeback?

Yes. But then the data in your database will no longer match the data in the files, which can cause problems later.

Is it possible to determine what metadata is being written back to files?

Just point the mouse cursor at the pending write-back icon.

[Aubrey]

I'm thinking of going down the sidecar file route for now, as the uploading of information to Backblaze will be significantly reduced (my upload rate is slow). When I have the opportunity to have a higher upload rate I would like to get away from cumbersome sidecar files.

One question still to be answered:

if I go to using sidecar files, is it possible to eventually change and have the sidecar information embedded into the NEF file?

Thank you,
Aubrey.

[Mario]

I write about the pro's and con's of using XMP embedded in RAW files in the help. Basically, most applications don't support embedded XMP in NEF files. As far as I know, only Nikon software does support this (and IMatch 5 via ExifTool). All other software uses the safer XMP sidecar way. Unless you have to support Nikon software, using the sidecar file is the better solution IMHO.

IMatch 5 does not offer a command to copy XMP data from sidecar files into NEF. You can do that with ExifTool if you have to. Then delete the sidecar file and re-import the files in IMatch 5 to pick up the XMP metadata embedded in the NEF (don't forget to set the NEF-specific metadata options to match your workflow).

[Aubrey]

I have read through the help section on the database converter.
It appears to state that if there is embedded xmf data in the IMatch3 data then this will be read from the files - why then does IMatch5 need to rewrite xmp data that already exists and was stable when using IMatch3?
The IMatch 3 database was stable with no pending writes before making the conversion.

As I only use Nikon software (Capture NX2 and ViewNX2) I was not wanting to go to buddy XMF. As I understand Nikon will only write back embedded xmf data.

Thanks again

Aubrey.
(Apologies for taking time out of your bug  fixing with all these questions)

[Mario]

1. XMP data embedded in your image files will be read be IMatch 5.

2. It is very likely that IMatch 5 produces "new" XMP data during the import because it generally produces much richer XMP data, supports all the latest XMP extensions etc. My initial post explains this.

3. IMatch does not write back the "new" metadata when you disable background write-back.

4. Nikon made the error in insisting in embedded XMP data in NEF files. Naturally they can do it because they are in control of the NEF format. Updating XMP or other metadata in RAW files is always risky (and as the past has shown, especially so with Nikon software). This is why the rest of the world goes the sidecar file route.

[Aubrey]

Thanks for taking the time Mario, appreciated your response.

Aubrey.

[Mario]

Do you consider this issue as successfully resolved?
Because this is an issue which may affect other users as well.

[Aubrey]

Mario,
Successfully resolved, probably not! I accept that there are going to be changes and plan to move on, Its simply with my preoccupation of backup to cloud. My upload link is not that fast (in Cyprus 750k, - download rate is not an issue).

As I only use Nikon software I want to continue with embedded XMF. I have decided to simply accept that after conversion there will be some rewriting of metadata and that most files will get touched and will require uploading to cloud. [I travel frequently and often have access to higher upload speeds and will eventually get all into cloud, in the meantime I'll be very careful to have local backups]

I appreciate you have had a lot of headaches trying to rationalize the metadata and have completely rewritten the code in places.

However there may well be some software issues so I illustrate the apparent change taking place in one file [AOC_D300_090525_8956.NEF]

I've looked at tags on a few files that have been "touched", it appears that there are rather minor changes that have taken place, for example, the before and after shown below [file size change is noted inside square brackets]:

Camera Categories.ISO;Camera Categories.Lenses used; ISPS.Music Evening 2009 [file size 11,024kB]
to
ISO; Lenses used; Music Evening 2009; Camera Categories.ISO; Camera Categories.Lenses used; ISPS.Music Evening 2009 [file size 11,017kb]

When I look in the file (using IMatch3) I see the keywords reading:
Camera Categories.ISO, Camera Categories.Lenses used, ISPS.Music Evening 2009

I attach the screen dump of the Metadata preferences, there is no request to replace or write hierarchical keywords, yet it appears that this is being done.

Thanks,
Aubrey.


[attachment deleted by admin]

[Mario]

The Metadata Options in your screen shot are only for IPTC data.
The fun is in Metadata 2. I can only re-iterate that IMatch 5 *will* most likely produce a much richer XMP record than IMatch 3 or your Nikon software (Capture does only the most basic XMP). So IMatch will have new XMP data after the import. It has to write that once. If you disable background write-back this will only happen when you explicitly write-back the files. So YOU are in control when and if your NEF files are updated.

The file size must not necessarily change when XMP data is written because part of the requirements is to allocate some extra space in the file to improve later writes.

XMP sidecar files are ideal (!!!) for cloud storage. Because a sidecar file usually has only a few KB and can be synched with the cloud in no time. The combination you use (RAW files with embedded metadata) is the worst case for cloud storage because it requires to shuffle massive amounts of data up and down the line. Just another argument against using Nikon software to maintain XMP data inside NEF files. I have given up Capture for several reasons, and Nikon insisting in using embedded XMP was one of them.

So: If you use IMatch in default mode, it will only update the XMP in the sidecar file (if your NEF files not also contain IPTC data or you disable IPTC). Ideal for cloud. No change to the RAW. Much faster. All you need to do is to switch your workflow in not using Capture to edit metadata. Only edit the image. Then manage the metadata in IMatch exclusively. When you switch to another RAW processor or LR or whatever later, your metadata will immediately be available too.

[Aubrey]

Mario,
Very good points made in your last email. I only edit the metadata in Imatch so the sidecar files will work well for me . When and if I go the LR then I will have everything available. Thanks for your patience.

I think that we can consider this thread closed.

Aubrey.

[jarraun]

Hi Aubrey,

Sorry for not considering the thread closed, I´m very interested y think that many others are.
In my workflow metadata area also embeded in NEFs via View NX2 but I´m considering seriously moving my images metadata to xmp sidecar files. I find two ways of doing so:
1- Import NEF files with embeded metadata to IMatch with Metadata 2 Options set to create XMP sidecar files, then it would be possible, erase embeded metadata with an empty XMP/IPTC preset in batch via VIEW or CAPTURE NX2.

2- Erase embeded metadata with an empty XMP/IPTC preset via VIEW or CAPTURE NX2 and then as I have JPGs derived from NEFs that have the same metadata as NEFs, may be would be posible via IM5 in batch mode copy these metadata from every JPGs to the XMP sidecars of the NEFs which have the same file name.

That way I will use IM5 to manage metadata and CAPTURE NX2 to photoedit NEFs.

I hope it won´t be too complicate due to my bad English. Any advice wolud be very welcome

Best regards

Javier from Pyrenees

[cytochrome]

You can easily delete xmp/iptc in NEF (or any other raw) from within IMatch with the Exiftool Control Panel.

Seems in a  short time I will be the only Imatch user left with an  "in image file metadata" workflow. Since I don't use the cloud but good old HDs to backup this is no real problem. I recently bought an USB3 dock, it is reasonably fast.

I use Photomechanic 5 for ingestion, Aftershot Pro (mostly), or ViewNX2 and  DxO9 for conversion, all this integrates well with IM and in file metadata. ASP and Dx0 simply pass the info to the derived JPG/Tiff, in ViewNX I can enter or modify all metadata of interest (to me), Imatch reads them fine. And vice-versa, no headache And it works also for my Panasonic RW2, so I have a unified workflow for all my cameras (2 Nikon and 1 Pana). Simple enough.

As Mario suggests, one can delay the write-back to limit the number of metadata read/write, this saves time but then the database and the image files are not in synchrony, what the metadata panel shows may be different from what the image file contains which may be confusing. It took me some time to realize that this was normal and I wrote some angry posts that Mario answered with great calm and serenity... The zen of Imatch is his next book.

Francis

[Mario]

It took me some time to realize that this was normal and I wrote some angry posts that Mario answered with great calm and serenity... The zen of Imatch is his next book.

[jarraun]

Francis,

Thank you for you suggestion:

You can easily delete xmp/iptc in NEF (or any other raw) from within IMatch with the Exiftool Control Panel.

My point is let only Nikon propietary software get into my NEF files, because of that I would delete embeded metadata from View or Capture NX2.

I´am always surprised by the variety of workflows, for me is very curious that you prefer embed metadata in NEF files and let Aftershot Pro or DxO9 generate sidecar files of your image conversions. Any way I´m glad this works fine for you and it is always fine to know others points of view.

Javier from Pyrenees

[cytochrome]

Hello Javier,

Yes there are many workflows. To me the sidecars written by ASP or DxO are of little interest, if I loose them it is no drama at all. They contain edits. These reflect mainly the presets I use to ingest the raw (ASP is better than DxO because I have a profile for NEF and another for RW2 and ASP picks the right one). Anyway the JPG reflects what I wanted at that time. In my experience, when I go back to a raw I start from fresh and do something quite different.

Metadata is something else, it is all I need to find the image back, whatever  software I use, they are written in the raw and jpg, so I find ALL images with this set of metadata, event with very simple programs. To move from 3.6 to IM5 I did not use the nice tools Mario provides, simply dragged the photo folders into iM5 and then the categories saved from 3.6 and that was it, IM5 reconstructed the database and indexes for me.

If wanted to drop Imatch (I certainly won't) I could use Ferdinand's script to write the cats as keywords to the files, and would be ready to switch to x or y. It gives me the impression that my choices are open, nice feeling.
And it gives me the liberty to edit metadata in the Nikon programs and in Photomechanic without bothering about integration in IM, this works well.

Francis from the Alps 

[DigPeter]

I use Photomechanic 5 for ingestion.....

@Francis - I would be interested to know what you do with Photomechanic in your work flow:  rename, assign to folders, write metadata, edit?

[cytochrome]

In PM5 I do

- ingest from SD to folders using for example {year4}\{year4}-{month0}-{day0} for folders and D7K_00{filenamebase:-4} for file names

This distributes  the raw on the SD  to folders 2013, 2014.... with subfolders 2014-01-01, 2014-01-08 (one subfolder per shooting) with names like D7K_013198.NEF. Of course the names can combine any exif etc info taken from the image file. I keep it simple, it is Imatch that takes care of the rest.

- if I have the data I import an gpx file (or a whole folder, it finds the corresponding one but thta takes time). It works like in geosetter, not better but it is integrated in the process.

- I fill the metadata template (it is a very complete and handy one) and apply. There is are settings to write it to sidecars or to the image with all the finickery needed to read first from file or sidecar, from xmp or iptc, to overwrite or not

- there is also a keyword panel that I have never used,, when the day comes I suppose I will do it from within IMatch to ensure good integration

- I have used it to copy xmp/iptc from jpg to raw. It is very rapid but requires that jpg and raw are in the same folder, now I prefer Imatch or command-line Exiftool for this (once I got it to work to my liking)

- I used it to convert files to UTF8 very fast again

- there is much more, it is really geared toward pros shooting sport (code replacement for players, rapid upload),  I don't use it and cannot comment.

I bought PM when I was using Im 3.6. With all the capabilities now built into IM5 I am not sure it is worth the money, but it is fast and easy. Good program and good forum and support.

I have seen in other posts that I am not alone to use PM and IMatch. Curious to learn what they do (althought it is off topic, but still...).

Francis

[DigPeter]

In PM5 I do....

I bought PM when I was using Im 3.6. With all the capabilities now built into IM5 I am not sure it is worth the money, but it is fast and easy. Good program and good forum and support.

I have seen in other posts that I am not alone to use PM and IMatch. Curious to learn what they do (althought it is off topic, but still...).

Francis

Still OT:

Thank you Francis for your full and interesting reply.  We are all having to review our work flow in consequence of IM5.  With IM3, I use an old application, Downloader Pro for initial ingesting from SD card, when I rename and place in folders as you do.  I also write standard XMP/IPTC data with it.  Having ingested, Downloader then automatically transfers to a complementary browser application, Breeze Browser Pro, which can display and convert from Raw to jpg.  I only use Breeze Browser to do an initial trawl to eliminate the no-hopers.  The convert facility is a bit clunky, so I use Lightroom to edit and convert raw images.  All categorising, keywords and properties is done In IM and these are written to metadata with IPIW script.  I do not think however I will change  this workflow significantly for IM5.  The main change I think is to use IM5, power to write initial metadata and rename after converting with Lightroom.  From what you say and unless someone comes up with a compelling reason to move to Photomechanic, I will probably stick to DL & BB Pro for the intial ingest from SD.

[cytochrome]

Hello Peter,

When using IM3.6 I found the IPTC editor fine but the xmp editor was a terrible thing  (sorry Mario but it was a pain to use). So I had a look into Downloader Pro and Breeeze Browser, they do a lot and cost less than PM but (at least when I looked them up) BB would not write into NEF or RW2.

If you are in an "Adobe workflow" the capabilities of PM to write into raw are not needed and anyway IM5 does it very well albeit slower.

Francis

[DigPeter]

Hello Peter,

When using IM3.6 I found the IPTC editor fine but the xmp editor was a terrible thing  (sorry Mario but it was a pain to use). So I had a look into Downloader Pro and Breeeze Browser, they do a lot and cost less than PM but (at least when I looked them up) BB would not write into NEF or RW2.

If you are in an "Adobe workflow" the capabilities of PM to write into raw are not needed and anyway IM5 does it very well albeit slower.

Francis

Hello Francis - BB does updates, but they have to paid for after a year on a previous issue.  My version does rw2, but I do not have Nikon.  I agree about IM3's XMP editor - I do not understand it. I think Mario is aware of people's problems there.

I will be off line for 10 days.

Peter