Reject files not working after some environment changes

[hkae]

I have a "strange" behavior since I moved from JPG to RAW images. 
To be honest I guess I have overlooked something therefore I try to describe what I have changed:

- upgrade from 5.4 to iMatch 5.5 (latest version 5.8.4)
- added a second SSD disk. Now all new images are stored on this new disk.
- all new images are stored as RAW (NEF) images including a sidecar xml file – that's new as well
- enabled automated write back metadata within iMatch as described within the article <USING ADOBE® LIGHTROOM AND IMATCH 5 TOGETHER>

Now if I reject one or more RAW images, iMatch starts a background process that saves this new state. A second process follows which reads the metadata. This is normal but in case the rejected file is a RAW file (located on my new disk) the reject state is lost as soon as these two processes are finished. From my point of view this has nothing to do with the iMatch upgrade because iMatch still works as expected if I reject a JPG image on my older disk.
Therefore I'm sure I have overlooked something.... as I switched from JPG to RAW images.

I wonder if someone may have a guess, based on my short description, what I could have overlooked or what I may have done wrong.

Thanks for any help or suggestion

Regards Hanspeter

[thrinn]

This sounds like the issue described in [https://www.photools.com/community/index.php?topic=2624.0.
Might be worth to check if your problem is related,

[hkae]

@thrinn:  Thank your for feedback.

If I understand the linked article correct this apples only if CR2 files from Canon cameras are used. So I guess this article applies not to my problem because I use/have a Nikon camera. But this is really new to me and therefore I may be wrong.

[Mario]

1. Did you relocate all folders/disks in your database after moving your files to the new disk?
2. Check the ExifTool output panel (View > Panels > Output) when you write-back a file. Error messages and warnings logged by ExifTool will show there.
3. Include an IMatch log file in your post, because this will allow us to see if IMatch is running into problems or errors. See log file in the IMatch help index for details.

[hkae]

Thanks Mario for your checklist:

1) relocate
No. I did not relocate my old iMatch image folder. I added a new disk and I added the new folder structure to iMatch. All old images are still on the same disk and within the same folder structure. But I copy all new images with a small script on the new disk. IMatch then picks up these new added files automatically. I have JPG's on the "old, original" disk and RAW files on the new disk. As far as I am aware of I use the default settings of iMatch with regards of sidecar files, which are new since I store RAW files. So I guess that I need to change a setting to prevent rollback the reject flag.


2) This is the ExifTool output (I removed the empty lines):
-overwrite_original_in_place
-charset
FILENAME=UTF8
-m
-use
MWG
-charset
ExifTool={PTETCHARSET}
-ex
-sep
-XMP-xmp:Rating=-1
-XMP:CreatorTool=photools.com IMatch 5.8.0.4 (Windows)
-xmp:InstanceID=xmp.iid:2fe387c4-43db-4669-b600-26667ad6364e
-XMP:MetadataDate=now
-XMP:ModifyDate=now
D:\Apps-Daten\iMatch\Bildarchiv\Zeitachse\2017\Mrz\20170305-1248-DS5_1206.xmp
-execute
-execute9999
1 image files updated

3) Logfile as attachment

[Mario]

The reject is a standard XMP rating with the value -1.

According to the log, IMatch has successfully updated the .XMP.
But it dit not update the RAW file. If your  RAW file contains an embedded XMP record, this may cause the effect you see. The XMP data in the RAW overrides the XMP in the sidecar file.
If your camera writes a rating = 0 into the RAW (some Nikon bodies do), you will have the effect.

If you did read the post thrinn has linked above, you will be aware of that.
Make sure you allow IMatch to update embedded metadata in your RAW files.
Edit > Preferences > Metadata 2 ...

Alternatively, remove the embedded XMP record from your NEF files using the corresponding preset ("Delete XMP Data") in the IMatch ExifTool Command Processor.
You can also use the ECP to check if the NEF file has an XMP record ("List Metadata").
Select one NEF file first to test.

Some Nikon bodies (Nikon's wisdom again) insist on writing an XMP record with only a rating=0 value. Which is nonsense and breaks compatibility with all applications which expect XMP data for RAW files to be in the sidecar only (as it is standard).

[hkae]

I tried to understand the impact of the article thrinn referenced. This is all new for me and therefore I followed the recommendation to leave the iMatch defaults so far.

I use iMatch and Lightroom. Based on my understanding the XML records within the sidecar file will be used to exchange the latest settings between this two software packages. So far I understand that my current setting updates the sidecar file but reads the NEF internal XML record and as a result the reject rating will be overwritten or ignored because the internal XML record has higher priority.

To change this behavior I would have to change the following setting - if my understanding would be correct:

Edit > Preferences > Metadata 2  > customize file format > select NEF  > uncheck <use default settings>
--->   change <XML sidecar file> from <Default> to <Favor XML sidecar file>

Did I understand your and thrinn's advice correct if I update my iMatch setting as described above?

Again thanks for your help to set up my environment correctly

[thrinn]

If I understand the linked article correct this apples only if CR2 files from Canon cameras are used.

While the article describes a specific situation using CR2 files, the reasoning behind should also apply to other combinations of Raw files with XMP sidecars. In essence, problems arise if the same meta data (rating, for example) is stored at two or more different places (RAW file + XMP, plus IMatch DB), and with different values. I think the FAQ post describes this quite well:

The way IMatch imports metadata from RAW files and sidecar files via ExifTool merges XMP data embedded in the file (if any) with the XMP data from the XMP sidecar file (if any) and XMP data produced by IMatch when importing legacy IPTC, EXIF and GPS data into the XMP record.

If a RAW file has embedded XMP data, this data is considered more important that the XMP data read from the sidecar file. This means that embedded XMP data overrides XMP data in sidecar files. For example, the XMP sidecar file indicates a rating of 3, but the XMP record embedded in the CR2 has a rating of 'none', IMatch displays the file without a rating.

[Mario]

Edit > Preferences > Metadata 2  > customize file format > select NEF  > uncheck <use default settings>
--->   change <XML sidecar file> from <Default> to <Favor XML sidecar file>

This could work. But in general, it's better to have only one XMP record for a RAW file. Lr does ignore embedded XMP records completely, IMatch leaves you options. Mainly for users who used Nikon Capture, which insisted on writing XMP into the RAW file instead of using a sidecar file like everybody else. Now Nikon continues this nonsense by making their cameras embed an XMP record into the NEF files. Usually this XMP record has only one entry: rating = 0. Which means "no rating". Nikon seems to be unwilling to support the XMP standard or at least give their customers the choice to not embed an XMP record in NEF/NRW files. Sad.

I would remove the XMP record from the NEF with the ECP. Everything will then work as expected. Exchange between lr and IMatch takes place via the XMP sidecar file always.

[hkae]

@Thorsten and @Mario:
Thank you for your explanations. With this new knowledge I'm (now) able to understand what the article describes and I agree that just one XMP record would be the preferred solution.
As suggested I tried the ECP tool on a single image. After I close the ECP tool iMatch updates the metadata and the image got the reject rating. I guess this happens because iMatch tries to find the embedded XMP info which is now gone. If I remove the initial reject rating I can toggle the reject rating on and off afterwards.  This behavior happens with both settings <Default> and < Favor XML sidecar file>

Right now that's what I do to import my RAW files:
I simplified my import script as I moved from 5.4 to 5.5.: I read the last imported file from a file before I start a transaction, which loops trough all files on the camera memory card, which is placed in a USB card slot. New files are renamed, the database.folders object is called to validate if the folder (year/month) already exists before the image is transferred to the SSD disk. The progress bar is updated and then the loop starts again. Quite simple and very fast, with a short delay iMatch picks up and processes all new images... Straightforward and easy to handle.

But now I'm wondering at which stage I should/could remove the embedded XMP from the RAW image. If I load the files to iMatch, as described above I have the issue that I would have to call a second script which would fire up ECP and remove the now set rejected rating after iMatch process those images. That could work with some risk because of the iMatch background process time delay but it would revers my goal to simplify the upload process for my wife.....

Calling ECP within the loop would slows down the transfer process and right now (my first test) it opens each time a black window.

So I guess I have to build a list of the new files, split them (because I have no clue how big the parameter puffer of ECP would be) and pass them to ECP before I move the images to the iMatch folders.

Right now I'm wondering how other Nikon camera owner import their RAW files.
Therefore I'm thankful for any advice, suggestions, tips, hints and/or best practice to get rid of this embedded XMD record.

Regards
Hanspeter

[Mario]

After I close the ECP tool iMatch updates the metadata and the image got the reject rating

This is because the XMP sidecar file remembers that you have rejected the file. The rating is no longer overridden by the embedded XMP record and hence applied. Correct.

Right now I'm wondering how other Nikon camera owner import their RAW files.

I copy from the card/camera into an "Incoming" folder that is monitored by IMatch.
Match brings in the new files automatically, runs a metadata template that sets several XMP tags to standard values.
After this, I just run the Renamer to give each file a unique file name.
All very simple.

If I would have to remove XMP data, I would run a batch that uses ExifTool directly to strip the XMP records from the RAW files.
ExifTool can do that in batch for all files in a folder. Just one call to ShellExecute.

ShellExecute will also be available in IMatch 2017 (no Basic scripting anymore, mind!).

[sinus]

I copy from the card/camera into an "Incoming" folder that is monitored by IMatch.
Match brings in the new files automatically, runs a metadata template that sets several XMP tags to standard values.
After this, I just run the Renamer to give each file a unique file name.
All very simple.

I do also exactly this.
Works fine.

[Jingo]

Right now I'm wondering how other Nikon camera owner import their RAW files.

I do things a bit differently so figured I would list workflow in case it helps you or someone else.  Since I RARELY go back to ever edit a RAW file once it has been processed, I only use JPG files in IMatch.  So, I do the following (for a variety or RAW formats from 4 different cameras including NEF's):

1 - use breezebrowser to load the RAW files from memory cards onto local hard drive (files are renames with camera model/date/time/image# and stored by Year/Year-Month).
2 - import RAW files into Capture 1, apply some import develop options and process the files.
3 - Batch export the RAW files to full resolution JPGs out of my NAS machine (also stored with same filename and in same Year/Year-Month structure).
4 - Load up IMatch which is monitoring the NAS folders so the images are auto-imported into the database.
5 - Review newly imported files, cull losers/duplicates, add metadata (keywords, ratings, labels) and write metadata out to files.
6 - Rinse /Repeat as needed.

I do this for a number of reasons... the most important being that I can see ALL edits made to the RAW images in IMatch since they are processed JPGs.  This is even more important now that family members are only using IMA to view and download images.  I used to use DNG solely for this purpose (LR embedding an updated preview) - but I find the JPG's much more convenient for end-users that just want to view and download a file for printing or upload to social media. 

Since my RAW and JPG "trees" and filenames are the same - it is simple to locate a RAW file if I want to tweak further or print directly in Capture One from the RAW file.

Anyway... this process is working great for me and family and the workflow is fast!  IMatch runs super quickly across these JPG's and I don't need to worry about versioning JPG's and RAW's together.  Hope this info can help you or someone else along the way...

[Mario]

1 - use breezebrowser to load the RAW files from memory cards onto local hard drive (files are renames with camera model/date/time/image# and stored by Year/Year-Month).

You can do this with the IMatch Renamer as well. Just to mention it.

[sinus]

1 - use breezebrowser to load the RAW files from memory cards onto local hard drive (files are renames with camera model/date/time/image# and stored by Year/Year-Month).

You can do this with the IMatch Renamer as well. Just to mention it.

With IMatch 3 I did this with a script.
Now, with IMatch 5 the renamer does a really good job.

First I copy the files from my card in a folder, scanned by IMatch automatically.
I let the files on my card until the next job, because it gives me an additional "backup", just in case my drives on my PC does both died or whatever. Since I have several cards, this is for me very good. If I reformat such a card, the job is mostly already delivered to my client.

Once automatically scanned in IMaltch, the renamer does this with one click:

- give a uniqe number (variables), count upwards
- depending on the date, creates a YYYMMDD and the time
- let me put in a box some abbreviations like client and event
- does change some umlauts (äöü) to ae and so on (if there are in the event)
- does change spaces in _ and big to lower (if there are in the event)
- at the end give "_a" to the file (means not used, if used, this will change later)
- moves the files into the correct folder, with YYYY-MM-mm
(if there is not such a folder (a new month), it creates one
-make a backup on anther drive

Hm, I guess, that's it.
This gives me a filename, in the correct folder like

20170310-1843-309264-s-coo-nightparty-steve_a

I have this system a long time and it works fine. I like specialy the "event-Text", inlcuded in the filename.
Like now, as one example, with IMatchAnywhere (well, I am trying  ) searching a file with my 240'000 files makes fun, if I can search in the title, because I have a result VERY quickly.
Of course also in IMatch.

Searching in the metadate is of course also possbible, but it takes much longer (with 240'000 files).
And mostly, even since years, I can remember a word like above "nightparty" or "steve".
Or, if I search apples and I did some shots in the studio, this word will also be in the filename.

The only drawback is, that it is a quite long filename, but IMatch does handle them very good and Windows meanwhile also.

Hence the renamer does a really good job.

[hkae]

Thank you all for your explanations about how you import your images.

Yes I used the same method, resp. I got a "starter kit" from Martin Brunner a long time ago called GIMB if I recall it correctly.
Till end of November 2016 I worked just with JPGs, which allowed me to remove big parts of the WF1 part. I kept the renaming part. With WF2 I used/created "similar" functionality as we got with the iMatch 5.4 collections with regards of years, months, etc categories and of course many more. My wife likes gardening and she takes a lot of pictures from the garden. So I added a graphic to the WF2, which represents the garden. Now she can just click on the screen to categories the exact point, direction, macro, single or multiply plants, small or big overview, weather condition, plant or tree names to name just a few of the categories which will be feed as a result of this clicks.
Yes iMatch is a great tool for such requirements.... And a search script with a same graphical representation allows finding whatever plant, tree, weather condition she is looking for again with some simple clicks. Really powerful and great based on the unlimited possibilities the categories over.


Because iMatch offers now a lot of this functionality out-of-the-box I started to simplified my scripts. My import script now reads just the image file creation date and with this info I create a unique filename (date, time and original image name) before the image is copied directly to the final iMatch folder.
I used the import folder to remove (reject) images before I renamed and moved the images to the final folder. My old WF1 script added a unique file name including a short description (vacation, garden, etc and including a picture number) to the filename. Because I never had any issues with iMatch I decided that I could eliminate the import folder.... The collection today, etc is really very helpful and the iMatch indexing tool adds all new imported files to a certain category which I need for my adapted WF2 workflow (just to categories the garden images as described above)

And yes I leave the images also on the camera card as a backup. My import script bypasses the images, which are already uploaded to iMatch. I backup iMatch to an offline NAS system and to a USB Drive (two in fact and yes I'm paranoid about backups) before I erase the camera cards.



But I got also some great hints, which I will seriously consider to solve my Nikon RAW data "issue".
I like the following hint: 

ExifTool can do that in batch for all files in a folder. Just one call to ShellExecute.

I still try to find a solution about how the import script could keep control. If I'm assume that a script won't block background processes I could try if I could pause the script for some time, then count the images the iMatch indexing task added to a temporary category to know when the ECP call could be executed. And I will try if I could pass the resulting file list to ECP because the uploaded images could be stored in more than one folder (Months end). If this file list should not work I will have to change back to an upload folder....

[Mario]

Please note that IMatch 2017 no longer supports Basic Scripting.
Try to use as much of the built-in functionality in IMatch. The Renamer can rename your files based on metadata, no script needed for that. The Renamer can create folders based on metadata and move files into these folders. No script needed for that.

[Jingo]

1 - use breezebrowser to load the RAW files from memory cards onto local hard drive (files are renames with camera model/date/time/image# and stored by Year/Year-Month).

You can do this with the IMatch Renamer as well. Just to mention it.

Yes.. and it is a super renamer tool!  But since I don't want Imatch to touch my RAW files - I need to use a different program to get things rolling in the workflow.

[Mario]

IMatch does no "touch" you RAW files. Unless you rename files or update metadata, it will not modify your files at all.
Thousands of other users don't have a problem with using the Renamer. Running scripts should always be a last resort, when the built-in functionality is not sufficient.
All scripts written in Basic will no longer work with IMatch 2017.

[Jingo]

Hi Mario - no.. that's not what I meant (I didn't explain clearly).  The reason why I don't use the renamer is because I do not wish to include the RAW files in the database.... This is because I only want "processed" JPG's in IMatch which represent the final state of the images.  All editing of the RAW files are handled outside of IMatch so I do the renaming as an initial step to ensure RAW and JPG match each other for easy locating later.

The renamer is a super powerful tool... like many of the other modules in IMatch - I can easily see a small market for you to publish these tools as separate programs to sell... that is what Hert from Idimager did with some of his modules - no idea if he has made any add'l revenue from them though.

[hkae]

Please note that IMatch 2017 no longer supports Basic Scripting.
Try to use as much of the built-in functionality in IMatch.

Using as much standard functions as possible is of course the goal. That's why I already started to simplify my old scripts (thanks to all new possibilities iMatch 5.x overs).

But coming back to my original "problem" it would of course help if I could call ECP for all new loaded images within the iMatch indexer like a template or something similar.

If something like that would be on the roadmap, that would really be cool  

[Mario]

This will not be possible. Running the ECP causes IMatch to rescan the files, probably even a write-back if the re-import produces new metadata and your IMatch installation is configured to write back immediately.

And all this is done asynchronously, in the background, while the Renamer is trying to rename files, basically pulling the files out under ExifTool...and this is just the simplest case. Not speaking of additional files which are also renamed at that time, because they are buddy files of files you rename etc.

You could simplify your workflow by creating a small script that runs ExifTool for the current folder or something.
You may also ask Nikon to give you an option not to embed a "no rating" in your NEF files.

[hkae]

@Mario: Thank for your explination and yes I was expecting that this can't work. I'm sure you would have already included this functonality if this would work.

Nikon won't change anything because of me and therefore I will call ECP from the input script before the script transfers the images to the final iMatch folder.
Again thank you all for your feedbacks and infos about your workflows and how I could solve this "issue".

Hanspeter