Inconsistent Thumbnail generation for D810 NEF raw files

[mking]

Hi Mario,

I am using 5.3.14

My recent D810 NEF raw files are not all generating thumbnails, even when I force an update on the whole folder. Some are and some are not creating thumbnails. If I select a few files and force an update, then they all generate thumbnails, even if they didn't when I selected the folder or a large number of files.

My Max file size is 0 and I have 40GB of unused Cache.

Is there anyway to fix this so it generates Thumbnails reliably without forcing an update on a few files at a time?, as I have 9730 new images and I do not want to have to select a few at a time and force an update.

Many Thanks,
Mike

[Mario]

Do you have an up-to-date WIC codec installed which supports the NEF variant used for these files?

Select one of the failed files in a File Window, then go to Help > Support > WIC Diagnostics. This test checks the installed WIC codecs on your system and tells you if the installed NEF codec is able to handle the file.

An IMatch log file (see log file in the help for details) from a session where you did a force update on some of these files would also be helpful, because it will show us the error messages returned by the image library / WIC codec.

See also this article in the IMatch knowledge-base for more info.

[mking]

Hi Mario,

Txs for the prompt reply. First to answer your questions - yes I have tried both the latest Nikon WIC and Fastpicture - one at a time. Both show the same Thumbnail generation problem. Though using your diags - Nikon's WIC can't read the embedded thumbnail and FastPicture can't generate previews - so I am using Nikon and letting imatch generate the thumbnails.

I've attached both the log from a forced update and a screenshot of a small section of thumbnail status after the update has completed, so you can see files that did and didn't generate thumbnails.

Finally because thumbnail generation works fine when I force an update of a small number of files but not a large number, I woke up this morning thinking what I might try to find the cause/fix. I did wonder if there was a threading problem that was highlighted by a large number of files being processed by the WIC at the same time. I was running before with 8 threads (on a dual 8 core machine). So I tried setting imatch to just use one thread - and now with force update, after a much longer wait, it generated all the Thumbnails on a 756 file folder where before with 8 threads it was not generating 30%+ of them.

So hopefully you can figure out a solution, or if the WIC is just not thread safe, give us the option of setting a thumbnail # thread separate to other # thread active.

Many Txs,
Mike

[attachment deleted by admin]

[Mario]

WIC codecs are support to be thread safe. IMatch processes multiple files at the same time, but each thread uses separate instances of the image library, separate instances of WIC codecs etc. Unless the Nikon codec just breaks when it gets under stress (8 threads is a lot!) this will work. I cannot do anything about the Nikon codec, which is known to have stress issues. At least I had several reports. The FPC codecs seem not to have any problem.

It's also not a memory issue, IMatch uses only 250 MB RAM at the time.

I suggest you just dial in 4 threads and let it at that.

[mking]

Mario,

Appreciate that you can't do anything about the Codec stability, but as I said earlier I get the same problem with Nikon and FPC, with FPC being worse because it (latest version) can't generate all the thumbnails or any D810 previews either.

Anyway I am busy rebuilding all 9730 files with a single thread - after 90mins getting metadata it now needs 8hrs to generate the thumbnails.

Also 8 threads is nothing. With our sim software we regularly run 32+ threads - but we spend a lot of effort making sure everything we touch is thread safe. I only limited imatch to 8 threads because that looked like the right number to match my SSD raid 0 array bandwidth.

The problem is, as we all experience, the camera companies are not software companies. I tried the new Nikon View i-NX at the weekend and it just crashes before I can even display anything !

Next time I add a reasonable number of images I'll try with 4 threads rather than just one. Right now I just need it to work and one seems to work - I'll know for sure in 8hrs or so

Cheers,
Mike

[Mario]

With our sim software we regularly run 32+ threads

Threading is a difficult business. 32 threads which all do in-memory parallel computing are usually not problematic. As soon as you add asynchronous disk I/O, database read and write, 3rd party libraries, Windows Imaging Components, WIC codecs developed by camera vendors into the mix, things become really difficult.

IMatch by default uses <number of cpu's in system>-1 background threads for processing images, with a maximum of 4 threads. This has proven to work over a wide range of machines. Quite a few customers have computers with 8 or 12 cores now, but using one thread per CPU will slow things down because neither the database nor the file system can keep up - not even on a SSD. 4 max is the tested scenario.

I suggest you set the value to the default 0 preset or at least stay at 4 threads. If this still causes problems, there is some other issue involved I have not yet experienced. Virus checker or something?

Maybe run the sysinternals FileSystemMonitor along with IMatch when you can reproduce the problem. Perhaps we see some error messages there which give us a hint about what exactly is preventing IMatch from loading the image...

[mking]

Just to update the situation - thumbnail generation worked fine for all 9730 images with just one thread enabled - just slow.
I then found some older D810 folders where thumbnail generation had partially failed and I forced an update with 4 threads and that seems to work fine as well. So it seems 4 threads are fine, 8 are not... Again for people reading this in the future, here I am using the Nikon WIC 1.26.0

Mike