Use multiple cores / processes when decoding images

Started by monochrome, October 01, 2018, 06:25:15 PM

Previous topic - Next topic

monochrome

IMatch is unsurpassed when it comes to organizing photos. I have something like 130,000 images in the DB and it just breezes through them.

There is really just one part of the photo workflow that is a true time-sink for me: culling. After having shot a stage performance I have something like 1000-1500 photos, all of them RAW, and usually end up with 50-60 really good ones. Being able to hit "delete" to mark for deletion is excellent - but when it takes 3-4 seconds to step through each image it's still a slow process. (I've taken to starting a slideshow with 0s delay just to preload images in the cache. I start the slideshow, and come back an hour later when it has finished.)

Then I looked at the Windows Task Manager and noticed that there's only one of my eight CPU cores that are working when going through images in the viewer.

Now, I don't expect a single image to decode faster (as that would require parallelization of the image codecs which is way beyond what is practical), but if you're going through a bunch of images then it should be possible to decode several in parallel. Having those little green dots in film strip that tells me that the image has been preloaded show up eight at a time would be a great step forward for sure.

Mario

#1
I don't know how it can take 4 seconds to step through images. Please attach log file, give details about file format used.
Are the cache images already created or has IMatch to decode your files on the fly? Do you show multiple files or one? Full-screen or windowed mode?
Which overlays are open? Overlays may take longer than the actual image to update, mind!

I have a typical switch time in the Viewer between 0.2s and 1s- with existing cache images or JPEG files.
~ 1 second is for 80 MP (!) files (10,000 x 8000 pixel).

My system is over 3 years old and has a 150US$ graphic card.
Cache is on SSD of course. JPEGs loaded from spinning disk.

IMatch pulls files from the cache using WIC and stores them in the Graphic Card buffer using DirectX. WIC loads 36 MP JPEG images in 0.3s and then I let the GPU do color management and stuffs.
Multiple CPUs don't do improve anything here.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

monochrome

When the cache is preloaded it's fast - what I'm talking about is the time when IMatch has to decode 24 MP NEF files.

Mario

The FPC codecs decode a 850D NEF file in < 1s. Same for LibRaw. 4 seconds seems a lot...
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

monochrome

I don't really know which codecs I use - but the main point here is that I think it'd be a good idea to use all available CPU cores as it seems like there's some untapped performance there.

Mario

You can check with WIC Diagnostics

The Viewer preloads the images from the cache in the background. A lot is going on while you are in the Viewer and throwing more CPUs at this will not only make this a magnitude more complicated but may also cause other side-effects.

If you are really affected by this, I suggest you switch IMatch to create cache images at the time files are added to your database or updated. This means that the cache images are already there when the Viewer needs them and the cache creating delay will be gone.

Of course IMatch uses all available cores when ingesting files. This does not mean that all CPUs will be at 100%, because a large share of the work is spent waiting for file I/O.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook