Faster culling in imatch

[stefanjan2]

I normally do my culling in imatch but yesterday came back with around 4,000 images from a bird of prey workshop.

In the viewer when scrolling through single images and rating or deleting them, imatch lags, takes time to catch up which which makes the process clunky as I have to wait for imatch to catch up.

I've seen that other imatch users use other software for culling such as Fastrawviewer. I have a licence for Fastrawviewer which has no lag at all as does Xnview MP.

I would much prefer to do all my culling in imatch rather than use another piece of software.

Are there any settings I can change temporarily change in imatch when I have a large number of images to cull? Or how feasible would it be to provide an option for culling in the viewer without a lag? For example would perhaps temporarily turning off database update help? 

[Jingo]

I just ran through this a few weeks ago with a recent birding trip myself having used my new Om-1 camera and 40-100 burst mode craziness... and came up with the following conclusions:

• Don't use 100 FPS unless you enjoy the culling process. Reviewing 80-100 of the same image with minute differences isn't fun.  
• There isn't any software on the market that I could find that let you review more than 4 images at a time together...other than IMatch at 8(!).  I tried FastRawViewer but found it clunky... also tried: Photo Mechanic, XNView, Irfanview, and many many others..  No dice!

Do - my process has become the following:

• Select groups of images to compare in the File Manager and then open them in the viewer.
• Highlight the first image (unless you did this in the FM - the last selected message also selects in the viewer) and choose 8 images.
• Use zoom to compare the 8 images and reject via delete the images that don't make the cut.
• Rinse and repeat through the set of selected images and then exit back to the file manager.
• Set a quick filter (or filter if not using experimental mode) to Hide Rejected images.
• Rinse and repeat with next set of images to compare

At the end of the cull, invert the filter, select all and delete the rejects..  then, rate, keyword, etc.

I was quite surprised not to find a program that let you compare 10-15 images together.. but I've found the Imatch 8 to be sufficient for now.

I too welcome anyone that has a different process to share... and hope this helps! - Andy.

[Mario]

myself having used my new Om-1 camera and 40-100 burst mode craziness... and came up with the following conclusions:

My first camera had a film which had space for 36 photos (only24 sometimes). And it was very expensive to develop.
I wonder how we I managed

I see reason to have 3 or maybe 10 photos per second-burst in a bracket in some situation (for my case: motor sports). But 40 to 100 in bird photography? Wow. You might get the "one" short that way, but my goodness, what an effort to review all these files and cull them. 40 to 100 photos per shot / motive...

I would myself write a software that overlays maybe 2, 4 or 8 of these images at a time and highlights the differences using math for pixel subtract or add. Basically walking though all shots of a burst that way.
I'm not surprised that there is no software for that. It's just to niche.

[stefanjan2]

I'm not surprised that there is no software for that. It's just to niche.

Are you answering Jingo or my original question. For my initial cull I want to be able to scroll quickly through a large number of images selecting those for a more detailed look.

Xnview MP can do that well, imatch can't. Once I have selected the best from a short burst I can then use imatch multiple image view to select the one or two I wanted to keep but it sounds like I have to continue XnView MP or Fastrawimageviewer for the intitial cull.

I was hoping I could do this initial cull in imatch but from your reply I understand that imatch in it's current iteration is not able to do this and there is no chance of adding this capability.

To be clear, I am not talking about bursts of 40-100 shots. We had 8 birds of prey with multiple flights and static shots. I don't each of my bursts was more than 5-10 shots.

Imatch works well for small shoots but for larger shoots I guess I'll have to use other software.

[Jingo]

I'm surprised you find IMatch not fast enough for an initial review given your mentioned shoot parameters... as mentioned, I'm often looking at 500-600 images from a bird outing with groups of 10-30 similar photos. 

For groups of 5-10 shots to compare, I find the viewer to do a great job - even if you don't use it for comparing with each other in groups of 8 like I do.. just open them in the viewer, set zoom to the right fit for your image and then use the arrows to quickly review, rate, reject each image as needed.  For me, I do this in groups of 8 against similar shots.. then do a 2nd round for the keepers until I have a much smaller set for final review later.

What is it that you find IMatch is not letting you do for the culling process?  Speed?  #of images to compare?   Just curious...

[Mario]

In the viewer when scrolling through single images and rating or deleting them, imatch lags, takes time to catch up which which makes the process clunky as I have to wait for imatch to catch up.

Are you working with RAW files?
Do you make IMatch produce cache files on-demand or do you let IMatch create cache images when it indexes the file?

What kind of computer and graphic card?
Which overlays and panels do you show in the Viewer?

IMatch can preload cache images into the graphic card very fast in the background using predictive caching based on your scroll pattern. But when you scroll faster as Windows can load cache files into the graphic card, you'll have to wait a bit until the background cache catches up.

As always, a log file in debug logging mode from such session would be helpful. Because it shows precisely, how fast WIC/IMatch can load images, if the work you to triggers costly database updates and whatnot. If there is enough memory available, if the graphic card is happy etc.

[stefanjan2]

Are you working with RAW files?

Yes CR3 and RW2

Do you make IMatch produce cache files on-demand or do you let IMatch create cache images when it indexes the file?

See cache settings attached

What kind of computer and graphic card?

Desktop:
AMD Ryzen 7 2700 Eight-Core
32GB Ram
NVIDIA GeForce RTX 3060

Laptop:
AMD Ryzen™ 7 6800HS
32GB Ram
NVIDIA GeForce RTX™ 3050

Which overlays and panels do you show in the Viewer?

Collections, Visual Proxy, Filmstrip

when you scroll faster as Windows can load cache files into the graphic card, you'll have to wait a bit until the background cache catches up.

Why does imatch wait but other software doesn't? My original question was, or suggestion was, would it be worth adding a culling mode to imatch so it did not lag? Avoiding the need to use dedicated culling software for large batches?
Maybe an enhancement which would make imatch even better and more attractive to new users. It's already fantastic!

As always, a log file in debug logging mode from such session would be helpful.

log file attached

[stefanjan2]

What is it that you find IMatch is not letting you do for the culling process?  Speed?  #of images to compare?  Just curious...

If I am scrolling a lot of images, quite a few display fine in the viewer but then I get a "loading" message and have to wait for imatch to catch up. I'm scrolling quickly in the first cull to select those I want to view for a more detailed cull.

[Mario]

Why does imatch wait but other software doesn't? My original question was, or suggestion was, would it be worth adding a culling mode to imatch so it did not lag?

I don't know. In my experience, the Viewer can load files as fast as I scroll.

My original question was, or suggestion was, would it be worth adding a culling mode to imatch so it did not lag? Avoiding the need to use dedicated culling software for large batches?

Was never needed nor requested. The Viewer is in constant culling mode for up to 8 images.
It preloads images in the background depending on your scroll direction. If you scroll back and forth, you may run out of preloaded images and then you'll have to wait until IMatch has loaded the next batch.

Your database has less than 70K files, which makes it a smaller one.
Still, IMatch reports that it takes 17 seconds to initialize, and 9 seconds to load the database alone.
That's on the slower side but OK.

Your cache is on the D: disk in your system. Which kind of disk is this?
Because the log shows:

D:\imatch Cache\C1B4B049-CF54-4238-978A-9B1F207DA12C\46\462176.jpg loaded in 469ms
...
D:\imatch Cache\C1B4B049-CF54-4238-978A-9B1F207DA12C\46\462177.jpg loaded in 516ms.
...
D:\imatch Cache\C1B4B049-CF54-4238-978A-9B1F207DA12C\46\462203.jpg loaded in 937ms.
...
D:\imatch Cache\C1B4B049-CF54-4238-978A-9B1F207DA12C\46\462204.jpg loaded in 1297ms.

Loading a cache image takes 0.5 seconds to over a second.
That's quite slow. No wonder the Viewer cannot scroll faster when it takes that long to load a cache file.

For a cache on a SSD, the typical load times are < 0.1s if the cache image exists.
If the cache image must be generated first, load times may vary between 0.5 s and 10 s, depending on the RAW format used, whether it has a suitable embedded preview and if WIC or LibRaw are used.

The load times we see in your log indicate that either a) D: is a rather slow disk or b) (more likely) cache images are missing must be generated first. Or maybe your virus checker is first scanning each cache file before it allows IMatch to load it?

[stefanjan2]

The load times we see in your log indicate that either a) D: is a rather slow disk or b) (more likely) cache images are missing must be generated first. Or maybe your virus checker is first scanning each cache file before it allows IMatch to load it?

a.
Could be the issue as I'm running microsoft storage spaces on spinning disks with mirroring on my desktop pc drive D. I do have a couple of m.2 ssd and conventional ssds in my desktop. I will have to try and reorganise my desktop so that the cache is on an SSD or m.2 . Would an m.2 SSD be much faster than a normal SSD?

My laptop has a single 1TB M.2 2242 SSD and displays less lagging when scrolling than my desktop

b.
On the settings I have shouldn't cache images be created automatically when importing images into imatch. All the images have been indexed. Am I missing something here?

c.
My only virus checker is Microsoft Defender. Should I change any of Defender's settings?

Another thought
I believe the embedded jpeg from my Canon R7 is 6960x4640. I don't need this size for culling, would it make sense to reduce the cache image to say a maximum of half this size?

Many thanks Mario for investigating this for me. It's reassuring to learn that the issue is probably with my setup rather than imatch.

[Mario]

Could be the issue as I'm running microsoft storage spaces on spinning disks with mirroring 

There you have it.. Looks like your configuration for cache storage is about 10 times slower than cache storage on a SSD.

Would an m.2 SSD be much faster than a normal SSD? 

Yes. Much faster interface.

On the settings I have shouldn't cache i

You use the default setting so cache images are created when the file is indexed.
You also don't purge the cache so all cache files are kept forever.

c.

Defender usually does not get in the way.

I believe the embedded jpeg from my Canon R7 is 6960x4640. I don't need this size for culling, would it make sense to reduce the cache image to say a maximum of half this size?

This depends on your requirements. The default cache image size is 2,000 pixels, which is "good" enough. If the embedded cache images are < 2K, IMatch will ask WIC/LibRaw to develop the full raw for the cache file.

If your camera produces 7K embedded previews (which is usually something you can configure in the camera) and you cull on a 2K monitor, setting the cache image size to a value matching your screen is perfectly fine. It won't do much for the image loading speed, though. From a SSD, a typical cache JPG is usually loaded in < 0.1 seconds.

And you won' be able to zoom to 100%, which I find usually beneficial while culling photos to check for sharpness etc. IMatch only displays the cache file in the Viewer, Quick View Panel, Peek View and Slide Show. Having the image at 100% is usually best.

Check the size of the cache in the Info & Activity panel.
If possible, move it to an SSD. Big speed improvement.

to learn that the issue is probably with my setup rather than imatch.

A spinning disk is a lot slower than SDD. Usually this is not important for things like cache file storage. Unless you want faster display sizes in the Viewer, where 0.07s vs. 0.5s (or, in your case, even 1.2s) for loading a cache images makes a lot of a difference.

[stefanjan2]

I've moved the cache to a Samsung 970 EVO NVMe M.2 SSD, it seems much better.

There is an initial lag when I open the viewer for the first images but then if I scroll at normal culling speed it's much improved.

Attach screen grab of the info panel also if you feel inclined to take a look I've attached the debugging log after moving the cache.

[Mario]

120 GB for the cache is a lot, especially when you don#t frequently look at old files.
Maybe enable purging and limit the size of the cache to something that covers your typically work scope.

[Mario]

The typical load time for cache images is now 0.5 seconds, which is OK for 7K cache files.

[stefanjan2]

Thanks again for your excellent support.

I have set purge on and set max size of the cache to 100GB. Actual cache has come down to 90GB.

Does cache size make any difference to performance. If I reduced cache size would it be faster or not make any difference?

[Mario]

Does not make a difference, unless it makes a difference for the Windows file system.
IMatch splits cache files into multiple sub-folders to keep access times (e.g. checking if a cache file is still current) down.

But a 120 GB cache folder is usually just wasted disk space. If you have cache files for images taken over a period of 10 years, how likely is it that you want to look at images taken 10 years ago and don't have the time to wait a short while while IMatch recreates the cache files when they are needed?

If you have disk space enough and a database small enough, you can of course let all images sit in the cache. If you want to conserve disk space and reduce backup volume, set a purge strategy that fits your typical usage pattern.
It all depends, which is why this is configurable.

[stefanjan2]

Thanks, all makes sense now

[philburton]

myself having used my new Om-1 camera and 40-100 burst mode craziness... and came up with the following conclusions:

My first camera had a film which had space for 36 photos (only24 sometimes). And it was very expensive to develop.
I wonder how we I managed

As a Kodachrome shooter, I remember thinking that ever shutter press cost 50 cents US, half a dollar. 

I would myself write a software that overlays maybe 2, 4 or 8 of these images at a time and highlights the differences using math for pixel subtract or add. Basically walking though all shots of a burst that way.
I'm not surprised that there is no software for that. It's just to niche.

Overlay, and not side by side comparison, is the key, I think. 

If you wrote something as a standalone utility you might get some sales.  I for one would be happy to publicize that program in the Lightroom forum where I am an active member.

[stefanjan2]

Overlay, and not side by side comparison, is the key, I think. 
If you wrote something as a standalone utility you might get some sales.

I am struggling to understand how overlaying images to highlight differences would help.

Now that I have solved the speed (cache) issue, I find imatch superb for culling.

1. Quickly delete the obvious failures

2. open the viewer and select up to 8 similar images to compare using the available tools if necessary e.g. focus peeking as per attached

3. Rate the keepers 5 - 1, send the 5 & 4s to Photolab for processing. Delete the unrated and possibly 1/2s depending on numbers of 5,4,3

I'm really happy with this workflow. As a previous Lightroom, ON1, Bridge user I'm really happy with imatch.

[Mario]

In your case, the 8 side-by-side images comparison with synchronized zoom & pan in IMatch should work very well.

I am struggling to understand how overlaying images to highlight differences would help.

That's a very particular problem very view users need to deal with.
Some modern mirrorless cameras shot up to 100 images per second in bursts. The differences between these 100 frames are naturally minimal so that overlaying several shots to indicate the actual differences will be helpful. The differences may be a few pixels only between frames, or lots more, if fast-moving objects like rare cars or animals in flight or similar are the motive.

But that's really some niche problem. I think most IMatch users can work just fine with 2, 4, 6 or 8 images side-by-side.
I could probably even increase this to 10 or 12 in a future versions, for users with very large monitors and beefy graphic cards with lots of RAM.

[stefanjan2]

I could probably even increase this to 10 or 12 in a future versions, for users with very large monitors and beefy graphic cards with lots of RAM.

Yes please, 12 would be great!

[Mario]

I could probably even increase this to 10 or 12 in a future versions, for users with very large monitors and beefy graphic cards with lots of RAM.

Yes please, 12 would be great!

Add feature request so we can see how many users are interested.
Increasing the number of side-by-side files is probably doable, but will require one or two days of work. Or more, if I have to deal with things like DirectX playing games. And for how many users?

When I look at telemetry to see how many images users display side-by-side, I see this:



97% of the sessions use only one image in the Viewer.  The 3% are 2 to 8 files.
Of course comparing multiple files side-by-side is something most users don't need very often, which has to be considered. Still, side-by-side display is not used that much.

[PandDLong]

That is a very interesting statistic and I find it quite surprising - the value of telemetry!

I almost always use the viewer for multiple images - culling and confirming/correcting face recognition.   It was one of the features of iMatch that I really liked right from the first day of evaluation. I also have a Quick View panel open most of the time as I find that much more efficient to look at a single image than opening the viewer.  (I am a single monitor user, that may make a difference to my use of the Viewer).


My 2 cents.

Michael

[Mario]

as I find that much more efficient to look at a single image than opening the viewer. 

You will like the Peek View in IMatch 2024/2025 a lot.
Middle mouse button click to open the clicked image in full-screen as long as you hold down the mouse button. Very fast and convenient. SPACE+left-click also works and Shift+SPACE for keyboard only types.

If you want more side-by-side images, open a FR so others can like it.

[philburton]

Overlay, and not side by side comparison, is the key, I think. 
If you wrote something as a standalone utility you might get some sales.

I am struggling to understand how overlaying images to highlight differences would help.



1. Quickly delete the obvious failures

2. open the viewer and select up to 8 similar images to compare using the available tools if necessary e.g. focus
peeking as per attached

For an action shot sequence, compare minor differences that might be hard to perceive in a side by side view without a lot of effort.

[stefanjan2]

Add feature request so we can see how many users are interested.

Nice to have but in view of the small number of users and effort to implement, probably not worth it.

[Mario]

An example of a feature that compares images to highlight differences would be something like this:



The third image shows the pixel difference between the two images. These are visible when you zoom in and know where to look for, but not that easily to spit. A feature that compares the pixels in two images and directly highlights them makes this easy.

If there is a feature request for this with sufficient Likes, I can look into this.
I envision this would work similar to the shadow/highlight overlays the Viewer offers and in a 2 images side-by-side view.

[Mario]

I've had a look at the code and it actually turned out that I (at some time) prepared the Viewer to show up to 16 (!) files side-by-side. It was just disabled / not implemented  in the code. Probably due to issues with graphic cards at the time. Not sure.

I have implemented additional 9 (3x3), 12 (4x3) and 16 (4x4) side-by-side modes for IMatch 2024/2025.
This is how it looks with 16 files:

[stefanjan2]

I have implemented additional 9 (3x3), 12 (4x3) and 16 (4x4) side-by-side modes for IMatch 2024/2025.
This is how it looks with 16 files:

That's sounds great. Can't wait for the upgrade. Something to look forward to. Both my desktop and laptop have dedicated graphics cards and decent resolution screens so I should be good to especially on the desktop large main screen.

The icing on the cake would be to have an option which meant images marked for deletion would vanish from the viewer but I think in the past you have said not easy to implement.

Anyway thanks for again responding so exceptionally.

[Mario]

images marked for deletion would vanish from the viewer

The Viewer skips marked images when navigating, so they usually don't get in the way.

[stefanjan2]

The Viewer skips marked images when navigating, so they usually don't get in the way.

I appreciate the view skips them but I think it would be even better if they disappeared so that when choosing images to keep images which were not deleted populated the 8 image view.

My workaround is to open a selection in a results window. Open selection in viewer grid. Delete some images. Press escape and then open the selection again with just the images which have not been deleted.

I appreciate that I can control click images in the filmstrip but I find it a bit trick to select using that method. Often find I end up with deleted images in the mix.

[Mario]

Fee free to add a feature request.

[Jingo]

Thanks for the bump Mario... this is PERFECT for me as mentioned in my recent viewer post. As also mentioned in that post, I too am in favor of a way to remove deleted images from the viewer so I can keep comparing images after a first pass without having to reload them into the viewer each time.  +1 from me for sure!