What is interruptable, and how best to do that?

[MrPete]

Now that I have a database with a few hundred thousand images, it's quite possible to get into situations where the wait icon is spinning for a while. Two examples:

• Accidentally click on the top of a file tree, with "show all levels" enabled, and forgot to press Pause
• Start writing a bunch of metadata, only to discover that this can take quite a long time
• (Even with a small DB, it feels "hung" while processing a few bad files ie w/ bad attributes etc.

The first one is something I am running into quite a bit. Any time I accidentally touch the wrong thing in the file tree, category tree, etc, that implies looking at a ton of images, I'm more or less stuck for a while. (Most of the tree has only a few files, but I'm learning there are places I do NOT want to touch because I'll be stuck...)

I searched here and in the help for "interrupt" and a few other words, and didn't find anything particularly useful.

My questions:

• Is there a recommended way to interrupt "busy" actions?
• Does iMatch intentionally break down large tasks into small interruptable chunks?*** 

If not, this would be one of my top suggestions for improved usability. An app that's nonresponsive, pretty much ever, can quickly feel frustrating.



***(The most sophisticated thing I've seen is something like this: a) keep a timer and two key params: sizeBatch (how many per batch) and timeBatch (seconds), usually preset to 1-3 seconds; b) begin with batch size 1, see how many it takes to go over timeBatch, if get to 10, scale sizeBatch up until time is reasonable: does 20, 50, 100, 500, 1k, 2k, 10k, 50k exceed timeBatch? Then let it go from there.)

[Mario]

Accidentally click on the top of a file tree, with "show all levels" enabled, and forgot to press Pause

You click on a folder. IMatch to load the folder into the File Window.
There is no "load half of folder" or an interrupt for loading a folder.

IMatch implemented a semi-working File Window "load cancel" via ESC years ago. It was complex, costly to maintain and according to telemetry was used 0 (zero times) in an entire year. So I removed it and could make loading files into File Windows much faster as a reward.

Not sure if your problem is a problem for many. I have a database with ~900,000 files. When I accidentally click on @All or the Database note, I have to wait maybe 10 or 15 seconds. I find that a good "punishment" for clicking the wrong thing. I don't to that often, though.

IMatch offloads many tasks and spreads them across all processors available in the system. It does most of it's work in the background. But, sometimes, users design a workspace so performance-hungry that not even IMatch can cope.

Showing dozens of expanded categories, for example. In the category View, Filter Panel or Categories Panel. If they are visible and become invalid, IMatch must recalculate them. And if the category is expensive to calculate (data-driven! variables!!) this can take some time.

Tip: Search the IMatch log file for #sl to find slow tasks. And slow categories. Having too many slow categories is a small problem that becomes a big problem when your database manages 500,000 files instead of 100,000.

Make sure your virus checker is leaving the folder containing the database alone.
Store the database on the fastest SSD you have (usually the C: drive).

Start writing a bunch of metadata, only to discover that this can take quite a long time

The progress dialog shown when you write-back metadata has a Cancel button.
Write-back speed is 0.5s per JPEG and maybe 1 or 2 for a RAW file. IMatch writes multiple files at the same time.
If your files are very large or on slow NAS storage, things may be a lot (!) slower. Compared to a local SSD NAS storage is 100 to 1000 times slower. Even slower if you have only Wi-Fi instead of 1GB/sec or 10GB/sec cable network.

(Even with a small DB, it feels "hung" while processing a few bad files ie w/ bad attributes etc.

When you mean "indexing files" (adding or updating files), a bad file may slow-down one ExifTool instance. Maybe even cause IMatch to terminate an instance after 2 minutes of inactivity. ExifTool seldom hangs, though.
But IMatch infests 4, 8, 16, or more files at the same time in background threads. The occasional bad file will not make a blip in the overall indexing.

IMatch does all this in the background, without blocking the UI. Except you use the loader overlay to speed up processing by allowing IMatch to not update the UI all the time (default and recommended).

Of course, if you have dozens of bad files which are all processed at the same time or your virus checker interferes and blocks IMatch, things look different.

Now that I have a database with a few hundred thousand images 

Something to keep in mind:

1.  Many processional stock photo agencies manage far less images. And they use potent hardware.

2.  I don't know what kind of computer you use, but I work speedily with databases managing about 300,000 files on my Dell notebook with a very fast .m2 SSD, 32 GB RAM and 16 processor cores with 22 threads. And a NVIDIA 4060 8GB graphic card for AI loads. Did cost me less than 1,200€ on last Black Friday.

3.  Physics!
Searching 200,000 files takes about twice as long (plus a bit) than searching 100,000 files.
If you have many data-driven categories, for example, it will take twice as long to update each of them when you go from 100K files to 200K files. Same for collections. Filters in the Filter panel.

IMatch is extremely fast and I constantly look for ways to make it faster.

For IMatch 2025, for example, I have redesigned and rewritten the entire background scheduling which is responsible for indexing files, reading metadata, face recognition etc.

The new scheduling is capable of utilizing and maxing out modern processor architectures with 12, 16, 24, 32 or more cores. And utilize modern super-fast .m2 SSDs to 100% when there are enough processor cores to generate enough work. It's fully dynamic, adaptive and quite beautiful. I sometimes see 50% to 300% better performance.

If you try to run a 500,000 files database on a several year old notebook, you will experience more performance issues than running the same database on a modern high-end notebook or even a workstation computer.

Reducing expensive operations by e.g. removing unneeded data-driven categories or setting them to manual is a good measure.
Closing or hiding unneeded expensive panels like Categories or Filter is also a good idea if you experience performance issues.

[MrPete]

Mario,
VERY helpful insights! Thank you.
I just spent a week flying to/from Korea, and attending intense meetings. Had no idea I would have zero spare time to go online. My apologies for the null response until now.

Pretty sure my issue is not HW performance at this point. Yes, yours is 2 years faster and smokes mine in specs, but I do have plenty of RAM (can allocate up to 80GB if needed - it doesn't), and am using RAID m.2 NVMe SSD for even more performance in most situations. These newer computers are NOT slackers

I'll dig through your several hints as I'm able. 

THANKS AGAIN!!!
Pete