SPEED of IM 5 - tools

[sinus]

Hi folks

In IM 5 we have a LOT of possibilites to work with - or play

I would like to know something about the speed of some of these possibilites.
This is, because if something is known - or we can suspect - to quite slow, then I will use this not or only very reserved.

I can only guess, what is known as a bit slow, some Mario has pointed out, but I cannot find it now.

Hence, if you know something about the following things, it would be great, if you could write some comments or your observations.

The target is, let run IM 5 as quick as possible

I think about a "normal" database, with about 50'000 - 100'000 images.

Here are my guesses, mostly are really guessed, please do correct me, if you think or you know, that I am wrong.
Bear in mind, it is only a rough "table", but could maybe also help some other users to avoid, to take for example a lot of very slow "tools":

Say, I am correct, that the more panels we have, the more decreases the speed. If I know this, then I would not open a lot of panels in my workspace.

These "tools" are slow, the more we use of them
-data-driven categories   (this is specialy of interest for me)
-write metadata
-write keywords
-Panels like History, Statistics
-Stack panel
-Version Panel
-Apps Panel
-scripts (depends of course)
-Map panel
-Attributes panel
-File relations

These "tools" are medium in speed
-Rating
-Label
-Annotions
-Attributes

These "tools" are quick
-categories
-flags
-Dots
-Pins
-Bookmarks
-read metadata
-read keywords
-Variabels

-----------
Hmmmm, I am sorry, I guess, this is not a very clever posting, nevertheless I sending it 

Finally, I want simply to know, for example, does data-driven categories slow down IM, if I use a lot of them.
Or, do open panels and apps slow down IM, if I use a lot of them.

[Jingo]

Thank you for posting this topic - it is an important point of discussion because it tempers everyone expectations when using the program.  Though an exact timing might not be known, if Mario knows that certain functions/features can take longer than others - that is good to keep in mind when using (or testing out) the program.

I look forward to further discussion on this.

[jch2103]

I see one complication for assembling a 'speed' list: some feature have more than one 'speed' component. For example, data-driven categories must be refreshed or recalculated ('slow') when new files are added to the database or metadata is changed but after the refresh/recalculation they are immediately available ('fast'). In this case, if one's database is largely static, the data-driven categories will seem very quick indeed, but if one is constantly adding/changing metadata then data-driven categories may be perceived as slowing things down. I'm not sure how to factor that into a 'speed' list, but it needs to be taken into account.

[ubacher]

I have quite similar interests.
I have avoided data driven categories so far because of this and I am about to delete all my stacks
so that they can not be the cause of any slow-down.
The slowdown due to updating of open panels Mario has eliminated thru the use of the dedicated load module.

Leaves the versioning. How much speed penalty due to it?

One difficulty is the evaluation of speed: it depends so much on what went on before. I have started to look at the log
file after I had an unusual long delay to see if I could see where the delay was.

My db has almost 200 000 files and performance is generally good but at times there are (unexpected, for me) long delays.

[Erik]

I see one complication for assembling a 'speed' list: some feature have more than one 'speed' component. For example, data-driven categories must be refreshed or recalculated ('slow') when new files are added to the database or metadata is changed but after the refresh/recalculation they are immediately available ('fast'). In this case, if one's database is largely static, the data-driven categories will seem very quick indeed, but if one is constantly adding/changing metadata then data-driven categories may be perceived as slowing things down. I'm not sure how to factor that into a 'speed' list, but it needs to be taken into account.

This is important to note.  I find that items like the Quick View panel can be slow if images are not cached yet. 

Many of these items can be quite fast though if they are up to date and cached.  I'm actually in the process of trying to figure out whether it might be worth setting up the cache to cache all my images since I have a modest database that is around 40k images, and I suspect that a cache of all images would not be terribly large when compared to the overall storage of the images.  More importantly my time is worth more than the time it takes to load an image into the viewer or slideshow.

I do wonder about the identification of panels as a source of "slow" behavior.  THe app panels are going to depend on the apps.  The stack and version panels seem like obvious items of slowness, but I'm not sure that having stacks and version sets slow things down.  I'd imagine that once they are in the database they are there, unless it takes IM a lot of effort to think about what images need to be collapsed for your current file window.  I'm not noticing any big issues.

Anyway, it will be curious to continue to see people's experiences.  I'm somewhat set because my db is stored on an SSD.  THe biggest slow downs on my system seem to mostly relate to having to access my files, specifically writing, rescanning, and generating image views (i.e. generating the cache for an image).

[Mario]

My db has almost 200 000 files and performance is generally good but at times there are (unexpected, for me) long delays.

As you pointed out, the trick in such a situation is to make a copy of the log file (can be done in Windows Explorer with a Quick <Ctrl>+<C>, <Ctrl>+V>) or taking note of the time. This makes it easier to find the spot of trouble.

IMatch users have databases with 50,000 files. Other IMatch users have databases with 200,000 files. What is never a problem for a smaller database may cause delays of several seconds (or longer) with databases having 200,000 files.

There are many powerful features in IMatch, like data-driven categories, versioning, powerful configuration features for file windows etc. All this luxury comes at a price. But it is not always easy to tell in advance what, if and when a feature has a performance problem.

Versioning can be expensive (in terms of CPU usage) when too complex or too many rules are added. But it may be super fast if the user has folders with only a few dozen or hundred files. For another user who has only one version rule, but folders with ten thousand files, versioning may be slow too.

Data-driven categories are awesome to organize even large databases automatically. But setting up a data-driven category like the sample "ISO" category is complicated and can be slow. If the category needs to be updated, IMatch has to load the metadata value for ISO for all files in the database. Group all files for each unique value and then update the category tree to add/update/remove files. If you have a database with 200,000 files this means many disk operations. Combine this with a "slower" disk and you can end up in trouble.

IMatch processes data-driven category in the background so usually you don't notice this. At least not when your PC is reasonably fast. But if you switch from, say, the Media & Folders View to the Category View and a data-driven category is selected in that View and that category is currently being updated, IMatch can only continue and show the file window when the data-driven category has been completely calculated. You won't notice this on smaller database, but if Metadata for 50,000 or 200,000 files has to be analyzed, and maybe there are also filters and other data-driven operations, IMatch may block for 10 seconds or longer...

(Note: The automatic update of data-driven categories can be disabled under Edit > Preferences > Background Processing)

It all depends on what you do. Metadata write-backs to JPEG files are fast.  Write-back to XMP files is fast. Updating embedded IPTC data in a 500 MB TIFF or PSD file can take a long time when ExifTool has to split the file into parts to make room for the IPTC data. It all depends.

When I get feedback about slow performance with certain features (ideal if combined with a log file in debug mode which shows what IMatch was doing) I can try to find ways to optimize things. The trick is to find the hot spots, which is not always possible before a software is released. Especially not a software as mighty as IMatch, with so many options users can fiddle with.

The bug report forum also accepts bug reports about slow performance. The key is to be specific, e.g. telling me exactly what you did, how, and what was slow or even caused "ghosting" (when the IMatch window becomes whitish). Add a log file taken at the very moment (see info above) and I'll see to it.