bad performance and missing ratings

[Gerhard]

Since the newest version (2021.15.2), it takes almost 1 second to switch from one picture to the next (auto advance) on rating the picture and some ratings are missing.

Files are as always on a network share and the time i use to compare is for size of 2 mb or smaller, I know that 5 mb+ takes more time, but also the small images now take ages compared to the past. there is almost half a second for each image only "InnerCalculate"
04.10 03:44:55+  453 [1180] 10  M>    <  3 [453ms] CIMCollection::InnerCalculate
04.10 03:44:55+    0 [1180] 10  M>   <  2 [453ms] PTMetabase::SetRating
see also attach file.

there are about 150 ratings missing from about 7000 rated files. Metadata auto writeback is disabled but normally the application saves the rating within the DB. This occurred first with the new version and rating system. I can identify this, because every time I continue with rating (unpause file window to open viewer or restarting the application and unpause ...) the first files I see are some, which were already rated or rejected  (the folder I work in has with subfolders about 12000 new files of about 66000 files)


Currently my problem is, that I already rated too much files with the new version and I'm not finished with everything to writeback the data. So switching to an older db version is not preferred...

Is there a way to identify the db version, because I have some manual backups and don't know which I should test, when I uninstall and reinstall the older version to compare.

Thank you.

[Mario]

The log file is incomplete and important vital data. Please always ZIP and include the full log file.

The only rating-related change in the last update was that the rating now toggles. This has no impact on performance at all.
When the user presses a rating keyboard shortcut, the Viewer now just checks if the rating is already the same, and then resets it to 0. Else it sets the rating. In both cases, a rating is written to the database.
If the rating is different, the old and new rating collection are updated. If the rating is toggled, the "0" and previous rating collection are updated.
This is an in-memory / in-database operation and very fast. I don't see why this should make a difference.

Have you tried setting the rating with the Collection control or the keyboard shortcuts which increase/decrease the rating, or by explicitly resetting the rating?
Nothing with these commands has changed for a long time and there should be no difference in performance at all.

Changing a rating also triggers file relations, if the rating is reject even buddy file relations. Which means file system access, which means slow operations, especially with files on a network computer. Maybe this is the case? Which kind of rating are you toggling?

When I bring 200 files into the Viewer for 150,000 files database, and then hold down <3> (with auto-advance enabled), the Viewer rates forwards so fast, I cannot even see the individual images, at least until the cache needs to be refilled in the background. When I set the rating to these 200 files to 5* and then press <5> to toggle, the Viewer operates at the same speed.

I sometimes see timing data for SetRating/ToggleRating, at around 30ms. Most of the time I don't see any, which means the duration is below the threshold.
For your database, the duration is about 5 - 10 times as high! For SetRarting as well as ToggleRating. This would either indicate a very huge database (say, 500,000 to 1 million assets) or a rather slow performance.
Is the database on a network? How large? Virus checker interfering? Is the performance also as bad when you explicitly set ratings? It should, 'cause SetRating is as slow as ToggleRating.

The CIMCollection::InnerCalculate is logged when IMatch recalculates a collection in the background. This does not interfere with the Viewer or anything, it runs on a separate thread from the thread pool
The duration of this operation depends on the number of images in the database, and the type of collection and the system performance (disk throughput, mostly).

[Gerhard]

I use the keys 1 to 5 above the letters and x for rejecting.
Using the arrow keys right/left is fast, but pressing x,1-5 is slow.

The DB is on a local SSD.
Overall DB size is ~120 000 images/ 18GB disk size.

DB folder and application are excluded from virus scanning/realtime engine.

I'm not sure what you mean with collection control, but I rate within the viewer window. Also I'm not aware  (I mean I don't know what else keys to press) of set and toggle and how to "change" it. I preferred the way, where it was before the patch, clicking 2 times on the same image the same number(e.g. key "3"), keeps the number. So if the new is toggle instead of set, how can I "set" it?

I usually do not add the full log, because it takes time to remove data from it...

[Mario]

So if the new is toggle instead of set, how can I "set" it?

No need to set it if it is already set. Instead of doing nothing in this case, IMatch now toggles, which makes for a much smoother workflow. And makes all ratings behaving the same, from reject to 5.
You can also set ratings from the menu and the The Collection Overlay
My questions about this was to isolate the toggle from the rest. Since nothing has changed for all the other methods to set/remove ratings, there should be no performance difference.
I cannot measure any performance difference between the new toggle mode and setting a rating and removing a rating. The code is the same as before, except for the nanoseconds needed for IMatch to check if the rating the user is currently setting is also set, and then set the rating to 0 instead of doing nothing.

Starting with 200 unrated files, I set them to 3* by pressing <3> and keeping it pressed. The Viewer zaps through the images as fast as it can load them.
I then repeated this test, starting with the first file again, keeping <3> pressed to remove (toggle) all ratings.
The two tasks take the same time. Only occasionally a ToggleRating with more than 10ms duration is logged, never over 30ms.
Database size similar to yours.

Do you have any data-driven categories based on collections? Or maybe formula-based categories?
File relation rules? Buddy file rules (only when you reject files)?
What puzzles me is why your database is performing so slow and shows timings or 300+ ms. for a simple Rating toggle.
This is usually only the case when the database is huge and on a spinning disk or, more likely, on a remote disk / NAS.
But on a local SSD?
The timings seem to correlate with recalculating the "None" collection (unrated files) (in the case you set/toggle an existing rating to none).
And this is the culprit I guess. This takes 0.5 seconds.
When setting a rating, IMatch updates the rating collections directly (adding/removing as needed).
This invalidates many other collections, from the Collections root to pending write-back etc. But the collections are just marked, not immediately recalculated.
Then the collection is immediately queried by "something" and this causes the recalc and the rather sluggish performance...

Do you run the Viewer on a second monitor so the IMatch UI is visible all the time?
Do you have the collection filter loaded in the Filter Panel?
Are you in the collection View?

[Gerhard]

If the reject behavior is as before, we can ignore the rating speed, as rejecting is the same bad performance.

I have no own collections, but there is of course some default ones like "Pending Metadata Write-back" (currently 10 000).
I have the main window either open in the background or minimized on the same monitor. "media & folder" is selected and the view is paused.

There is nothing set to create buddy files. The folders I work on do not contain raw files (only jpg and png).

I select a folder on a share and have the filter panel filter on unrated files and file format not txt, but I thought this is paused, when the main window is on pause.

For testing, I just (right now) disabled everything on the "background processing" tab in the settings and removed all filter after opening the viewer(no noticeable difference).

[Mario]

This is all unrelated to background processing.

The problem is that recalculating the "None" collection (or any other collection) takes 0.4-0.6s on your system and database. And this is interfering with the UI fluidness.
So I thought about this and I could improve it quite a bit, avoiding the recalculation of the collection while setting or removing a rating - delaying it downstream until it is really needed.
For you (and all other users) this means that setting ratings or labels or other collections is now much faster - although if you notify this depends on the size of your database, the speed or your computer and your workflow.

Since all this is unrelated to the new rating toggle feature, I'll close this bug report now.