Speed of searching

[sinus]

Hi Mario
ok, here I am to write something about the speed of searching. It was not my intention, but I can of course say something.
It is, like the old search in IM3, really not that fast. It depends on where you search.
And MAYBE it is the same thing like with IM3. The first search over the whole db is slow.
But the second round is much quicker.

first run
in a folder, 2038 files in it, search for filenames only, search "menziken"
found 357 files, time: about 47 seconds

in a folder, 2038 files in it, search for everywhere, search "menziken"
found 357 files, time: about 5 minutes, 45 seconds

So, if I am not wrong, IMatch searches for the images in the scope, I am in a folder, so IMatch searches only for 2038 files.
If so, I would say, this is a not a very quick search.

That is why I understand ubacher: if I stay up for a cup of coffee during the 5-minutes-search and come back, I will not see on the first view, is the search still on or not. (see attachement 1 during the second search).

And that is, why I came up with the question, that during such a search would also be helpful a "timeline-box", with a moving bar and the roughly time, what IMatch will be searching.
We had such a box in the searches of IM3, and I knew always ... a search is running.

If this makes here no sense, a solution like ubacher asked, is of course also good. It should simply be something, what we can see on one view, that a search is performing.
That is all, what I asked for.

I want not complain about the speed. I guess, in a second run the search would be ways faster (it was so in IM3), so I try it again now with exactly the same preferences.

second run
in a folder, 2038 files in it, search for filenames only, search "menziken"
found 357 files, time: about 0.5 seconds

in a folder, 2038 files in it, search for everywhere, search "menziken"
found 357 files, time: about 1 minutes, 20 seconds

So this seems to be ways faster. Though the search for everywhere I would also not call very fast.

OK, I do a third run, this time something else:

third run
in a folder, 754 files in it, search for filenames only, search "eckenheim"
found 2 files, time: about 0.5 seconds

in a folder, 754 files in it, search for everywhere, search "eckenheim"
found 2 files, time: about 1 minute, 17 seconds

in a folder, 754 files in it, Limit search for frequent used tags, search "eckenheim"
found 2 files, time: about 31 seconds

Now I change to the category-view, there I can easy search for more images.
in a folder, 84'996 files in it, search for filenames only, search "elux"
found 2'703 files, time: about 0.5 seconds

in a folder, 84'996 files in it, search for everywhere, search "eckenheim"
found 3'842 files, time: about 1 minute, 17 seconds

in a folder, 84'996 files in it, Limit search for frequent used tags, search "elux"
found 3'166 files, time: about 33 seconds

I would say, for searching with over 80'000 files, this is very good for me, looking at the speed.
I have the advantage, that all my most important things are in the filename, so for me the search for filenames is really VERY fast
 

So as I pointed out, FOR me the speed is not that important (if you remember, IM3 has in its first run to search the whole db quite a while, so I can do in this time something else).

But nice to see would be on one view, if the search is still running or not.
And I think, you as the chief has the choice, what you want do, but from my point of view, only dimming the window is for some users not clear enough.

Another colour somewhere, a small box (like attachemet 2 from IM3) or a big box, like you use for indexing images, it does not really matter (for me), but it should be better viewable.

Because ubacher made already a feature request, I must not make another.
And for the speed, well, maybe this here helps you, but for me personally it is ok.

Maybe I did also something not very good, and for sure my computer is not the best and for sure also I have quite a large db, but for me it is ok.

Actually I have 169'675 files in the db, the size of the db is about 9.5 GB.
Thanks.

hmmm, just before I posted this, I wanted attach the log, but I closed IMatch and so the log was gone (did not find it).
To have a log, I opend IMatch again and tought, ok, I let run the searches again, and guess, what?

The first run have been much quicker.
in a folder, 2038 files in it, search for filenames only, search "menziken"
found 357 files, time: about 0.5 seconds

in a folder, 2038 files in it, search for everywhere, search "menziken"
found 357 files, time: about 2 minutes, 30 seconds

So I do not know, if it makes sense to repeat the whole search, Mario?

[attachment deleted by admin]

[ubacher]

I tried a few searches and (at least for file names only) speed is not an issue.

I noticed that the X turns red - and I could not find any info on this in the help file.
It seems to stay red while it is searching. Mario?

[sinus]

I tried a few searches and (at least for file names only) speed is not an issue.

I noticed that the X turns red - and I could not find any info on this in the help file.
It seems to stay red while it is searching. Mario?

For seach only in Filenames, I think, speed is mostly not a problem in speed. Was not in IM3 and is not here. The speed is for such a search very good (except some exeptions, mavericks).

As I wrote, also for me speed is not a problem, specialy because I have the most important information in my filenames; I did this very long and I am happy in always again with this decision).

Sometimes I have seen the red X also, but from my point of view it is not enough clear for me. Search is exentiell for a DAM, and hence a user should be able to see, if a search is running or not, very quickly .

We have such a LOT of information in IM5, from my point of view this information for the searching could be better viewable.

[Mario]

I've made a few tests now.
Database with about 70,000 files. Mixed image (all formats + RAW), PDF, video, audio.

Select the database node and enter

studio AND mario

into the search bar. Search bar is configured to search frequently used tags.
Search time: 2.4 (!) seconds. File Window displays 890 results.

Perform a few searches on your system and then search the log file for

CIMQueryManager::InnerSearchMetabase

in the rows starting with < you'll find the run time, e.g.

< 15 [2340ms] CIMQueryManager::InnerSearchMetabase

where 2340 is the time in milliseconds (1/1000 s).

[sinus]

Perform a few searches on your system and then search the log file for

CIMQueryManager::InnerSearchMetabase

in the rows starting with < you'll find the run time, e.g.

< 15 [2340ms] CIMQueryManager::InnerSearchMetabase

where 2340 is the time in milliseconds (1/1000 s).

Thanks, Mario, I did so.

I go to a node with 41'395 files (nef, jpg, tif, doc and others) in roughly 50 folders

Search-Option "Limit to search for frequently used tags (faster)"
Search for "Ausstellung"
found 711 files in 28.7 seconds
11.02 20:45:12+    0 [0B64] 10  I>                  711 results fetched (bv)
11.02 20:45:12+    0 [0B64] 10  M>                < 15 [28688ms #sl] CIMQueryManager::InnerSearchMetabase

Search-Option "Limit to search for frequently used tags (faster)"
means, the searched words must be in the same tag!
Search for "Ausstellung AND Verkauf"
found 98 files in 27.7 seconds
11.02 20:47:44+    0 [100C] 10  I>                  98 results fetched (bv)
11.02 20:47:44+    0 [100C] 10  M>                < 15 [28658ms #sl] CIMQueryManager::InnerSearchMetabase

Search-Option "Search file names only"
Search for "Ausstellung"
found 417 files in 2.3 seconds
11.02 20:48:28+ 2262 [0FA0] 10  I>                  417 results fetched (bv)
11.02 20:48:28+   16 [0FA0] 10  M>                < 15 [2325ms] CIMQueryManager::InnerSearchMetabase

Search-Option "Limit to search for frequently used tags (faster)"
enabled the option "AND/NOT consider all tags"
Means, the searched words can be in different tags!
Search for "Ausstellung AND Küttigen"
found 98 files in 1 minute and 3 seconds
11.02 21:03:06+    0 [0B6C] 10  I>                   98 results fetched (bv)
11.02 21:03:06+    0 [0B6C] 10  M>                < 15 [62962ms #sl] CIMQueryManager::InnerSearchMetabase

Search-Option "Search everywhere"
enabled the option "AND/NOT consider all tags"
Means, the searched words can be in different tags and the search goes over ALL tags, this means, this should be the slowest.
Search for "Ausstellung AND Küttigen"
found 24 files in 1 minute and 21 seconds
11.02 21:59:12+    0 [0FA4] 10  I>                  24 results fetched (bv)
11.02 21:59:12+    0 [0FA4] 10  M>                < 15 [80622ms #sl] CIMQueryManager::InnerSearchMetabase


Now, this means to me:
the filename-search is VERY fast.
And the "search everywhere" is a bit slow.

But the ultimate search for things, where we cannot fully remember all, this search is the "messy brain-search"  and this are FOR ME these options:

"Limit to search for frequently used tags (faster)"
and enabled the option "AND/NOT consider all tags"

With this preferences I can simply add one or several words and IMatch will find them, and it doesn't matter, in what tags these words are, they must not be in the same tag.

The only difference to the slower search "Search everywhere" is, that the "messy brain-search" searches not all tags, but for sure the most important tags.
So this will be fine for me.

I do a real test and try to find some files in my whole database with 168'537 files.
So I go to the Database-node in the "Media ¬Folders"-view.
I want to find all files, what have the word "Zürich" (a city) somewhere and because this must be quite a lot, I want search for a second word, "Kirche" (church).

Zürich alone would bring too much files.
Kirche alone would bring also files from other towns, hence I tpye in

Zürich AND Kirche
Well, after quite a short time (all 168'537 files will be searched!) I have my results:

20 images from Zürich, all with a kind of church.
found 20 files in 46 seconds.
11.02 21:24:02+    0 [04B4] 10  I>                  20 results fetched (bv)
11.02 21:24:02+    0 [04B4] 10  M>                < 15 [45771ms #sl] CIMQueryManager::InnerSearchMetabase

--------
So, finally, this means to me: the search-speed from IMatch is very good. It makes sense, when we think a bit about what we are searching.
Remember, the search is over the files in scope, so if we know for example, the results must be in the year 2013, so it would make sense, to go first to this year (or to a category and so on), to limit the files in scope and then let run the search.

Thanks, Mario, for a very good search. And yes, we have also still the powerful filters!