Search for file names optimization - urgent request

[lightchaser]

Dear Mario,
I would really urgently ask you to give back to us the  quick search function for file names as we had in IMatch 3.6.
As i regularly have to search for instance for the raw file of an image or for the big version of a small jpeg in my portfolio I can`t stand the time loss I have to realize with the new version. It starts with the possibility to quickly copy the file name in the file window. Okay, I know: Strg +Copy. But then I have to delete the part of the folder name. Uuurggh. Then I enter search and it takes a minute to get the result. As I still have Imatch 3.6 installed i did a side by side comparison to search the same file:
In Imatch 5.6x it took nearly a minute
Imatch 3.6 it did hardly take a second.
That`s a point that drives me crazy.. Its hard to recommend the software if we get worse results than we had in the old software.
Please Mario, I`m only calling for what was reality in the old version: A quick file name search.
Regards
Franz

[Mario]

Did you try to restrict the file window search to file names only? It's as fast as IMatch 3 when it does not need to search all metadata in your database. Drop-down button next to the file window search bar.  And the "file name digit match" script, many users find useful.

It takes here much less than one minute to search all files in a 300,000 files database, for all metadata not just file names. You did not mention how many files you search, but when the search takes one minute for less than 100,000 files, something needs to be optimized on your system.

[MyMatch]

I was so far not able to find a "file window search bar" :-O
Also not in the "View" menu.

I always used <Ctrl>-f for a search.

And i am not sure if this is, what the OP wrote, but i miss the IMatch 3 version alot!

Im IMatch 3, when i did this, i got an window the the list of any and all directories (or file?!?) that contain the string i entered.
Great.
I could see all of them and act accordingly.


In the new version, i do not get this - instead, i need to klick "Next" and the result is, that i JUMP to that directory!
This is not what i need.

I need such a list - to see where and in places such directories exist.

If i need to visit them, i can still doubleclick them

So, just in case i did not overlook such an option, i very much would like to get the same search and find feature as from IMatch 3!
So, i clicked "Like" ;-)

[Mario]

I was so far not able to find a "file window search bar" :-O
Also not in the "View" menu.

It makes usually not much sense to reply to posts which are six months old.

The file window search bar, as explained in the help, is the box directly above the file window where it reads "Search the file window". This is quit hard to miss I guess.

I always used <Ctrl>-f for a search.

Yep. This is the shortcut command which focuses the file window search bar and allows you to enter something to search for. You can also just click into the search bar. After you do this, please press <F1> to see the corresponding topic in the IMatch Help System. It explains all about the search bar, how to use it, all the features you have probably missed so far etc.

And i am not sure if this is, what the OP wrote, but i miss the IMatch 3 version alot!
Im IMatch 3, when i did this, i got an window the the list of any and all directories (or file?!?) that contain the string i entered.
...
In the new version, i do not get this - instead, i need to klick "Next" and the result is, that i JUMP to that directory!
This is not what i need.

I need such a list - to see where and in places such directories exist.

I think you are confusing this with the search for folder names you had in the old IMatch 3 version. As you surely found out by having at least a glimpse at the documentation of the Media & Folders View (Tip: Click into the window you want to know something about, then press <F1>). I think you would save a lot of time if you would just give the help system a chance instead of just digging around for features you may remember from IMatch 3 from years ago.

To search for folder names, or to filter the Media & Folders View to show only specific folder names, use the features available in the property windows below the Media & Folder tree.



I'm quite sure this should more than satisfy your requirements for search for folders. Tip: It's all explained in the help. And you have something very similar in the Categroy View as well. This is explained in the help topic for the Category View in detail.

[DigPeter]

Are they asking for a search of the whole database, not just what is displayed?  Ferdinand produced a useful script entitled "Search from anywhere", which does this.

[Mario]

To search the entire database, you only need to select the Database node in the Media & Folders View, or the @All category in the Category View. No special script needed.

You can use both the filter bar and the Filter Panel on the entire database or individual folders - all the same. Both work with the current scope, which is what is in the file window.

[DigPeter]

To search the entire database, you only need to select the Database node in the Media & Folders View, or the @All category in the Category View. No special script needed.

You can use both the filter bar and the Filter Panel on the entire database or individual folders - all the same. Both work with the current scope, which is what is in the file window.

Yes - I realise that, but I find the script much quicker. I do not have a super fast computer (i7) with ssd storage.  I takes up to 2 minutes for my whole database (53k+ images) to come into scope and another 30 seconds to do the search.  With the script, 5 seconds (max) to get it and from 1 to 5 seconds for the search.  My database is regularly diagnosed and optimised.

[Mario]

This performance is real bad. i7 and a SSD is as fast as it gets these days.
And 50K files is small. I get much faster performance for my 300,000 files (!) test database.

1.  Did you try to pause the file window before you select the database node?
This avoids loading 50,000 files into the file window, just to throw them all out again when the search has finished.

2.  Where do you search? File names? Selected Metadata? Everywhere?
When you have configured the file window search bar to search everywhere, it may mean that it has to search 50,000 x (approx.) 200 metadata fields (10 million records - could also be 20 or 40 million records, depending on the file format you use , plus your Attributes (if you use them) plus 50,000 file names.

If you only want to search for file names, configure the file window search bar to search only in file names (drop-down menu in the X button). This can be done in-memory and takes about 1 or 2 seconds for 50,000 file names.

If you already search for file names only, I would really like to see a log file in Debug mode from a session where you performed the search. I also have an i7 and the database on a SSD, but I use a 300,000 files database and my search times are more than 10 times faster than yours.

[DigPeter]

This performance is real bad. i7 and a SSD is as fast as it gets these days.
And 50K files is small. I get much faster performance for my 300,000 files (!) test database.

Thanks for the response, Mario.  I said "I do<b> not</b> have a super fast computer (i7) with ssd storage"  ( I wish I had  ) My set up i5 with HDD.

1.  Did you try to pause the file window before you select the database node?
This avoids loading 50,000 files into the file window, just to throw them all out again when the search has finished.

No - I did not realise that this could work.  I thought that all the files had to be displayed for them to be in focus. I have now done this and it is very quick.

2.  Where do you search? File names? Selected Metadata? Everywhere?
When you have configured the file window search bar to search everywhere, it may mean that it has to search 50,000 x (approx.) 200 metadata fields (10 million records - could also be 20 or 40 million records, depending on the file format you use , plus your Attributes (if you use them) plus 50,000 file names.

Frequently used tags.  I must admit that I overlooked the tiny drop-down menu arrow.  It is quicker to search on file name.  Under the circumstances, I do not think  that a log is needed.

Thanks for your help.

[Mario]

i5 and i7 does not make much of a difference.
And when you only search for file names, IMatch does not need to load any data from the disk, it can do that in-memory only.

How long are your search times now?
I still wonder why it takes 2 minutes to load only 50K files into the file window.
Do you perhaps use a customized file window layout with lots of metadata?

I've made a test with a database with 54,000 files here.
Mix of mostly RAW, JPEG, some videos, audio files etc.
Database size about 2 GB on disk or so. My i7 with fast SSD.

For a "cold" load (database not loaded before so nothing in the Windows file cache) it takes IMatch 6 seconds to load the database.

I now click on the database node to load the entire database into the file window (54,000 files).
I use the "Default" file window layout. Hierarchical mode off or on makes not difference here.

IMatch reports that it took about 2 seconds (!) to load the data into the file window. Maybe add an extra second to shuffle everything into place and to let the UI settle.
That's a typical performance for such a small database.
Even if your PC would only be 1/4 as fast as my machine it should take maybe 5 or 8 seconds. Not minutes.
I would really like to see a log in debug mode to see where IMatch is spending all the time on your system.

[DigPeter]

Mario - see attached for log, my window tips file and typical thumbnail image.

The initial, cold load took over 2 minutes.  A subsequent load took a few seconds.

[MyMatch]

Ah.

I found it!

I formerly just used the "Search" field as i did not want to "filter"something.
Search behaves as i noticed: It allows to jump to the next folder that you searched for.
So "search" (with <CTRL>-f) does something different now compared to IMatch 3!

And what i wanted is now in "Filter"!

It reduces the folders in the folder view to what i was searching for!
Great.
Thanks.

Mario, please do not always assume that all people read all documention.
People are not so ;-)
They prefer to look and try ...
Also, they assume that things work like before.
It realy is so!

Therefor, valid defaults are very important - and that things do not change easily.


BTW, you did not understand that i could not find a "file window".
Now, i CAN read, and i tried to read the names and headers of all sub-windows, even in the menus!
Most such windows say that they are called, they have a title bar.

But the "file window" has no such title!
I now found the string - but just from the image you provided.
And it is even written there: "Search the file window" ... but my brain filters out such greyed-out text. I did not notice it. Sorry!

So, why not give that subwindow it´s own title?
"File Window"

So, people can find it more easy.

Thanks for your attention!

BTW, i now like IMatch 5.5 more than 3.6 ... beside some few things!

[Mario]

Mario - see attached for log, my window tips file and typical thumbnail image.

The initial, cold load took over 2 minutes.  A subsequent load took a few seconds.

If the initial load took so long and IMatch then could load the database in a few seconds from the file system cache, your hard disk is the bottleneck. It seems to perform not so well.

The log file, however, reports that the database was loaded in 17 seconds.
But then I see that IMatch needs 28 seconds to load all "object ids" for the files in your database. This is not only slow but usually is optimized out entirely by a separate cache IMatch maintains in the database. If this cache is not available, the database is either new, has recently been compacted/optimized or the database was not closed properly the last time and IMatch thus re-creates all caches on the next load....but re-creating this cache should be required only once.

To load about 54,000 files into the File Window, IMatch needs whooping 119 seconds on your system.
I cannot tell which kind of information your file window layout  shows below the image. Could be keywords or a title or something. But it looks like metadata, which requires IMatch to load a lot of extra data.

Can you please switch to the "Default" layout or a "Thumbs only" layout and see how long it takes then to load the file window?
If your disk does not perform so well, loading 54000 extra records from the database may be the cause for the slow file window load.

[DigPeter]

Can you please switch to the "Default" layout or a "Thumbs only" layout and see how long it takes then to load the file window?
If your disk does not perform so well, loading 54000 extra records from the database may be the cause for the slow file window load.

I switched to "Thumbs only" layout and it still takes about 2 minutes to load the database to the file window.

[Mario]

Are you sure your virus checker is not scanning the folder containing the database all the time?
Is this a normal hard disk or a SSD?
Two minutes is really bad, especially if no metadata needs to be loaded. I get lots better times on my old notebook with a 2.5" spinning disk...
Do you use a sort profile perhaps that uses metadata or attributes?

[DigPeter]

Are you sure your virus checker is not scanning the folder containing the database all the time?

Both the program  and the data folder are excluded from virus scan.

Is this a normal hard disk or a SSD?

I use two normal HDD's.  C drive for the program and edited/converted images (about 43,000) and an external drive for original, mainly raw images (about 10,000).  Initial load for the C drive (43000) is about 100 secs. and for the external drive (10000) about 15 secs.

Two minutes is really bad, especially if no metadata needs to be loaded. I get lots better times on my old notebook with a 2.5" spinning disk...
Do you use a sort profile perhaps that uses metadata or attributes?

My normal sort profiles are capture time, which was used for the recent test, or title, from metadata.

[Mario]

All sort profiles who are based on metadata require extra database access and processing.
To rule that out, try with the "Default" sort profile, which uses only data that's already cached in memory. If this loads your 54K files faster, we know that the slow-down is caused by the additional database operations and disk access. I'm quite sure that's it is the apparently rather slow disk so nothing I could do to help here. Did you consider switching to a SSD? They are quite cheap these days and improve overall performance a lot, even for older computers.

[DigPeter]

All sort profiles who are based on metadata require extra database access and processing.
To rule that out, try with the "Default" sort profile, which uses only data that's already cached in memory. If this loads your 54K files faster, we know that the slow-down is caused by the additional database operations and disk access. I'm quite sure that's it is the apparently rather slow disk so nothing I could do to help here. Did you consider switching to a SSD? They are quite cheap these days and improve overall performance a lot, even for older computers.

Immediately after opening IMatch, with default sort and thumbs only, all files loaded almost witin 1 second!

Yes - I have thought about SSD.  I am not clear whether the internal HDD should be replaced, or an external SSD would suffice.  And should SSD be used for data and/or the program?  Could you please advise.

[Mario]

So it's the external disk IO caused by reading 54,000 titles or keywords for your sort preset. That's a disk-intensive operation.
Since you don't usually load that amount of files into the file window at once, it does not show that much. Loading 54K files at once with lots of metadata is of course a stress test.

SSD...

It is usually a good idea to move your entire Windows partition and all programs on the SSD. Most SSDs include software which makes this transfer easy.
Windows boot times go down massively, programs start much faster etc. Your IMatch database should go on the SSD as well. This will speed up things considerably.

I know many who did it and nobody ever regretted the switch to SSD. Especially not when you get 256GB models now 80 bucks ad 512 GB models for 130. And 512 GB means a lot of data. I have a 512 SSD disk for boot and OS, connected via SATA 6/GB. And a RAID with two 2TB hybrid disks. After working with my PC for 8 months (and developers have lots of stuff on their machines) the SSD is only half full.

If switching the OS partition to SSD is a no-go, a internal SSD connected to the fastest bus on your system (SATA-3, SATA-6, M-2) is preferable.  External SSDs connected via USB 3.0 or better are the 3rd best choice IMHO. They loose much of their Oomph! that way. In any case, the IMatch database on the SSD will improve everything a lot.

Mass data like images, videos etc. can stay on standard disks, because you won't notice that much of a difference.

[DigPeter]

Thanks very much, Mario for the advice, which gives food for thought.