Enhancement of the App "File Finder"

[sinus]

We have several ways to search for images in IMatch.
But almost all search options (filter, search bar) only work with files in the file window.
This means that if you want to search ALL files in the entire database, you first have to go to the All-category or to the top database tree (database) and then after the search again back.

And that is sometimes tedious.
Fortunately, there is an exception: the app "File Finder".
You open the app without changing the window (folder, category ...), enter a name and very quickly IMatch displays the files found in a result window.
In this search, you can set IMatch to search the entire database so that no file is "forgotten". Perfect.

There is only one problem: this search only searches file names.

Hence my feature request:
Please add a search option to this search (File Finder) that allows you to search one or more words in the entire database.
And the App would search in all Metadata-fields, this would mean, it would even find a file, where I have entered a word in the complete wrong metadata-tag, or if a file would be in the wrong folder ...

I know there is a great app by Andy/Jingo (Fancy Search), which I use very often, but unfortunately it takes quite some time (about 2 minutes) with my many data (over 300'000 files).
But I know from Mario's comments that you can optimise a search so that it is faster.
But I'd rather wait for 2 minutes than have to laboriously go through directories and then, above all, back again.

[Jingo]

Hi Markus - been a long time since I last looked into my Fancy Search code.. looks I am just using a standard Imatch.get('v1/search/metadata' call with the list of provided id.tags and then showing the results in a result window.   Of course, that is going to take a lot more time to search than the 'v1/search/filename' that the standard search is using.

The only thing I noticed is that perhaps I can add "tag groups" to the parameters and target only those groups that are part of the search list.. ie: no need to search IPTC if you are only looking to search through XMP tags.. not sure how much this will speed things up - but I can give it a shot! 

Stay tuned!!

[Jingo]

Hi Markus - can you please give this version of the App a try to see if it makes a difference on your system?

I just tried it on my end and searches are now taking a second or two... but with the buffering and such of lists and ids, it could be red-herring too.  I close/re-opened IMatch and searches that were taking 21 seconds before now open in 2 seconds... that is through a database of 88,000 images.  I do hope this simple change works!!

Probably a good idea to backup the original folder just in case - but I only made a single line change to the app so overwriting should be ok too..  Looking forward to your test results.

Fingers Crossed! - Andy.

[sinus]

Thanks, Jingo

I did not expect, that you looks again into your code 

I can only remember, quite some time, maybe a year or so, Mario wrote something about your app, that it could be quicker using something others ... but I am so sorry, cannot remember.

IMatch can search very, very sophisticated (filter and so on), but the only reason for my post is, finally, from my point of view, it should also be possible in a DAM, hit a button, enter a word and let search over the whole DB.
Finally, what your app does. 

I can remember, I could this looooong time ago in a product from Fuji (DAM) and in Portfolio from Extensis. Simply hit a button, enter a word and the search went over the whole DB.

And I can remember, I had such a search also in IMatch, also a long time ago, with a scritp with visual basic, what I made. But to be honest, over all, Java Script is great, but too complicated, if you have also to earn money with something else 

Hence my feature request, because, from time to time, I want simply a search over the whole DB and what I do, as quick as possible? Yep, use your App!  Thanks for this!

[sinus]

Hi Markus - can you please give this version of the App a try to see if it makes a difference on your system?

I just tried it on my end and searches are now taking a second or two... but with the buffering and such of lists and ids, it could be red-herring too.  I close/re-opened IMatch and searches that were taking 21 seconds before now open in 2 seconds... that is through a database of 88,000 images.  I do hope this simple change works!!

Probably a good idea to backup the original folder just in case - but I only made a single line change to the app so overwriting should be ok too..  Looking forward to your test results.

Fingers Crossed! - Andy.

uppp, just sent a post .... I am back from a photosession (pumpkins on the field) ... I will try your new code just now! Takes of course some time. 

[Mario]

The ultimate solution for this would probably a button in the FW toolbar to search the entire database instead of the scope, and redirect the results into a Result Window.
This is unfortunately slightly more difficult as it sounds, since many optimizations that result in the search being so fast depend on the scope and background pref-etch and caching.

[sinus]

Hi Markus - can you please give this version of the App a try to see if it makes a difference on your system?

I just tried it on my end and searches are now taking a second or two... but with the buffering and such of lists and ids, it could be red-herring too.  I close/re-opened IMatch and searches that were taking 21 seconds before now open in 2 seconds... that is through a database of 88,000 images.  I do hope this simple change works!!

Probably a good idea to backup the original folder just in case - but I only made a single line change to the app so overwriting should be ok too..  Looking forward to your test results.

Fingers Crossed! - Andy.

Hi Andy, just tried it!
The search alone (only the bottom part) for one word took even longer then before (2.30 min instead your "old" fancy search, 1.50 min).
BUT your part above is interesting.

If I use there only headline or another tag, if finds very quickly, really quickly enough.

If you do not mind a quick idea: If there would be at least one radio button (or something else), what I could click, with e.g. "headline", then this would be great.
(maybe I am able to add then, using your code, add some more "predefined" buttons).

Because usually you search for headline, description, maybe city, keywords and mostly that' it.
Then we could click quickly on such a button and enter the search value.

If we are not sure, we could enter no tag, and fancy search would search all (like now).

This is a great enhencement, thanks a lot. These "find tags" is cool.

If you have time for makes it a bit easier (like I tried to write above), than this would be cool and if not, at least you made it really better.
Thanks.

hmmm, if your app envolves so quickly and good, I have to redraw my feature request. 

[sinus]

The ultimate solution for this would probably a button in the FW toolbar to search the entire database instead of the scope, and redirect the results into a Result Window.
This is unfortunately slightly more difficult as it sounds, since many optimizations that result in the search being so fast depend on the scope and background pref-etch and caching.

Yes, Mario, thanks for your input.
You are right, absolutely.

Would be great, hmm, of course, a simply user like me thinks, must be easy to create. 
But you are the master and you know exactly the problems and so on.

But on the other hand, the app from Andy is really almost the solution for my problem, a bit easier and it woud be great and I could not wish more.
He has enhanced his code already, and it helps me for sure.
A bit easier (like I wrote above, some predefined buttons or so) and this would be the solution for my RF.

[Jingo]

Hi Markus - can you please give this version of the App a try to see if it makes a difference on your system?

I just tried it on my end and searches are now taking a second or two... but with the buffering and such of lists and ids, it could be red-herring too.  I close/re-opened IMatch and searches that were taking 21 seconds before now open in 2 seconds... that is through a database of 88,000 images.  I do hope this simple change works!!

Probably a good idea to backup the original folder just in case - but I only made a single line change to the app so overwriting should be ok too..  Looking forward to your test results.

Fingers Crossed! - Andy.

Hi Andy, just tried it!
The search alone (only the bottom part) for one word took even longer then before (2.30 min instead your "old" fancy search, 1.50 min).
BUT your part above is interesting.

If I use there only headline or another tag, if finds very quickly, really quickly enough.

This is a great enhencement, thanks a lot. These "find tags" is cool.

If you have time for makes it a bit easier (like I tried to write above), than this would be cool and if not, at least you made it really better.
Thanks.

hmmm, if your app envolves so quickly and good, I have to redraw my feature request. 

Hi Markus - ok.. almost there then!

So, I didn't add much to the app other than restricting it to the XMP tag group - the ability to define and narrow tags existed before and should be used if possible to narrow down the fields to search - as you noticed, this makes a HUGE difference in speed.

I think I am going to take the time and add some features to this... what I propose is similar to your idea:

  1 - allow the user to find a tag and create a list of such tags to search for
  2 - allow the user to save the tags and create a "Search List" so future searches can be done with a single button click
 
Stay tuned!

[sinus]

Hi Markus - ok.. almost there then!

So, I didn't add much to the app other than restricting it to the XMP tag group - the ability to define and narrow tags existed before and should be used if possible to narrow down the fields to search - as you noticed, this makes a HUGE difference in speed.

I think I am going to take the time and add some features to this... what I propose is similar to your idea:

  1 - allow the user to find a tag and create a list of such tags to search for
  2 - allow the user to save the tags and create a "Search List" so future searches can be done with a single button click
 
Stay tuned!

Andy, that sounds promising and would be really great!
Thanks a lot in advance, but of course, take your time, and do, what you think, is good.
Your two points looks really like a perfect solution.

[Jingo]

Thx Markus - Almost done.. spent the better part of the day on the APP instead of doing book design work... equally fun though and I'm thoroughly enjoying it!!   

[Jingo]

And.. I'm headed to bed.. but just have one more bug to work out tomorrow am and then I'll share the updated app.. here is a sneak peak of the additional features.

[sinus]

Thx Markus - Almost done.. spent the better part of the day on the APP instead of doing book design work... equally fun though and I'm thoroughly enjoying it!!   

Jingo, I am sorry for you ... but if you also enjoying your work, than it is very good!
I am sure, you do a very good book-design job, but from time to time it is also relaxing and interesting, to do something else.

Your screenshot looks great, really, I am sure, this will work very good. 

[Jingo]

Hi Markus - ok... this is now all set!  So, I made a bunch of modifications so please backup your old copy before using this new one... just in case.

Here are the changes:

1 - The search now restricts itself to the 'XMP' tag groups so it should run much much faster as the search scope is reduced... searches for me are much faster.. but I'd be curious to hear your results.

2 - You can now search on multiple fields across 'XMP' groups.  For example, if you type in 'description' and click the Find Tags button, all XMP tags with description will display.  You can choose 'XMP::dc\description\Description\0' and click the new ADD TAG TO LIST button to add to the search list.  Then, you can search for 'headline', add 'XMP::photoshop\Headline\Headline\0' and click the new button to add it the search list.. etc.  The system will now search all the search list tags you've added rather than just those for a specific single item!   

You can also clear the entire list and/or remove selected list items using the provided buttons in that section.

3 - You can now save and load search tags for easy future searches! Right now this is a one shot... but, I want to add a tag list feature as well so you can have multiple search lists available to choose from.

So.. that's about it.  I tested it pretty well last night and this morning and it seems to be working ok.  Please let me know what you think... and I do hope the speed is improved for you as well!!  If not, I'll need to look into things a bit closer to see what is going on.

Enjoy! - Andy.


The New Screen:

[sinus]

Hey Andy,

First of all: Congratulation, super work!!!
Thanks a lot!

I tried just now you new App, here are my remarks:

1) This first question is hmmm, maybe a curious thing, but does not really matter.
I tried two different times, each time only one search-word (Käse and in the second run hinwil).
I tried the "old" fancy search, what I used (without the possibility to choose tags).
I closed and reopend IMatch each time.

Käse with the old fancy:
found 1426 files and 7493 stacked in 1 min 49 sec

the newest fancy:
found 1424 files and 7450 stacked in 3 min 20 sec

hinwil with the old fancy:
found 135 files and 314 stacked in 1 min 51 sec

the newest fancy:
found 135 files and 314 stacked in 3 min 22 sec

That the new fancy found less files in the Käse-run, makes sense, because you restricted the search, if I understood it correct. Seems to me correct.
Now that the new fancy uses far longer for the search, I have no clue, what this could be.

Means in short, for me here was the old search better, because quicker and found for sure all files.

But now comes the fun part:
I used now tags, what works wonderful, see attachement:

Searched the new fancy with these tags:
Käse
found 1488 files and 7142 stacked in 32 sec!
And, without closing IMatch, found the same and also other new search-terms in roughly 4 sec!!!

I guess, IMatch holds somehow a list or so, no clue.
But this works not with the search over all, it worked here only with the tag-search, but I tested not very long.

hinwil
found 124 files and 267 in 32 sec

After this search I tried
search for "Berikon OR salt"
found 136 files and 286 stacked in 5 sec!

This means for me, Andy, problem solved! You solved it!
Thanks really a lot.

Now, allow me some remarks, what not mean, that you must work on them, only that I mentioned it, and please, take them really only as rough ideas or remarks, I am happy with that, what you have uploaded now! 

- you can see in my attachement, that I cannot see the search-field, I have to scroll down. I could imagine, to put the search-field on top would be a good option (btw: your search in progress - info is very cool).

- I do not know, if this is complicated, if not, it would be nice, that fancy search would remember the last choosen tags. Mostly I think, if we have choosen some tags, we do not change them often, hence they could remain until we want change them again.

- on the left side near "Tags to be searched" is an square-icon, I cannot see, if this is maybe a icon, what is not loaded?

- also no clue, is this complicated: a running time, during the search, would be of course very cool.

- with such a great app and a lot of work, maybe, if you are interested, it would be also nice, somewhere have written your name or so, like "created by Andy ..." I do not know, if this is common or not, but after all, sometimes it is cool to see, who made it.
I can remember, in the past, when I have used a lot of Apps, I did sometimes not know, it this a official App (from Mario) or is it from a user. Hence my "idea".

You made really my day!

[Jingo]

Hey Andy,

First of all: Congratulation, super work!!!
Thanks a lot!

Thanks Markus... app development is fun and relaxing for me.. so always happy to offer up solutions!

To answer some of your questions/feedback:

1 - Yes.. you will always want to narrow down the search to a set of XMP fields as this SUBSTANTIALLY speeds things up.  Otherwise, the system is searching through every XMP field... narrowing down to the fields that are likely to hold the info is much better.

2 - I think designing the app to fit most screens is hard... On my wide monitor, it fits just fine because I have a lot of room available.  I could remove some of the result box lines.. but 20 in each seemed like the sweet spot.  I could move the search info the top.. maybe that makes the most sense?

3 - So.. Fancy search will remember the last used tags if you click the Save Tags button... then, the next you use the routine, just click the Load Tags button... voila!

4 - I didn't include a running time because the search is so very fast now.. :-)  But, I can add one to the screen.. along with a results summary perhaps.

You made really my day!

Thanks Markus... glad I could bring some joy to the world! 

[Mario]

Tip: The Chrome debugger has options to display your app at various mobile/desktop screen sizes and DPI. Always good to give it a quick check there, especially when you work with Bootstrap and responsiveness.

These are the Top screen resolutions used by IMatch users participating in telemetry (based on total users over the past 6 months):

1920   48%
2560   25%
3840   15%
1680   2%
1600   2%
3440   2%
1280   2%

[sinus]

2 - I think designing the app to fit most screens is hard... On my wide monitor, it fits just fine because I have a lot of room available.  I could remove some of the result box lines.. but 20 in each seemed like the sweet spot.  I could move the search info the top.. maybe that makes the most sense?

For me that would be perfect, simply move the search info at the top.

4 - I didn't include a running time because the search is so very fast now.. :-)  But, I can add one to the screen.. along with a results summary perhaps.

If you could do this, would be great. I think, Info is always good.
For example, since Mario added the time for "compact and optimize" this helps me really a lot and give my, btw, also a kind of control.

I think, the same would be for searching, specially in my case with over 300'000 files.

Thanks, and have a good day!

[Mario]

For example, since Mario added the time for "compact and optimize" this helps me really a lot and give my, btw, also a kind of control.

Keep in mind that for databases on SSDs this is rarely needed, only when you remove tens of thousands of files to shrink the database.

[BanjoTom]

Wow!  Andy, the new version of your File Finder is GREAT!  Works very quickly and is quite helpful, particularly when I have to do a search across my entire database (~80,000 files).

But I have a question or two...

1. When I first load IMatch, and use the new "Fancy Search" file finder, it opens right in the App Manager window.  Subsequently when I try to use it again, it only offers me the choice to run it in a web browser or Windows Explorer.  Can't it always open in the App Manager window?

2. I wanted to load a tag to search for "location."  In the Metadata list that tag is shown as "{File.MD.Composite\MWG-Location\Location\0}"  but no such tag shows up when I search for "location" in your file finder.  What am I missing or misunderstanding? 

[Jingo]

Hi Tom - thanks for the kind words.. glad the app is useful!

1 - When you say it only lets you run it in a web browser or explorer on subsequent tries... what do you mean?  Are you closing the app and then trying to load it again from the App Manager?  It should just load a new app window again if it is closed (don't access the drop down overlay.. just click the app button from the app manager to re-open it again).

2 - The reason the File.MD tag isn't shown is because only XMP tags are included... I figured the majority of tag info is stored in corresponding XMP tags so that would be sufficient.  But - if we know certain tags don't migrate to XMP, I'm happy to explore opening up more tag data.. just let me know!

[BanjoTom]

Jingo, I just tried again, and was going to do a video capture of the screens I see, but this time it all worked fine! 

So, I'll play around more, and will contact you again if I have any problems. 

And using the suggested tag for "location"  XMP::iptcCore\Location\Location\0  it works as I thought it would had the search app accepted {File.MD.Composite\MWG-Location\Location\0} instead... 

Again, this is VERY useful for me!  I'll follow along in case you create any further enhancements, but I'm definitely a fan of "Fancy Search"!   

[Jingo]

Thx Tom - glad it is working this time!!

Markus - I just modified the routine again to include a "result bar" that will tell you the number of seconds the search took in addition to the result found... I hope this is a helpful addition!  I also moved the search bar to the top of the screen so this allows you to type in your term without scrolling. 

In essence, once you have setup and saved your "most searched" fields, you can literally perform a database search across those fields with 2 clicks (Load Tags/type search phrase/hit Enter or click Search button).  I'm quite proud of how this enhancement turned out...

So - please download this new version which can safely overwrite the earlier version and replace all the files.  I'll update the original Fancy Search App thread to include this latest zip file to keep things recent in that thread.

Enjoy all!! - Andy.

[jch2103]

@Jingo - Very nice! Thank you! Works well and should be quite useful.

[Jingo]

Thx John... glad the APP is useful!  Enjoy..

[wolboe]

Für mich schon seit Längerem die perfekte  Suchmaschine zum täglichen Gebrauch, jetzt noch schneller.

For me it has been the perfect search engine for daily use for a long time, now even faster

Wolfgang

[Jingo]

Thanks Wolfgang... I greatly appreciate your feedback!

[sinus]

Markus - I just modified the routine again to include a "result bar" that will tell you the number of seconds the search took in addition to the result found... I hope this is a helpful addition!  I also moved the search bar to the top of the screen so this allows you to type in your term without scrolling. 
...
Enjoy all!! - Andy.

You did it, Andy!!! 

Really great stuff, what you created. I have often used your App, but now, it will be my daily search-tool!

Thanks a lot, you made my Feature Request not more necessary, Mario can see it as "solved"! 

And your "result bar" give me the possibility to show you my first results:

First search after start:
Search time was 32.2626 seconds long and we found 32 matching files from searching 305061 files (search: Matterhorn)

Then, please notice, from now on the search is even much quicker. The first run ist always the slowest.
Search time was 4.4207 seconds long and we found 58 matching files from searching 305061 files (search: Thunersee)
Search time was 4.4760 seconds long and we found 15344 matching files from searching 305061 files (search: Zürich)
Search time was 5.8191 seconds long and we found 20314 matching files from searching 305061 files (search: Bern OR Erwin OR Zürich)
Search time was 4.4938 seconds long and we found 3 matching files from searching 305061 files (search: Clara AND Ursula AND Joseph)

Is this not great?!
Means, first search, when IMatch is started, about 30 seconds for over 300'000 files.
And from then on each search roughly 5 seconds.
I used only the tags City, description and headline.

Thanks Andy, as I have seen from other comments, I am not the only one, who uses your great App! 

[Mario]

The ultimate solution for this would probably a button in the FW toolbar to search the entire database instead of the scope, and redirect the results into a Result Window.
This is unfortunately slightly more difficult as it sounds, since many optimizations that result in the search being so fast depend on the scope and background pref-etch and caching.

After posting this abive a couple of days ago I have been thinking about this. How difficult it would be (quite so), how to best integrate it etc.
And then spent yesterday to implement this 

The File Window Search bar now has a new button:



When you click this Global Search button and then perform a search, IMatch searches the entire database and opens a new Result Window for the results.

In the new Result Window you can now use the same button to either perform additional global searches (which replace the contents of the Result Window) or, by disabling the global search button, search within the current result.

All search options available for the File Window search bar are also available for this new feature.
The state of the "initiating" File Window is not changed by running a global search.

I'm currently running tests with this and looking for ways to make the intelligent query cache IMatch uses work efficiently also in this scenario.

The search entire database is the worst-case scenario, performance-wise of course.
Depending on the search settings the user has chosen, IMatch may have to retrieve and search millions, if not hundreds of millions, values.

IMatch uses an internal query cache which keeps the tags and attributes values used for the previous search and scope in a fast-access data structure.
This is why searches following the first search are so much faster. The assumption is that the user searches the same scope, with the same options, for different things.
If the scope changes or the user changes the search bar settings, the query cache needs to be re-created. IMatch does this in the background, and usually the cache is ready before you even start to search 

But this makes sense only up to a certain amount of data, else it would consume far too much resources, without gaining performance.
Searches on the database-level are not (often) suitable for caching, because it takes longer to build the cache and search than it takes to search.
I will test this new feature over the coming weeks and see what I can do.