Filter apply taking over 90 seconds.. am I doing something wrong?

[Jingo]

Hi Mario... just created a brand new database that contains 52,719 images (took about 20 hours to create with standard cache settings).  Ran a DB Compact/optimize afterwards.  DB is 2.12GB (Total size=236.75GB).. cache size (53.52 GB).

I created a filter with the Value Filter: File Extension = .jpg   to try and isolate those from the other files in the DB.

I choose Categories: @All (52719) and apply the filter... it takes 90+ seconds to filter the images and the program hangs during this time.  Is this filter time "normal" and expected behavior for a database of this size or am I not using optimized filter settings?   Is the File Extension not using fields in the DB and instead using the filenames out on my NAS?

If I invert the Filter, the non-JPGs display very fast (about 1000 of them).. within a couple seconds...
If I invert the Filter again, the JPGs display after but take some time again (51813 of them).


Also, should the setting for the Filter "auto" box be maintained between sessions?  If the Filter "Auto" box is not checked when the program is shut down.. but the panel is left open.. when the program restarts, the filter is applied because the panel is open (expected behavior) but the auto button is re-checked (not-desired)..  is there a setting one can set to have the default on the filter be "auto-off" on restart or should the filter panel always be closed upon shutdown?

Log file attached...

Thanks as always...!

[Mario]

20 hours for only 50 KB files? This seems to be extremely slow. I create databases of this size as part of the installation test routines for IMatch, several times a day...
Either you process extremely large files which are extremely slow to load, or your computer is very slow or something is interfering with the process.

But your database is loaded in about 7 seconds, which is normal for a first (cold) load on an average computer. Nothing exceptional here. Total load time until UI is ready: 13s. A bit on the slowish side, but still OK.

Now...


First of all, I think it would be easier to use the File Format filter, which has been designed for this purpose.
You can show this filter in the filter panel by clicking on the gear icon in the filter panel toolbar.
It takes only a second or so to filter out files in a specific format from 60,000 files.

I've made a quick check here is a 60,000 files database to have some numbers.
Showing @All 60,000 files in a file window takes (after disabling the auto pause) 1.2 seconds.

Your system takes 96 seconds (!) for the same (51813 files in your case). This seems awfully long, especially because I cannot see in the log file what exactly is taking so long. There is a long gap of inactivity between 11:58:16 and  11:59:51 where nothing is recorded in the log file. IMatch is apparently waiting for something, probably the database coming back with data - this looks like something was blocking the database perhaps. Virus checker?

I've repeated your test here:

Still having selected @All I setup a value filter for file extension. The file window switches back to pause to I un-pause it.
This loads the 60K files into the filter in less than 2 seconds. I can see all file extensions and how many files there are for each extension.

Now I click on PDF to see only PDF files (124 files in this case). The filter is applied in less than one second, leaving only 124 files from the 60,000 files in the file window.

I have a fairly fast PC so on a normal PC it may take 5 seconds to load the filter, and maybe 3 to apply it when you choose formats. But not 90 seconds, as in your case.

Do you use a custom file window layout perhaps? Something that uses variables or loads a lot of metadata? A complex sort profile?

[Jingo]

Hi Mario.. thx for the info and quick response as usual....!

Files aren't huge.. not sure what it going on.  IMatch DB and preview cache is on an SSD drive local to the PC. 

No virus scanner/firewall on my machine.  Tried with the File Format Filter and it took the same amt of time...

Seems to be an issue locally though because I turned off ALL panels and still see this slowness even just loading all 50k images. not sure what my system is doing but I will try this test on another of my PC's to see if it is related to something on the machine.  Been having some strange display driver issues and think I might have a faulty video card... perhaps that is causing some system issues though my CPU and memory monitoring during the load shows just IMatch doing the lionshare of work..

Will do some more monitoring on my end.. not sure why the software would be non-responsive so long unless there is something also wrong with the SSD...

Thx Mario!

[Mario]

It should just fly on an SSD.

Check Task Manager (Performance tab) and see how high the CPU and disk utilization is. If it takes 2 seconds here, it should not take 90 seconds on your machine - except something is interfering.
If you use a custom file window layout, switch to "Default", just in case.

[Jingo]

Will do some more testing when I'm home this evening... something bad is going on but I'm not sure what... even just trying to load the 50,000 images with no filter takes that 90+ seconds...  already using the defaults but still an issue (all panels closed too).  More details to follow!

[Jingo]

Ok.. so I WAS doing something wrong... I had setup a custom sort profile using XMP::photoshop\DateCreated.... once I switched to Default or Capture Time - we are now looking at 3-5 seconds to load the 50k+ images, filter by filetype, etc....  Thanks Mario.. sorry to bother you!

[Mario]

When you use a sort profile based on metadata, IMatch has to load the metadata for all files in the current scope in order to sort them correctly.

Usually it loads only the metadata required to displays the visible thumbnail panels, plus an extra page. This means that it maybe needs to load metadata for 100 or 200 files. Loading metadata for 50,000 files of course takes a lot longer.

The default "Capture Time" sort profile used data cached in memory. But it's actually the same as the one you have created manually - it's based "File Date and Time" standard IMatch value, which is produced from the EXIF timestamp(s). The same as the photoshop Subject Created timestamp in the metadata.

Your profile requires IMatch to load metadata for 50,000 files, which is slower.
On my computer, loading 60,000 files into the file window with a custom sort profile using XMP::photoshop... DateSubjectCreated takes less than 5 seconds. Not 90. Something seems to thwart your database nevertheless.

[Jingo]

Tx Mario.. I was having a bunch of other issues with my system right around the same time.. for example, my computer would freeze and display red dots all over the screen.  Looking at the event viewer - this was preceded by some driver failures.  In the end, I noticed my case was full of dust.. so I took the PC apart, vacuumed all the components and put it back together... no more issues.   I think the system was overheating... my temps are now in the 60's where before they were close to 100...

Now - that same custom filter on the 50,000 images takes about 4 seconds to complete...

Dust... go figure!   

[Mario]

Dust... go figure!   

I vacuum the filters in my PC (High-end workstation) every two weeks. I picked a case which has proper filters in front of all case and power fans. Takes two minutes but is worth it.

[Jingo]

Yup.. I have a high end case with those filters too... and some VERY large blue LED fans... guess I should take the time to clean them a bit more often!  Sorry to bother you... but thx as always!