Imatch performance

[MichaelR]

Imatch performance

We are having problems when we want to make a search in our database. Maybe there are ideas on how to solve this?

Database:  450.000 images
Imatch database imd5  20GB

PC
Windows Pro 64 bit
Intel Core i7 - 8700K CPU 3.70 GHz
Installed RAM 16GB
Samsung SSD 970 EVO

A search in Media & Folders results in: photools.com Imatch Application is not responding.

When we open the Categories view (what takes a very long time) we sometimes can use the search function. We use regular expression. When it works it's really fast

We did run a database diagnostics and the message is all is fine.

[Mario]

450,000 files is quite a large database.
Which version of IMatch do you use?

Search in the Media & Folders View" means what, exactly?
I assume you use the File Window search bar? How large is the scope? Do you search in a folder, several folders or the entire database.
What search settings do you use for the Search Bar?

Usually the first search in a given scope takes a bit longer, while Match is caching the data to search. Subsequent searches are then much faster. Except when you change the scope again, which requires a reload of the index.

On my 4 year old system IMatch searches 500,000 files in less than 2 seconds (2nd and subsequent searches). The first search takes 8 seconds, which means building the index takes 6 seconds for 500,000 files.

Please produce a log file in debug mode from an IMatch session where you loaded your database, performed some searches. ZIP and attach.
This will show us exactly what you are doing, what takes how long and how your system performs.

See log file for details.

For a database that size I usually recommend to remove all non-used data-driven categories etc.

[MichaelR]

We use Imatch 2019.2.3 (64 bit)
We rebooted the PC, we did run database diagnostics and compact and optimize. All is fine again and a search on the full database (450.000 images) takes 6 - 8 seconds.

To open the database about 20 minutes is needed.

The system has at the moment 16GB RAM. Would it be of use for Imatch to add more RAM? 32, 64 or more?

We expect to add another 500.000 images to this database.

We would like to install a second database for images not ready for publication. It will be used to make searches and will also hold one million images. My question is where do we place this database? As a second database in the same application? Can the two databases be open simultaneously? (I mean to avoid the 20 minutes start up time)

Or do we need to install the second database on a second machine?

[Mario]

To open the database about 20 minutes is needed.

This is ridiculous. My 600,000 files test database loads in 22 seconds (!) from a SSD installed on my local PC.
Maybe your virus checker is badly interfering with IMatch? Did you exclude the folder (!) containing the IMatch database and all files within from your on-access virus check?

You did not answer all of my questions (see above).
For example, it is important to know where your search and which options you use.
You did not attach a log file in debug mode so I cannot even start to check what takes so long on your system.

1 million files is a huge database. Many stock photo agencies have less files to manage.
If you really need to manage that many files in one database, make sure to not further reduce performance by keeping many unneeded data-driven categories etc.

16 MB RAM is what is considered minimum these days. But IMatch does not really need much RAM (a 600,000 files database needs less than 1 GB RAM).
Unless you run many memory-hungry applications there should be no problem (see Windows Task Manager to learn how much of your RAM is used by which applications).

You can create any number of databases. Always store them on your fastest disk (SSD preferred).
You cannot open two databases at the same time.

[MichaelR]

Hi Mario,

See answers between your text.
I assume you use the File Window search bar? Yes
How large is the scope?  Entire database, about 450.000 files
Search setting:
Description, Title, Headline
Keywords
Enable Regular Expressions

Imatch log file attached

Friendly greetings,

Michael

[Mario]

Let's see:

You are searching 460,000 files.
You are searching 4 metadata tags, which means that IMatch has to load and search about 1.9 million metadata fields for each search.
You are using the slowest possible search mode (regular expression).

This explains that the search is a bit slow. On my system it takes several seconds for each search (a few seconds more for the first search, when IMatch is building the search index into memory).

There is no free lunch.
Searching about 2 million text values with regular expressions will take time. The faster your disk is that the faster your processor is, the better.
If you don't need regular expressions or you can reduce the number of tags to search you can reduce the search time considerably.

The load time for the database is abysmal, with  1400 seconds, about 23 minutes. Yikes!
The typical load time for a 500,000 files database from a SSD is 20 to 30 seconds. Maybe a 50 seconds from a normal fast hard disk.

What kind of system do you use
Notebook perhaps with a slow spinning disk?
A database that size should definitely go on a SSD.

Did you disable your virus checker for the folder (!) containing the IMatch database and all files within?
Such super-slow operation times are often caused by virus checkers interfering badly with IMatch's file I/O...

[MichaelR]

As mentioned before this is the system: 

Desktop PC
Windows Pro 64 bit
Intel Core i7 - 8700K CPU 3.70 GHz
Installed RAM 16GB
Samsung SSD 970 EVO


The load time for the database is abysmal, with  1400 seconds, about 23 minutes. Yikes! This is the issue...  Also when we go from the Media & Folders tab to the Categories tab, it takes maybe between 30 minutes and one hour
The image search takes maybe 6 - 8 seconds and we are happy with this

We will check the virus scanner

[MichaelR]

We disabled Bitdefender and the issue is solved.

Thanks for helping us out.

Friendly greetings,

Michael

[Mario]

Very good 

But please do NOT disable your virus checker 

Just configure an exception for IMatch (remember to update this after each IMatch update, just in case).
Also make sure the on-access scanner of your antivirus leaves the folder containing the database alone. This is very important because IMatch and the database system create and delete temporary files in that folder.

This is also mentioned as important in the The IMatch Database topic in the Help System, which also offers other important tips.

[Michael2019]

Hi Mario,

We have the same issue again and removed Bitdefender. This doesn't help and it still takes one hour to open a database. In the attchment you will find a debug file. Maybe this will be of help.

Thanks and friendly greetings,

Michael

[Mario]

Hi. Michael

I have replied to your email.

I think the problem is that your database has almost 500,000 files (which is OK) but it also has 630,000 (!) categories, and this is way out of spec.
Not even scientific users with large taxonomies or stock photo agencies need more than 50,000 to 100,000 categories.