performance question: too much metadata?

[kkiel]

Hi everybody,

first of all congratulations for IMatch5 alpha! As solid and reliable as ever. I've been looking forward to this for months!

I happily managed to transfer my large database (as imd5 it has 1,6 GB, ca. 65.000 images plus standard iptc metadata + about 30 attribute entries per image) to IMatch5. I deaktivated all immediate metadata actions. After building the index it is still partially very slow: search (everywhere) takes several minutes (!) each time; marking two or more images for batch keywording takes several minutes, too (sic!), until the machine has read their metadata.
Did I overlook some presetting? I think this is  not an IMatch5 performance issue but must be somewhere in my settings? Or do I have too much metadata?
Most of my relevant meta information is in the attributes (english and french translations of captions and keywords, workflow information, sales history etc.) because IPTC doesn't provide a space for them or else I don't want that kind of data in the images.

Thanks for answering,

kkiel

[jch2103]

Be sure to run a database Compact if you haven't already.

Database hardware can make a difference; what's your computer configuration (especially storage)?

[Mario]

A database with 65K files is still quite small. The largest IMatch 5 database I'm aware of has about 500,000 files.

You did not add a log file so I cannot say anything about the performance of your database (see How to report bugs for more info). I cannot even say if your database is on a local disk or a network, which can mean a 1000% performance difference.

I suggest you first run a database diagnosis (Database menu > Tools) and also a Compact (Database menu > Tools > Compact). IMatch should have prompted you to run a compact when you have added 65,000 files.

[kkiel]

Ok, I'll do the compact first. Database and Images (460 GB) are on a usb3 harddrive, running on a thinkpad laptop e330 with 4 GB RAM, 64 bit windows 8 system. This pc will have to do for the moment ...
Thank you very much for your fast answers!

[kkiel]

three compacts failed. I attach a logfile.
There is enough space on the harddrive (650 GB).

[attachment deleted by admin]

[kkiel]

Compact failed again after restart, with a hint that there is not enough space on the disk where the database is stored. This cannot be ...

[Mario]

1. IMatch needs as much as 2 times the size of the database file temporarily during the compact.

2. Your log file is full of low memory error messages. Your system memory (RAM) is always utilized to 90%, which means that Windows has to swap pages from memory to disk all the time. A real performance killer (trashing). Please check which applications you run and how much memory they consume in Windows Task Manager. If you run an Adobe product like Photoshop, try to reduce it's memory allowance.

If there is not enough disk space on your SSD to compact the database, do this:

0. Close the database in IMatch
1. Move the database to your regular hard disk.
2. Open it from there via Database > Open
3. Run the compact
4. Close the database
5. Move it back to replace the one on your SSD.

[kkiel]

Hi Mario,

optimizing and compacting worked after database diagnosis, search is much faster now. I'll try your tips next!

Thanks a lot!

kkiel

[Mario]

Did the diagnosis report warnings or other problems?

[kkiel]

Hi,

after following your tips (esp. putting the db on the internal harddrive) search has immensely sped up.  Database diagnostics runs without errors.
I can't successfully run a compact, though: I deleted several autostart entries and cut photoshop's RAM drastically back, but I still get the message that there is not enough space on the disk where the database is (now the harddrive). I attach the logfile again.
After unsuccessful compacting IMatch5 launches the database immediately without problems and some minutes later there is a message, that the optimization process was successful.

Thank you for your help!

kkiel





[attachment deleted by admin]

[Mario]

This log file is not in debug mode (Help > Support) so it contains only minimal info. Please switch and retry.

It also does not show a run of the Compact operation. Is this log file from a session where you used Database > Tools > Compact & Optimize?

Your system memory is still used up to over 83% with 0 bytes physical memory free. Your system needs more RAM or you need to close some applications. IMatch needs only 300 MB, but still Windows will need to swap memory in and out which costs performance. May even be the problem why the compact does not work because the database system has not enough ram.

[Richard]

This thread brings a question to my mind. Much has been said about using a high speed USB 3 stick as a drive for the database. Some have also used memory as a drive. What I am now wondering is if Windows can be configured to use a USB 3 stick as additional RAM? Or at a minimum, to use the USB 3 stick to swap pages to and from memory.

[Mario]

I think Windows does not allow the page file to be on a removable media, for good reasons (the media failing or being removed would cause a BSOD immediately).

A typical new PC these days comes with 8 or even 16 GB RAM. RAM is very cheap.
Some users create RAM disks like in the old days to speed up their system. Some say it makes a difference, some days it's better to leave the RAM to Windows which will use it to create a very large file system cache anyway.

Even a rather small 128 GB SSD to hold Windows, the page file and some frequently used and performance-critical will boost every system massively.

I know that my next computer (it's about time) will have a mix of a SSD (or SSD RAID mirror for safety) and normal disks for mass data storage. And 16 GB RAM :-)

[kkiel]

I'm gonna get more RAM, too, but search speed is ok at the moment.
Another observation: I get the impression that IMatch does the compact correctly (thus no  log file in debug mode) but complains about missing space anyway ... I start the compact, IMatch complains, but generates no downloadable logfile, then the program starts fine without problems. Is that possible? Diagnostics show no remaining problem either.

[Mario]

When IMatch compacts a database it writes corresponding entries into the log file, even when it just starts the compact.
In the log file you had attached there is no such entry.
Please switch to debug logging (Help > Support > Debug Logging) and run Database > Tools > Compact & Optimize. Then attach the log file resulting from that session. This may help finding the reason for this.

[kkiel]

Ok, I'll attach one. I had expected that IMatch prompts me to download the log file (I think I've seen that before?)


[attachment deleted by admin]

[Mario]

Thanks for the log file. IMatch offers a DUMP file when it crashes. The log file is always there. See the help on logfile respective dump file for details.

I see that the Compact operation returns after less than two seconds with an error. No specific error messages from the database system are logged because this has never failed before. I have added more log info for the next built. Please retry when version 5.1.6 is available.

I'll move this thread into the General forum because it does not give tips to other users.

[kkiel]

Thank you very much! I'm happy with the faster search anyway.