Database Diagnostics & Compact/Optimize Revisited

[Darius1968]

Right now, my database has a total of about 88,000 files.  This post, I know, is a little repetitive of another one in which I ran the database diagnostics and it didn't complete successfully.  This time, it did, however.  My concern, though, is that the time it took for the process to complete (in the neighborhood of 45 - 50 min.) may be too much, as from what I gather from other posts on the forum, others get much shorter completion times. 
Some time ago, I tried to unsuccessfully attach my log file to this forum, but file size exceeded limits.  However, this time, I made use of wetransfer.com that has allowed me to make available the log file "im5diag (20130809).txt" to anyone on this forum who cares to look at it.  This log file, I have to say, was generated in the normal running of IMatch, not in any debug mode.  If that makes a difference, then I'll be happy to regenerate the log file in the proper debug mode, after being told how to do it.  Thanks!

Here's the link to follow to get the file "im5diag (20130809).txt":
https://www.wetransfer.com/downloads/054c509e60b3466698c5e5f18b4b750520130809233236/4b881eda319053d2000075b23abcfaa620130809233236/61a598

[Richard]

I do not see even an hour of database diagnostics as being excessive for 88,000 files. On just 3,846 files I have had times from a few minutes to nearly and hour. The longer times were when I was having problems and it is likely that things got messed up and had to be fixed. Metabase and Optimizing take the most time for my databases. It often seems that nothing is happening. If I just leave it alone it always finishes. Once it does finish there is a log you can view. You can also save that log in case Mario needs it.

I would recommend that everyone doing Beta testing set IMatch to debug mode. To do so, click on Help > Support > Debug logging.

[Mario]

1. Your log file is zipped only 256 KB (!). You can upload files up to 1 MB so if you just ZIP the log file you don't need to use external services.

2. You can also send me files up to 10 MB per email. Make sure you include a link to the community post you are referring to, and your community user name.

3. For even larger files, e.g. DUMP files produced when IMatch crashes, I maintain a FTP server. You can contact me to get a user name and a password if you need to upload very large files.

You've sent a normal diagnosis log file. Do you also have the standard IMatch log file from that session? Because the application log file contains detailed timing information which tells me which phase of the diagnosis takes how long. From the diagnosis log I can see that the entire process took about 58 minutes.

Tip: To run the diagnosis without Optimize hold down <Ctrl> when you press the Start button.

When IMatch optimizes/compacts the database (during the diagnosis or when you run a Database > Tools > Compact) it basically copies the database file to a new file. During this copy operation it removes slack data, rearranges and streamlines index data an suchlike. This process almost entirely depends on the performance of your hard disk, how much memory is available for the Windows disk cache and the size of the database of course. Virus checkers or other software which interferes disk I/O can have a real bad impact on the performance of this step.

Note: Optimize is usually only required when you have added/removed thousands of files (maybe 10,000 files or so). It can restore performance back to optimal levels. But under normal conditions (e.g. you just add a few hundred photos per day) running Optimize separately is not needed. Just let it run when you run the Database diagnosis (weekly).

[Fellwalker]

In iMatch 3, diagnosis and optimise only used to take a maximum of 10 minutes. In the new iMatch 5, on virtually the same files (a few hundred added since the move) it just took an hour.

Please could you give a thought to changing the operation to do the diagnosis without optimise as standard, and maybe a tick option to include optimise. Perhaps it is not even necessary in the diagnosis menu, given that immediately below it is compact and optimise?

[Mario]

How large is that database?
Can I see the application log file and the diagnosis log file?

The diagnosis routines in IMatch 5 are much more throughout than what we had in IMatch 3. Plus, IMatch carries a lot more payload in the database.

I just started a diagnosis on a 120,000 files database while I write this - to get a feeling for the problem.

You can skip the optimize by holding down <Ctrl> when starting the diagnosis in the dialog. I think this is documented in the help as well.

The optimize step is important, though. It not only speeds up the database, but also copies the database file to a new file. Removing unneeded slack data, de-fragmenting the file. But it also touches every byte in the database during that step. This is the only way to detect silent errors in seldom used areas of the database, or read/write problems caused be temporary and unnoticed problems with the disk, file system or network.

Such errors can go unnoticed for a long time, until the database sometimes needs to write to or read from such a damaged section - and then it's too late to recover a working backup usually.

Wait...Wait...

OK, diagnosis and optimization completed.

120.000 files database on an USB stick (my internal disks are pretty old and slow).

Total time: 14:32.

That's about as fast as it can get, except maybe if the database is on an internal SSD with high-speed controller.
Once a week 15 minutes for the diagnosis is OK I think.

[Darius1968]

Okay, I've attached both log files at your request.  The database is about 90,000 + files.    This is after the database optimization, which completed in about a little over an hour. 

[attachment deleted by admin]