Database compacting, is it working?

[JohnZeman]

As I recall a few months back Mario found a bug that prevented a database from being properly compacted and I think he fixed it.  Lately my database has been running around 7.3GB in size and I generally compact it daily.  Last night due to some self-inflicted problems I decided to start all over with a brand new database.  I loaded it with the exact same images, attributes, categories, everything, and the result is the new database is 1.3gb smaller in file size than the original.  Furthermore lately when I do a database optimize/compact operation I'm not seeing any database file size reduction at all.

I'm not calling this a bug because I'm not sure that it is, I'm interested in finding out if others are seeing the same thing I am.

[Richard]

Hi John,

Keep in mind how space is allocated in blocks for a database. That may account for some of what you are seeing. If the amount of space reserved for a database becomes excessive, when is the extra space removed?  I also question the true meaning of "compact". Writing files in a more logical order will not make the files smaller. What gets compacted? Wasted space? When I de-fragment a drive, efficiency is improved but only because the files are in a more logical order. Nothing gets much smaller.

[jch2103]

All true,  but deleting a large number of records from a database can leave empty space in the database file.  Compacting in this situation should reduce the size of the database file.

[Mario]

The compact which is part of the Diagnosis or the stand-alone Compact under Database > Tools > Compact...?

[Richard]

Compacting in this situation should reduce the size of the database file.

Let's say that I have a 10 GB allocation but only 9 GB is used. Now I remove files from the database. At what point should the amount of space allocated be reduced?

[JohnZeman]

The compact which is part of the Diagnosis or the stand-alone Compact under Database > Tools > Compact...?

Mario I always the compact which is part of Diagnostics.

[Mario]

Try the Database > Tools > Compact.

[DigPeter]

I see that the word is "Optimising" only for diagnostics, but "Compact & optimise" in Database tools.  When I do the latter I get a significant reduction in Db size, but not with diagnostics.

[sinus]

... Lately my database has been running around 7.3GB in size ...

OT: John, beside your thread, I really wonder, if it would be wiser to cut such a huge DB into two pieces. My DB with 3.6 is about the same size, but for example to click on a category, what has a lot of images, it takes time. When IM5 is out, I will have to decide this. But of course... to have all images in one DB has advantages. Sorry, I do not want to "hack" your thread, wanted only to say this, think, it is not worth to create a new thread.

[Mario]

The Optimize and Analysis step in the diagnosis no longer compacts the database.
This was superfluous and did not help in analyzing and checking the database.

I have documented this now  more explicitly. If you want to shrink your database, use the Database > Tools > Compact command.


@sinus:

Please define "it takes time" and let us know if the category you talk about has 5,000  or 50,000 files.
And a log file from a session where you did that will help.

Usually loading a file window is very fast, but I think I remember you are using custom file window layouts with variables and metadata, and this can take a long time to load. The standard data is usually cached in memory. But if you use variables and metadata in your file window layout, IMatch has to load the data from the database file on disk, and this can cause a 100% to 1000% slow-down.

Or when the category you want to look at is "dirty" and needs to be re-calculated before the file can load the files.

Many factors play a role here. I can load a 20,000 files file window in about 4 seconds for a 120,000 files database... how long does it take on your system?

[JohnZeman]

OT: John, beside your thread, I really wonder, if it would be wiser to cut such a huge DB into two pieces. My DB with 3.6 is about the same size, but for example to click on a category, what has a lot of images, it takes time. When IM5 is out, I will have to decide this. But of course... to have all images in one DB has advantages. Sorry, I do not want to "hack" your thread, wanted only to say this, think, it is not worth to create a new thread.

Markus I really haven't had many speed problems with IM5, it seems quite a bit quicker than version 3 for my jpg photos.

[JohnZeman]

The Optimize and Analysis step in the diagnosis no longer compacts the database.
This was superfluous and did not help in analyzing and checking the database.

I have documented this now  more explicitly. If you want to shrink your database, use the Database > Tools > Compact command.

Thanks Mario.  The Database Tools > Compact and Optimize seems to work as it should.  It was the compact and optimize operation that had been part of the diagnosis that didn't seem to do anything.