IM5 Diagnostic

[Gerd]

Dear Mario,

due to my "Challenge of the year, IM3 -> IM5", today mornig IM5 has finished reading-in Metadata and Updating Files ...

After that, I did an "Optimize" of my database ... no problems, only that it takes again 2,5 hours ... I have learned, to be patient, if I work with IM5 

Now I want to make a Diagnostic-Check, that's fine and I got no error, but at the end, the last point is also "Optimizing" ... no! ... not again 2,5 hours waiting for what I have just done! ... so I clicked on Abort ... and now again waiting ... since roundabout 30 min, that it will finish to abort!

My wish is, that you please complete remove this function from the Diagnostic (it is not a diagnost-function!)  or make it selectable via a check-box.

Regards
Gerd

P.S. since 13:00 IM5 ist still busy with 1% till 3% CPU-time to finish abort ... shell I wait or can I do some other trick to finish this?

[Mario]

When you hold down <Ctrl> when you press Start, the Optimize step is skipped.

The Optimize is a built-in function in the database system. It depends completely on the performance of the disk. Just made a test, My 100,000 file sample database optimizes in about 20 minutes, on a regular RAID hard disk.

Do you have perhaps a virus checker running or something that interferes with disk operations?

[Gerd]

Hi Mario,

I saved at different steps the log-file. Can you see there something, why ist is so slow? At moment I wait till the optimizing-process has finished, due to the fact, that I closed earlier today via Windows-application-stop. It's now running since 15:30 and still not finished.

But again, back to my proposal: a diagnose is a diagnos and show only where it os good or bad. Optimizing can be an action after the diagnose, but normaly not part of the diagnose. If you want to keep it there, please put the text, how to skip it, into the diagnose-window.

Regards
Gerd

[Richard]

please put the text, how to skip it, into the diagnose-window

+1

[Mario]

Optimizing can be an action after the diagnose, but normaly not part of the diagnose.

The compact/optimize step is part of the diagnosis because it accesses every part of the database while producing a new, compacted database as output. It is very likely that any physical damage in the database is detected by this step, including damage in spare ranges of the file, index structures, unused segments and the like. I consider this an important step because the other analysis steps will not reveal such problems.

I documented the <Ctrl> trick in the help, but I actually want that users let the diagnosis run with the final optimize phase.

[Richard]

My experience is that the compact/optimize step can finish rather quickly if the database is in good shape. When it takes a long time, it likely means it was needed. Right?

[Mario]

My experience is that the compact/optimize step can finish rather quickly if the database is in good shape. When it takes a long time, it likely means it was needed. Right?

Yes. If you run optimize two times in a row you can directly see the effect. The file system cache also plays an important role. It's like loading a database two times in a row. The first one will always be slower because the second time Windows will have most of the file in the cache already.

In my experience if the Compact is very slow, the database is very fragmented on the hard disk. The Compact collects all these spread fragments and writes a new file, which allows the file system to arrange the parts of the database nearby. It's 97% hard disk and only 2% CPU so the speed of the disk is the main factor. It usually just flies on a SSD or even a USB 3.0 Stick  but performs more or less fast on regular disks.

[Gerd]

I let now run a defrag-program.

My IM5-database has now a size of 8,3 GB, is there a most optimal size for the IM5-cache?

Regards
Gerd

[Mario]

I let now run a defrag-program.

Windows 7 and later usually automatically run a defrag when the system is idle. This is usually sufficient to prevent excessive fragmentation. When you type defrag in the START menu and then Disk Defragmenter Windows will tell you how much fragmentation there is on your disks.

The other fragmentation (the more important one) is inside the database itself. Especially when you add thousands of even tens of thousands of files, IMatch has to add/write/delete millions of data records, growing the database all the time. This can lead to internal fragmentation and non-optimal structures. The Compact takes care of that.

8.3 GB is quite a large database already.
I have a 100,000 files database here which has only 3.5 GB.

How many files do you have in that database?
What thumbnail size do you use (unless you changed it, it will be 300 pixel).
Which file formats are you using? I guess that a good part of the database is filled with metadata.


I don't understand your question about the cache. You cannot configure a cache for the database...

[sinus]

From my point of view this was also a the case in 3x - this task took very, very long.

Here in IM5 we have also the points

- Database - Database Tools
-Database Diagnostics
- Compact and Otimize

Specially in the first point, like Gerd pointed out, the user has no clue, how long would it take (this was, why I made this in 3x very seldom  )

Would it not be possible, that you could get the user some ROUGHLY information, before he push the "healing button"?
If this is not possible, maybe it would be possible in a second step?

I mean, I push the button, and then IM5 takes some secondes, and it uses this time to check how long this will take. And then IM5 would informe the user, how long it would roughly take.

For a user, I think, it is a difference, if IM5 takes 3 minutes or 30 minutes. In the first case a lot of users would push the button, in the second case, users would decide to push the button later - in the coffee-time 

BTW: MAYBE you could add a button "close IM5 after finnishing and shut down the computer"? In this case I could push the button and go at home ...

[Mario]

1. There is no estimate possible. The optimize is a atomic operation of the database system, and it does not provide any estimate.

2. Shutting-down the computer requires administrative rights. I don't want to add all the elevate and ask user for Admin password stuff into IMatch just for this purpose.

[DigPeter]

Windows 7 and later usually automatically run a defrag when the system is idle. This is usually sufficient to prevent excessive fragmentation. When you type defrag in the START menu and then Disk Defragmenter Windows will tell you how much fragmentation there is on your disks.

I did not know this.  Is there some set-up option in Windows 7 that activates an auto defrag?

[sinus]

1. There is no estimate possible. The optimize is a atomic operation of the database system, and it does not provide any estimate.

2. Shutting-down the computer requires administrative rights. I don't want to add all the elevate and ask user for Admin password stuff into IMatch just for this purpose.

I thought something like this. For me personally this is no problem. I did and will do let run this task, when I know, I have something else to do. Coffee break or simply, ahem, working. 

[herman]

I did not know this.  Is there some set-up option in Windows 7 that activates an auto defrag?

In W7:
Windows Start button - Computer - Right click the drive you want to defrag - Properties - Tools tab - Defragment Now button.
There you can set options, schedule etc.

For SSD users: make sure defrag is set to OFF for the SSD.
Windows is supposed to do that automatically, but for some reason it sometimes fails to do so.

Hope this helps,

[Mario]

I did not know this.  Is there some set-up option in Windows 7 that activates an auto defrag?

START menu, search for defrag. Open Disk Defragmenter. There look for the Scheduled Defragmentation. It's usually on by default.



[attachment deleted by admin]

[DigPeter]

Thanks Mario

[Richard]

For SSD users: make sure defrag is set to OFF for the SSD.
Windows is supposed to do that automatically, but for some reason it sometimes fails to do so.

I have read that running defrag on an SSD can ruin the SSD. At the least it does no good so Herman's advice should be heeded.

[DigPeter]

I did not know this.  Is there some set-up option in Windows 7 that activates an auto defrag?

In W7:
Windows Start button - Computer - Right click the drive you want to defrag - Properties - Tools tab - Defragment Now button.
There you can set options, schedule etc.

For SSD users: make sure defrag is set to OFF for the SSD.
Windows is supposed to do that automatically, but for some reason it sometimes fails to do so.

Hope this helps,

Thanks for the advice Herman