Corrupt database including several backups!

[Tallpics]

I have just experienced problems with my Database. Can anyone help?

I am getting this text at the bottom of a diagnostics report:

Checking file history:
      Entries:  1,677,918
Completed.

Checking time line:
      Entries:  3,224
Completed.

Checking Metabase:
Completed.

Checking Cache:
    331,092 files in cache folder.
Completed.

Checking Annotation Objects:
      Containers:  0
Completed.

Checking Visual Query Data:
      Data items:  1,326,832
Completed.

Checking Favorites:
Completed.
    Clearing oid cache.

Analyzing database:
    ERRORS were found in database file structure!
This usually indicates that the database file is physically damaged on disk. Such errors cannot be repaired.
You should RESTORE THE LAST KNOWN WORKING BACKUP of your database!
    '*** IN DATABASE MAIN ***
CORRUPTION DETECTED IN CELL 33 ON PAGE 5027725
FRAGMENTATION OF 11 BYTES REPORTED AS 0 ON PAGE 5027725'

Optimizing Database, rebuilding optimial index structures and query plans:
Completed.

WriteTest:
Completed.

Results:
    Errors:    3
    Warnings:  0

  + + IMPORTANT NOTICE TO USER + +

  If the above error count is greater than 0 you should run the diagnosis again to ensure that all errors have been repaired.
  If errors remain in the database you should definitely restore your last known backup of the database file!


IMatch database diagnosis logfile closed: 26/02/2015 16:33:48 (00:17.17)

I make regular Diagnostics of my database before backing up on a regular basis. All my backups reported no problems before each backup.

However I have just had to restore five separate backups (going back about one month) to find the first database that didn't report that it was damaged. Even though they were OK at time of backup!

It now seems I have lost a lot of categorising (the winter is when I put a lot of hours into this).

Whilst this confirms the need for regular backing up does anyone have any idea what might have gone wrong?

Also, am I dreaming when I hope that the original database could be repaired? It seems that the error is just down to a tiny number of bytes! It seems so simple

CORRUPTION DETECTED IN CELL 33 ON PAGE 5027725
FRAGMENTATION OF 11 BYTES REPORTED AS 0 ON PAGE 5027725

I have run a CHKDSK just in case but that shows no errors.

Are there any other processes/repairs that I could try?

Thanks

[Mario]

Which version of IMatch are you using?
Recent versions of IMatch perform more rigorous checks on the database, revealing potential damage.
This error comes straight from the database system which found invalid data somewhere in the file.
Is this database stored on a local disk?
Any problems in the past? Power failure hard Windows shut-down?

You can contact me via email and I will send you a tool which maybe allows you to export your database and then create a new one. This sometimes works when the database damage is not in a sensitive area, or only in an area used by the index data - which are re-generated anyway.

Before you do this, please run a Compact & Optimize on the database. This copies the entire database file and re-creates the internal structures. If the error is in an area not needed, this will repair the damage.

[Aubrey]

Have you recently installed the latest version of Imatch?

I have had database errors when I loaded a new version (I cannot be sure of which version now), I loaded a backup and still had issues... I reinstalled previous version of Imatch performed a check on database and then saved database. Reinstalled newer version loaded database and everything was fine!
I did not report it at the time as I did not see how I could reproduce it. Everything is now fine I regularly make pack and go backups and move database between desktop and laptop without issues.

Aubrey.

[Tallpics]

Thanks for the quick response Mario.

I am using v5.3.6 of IMatch. I always update after each new version has been released for a couple of days.

The database is stored local on a SSD.

I have had occasional database corruptions but haven't been able to isolate/reproduce the cause. In the past the backups have always come to the rescue

I have tried a Compact & Optimize on the database however this reports:

Operation Failed. Most likely there was not enough disk space available on the disk containing the database.

A check on the database disk states that 78GB is free, which seems as though it should be enough?

I have attached the Log File created after the failure to Optimize and I will e-mail you with a request for the export tool.

Would it be 'good practice' to try Aubrey's suggestion of re-installing an older version of IMatch?

Thanks

[attachment deleted by admin]

[Mario]

If the Optimize fails, the database is truly corrupt. The database system aborts the operation because it runs into an unrecoverable error while copying data from the old database file to the new database file.

I don't see why an older version can magically repair a database file with physical damage. Damaged is damaged. Maybe the error is just suppressed or not detected by the older database kernel. Or maybe this causes other issues which then show up in several months time, and nobody knows where the problem stems from.

In older versions of IMatch, the database may have overlooked a certain kind of corruption in the file for a long time. No IMatch performs more rigorous checks and also remembers if the database returns an error during an operation (IMatch may perform thousands of database operations per second, from 2 or 10 separate processing threads). When the database returns "physical damage" detected, IMatch sets a flag in the database so it won't forget this, and then informs the user at the earliest opportunity.

I cannot really test this 'live' because I never had a damaged database myself. I use a robust Dell workstations and I still use databases created byearly Beta versions of IMatch 5 for my test cycles. I need to forcefully corrupt databases by pulling out USB sticks containing databases in full run, and then deleting the rollback log files so the database system cannot undo the open transactions. My other process to damage a database is to open it in a binary editor and to overwrite some random areas with nonsense data. It's really hard to break the databases, which is why the same database system is used by Apple, Adobe, Spotify, Airbus and other companies.

[Tallpics]

Thank you Mario for the extra detailed information.

It is clear to me that you consider the 'safety' of our databases as your highest priority. This is illustrated by your use of an 'industry standard' database and your fast response to any problems.

I certainly don't put any blame on you or IMatch. You have to deal with so many different systems.

I am coming to the opinion that my problems are somehow specific to my setup.

The frustration is not knowing where my workflow/system is causing an error. Of course you cannot be expected to sort that out!

I do take the utmost care to keep a healthy system. I regularly use PC Tools to run low level disk/memory/processor stress tests and never seem to have an issue.

I am also the sort of guy who will sit and wait for processes to finish... Even closing all other programs to minimise any stress.

My backup routine includes daily backups (several each day when heavy work is on-going). These backups 'rotate' every seven days. All of them are on separate physical disks, two local, three exterior, two out of the house. I then also keep two week and monthly copies! Oh, I also have the old v3.6 database just in case

This backup system has always worked although this time only just as I have had to go back to the monthly copy!

I assume that it is very unwise to continue with a database that reports problems even though it is/seems to be working perfectly.... it will 'bite me' one day into the future.

Maybe I can still try the export utility that you offered.

Thank you for all your good work.

[Mario]

The database company supplements diagnostic routines which I use in the IMatch diagnosis. If these routines find no errors, the database is supposed to be in top shape. In your case, the routines found mismatched/corrupted data and IMatch reported that. I have no idea why previous diagnosis runs did not reveal that before you did the backups.