Database Error

[RainerG]

System Win7 64Bit
IMatch 5.1.14

After a couple of hard db-errors i take an old backup file.
After restore i made an db-check(diagnostic). The check runs without errors.
Then i run an optimizing (after the backup i had deleted pics)
After the optimizing i run the db-check. The check runs without errors.

And then after a few seconds IMatch sad there is an hard db-error!!!

(i have nothing do in imatch!!)
I start the db-check. The db-check reports errors!

This usually indicates that the database file is physically damaged on disk. Such errors cannot be repaired.

OK! Then take again the old backup and save it on another harddisk.

I start IMatch. I immediately made an db-check. No errors!
Then i start compress and optimizing (beacause the deleted pics)
I start the db-check and an error occurs:

Analyzing database:
    ERRORS were found in database file structure!
This usually indicates that the database file is physically damaged on disk. Such errors cannot be repaired.
You should RESTORE THE LAST KNOWN WORKING BACKUP of your database!    '*** IN DATABASE MAIN ***
FRAGMENTATION OF 20 BYTES REPORTED AS 0 ON PAGE 310854'
    'row 5367929 missing from index idx_md_tag_data_oid_tag_oid'
    'row 5367930 missing from index idx_md_tag_data_oid_tag_oid'
    'row 5367931 missing from index idx_md_tag_data_oid_tag_oid'

Optimizing Database, rebuilding optimial index structures and query plans:
Completed.

WriteTest:
Completed.

Results:
    Errors:    4
    Warnings:  0

  + + IMPORTANT NOTICE TO USER + +

  If the above error count is greater than 0 you should run the diagnosis again to ensure that all errors have been repaired.
  If errors remain in the database you should definitely restore your last known backup of the database file!


IMatch database diagnosis logfile closed: 25.08.2014 12:36:54 (00:01.57)

I made a next db-check and then come other errors:

Analyzing database:
    ERRORS were found in database file structure!
This usually indicates that the database file is physically damaged on disk. Such errors cannot be repaired.
You should RESTORE THE LAST KNOWN WORKING BACKUP of your database!    '*** IN DATABASE MAIN ***
FRAGMENTATION OF 18 BYTES REPORTED AS 0 ON PAGE 310854'
    'row 3809426 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809427 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809428 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809429 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809430 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809431 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809432 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809433 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809434 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809435 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809436 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809437 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809438 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809439 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809440 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809441 missing from index idx_md_tag_data_oid_tag_oid'
    'row 3809442 missing from index idx_md_tag_data_oid_tag_oid'

Optimizing Database, rebuilding optimial index structures and query plans:
Completed.

WriteTest:
Completed.

Results:
    Errors:    18
    Warnings:  0

  + + IMPORTANT NOTICE TO USER + +

  If the above error count is greater than 0 you should run the diagnosis again to ensure that all errors have been repaired.
  If errors remain in the database you should definitely restore your last known backup of the database file!


IMatch database diagnosis logfile closed: 25.08.2014 12:41:11 (00:01.51)

And a third db-check:

And now at the beginning of the protcoll:

CRITICAL ERROR: The database system reports physical damage in the database file. Such errors are caused by disk/network problems, power-failures, blue screens and similar problems outside the control of IMatch. Please restore the database from the last working backup.
IMatch database diagnosis logfile created: 25.08.2014 12:41:57

And at the end with this errors:

Analyzing database:
    ERRORS were found in database file structure!
This usually indicates that the database file is physically damaged on disk. Such errors cannot be repaired.
You should RESTORE THE LAST KNOWN WORKING BACKUP of your database!    '*** IN DATABASE MAIN ***
FRAGMENTATION OF 20 BYTES REPORTED AS 0 ON PAGE 310854'
    'row 5354680 missing from index idx_md_tag_data_oid_tag_oid'
    'row 5354681 missing from index idx_md_tag_data_oid_tag_oid'
    'row 5354682 missing from index idx_md_tag_data_oid_tag_oid'

Optimizing Database, rebuilding optimial index structures and query plans:
Completed.

WriteTest:
Completed.

Results:
    Errors:    5
    Warnings:  0

  + + IMPORTANT NOTICE TO USER + +

  If the above error count is greater than 0 you should run the diagnosis again to ensure that all errors have been repaired.
  If errors remain in the database you should definitely restore your last known backup of the database file!


IMatch database diagnosis logfile closed: 25.08.2014 12:43:52 (00:01.55)

Between the db-checks i have done nothing (only compress & optimizing)
There where no blue screen or powerfailurs, i see no disk errors.

The errors on two harddisks!

[Mario]

Just to reiterate:

You run a diagnosis and it reports a clean database.
Then you run a Compact & Optimize.
Then you run a diagnosis again and the database system reports errors in the database?
This would mean that the database system itself damages the database while compacting it!!
I'm sure the company creating the database system used in IMatch will be very interested in this - because this database system is also used by Adobe, Apple, Google and others.

Does this happen on any hard disk?
Can you upload the original (before the damage) database somewhere? My FTP server perhaps?
You did unfortunately not include the diagnosis log file and the IMatch log file so I cannot even tell how large your database is.

[RainerG]

I wonder once again with the last imatch 3.6 db version to start.

I startet to run with an imatch5 beta in january.

Is it better to start once again?

[Mario]

The database on the lowest level (which is where your error happens) is unchanged for a long time. IMatch uses a industrial strength database system for storage. It should never damage a database to run an optimize. I have never heard about such a problem, but it may be serious if you can reliably reproduce it.  I cannot say anything without having your database here for testing. How large is it? Can you upload it somewhere?

[RainerG]

The DB is uncompressed 1,6 GB. Compressed ~0,8 GB. I upload it to my space at the next days (before i go in holiday :-). I send you an email after uploading.

[Mario]

Just to make sure: I need the database version before you run the Compact on it - the "OK" database. I need to check of the Compact & Optimize run here also damages the database.

[ubacher]

For what it is worth: I also encountered this situation. After an IM crash and restart I ran diagnostics and it reported it as clean.
Did some recovery work - and IM started to report the db as damaged. Went to the backup to recover.

[Mario]

If damage is done to the database file in some area, the database system will recognize this only when it tries to access the data. This may not be immediate, when only rarely accessed data is damaged. But the database vendor told me that the routines I run during the diagnosis are supposed to find all physical database damage caused by disk/network problems or unwritten data stuck in the Windows file system cache during a power failure. Or by some driver not handling a hibernate/sleep cycle properly when the disk is under stress because IMatch is currently writing data.

If you encounter such a problem, I would need the original database.
I then run the diagnosis here and when it comes out clean despite the database being damaged, I can inform the vendor and send your database as a debugging aid.

[RainerG]

You got mail --> <email address removed>

With a downloadlink to the DB and problem description (in german  ).

[Mario]

Thanks. Please allow a week or two before I work on this. My "look at this file" queue is more than full. I have over 10 GB of DUMP files and image files to sieve through and I can stand this only for a couple of hours a day.

[RainerG]

Thanks. Please allow a week or two before I work on this. My "look at this file" queue is more than full. I have over 10 GB of DUMP files and image files to sieve through and I can stand this only for a couple of hours a day.

No problem! Next week i make new food for IMatch in danmark 

[Ferdinand]

I have removed the email address you included.  Email addresses inserted in forums like this are harvested by spammers.  While it might appear that this email appears on the photools web site, it is in fact inserted there using javascript that is (supposed to be) not harvestable.