Database corruption when searching with heirarchy enabled.

[mlavicka]

Get message "Severe Database Problems detected! - The database engine has reported potential damage to the database file on disk." message when I use the new filter bar at top.

I start in a single sub-folder, then enter a search word in the bar and it correctly show the filtered result.  I then change to my main photos directory with showing all hierarchy folders enabled and the database gets corrupted. I have over 50,000 photos in hundreds of sub-folders, and my search criteria are set to use tags "Creator and Rights", "File Name", & "Description, Title, Headline...".

I then run Database Diagnostics and get the message: "This database had errors!" with the log file showing
 "CRITICAL ERROR: The database system reports physical damage in the database file."

I then run Database Diagnostics again, and it seems to be able to make the appropriate repairs, showing "This database has no errors".  But the same corruption errors happens again if I do the same steps above.

The imatch log file shows:
03.15 13:28:53+19890 [2904] 00  E> 'no such table: scopefiles.tag_cache'

• for: SELECT DISTINCT c.oid FROM 'scopefiles.tag_cache' c WHERE     ((TRIMDC(c.tdata) LIKE '%todd%' ESCAPE '~'))   'V:\develop\IMatch5\src\imsq\IMSQLite.cpp(1328)'

03.15 13:28:53+    0 [2904] 01  W> 'no such table: scopefiles.tag_cache' [21]  'V:\develop\IMatch5\src\imsq\IMSQLite.cpp(3485)'
03.15 13:28:53+    0 [2904] 01  W> SELECT DISTINCT c.oid FROM 'scopefiles.tag_cache' c WHERE     ((TRIMDC(c.tdata) LIKE '%todd%' ESCAPE '~'))   'V:\develop\IMatch5\src\imsq\IMSQLite.cpp(3486)'
03.15 13:28:53+    0 [2904] 02  I> FWFind: 0ms for 0 results.
03.15 13:28:59+ 5891 [075C] 00  E> <<ENGINE WARNING/PANIC>>: major:80 minor:1 d1:0 d2:0 s1:'' s2:''  'V:\develop\IMatch5\src\IMEngine\IMEngine5.cpp(1455)'
03.15 13:28:59+    0 [075C] 00  E> The database file has been physically damaged. Replace from backup immediately.  'V:\develop\IMatch5\src\IMEngine\IMEngine5.cpp(1456)'

[Mario]

I tried to reproduce this, but I could not. There was also never a similar report, as far as I can remember.
This is what I did, does this match what you do?

I used a 100K files database.
Searched in a sub-folder for the word beach.
Then I switched to the top-level folder and enabled the hierarchical display mode to show all 100K files.
The File Window automatically applied the search and a couple of hundred files were found.
Did that a couple of times.
No errors in the database.

The no such table: scopefiles.tag_cache error indicates that something went wrong when the File Window tried to create a temporary table to cache metadata for faster processing.
These tables are created all the time, and dropped when no longer needed.  Each File Window creates these tables when needed for a search operation with the File Window search bar (which is not a new feature but part of IMatch for many years).

Please provide the fill log file from that session. ZIP and attach.


Physical database damage may stay undetected for a long time, unless the defective section of the database is used, or a database diagnosis runs. And physical damage cannot be repaired by running a diagnosis, only detected.

Which makes me believe that the error message is false and the database system somehow fails to create / attach a table and considers this a hard error? Or, IMatch dropping the temporary table automatically somehow fixes this. No idea, really.

Can you upload your database to your cloud space and send a link to support email address?
And detailed instructions which folder you start searching in, with which options and search term, and which folder you then select.

Also, if you have backups of your database, use the most recent one. Run a diagnosis, Compact & Optimize and Diagnosis again to ensure that the file is physically OK.

[mlavicka]

This is what I did, does this match what you do?

Looks right, but you should make sure the same search tags are used.

Please provide the fill log file from that session. ZIP and attach.

Done.  I sent it through with the email to the support email address

Can you upload your database to your cloud space and send a link to support email address?
And detailed instructions which folder you start searching in, with which options and search term, and which folder you then select.

Done.

[mlavicka]

UPDATE:  I tried to narrow down the issue, by systematically cycling through each sub-folder with the hierarchy enabled and search enabled. Everything went fine.  I then went back to the root directory to try and recreate the same problem I had before and now it all works fine (I can no longer recreate the issue).  So maybe it was something in the way the cache was created, or needed to be created, before doing the broader search.  Let me know if you see anything in the link to the database or the log file I sent you.

[Mario]

Interestingly, another user reported a similar issue:

https://www.photools.com/community/index.php/topic,15037.msg

I already know that the database has NOT been damaged. It's just so that a missing database table is considered a critical error (because this should not happen). And it has never happened before, afaik.

File Windows use temporary tables to cache metadata results, and these tables are created and removed as needed.
The other user did not provide a log file, but I guess he has managed to get the same "no such table" issue like you did.

I tried to reproduce the problem by searching for the same pattern, but I could not so far, on three different computers.
Something must trigger this, I just don't know why.

Update: I figured it out!

See my explanation in this thread: https://www.photools.com/community/index.php/topic,15037