Multiple PCs

[bobneedshelp]

I have 2 different PCs that I use iMatch on.  I have different user names on both PCs.  I'm not interested in using the PCs at the same time.  In both cases, I am the user.  They both have access to a shared NAS device.  I wrote a script to copy the main database (IMatch Database.imd5) from the NAS to the local PC for faster performance.  Both PCs can share the same imatch.pts file on the NAS.  The paths are the same on both PCs. 

While setting this up, I accidentally started iMatch on both PCs at the same time (I didn't think this could happen with my script).  Everything works fine on the first PC.  Now I get the following error on the second PC.

On one PC, everything works as expected. 
On the second PC, I get an error:
An error occurred while opening the database C:\iMatch\IMatch Database.imd5:
[17201] A database operation failed.

I don't believe this is due to permissions on the file.  The next place I looked is in the application log file.  It has the following issue stated:

CIMSQ: Physical database damage: 'database disk image is malformed'

If I run diagnostics on the first PC, everything is fine.

Any ideas how to get this working again?  Is it the .pts file that has the issue or perhaps some cached files? 

[Mario]

The database system reports that the PTS file is physically damaged.  It has to be replaced from the last backup. There is no recovery.

I understand that you keep the PTS file on a NAS and that you have allowed two instances of IMatch to write to the file at the same time. Concurrent write access to the PTS file may cause issues when the network locking mechanism implemented on the NAS is not solid. This cause is listed as one of the very few issues that can actually damage a SQLite database.

I recommend you keep the PTS file in it's designated folder (C:\...) and use Pack & Go to transfer the PTS file, database, config files, apps and other relevant files when needed. See Pack & Go

[bobneedshelp]

I restored the backup and have the same issue. 

Just as clarification, I don't run them on different PCs at the same time.  That was an accident that happened while I was adjusting the script.  I now have fixed that issue.

In any case, I still have the issue on the second PC.  I will try to use the Pack & Go method to see if I can get it to work that way.  Hopefully I can automate that via scripts. 

I guess my question is, since it works on one PC, is the PTS really damaged and I should figure out a way to start over so that I am not moving forward with a damaged file?  Is there a process for that?

[Mario]

If the PTS file (which is a SQLite database) is damaged, it is done. Physical damage is really rare and usually only happens during a hard power failure, when Windows is unable to flush file system changes in memory to disk.
Another reason for damaging SQLite database is a faulty file locking mechanism - which may happen on NAS boxes running some versions of SAMBA on some version of Linux. Hard to tell.

If you have  working copy of the PTS file, copy it to the other PC and see if it works OK. Don't put it on your NAS. If it failed once, it may fail again. IMatch runs a diag on the PTS file automatically to ensure it's valid.

[bobneedshelp]

I copied over the working PTS file from one PC to the other PC.  Still have the same error messages.  I made sure the PTS files were being read from their default locations on the PC. 

What is the process from here?  Do I need to start a new database and import all the pictures, ...?

[bobneedshelp]

I also followed the pack and go instructions.  I could get it to make an IMPAG file as long as I didn't verify or optimize.  When I tried to restore the file on the second PC, I got the following protocol error message

IMatch Pack and Go (c) 2001-2024 Mario M. Westphal. All rights reserved.
[C:\Program Files\photools.com\imatch6\IMPackAndGo.exe], version info: 2023.14.2

Restore started.
Intalled IMatch version is 2023.14.2.
Package IMatch version is 2023.14.2.
Original Data:
etFileSettingsDB    C:\ProgramData\photools.com\imatch6\config\
etFolderCache    C:\ProgramData\photools.com\IMatch6\previewcache\
etFolderConfig    C:\ProgramData\photools.com\IMatch6\config\
etFolderDictionaries    C:\ProgramData\photools.com\IMatch6\dictionaries\
etFolderPresets    C:\ProgramData\photools.com\IMatch6\Presets\
etFolderPrinting    C:\ProgramData\photools.com\IMatch6\printing\
etFolderUser    C:\Users\UserN\AppData\Roaming\photools.com\IMatch6\
etFolderWebRoot    C:\ProgramData\photools.com\IMatch6\webroot\
Restoring 1,387 packet
Restored 'C:\iMatch\IMatch Database.imd5' to 'C:\iMatch\IMatch Database.imd5' (2.88 GB).
Running analysis for restored database 'C:\iMatch\IMatch Database.imd5'
Analysis of restored database failed. See application log file for details.
Duration: 00:00:00

Restore completed with errors. Please see the application log file for additional details.
Duration: 00:00:29


ADMIN SAYS: PLEASE don't just post hundreds of lines from a log file.
This makes your post unreadable on mobile (which I often use) and also fills the community search engine with nonsense, which is bad for all users. I have this time removed the log dump and stored it in a separate text file which I then have attached to your post. In case users want to look at it. I won't do that again.


The Application Log is shown below but not much help?

See attachment.

[Mario]

Sorry. I did not recognize yesterday that you had posted only a segment of a log file.
Since the physical database error came so early in the log, I falsely assumed it was related to the settings database (the PTS file). Not the actual IMatch database. Pleases always ZIP and include the full log file.

The IMatch database file C:\iMatch\IMatch Database.imd5' is physically damaged ('database disk image is malformed') and needs to be replaced from the last backup. Although IMatch databases are extremely, extremely robust, they can be damaged by hardware or network failure (when the database is on a network server / NAS), power failures, computer viruses and similar causes.

Restore your last daily backup from before.
Open it in IMatch and run a database diagnosis. If it also reports physical damage, go back one day further in your backup, until you find a database without damage.

Note that sometimes database damage happens in rarely used sections of the database file. The database system then recognizes the error when it has to access the damaged portion of the database file the text time. Or during a database diagnosis which checks the entire database for problems.

[bobneedshelp]

I restored my last backup of the iMatch Database.imd5 prior to the issue and it works.  I separated the imatch.pts files to both reside in their local default locations.  I then used Pack and Go to move the files to the second machine.

I will start a separate post and do some research first on Pack & Go.  My question is that when I moved the file via Pack and Go to the second machine, there was nothing outstanding to write to the database from a Metadata perspective.  On the second machine, I needed to update the metadata on almost 500 files.  It also did a lot of updating when I brought over the files.  Is that normal?

[rolandgifford]

Note that sometimes database damage happens in rarely used sections of the database file. The database system then recognizes the error when it has to access the damaged portion of the database file the text time.

A hypothetical question based on this.

If damage is in a rarely used section of the database it can presumably lie hidden through several backup cycles during which lots of changes elsewhere have been made, making it very difficult/damaging to restore a very old backup (in terms of updates).

Are there any tools to repair this type of damage other than restoring from backup?

[Mario]

A hypothetical question based on this.

If damage is in a rarely used section of the database it can presumably lie hidden through several backup cycles during which lots of changes elsewhere have been made, making it very difficult/damaging to restore a very old backup (in terms of updates).

Are there any tools to repair this type of damage other than restoring from backup?

If Windows fails to write data to the database while confirming a successful write to the database system (Window file system cache unable to purge due to a power failure, disk failure, USB glitch, ...), bitrot happens, network problems etc. a database file may become damaged. It's really rare, but there is a theoretical possibility.
If the damage is in a rarely used section of the database, the database system may notice it only weeks later.

That's why there is a database diagnosis in IMatch which checks the entire database. That#s why Pack & Go suggests to run a database diagnosis when you also add a database.

When a database file is physically damaged, there is no repair. If you are really lucky and the damage is in an index, the built-in repair in IMatch may be able to fix it. But...

Like with all important data, make daily backups of your IMatch database. The entire "program data" folder where not only IMatch but all your software stores important data.

The SSD in your PC works fine today. Tomorrow, it fails and you won't be able to boot your PC.
Can you just replace the damaged SSD (or let somebody replace it), restore all your data and be back and running in two hours?
If not, you should reconsider your backup strategy.

[rolandgifford]

That's why there is a database diagnosis in IMatch which checks the entire database. That#s why Pack & Go suggests to run a database diagnosis when you also add a database.

Backup comments noted, backup is indeed vital as is knowing that what is being backed up is not broken.

There are two config options for database diagnosis which don't work as I would expect (I did have them turned off as I do manual checks, I'm considering doing them automatically instead). I don't normally leave IMatch running when I'm not using it and would have to change that for automatic checks to be effective.

I set a 1 day interval and 10 minutes idle. When starting IMatch I received an unexpected prompt to run diagnostics. I expected no prompt as the check will be run when IMatch is idle. Is there some way to set these so that I don't get prompted?

I also looked at the run time of diagnostics (I normally set it running and go and do something else so it doesn't matter) and found it took around 15 minutes, 12 minutes of which were optimizing. I don't want diagnosis to optimize at all, I see that as part of the compact process. Is there some way to turn that off?

[Mario]

Optimize is different  that compact & optimize.
Optimize run as part of the diagnosis just optimizes indices, pre-fetch information and statistics that can be used by the database to optimize index usage. Compact & Optimize creates a new database file, then copies everything from the old database to the new file, leaving out unneeded stuff. Compact & Optimize was a lot more important on spinning disks than on today's SSDs. It's still useful if you delete lots of data from the database, to compact it afterwards.

If the diagnosis is due, IMatch will show the prompt. If the application is not busy at the time, IMatch then runs the diagnosis automatically. If you do daily backups of your database (you should!) with a normal backup program and your database is on local SSD (not external USB or NAS) running a daily diagnosis is overkill. If something really bad happens and you notice it with your weekly diagnosis, you can go back a day and restore your backup.
I use four databases created with IMatch 3 (!) initially every day. The largest one now has over a million managed files. Never had an issue or database damage.

[rolandgifford]

Points noted, thanks. The differences between the two optimises was especially useful. Knowing what the Compact variant does means that I will use it far less than I have been doing as a full rewrite of the database is a risk I can avoid.

I have turned off automatic checking again as manual is more appropriate to my usage pattern. It was good to consider it though, re-evaluating working practices is a good habit to get into.

My usage of IMatch means that the optimum backup process for me is different than for you. Being able to get back to a specific point on my editing path is more important than getting back to yesterday as I bulk edit rather than routinely tweak. I do system wide routine backups for major failures which includes the IMatch files but IMatch backups are separate and additional to that.

[Mario]

ss than I have been doing as a full rewrite of the database is a risk I can avoid.

There is no harm. The database system creates a new file, copies over the data from the old database, leaving out discarded sections, and when everything works and is validated, it deletes the old database and renames the new database to the old name.

If you work with your database once every two weeks for a day, it's of course useful to do the backup in the same pattern.

I use "imager" backup software which already backs up entire disks on a sector-by-sector basis. If a database is not changed in two weeks, it adds zero bytes to the daily backup volume.

[philburton]

....may cause issues when the network locking mechanism implemented on the NAS is not solid. This cause is listed as one of the very few issues that can actually damage a SQLite database.

A sort of side comment and a question.  I have always believed that SQLITE was designed for same system use only.  And any use over a LAN will inevitably corrupt the database.  Is that true, or just a limitation of the applications (Lightroom, Quicken) that use SQLITE?

[Mario]

As long as the underlying system has proper locking implemented, there is no harm using SQLite over a network.
Cheap NAS boxes with old SAMBA versions or faulty network stacks may cause problems.

[philburton]

As long as the underlying system has proper locking implemented, there is no harm using SQLite over a network.

How can I as Joe User, verify that file locking is implemented properly?

In a standard window 10/11 network, using Network Shares, is file locking implemented at all?

[Mario]

You can't. When you run a "real" Windows network with a Windows Server edition, no problem.

If your NAS is stable under high and prolonged load as produced by IMatch when it processes files stored on the NAS or even with a database stored on a NAS, you'll find out when it breaks. Not so problematic when images are on the NAS, IMatch will just log errors and report.

But if the NAS/SAMBA implementation causes damage to the database file because data gets lost, things are grim.
IMatch and SQLite cannot know that the simulated Windows file system created by the SAMBA software running on the Linux operating system running on whatever NAS hardware has reported data as "written" but actually failed to do so.

The same happens (very rarely) during a power failure. IMatch/SQLite write data, Windows reports it as written. SQLite closes the transaction. But the data was still in the Windows File System Cache or the cache on the SSD/disk when the power failure happens. And thus never reached the actual storage medium. Very rare, but can happen.
The IMatch database diagnosis will find and report the damage.

[philburton]

Mario,

Thanks for the reply.  No way I want to run (and pay for) Windows server.  I enjoy working with and even tweaking Windows, but that's about it.  If I had to run a server, which I don't, it would be Linux.  Free and well-supported.  But I haven't even justified in my mind the need for a NAS.  My desktop is a full-size tower case with 2 NVMe drive, 3 SSDs, and 5 HDDs.  My wife is a very casual user, with just a medium power laptop.