IMatch says database not closed correctly

[Stevef48]

I closed the database and stopped IMatch, but when I started it again it told me that the database wasn't closed correctly.
This is the fifth time this has happened recently.
What must I do to stop this message appearing? If I choose to let IMatch run a database diagnosis, which takes several minutes its always tells me that there are no errors.
Thanks in advance
Steve

[Mario]

IMatch shows this message if, as it says on the label, the last session was not closed properly. This usually only happens when IMatch crashes and does not shut-down cleanly.
If you see this, check the IMATCH6_LOG_BACKUP.TXT file in the TEMP folder on your system. See log file

Maybe IMatch has logged some error messages or something.

[Stevef48]

I closed the database before stopping IMatch last night.
The message appeared again this morning. IMATCH6_LOG_BACKUP.TXT contains data from early yesterday morning.
The zip contains all the IMATCH logs that were in my Temp folder before I started IMatch this morning, plus the log from the running program.
Steve

[Mario]

When IMatch starts it renames the last logfile to IMATCH6_LOG_BACKUP.TXT before opening a new log file.
Your ZIP did not contain this file? Only a "b" log file, but which showed a clean shut-down.

The warning I see in the active (open) log is 80:2, which means "hot journals found".
This means that the database system did not shut-down properly, or that some external source kept the journal files open, preventing the database system from closing them.
Do you access the IMatch database with other software?

[Stevef48]

I grabbed the log file before IMatch started, added the date and b to the file name. To avoid confusion with all the other log files.
No nothing else accesses the IMatch database.
Steve

[Mario]

Have you rebooted your system recently?
Do you see files named <database name>.imsema or <datbase name>.imd5-shm or <datbase name>.imd5-wal in the database folder after IMatch has closed?

[Stevef48]

Have you rebooted your system recently? Yes
Do you see files named <database name>.imsema or <datbase name>.imd5-shm or <datbase name>.imd5-wal in the database folder after IMatch has closed? Yes, but that still doesn't explain Why they were left after IMatch said that the database was closed

[Stevef48]

I've just gone through the following steps:
Close the database;
Exit IMatch.

All of the files shown in your post are present in the IMatch folder and all have the same date/time modified as the current database. For some reason there are no files in appdata\temp?
Steve

[Mario]

These files should automatically vanish when the database system shuts down.
Else you will always see the error message about IMatch not having shut-down properly.

1.  Check in Windows Task Manager if the process "photools.com IMatch Application" (or IMatch2020x64.exe in the Detail View) are still running.

2. If not, some external application, server or, most likely, virus checker, keeps locks on the database.
If the IMatch process is closed, all file handles and locks created by it are automatically removed by Windows.

If you are an experienced user you can use software like the free Microsoft Process Explorer (https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer) to check which applications or services access the database file (or the -shm or -wal files). Run the tool after unpacking the ZIP file (no installation required) and then use "Find" in the main menu. Search for the database name to find processes holding the database or the journal files open.

[Stevef48]

The database has now opened without the message several times. I have no idea what caused that and of course there's no way to fix it if I can't duplicate.
Thanks
Steve

[Stevef48]

I now have the opposite situation, IMatch stopped responding after finishing its first run of metadata updates. I waited for several minutes, then closed the program.
I'm still trying to find out why my NEF files take so long to import, so fortunately have a debug log. That and the log from the next program start are to big to post, so I stored them in DropBox:- https://www.dropbox.com/s/z7k6pxlupk9c52v/IMATCH6_LOG%20200503.zip?dl=0

I'm confused, because I requested writeback for all pending files, but the log shows lots of files having no data to writeback. Why was IMatch even looking at them?
Steve

[Mario]

You limiting the number of threads to use for reading files and metadata to 2. On a system with 12 processors.
Usually IMatch would use between 8 and 12 threads, depending on some measurements. This would speed up things considerably.

ExifTool takes between 5 and 7 seconds to extract metadata for 5 files. This is a bit on the slowish side, especially considering that IMatch is only allowed to run 2 threads.

In the big log file I see many warnings like

Write-back Cannot write to off-line/read-only file D:\Photos\2011\.dtrash\files\IMG_4255-417abd0c.JPG

File system privilege problem? Virus checker interfering?  File really off-line or read-only on disk?

Quite a bunch of warnings from ExifTool about invalid timestamps. Apparently only JPEG files?
See the log file and search for W> to find all the files which have issues.

Some MOV files which cannot be updated.

Many warnings about broken maker notes or partially corrupted data, like

FaceDetInfo pointer references previous IFD0 directory - D:\Photos\2012\2012-07-July\P1140124.JPG

No idea how you created these JPEG files or which software was used on them before ExifTool analyzed them during write-back.

No errors in the log file. When the log file ends, IMatch was in full flight, writing metadata to XMP and image files.

The state of the files is a bit on the nasty side, and performance is slowish for D:, but OK for RAW files with about 1s per file.

Maybe process files in smaller batches of 5,000. Do not throw 100,000 files at once at the process. Not with so many problems int he files, which could cause unknown side effects everywhere.
Try to increase the more files at once (Process control settings), unless this makes your computer unstable.

I have a new PC with 12 cores and it rips through 100,000 files in my medium sample library of RAW and JPEG and other image files, videos, Office, etc. like a race car. Even running for hours no problems with the default settings fr process control (0,0,0) in IMatch.

[Stevef48]

You limiting the number of threads to use for reading files and metadata to 2. On a system with 12 processors.
Usually IMatch would use between 8 and 12 threads, depending on some measurements. This would speed up things considerably.

ExifTool takes between 5 and 7 seconds to extract metadata for 5 files. This is a bit on the slowish side, especially considering that IMatch is only allowed to run 2 threads.

In the big log file I see many warnings like

Write-back Cannot write to off-line/read-only file D:\Photos\2011\.dtrash\files\IMG_4255-417abd0c.JPG

File system privilege problem? Virus checker interfering?  File really off-line or read-only on disk? Trash directory created by Digikam I removed most from IMatch, but obviously missed some

Quite a bunch of warnings from ExifTool about invalid timestamps. Apparently only JPEG files? Yes, I've been aware for years that something is messing with dates and times. I noticed some images with dates in 1927, 2 decades before I was born. Haven't been able to trace the cause.
Is the solution to use the date wizard on these images? It sounds like ExifTool only reported the invalid timestamps, it didn't fix them.
See the log file and search for W> to find all the files which have issues.

Some MOV files which cannot be updated. I assumed that xmp files would be created for these, I didn't think IMatch would update them

Many warnings about broken maker notes or partially corrupted data, like

FaceDetInfo pointer references previous IFD0 directory - D:\Photos\2012\2012-07-July\P1140124.JPG Very Odd there is no face in this picture

No idea how you created these JPEG files or which software was used on them before ExifTool analyzed them during write-back. I think that I was using Adobe Photoshop Album then. Usually the files are as written by the camera. If I do more than simple lighting adjustments in Lightroom, I save as a new _edited file.

No errors in the log file. When the log file ends, IMatch was in full flight, writing metadata to XMP and image files. Great

The state of the files is a bit on the nasty side, and performance is slowish for D:, but OK for RAW files with about 1s per file. I noticed that it varied a great deal, the speed of processing the same type of files can vary up to 5 times e.g. I think NEF varied from 350ms to 6533ms

Maybe process files in smaller batches of 5,000. Do not throw 100,000 files at once at the process. Not with so many problems int he files, which could cause unknown side effects everywhere. Sorry, I didn't know the files were so bad, I became fed up with waiting for the metadata read/write to finish. As I said somewhere, 3000 files took 30 minutes for the first pass, then it restarted
Try to increase the more files at once (Process control settings), unless this makes your computer unstable. I changed all Process Control thread settings to 0 (maximum?), maybe after the giant debug log

I have a new PC with 12 cores and it rips through 100,000 files in my medium sample library of RAW and JPEG and other image files, videos, Office, etc. like a race car. Even running for hours no problems with the default settings fr process control (0,0,0) in IMatch. I need to fix all those errors, mine has 6 cores and all data is now on SSDs Database on C: and Photos D:

Thanks for your help.
Steve

[jch2103]

Welcome to the wonderful world of messed up metadata! You really don't know how messed up it is until you have a tool like IMatch to show you. It's taken me some time to clean up problems created by various programs I used many years ago. Good luck!

[Stevef48]

Welcome to the wonderful world of messed up metadata! You really don't know how messed up it is until you have a tool like IMatch to show you. It's taken me some time to clean up problems created by various programs I used many years ago. Good luck!

Thanks I see legacy data from ACDsee, Microsoft Gallery, Photoshop Elements...
Steve