RemoveCacheImage failed

[jelvers]

Since some weeks Imatch (current Vers 2023.3.6) seems to stop to react, so I had to close it via the task manager. New start-up of IMatch takes a very long time ("initializing...'). The log file shows lines like:

10.22 12:38:38+ 5062 [40D4] 01  W> CIMCacheManager::RemoveCacheImage failed for C:\ProgramData\photools.com\IMatch6\previewcache\3B1DB56B-3356-4A04-87EC-53B989E81925\29\291826.jpg  'V:\develop\IMatch5\src\IMEngine\IMCacheManager.cpp(1488)'

Any advice highly appreciate.

Juergen

[Mario]

Does the file exist in the file system?
You can delete it in Windows Explorer.

Under which circumstances did IMatch stop responding? What did you do at that time?How long did you wait?
Note that Windows often announces an application is not responding when the application does not accept user input for 5 seconds. Maybe IMatch was just too busy at the time?

Of more concern is the log entry:

Database 'C:\ProgramData\photools.com\IMatch6\config\imatch.tdb' for thread 39E0 (14816) shared cache 1  Database file read-only/on read-only media.
'attempt to write a readonly database' [8]

For some reason reason the telemetry database is read-only or something is blocking IMatch from writing to it. This is not good and should never happen. The database is automatically created by IMatch and is fully writable for authorized users and your user account.

Close IMatch and then delete the file 'C:\ProgramData\photools.com\IMatch6\config\imatch.tdb' in Windows Explorer.
IMatch creates the database automatically when it starts the next time.
If there is still this error message in the log file, something (virus checker?) is blocking access.

If your virus checker is blocking IMatch, this would also explain why IMatch cannot delete cache images anymore.

[jelvers]

Thks for quick response. Will further check. The issue is that it didn't happen all the time and I can't really remember what my last "actions" were.
In the meantime I have deleted the above mentioned file. Can't find a message about "blocking by a virus checker" or similar (I use Defender) in the log file. Looks good so far.

[Mario]

Search the log file for W> to find warnings logged by IMatch. If the telemetry database file could be created again and is now writable, all is well. Unless, something is suddenly blocking IMatch again...

[jelvers]

It seems to work now. At least the respective file is writable:

10.22 18:09:22+    0 [54A8] 02  I>    Database 'C:\ProgramData\photools.com\IMatch6\config\imatch.tdb' for thread 54A8 (21672) shared cache 1  Writable.

Again many thks for quick help.

Juergen

[Mario]

This database is created only once (when IMatch runs for the first time) and then used forever.
It is used to store telemetry data (if enabled) and is automatically emptied when IMatch closes.
I have no idea how this file can become read-only.

I had one similar case the other week. The user sent me a log file with the same problem.
But in his case, the database was completely empty, IMatch was unable to create tables after creating the file. But it could create the file...!

I've sent him an empty telemetry database (like IMatch creates it on program start) and told him to delete and replace the telemetry database on his computer. Again, IMatch was unable to write to the telemetry database, despite the database working fine on my system and the file system permissions were set to "all authenticated users full access" on his machine.
This ticket is still open. I have no idea what to do.

[rolandgifford]

This database is created only once (when IMatch runs for the first time) and then used forever.
It is used to store telemetry data (if enabled) and is automatically emptied when IMatch closes.
I have no idea how this file can become read-only.

I have had this happen on more than one occasion and when going to check the file (without IMatch running) have found a set of three files rather than one. I think that the others (shm and wal) are created as temporary files when the core file is open for writing. The main database file operates in the same way.

I have assumed that the shm and wal files existing while IMatch isn't running are due to an incomplete/faulty shutdown. When next running IMatch it can't open this file as a writable file (as the the temporary files already exist) and misleadingly records the file as read only.

I may be very mistaken but hope this helps

[jelvers]

Unfortunately it just happened again. The program stopped to react, when I used the Image Batch processor to reduce the size of 260 images. Everything went well in the beginning but after picture 199 it simply stopped to react and I had to close the program via Task manager.
After restarting IMatch it is "initializing...."
I use Defender and have defined Imatch as an exception.

Juergen

[Mario]

The additional files are used for "journaling" and transaction purposes. They are created by the database system.
When IMatch can close the database normally, these files are deleted. Otherwise they remain and are used to rollback (undo) open transactions / incomplete database operations. This ensures that the database is in a good state.

When you experienced this scenario, were there any E> (errors) or W> (warnings) in the IMatch log file?

When IMatch shuts down and telemetry is enabled, it sends the data recorded in the telemetry database to my server and then empties the telemetry database.
There is a time window for that. If the upload times out, the telemetry database is not emptied but still closed before IMatch finishes with the shut-down process.

I have tested this in many scenarios, including disconnected networks and super-slow file system operations. This does not mean that this works in all situations. Telemetry exists in IMatch since IMatch 2021 and so far I have seen a read-only telemetry database only two times.

[Mario]

No signs of a batch processor run in this log file.
But still many warnings about IMatch being unable to delete cache images.

I wonder why IMatch tries to delete cache images. The only reason would be that the cache size you have configured is exceeded...? Which settings do you use under Edit menu > Preferences > Cache?

[rolandgifford]

When you experienced this scenario, were there any E> (errors) or W> (warnings) in the IMatch log file?

I expect there will have been as the only times I've looked have been prompted by your comments on a log file posted in relation to some problem that I have been having. Presumably none relevant or you would have mentioned it. I don't routinely look at my log files and did turn telemetry off for a while to reduce potential noise in the log file but have turned it on again since.

I only mentioned the shm and wal files existing as a possibly reason IMatch was incorrectly reporting that the telemetry database is read only where it isn't. It hasn't ever been read only (from a file system point of view) on my system.

Occasionally IMatch tells me that it hasn't closed down correctly and I now usually tell it to carry on. I have used the option to look at the log file on those occasions but haven't ever seen what the problem is so now don't bother. These days I use task manager to check that everything has finished before closing and ignore the pop-up message the tells me background processes are complete. I don't remember having a 'not closed properly' message since I started doing that.

[jelvers]

Here is a screenshot of my Cache Preferences......

[Mario]

Caching JPG files isd usually not needed (Should be On) since it basically doubles all files - unless you work with off-line files often.

Minimal cache size is by default 2000 pixels (not 640) to avoid too small cache images.
10 Gig cache size should work nicely, unless IMatch is prevented from deleting cache files over that threshold for some reason.

[Tveloso]

Unfortunately it just happened again. The program stopped to react, when I used the Image Batch processor to reduce the size of 260 images. Everything went well in the beginning but after picture 199 it simply stopped to react and I had to close the program via Task manager.

I wonder if you might have acted too soon, and killed IMatch before the Batch Processor was actually finished...and it's that abnormal termination that then accounts for the subsequent issues related to the Telemetry Database?

In order to not muddy this topic with those Batch Processor details, I created another topic for that:

https://www.photools.com/community/index.php/topic,13706.0.html

Your Batch Processor Preset is of course doing different work from the one discussed in the above topic, but I wonder if you might actually be experiencing the same thing, leading you to think that IMatch has frozen?

If this happens again, perhaps you can monitor the target location in Windows Explorer, to see if any ExifTool Temporary files are being generated there during the period of time that you perceive IMatch to have locked up...and/or wait a while longer to see if the Batch Processor completes.

[jelvers]

I wonder if you might have acted too soon, and killed IMatch before the Batch Processor was actually finished...and it's that abnormal termination that then accounts for the subsequent issues related to the Telemetry Database?

You have a valid point, because somehow I got IMatch running again after waiting, waiting and waiting. But the issue for me is, what does "too soon" mean? In one case I waited more than 20 min before I stopped the program via Task Manager. But I also experienced that IMatch was back after ca. 10 min. So something is defintely different from what I used to experience before. I use IMatch since 2005!

But many thks for your contribution.

Juergen

[Mario]

Switch IMatch to debug logging (Help menu > Support). IMatch then writes out log file data immediately. You can watch the log file in Windows Explorer and open it with Notepad to see if and what IMatch is doing, even if the IMatch UI is not responding anymore.

[jelvers]

May I again bother you with this issue, which happend again. IMatch has been in 'debug logging'-mode. Yesterday evening the system froze again, when I used 'Slide Show'. It froze after I went through ca. 100 pictures. Actually I get the impression that the issue shows up, after having dealt with a higher number of files (e.g. 100-200), be it via the slide show or processing them via the 'Import & Export' tool or else
Anyway, I have attached the logfile ('IMATCH6_LOG_BACKUP') of yesterday night, when I had to close the programm via Task Manager. The second log file refers to starting IMatch again this morning. IMatch showed the opening screen ('initializing') and seemed to be frozen again. After ca. 10-20 minutes, however, the program was back, as if nothing had happened. 

Regards, Juergen

[Mario]

Many "CIMCacheManager::RemoveCacheImage failed" entries, but also many successful removals of old cache images.
You can find the file names IMatch tried to delete and failed by searching the log for W>. Maybe delete them manually in Windows Explorer?

When the cache manager purges, it determines the files to purge and then does a loop:

A) if file exists
  B) delete file

Since we see this warning in the log, the Windows function for checking the existence of a file succeeded, but the Windows function used to delete the file failed for some reason. I can only guess that there is a file permission issue or that the delete function succeeds but returns an error code nevertheless. I have already improved the message by logging additional error information to the next IMatch release.

Nothing much else happens. A dialog is opened (I cannot tell which one because of sloppy logging), a File Window loads, IMatch determines the associated application for a JPG file in D:\PHOTO\PHOTO CANON\2023\URLAUB_2023\SI_2023\ to show the icon in the toolbar, a file is loaded from the cache for display, then IMatch is closing (normally?).
The log ends in the middle of the shut-down procedure.

[jelvers]

Many thks for quick reply and advice. I checked the 'C:\ProgramData\photools.com\imatch6\previewcache' directory and it contains four sub-directoriesbut, but only one contains pictures ('C:\ProgramData\photools.com\imatch6\previewcache\3B1DB56B-3356-4A04-87EC-53B989E81925\29'). I can indeed trace the files as mentioned in the log file. But there are more. Shall I delete all of them and also all subdirectories, i.e. the one with the numbers under 'e\3B1DB56B-3356-4A04-87EC-53B989E81925'?

[Mario]

IMatch creates one cache folder per database, using the unique database GUID as the top-level folder under the cache root.
You can theoretically delete the entire cache folder. IMatch creates cache files on-demand again when needed. But that may cause IMatch to need longer to display files in the Viewer or Quick View panel.

If IMatch fails to delete a cache files (which usually does not happen), it will automatically try to delete it again when it has to purge the cache the next time. Maybe it just works then, who knows.

What puzzles me is why IMatch fails to delete so many files, but deletes as many other files without problems, during the same purge operation. The only logical reason for this would be file system permissions. Did you copy the cache folder from another computer perhaps? Check a few of the files reported as "failed to delete" in Windows Explorer and see if you can delete them manually.

[jelvers]

Deletion works, but it requires admin rights, which I have to approve in order to proceed (see attachment).

Re 'copy the cache folder from another computer' I reinstalled IMatch on a new one by using the standard procedure which worked fine. I just can't recall that I interferred with copying directories in this context. The issue came up recently and the pics I see in the previecache folder are of newer date.

[Mario]

Deletion works, but it requires admin rights, which I have to approve in order to proceed (see attachment).

Ah, that explains that, then. Your user account has no write access to the files, and hence IMatch cannot delete them.
Unless you run IMatch with an Admin account (occasionally), I don't know how this file system privilege issue could happen.
Copying files from another computer may mess sup the file system permissions, assigning he current user (you) only read and execute privileges.

Select the top-level cache folder in Windows Explorer and right-click. In the Securities Tab, select Edit and add your user account to the list and make sure "modify" is enabled.

[jelvers]

Will dive into the rights-issue, using your advice, and come back with feedback.
Many thks again for your kind help.

Juergen

[jelvers]

Quick feedback and good news: after changing the account settings to write access  the issue has been solved.

Thks for advice and support.

Juergen

[Mario]

Good result