Rescan Issues

[StanRohrer]

PROBLEM:
When I'm working in Photoshop and create a new file, I can Rescan an iMatch folder to detect and add that new file. Currently, when I do a folder Rescan, all folder files can take scan time and updating, not just the new file that I expect. The database/iMatch seems to be losing the as scanned status of all files.

BACKGROUND INFO:
iMatch 2020.14.2

Recent upgrades to desktop PC include CPU (now i7-11700K, Motherboard, 32GB Ram). Boot disk, with windows and programs, with iMatch database and working files (15,159 working files), was moved from SATA SSD to NVME SSD (and Relocated). iMatch historical archive drive stayed as SATA HDD (179,094 files) (and Relocated to the same location). Also updated a month prior was video card and monitor that was also brought from the old PC to the new PC hardware. This recent system update should be a very large speed increase from my 10 year old i7 (Gen 2) CPU and hardware. The new CPU includes more cores and threads.

TROUBLESHOOTING:
Coming out of this CPU hardware upgrade, I discovered all files within iMatch database needed a Rescan as observed by running Folder Rescans. The Rescan icon indicator does not initially appear on these files but the files are still included in the Rescan operation. The existing files are not quickly bypassed as having been previously Scanned, as I normally expect.

An overnight Rescan of all Archive HDD drive files (179,094) failed to reset the Rescan status of the files.
With some files successfully Rescanned (a subsequent Rescan didn't take time to access those files), an overnight Rescan left all files in the database requiring a Rescan. A Rescan within some folder will change the rescan status of other files not associated to the given operation.

A Rescan of 19,059 files seems to work. The files do not need Rescanned on a second Rescan attempt.

A Rescan of 22,000 files results in all files of the database needing a Rescan. Yes, the status changes of previous good status files and they will then require a Rescan.

There appears to be some number of files threshold limitation between a Rescan of 19,059 (which is successful) and a Rescan of 22,000 files (which leaves files that will require Rescan).

It appears I can do a single or multiple batches Rescan to get to 19,059 files. It seems the file count is significant, not the number of Rescan batches.

Is there a practical limit of how many files can be included in a single Rescan batch?

Is there a practical limit of how many total files can be included in multiple Rescan batches?

Following large Rescan batches, is there a practical limit the number of files to be updated via the Compact and Optimize command?

During my attempts in diagnosing this Rescan issue, I have occasionally done Database Diagnostics and the results were No Errors.

An overnight (13 hours) Rescan-Force Update of all Archive HDD drive files (179,094) failed to reset the Rescan status of the files (update icon was not showing but subsequent Rescan attempts reviewed every file).

Due to anticipated file size, I hesitated to capture a Diagnostic Log File on a 22,000 file Rescan. What level of Log file would be useful in troubleshooting this issue? Maybe I can Rescan 19,059 files (which I expect to work) and start a log file while Rescanning an added batch where I expect the issues to happen.

I have used iMatch since maybe 2003. I plan to upgrade from my current after I think my system and hardware changes are stable (didn't want to change too many things at one time). If the Rescan issue has a fix in a newer version of iMatch, just say the word and I will upgrade for the fix.

OTHER NOTES:
Somewhere in this hardware upgrade and Rescan troubles, I now have 31,191 files pending write-back. Collections tab / Pending Metadata Write-Back, appears to not be able to display this volume. I wanted to see what info was pending write-back and it would not display the files. A 6-8 hour Write back All is in progress as I type this message.

[jch2103]

You may want to post an IMatch log file (debug logging).

[Carlo Didier]

Sounds like similar issue that I have occasionally, but can't reproduce at will.
If I remember well, I wasn't the only one either.
Problem is that it doesn't occur often and I can't reproduce it. That's one of the reasons I leave debug logging always on, just in case it happens again.

[Mario]

Without the log file of that very session (see log file) we don't have not anything to work with.
When IMatch rescans a previously file, the "last modified" on disk timestamp has changed, or the user triggered a forced rescan. IMatch will log the files checked and if the file was modified and thus needs rescanning.

Which virus checker do you use? Exception configured for the database folder and IMatch?
Which other software do you run? And run while IMatch is running?
Are your files stored in a folder controlled by DropBox, OneDrive, Google Drive?
Are your files stored on a NAS?
Have you configured IMatch to write-back metadata immediately?

Collections tab / Pending Metadata Write-Back, appears to not be able to display this volume.

This cannot be right. Categories and Collections can contain hundreds of thousands of files and File Windows have no problem displaying this amount of files.

All this sounds to me as if something on your system is working against IMatch or doing crazy stuff in the file system...

[StanRohrer]

Collections tab / Pending Metadata Write-Back, appears to not be able to display this volume.

This cannot be right. Categories and Collections can contain hundreds of thousands of files and File Windows have no problem displaying this amount of files.

I entered a bug report, with more info, on this minor discussion. https://www.photools.com/community/index.php?topic=12039.msg85701#msg85701

[StanRohrer]

Without the log file of that very session (see log file) we don't have not anything to work with.
When IMatch rescans a previously file, the "last modified" on disk timestamp has changed, or the user triggered a forced rescan. IMatch will log the files checked and if the file was modified and thus needs rescanning.

Which virus checker do you use? Exception configured for the database folder and IMatch?
Which other software do you run? And run while IMatch is running?
Are your files stored in a folder controlled by DropBox, OneDrive, Google Drive?
Are your files stored on a NAS?
Have you configured IMatch to write-back metadata immediately?

[snip]

All this sounds to me as if something on your system is working against IMatch or doing crazy stuff in the file system...

Last Modified: Attached is a sample of Old Files which get rescanned during my testing. Note the last Modified date is about 19 years ago.

Virus Checker: I'm running Microsoft Windows Defender. Attached is a screen capture of my excludes.

Other Programs: I run quite a few programs on this machine. The last clean windows installation may be more than 10 years ago. So a lot of programs, and perhaps residual junk exist. During iMatch use for this testing session: Thunderbird Email, Microsoft Word, Google Chrome, Windows File Explorer, Windows Task Manager, Adobe Photoshop.

Cloud Storage: Only one folder from within iMatch domain is linked to Google Drive. Otherwise, no other files/folders indexed by iMatch are linked to a cloud service. Microsoft OneDrive and Google Drive is minimally linked to a few objects not within the scope of iMatch.

NAS: I do not have any Network Attached Storage. As noted in my first post, two drives operating with iMatch are internal to this desktop PC. Boot disk, with windows and programs, with iMatch database and working files (15,159 working files), was moved from SATA SSD to NVME SSD (and Relocated). iMatch historical archive drive stayed as SATA HDD (179,094 files) (and Relocated to the same location). Rescans worked as expected on the old hardware including SATA SSD for the boot disk. I don't know if something (setting) got corrupted during the hardware changes. I don't know if the much faster CPU and disk access (NVME SSD) had an effect.

Write-Back Metadata Immediately: Box is unchecked in the Edit / Preferences / Background Processing tab.

Attached is a log file of a 41,294 file Rescan. Some files were OK and got bypassed. They were left with good status from some of my previous smaller batch Rescans. It appears to me that 12,081 files got an actual Rescan in this run. At the completion of this run, I went back to just the 2002 folder and the Rescan of that folder updated all files (should have been no need to update).

[StanRohrer]

I rebooted Windows. Turned OFF Defender Anti-Virus Real Time. Opened IM and enabled data logging. Ran another large Rescan. At the end, I went back to folder 2002 and ran a Rescan (should have been scanned and not need a Rescan but all files again Rescanned).

ETA: When I went back to turn On Real Time Antivirus, it was already On. So I can't be completely sure it was Off during the run. I think Defender likes to turn itself back on after a period of being off. So I can't be sure when it came back on.

[Mario]

In the first part of the log, IMatch rescans many folders and over 70,000 files.
All files are logged as current (no change) and are not processed.

The first file IMatch actually processes is P:\Photo Archive\2006\20060109 north lima trip\IMG_28572z.jpg, starting at 18:55:07, because the cache image is missing.

Do you have configured IMatch to use cache images for JPEG files?
Have you moved or deleted the cache folder perhaps?

The log file has 500,000 lines, which makes it tough to analyze. But to me it seems IMatch is to recreate missing cache files. It does not process the original files at all, no metadata import or other processing.

[StanRohrer]

Edit / Preferences / Cache / Don't Cache JPG Files was set to Off. I have set to On and will make some more Rescans and see how it acts. Thanks for your review.

ETA: Yes the cache folder was cleared once I got onto the new hardware. With the changes in monitor and video card iMatch came up with some suggested changes to the Prefs for screen and text magnifications/scaling. About that same time I had thought the Preferences got lost and/or reset. There are so many settings that I didn't remember where they all were before. Some I recognize as being exactly where I left them and a few were not. Cache size I certainly remember playing with since I was now on a larger faster drive. Don't Cache JPG likely changed at that time to be Off.

[Mario]

IMatch by default does not cache JPEG files (waste of disk space for most users, since cache files are JPG files).
If you have enabled to cache JPG files, and the cache folder has been emptied or moved, IMatch will recreate missing cache files when they are first needed, or when the file is rescanned for a reason and IMatch notices that the cache file is missing. So it's perfectly normal that IMatch does what it does.

[StanRohrer]

No cache for JPG files seems to have helped my Rescan concerns. I'll keep watching as I still see a couple of places where the Rescans are taking time unexpectedly. Well, unexpectedly for me, as I've not studied the behavior in depth and rarely have Rescanned major segments of my collection. So I'll call it good for today with no JPG cache. Thanks, Mario, for your diligent and timely help!