Folders requiring rescan

[rolandgifford]

I don't update any image files which are in the IMatch database outside IMatch but I regularly have folders which IMatch tells me rescanning. I understand that this is because something has changed the contents of the folder outside IMatch and the database needs to be verified.

I have recently done a full rescan of all folders (as I had done a change to GPS data covering lots of images and I wanted to be sure that everything was in sync) but now less than 2 weeks later something like 50% of my folders are flagged as requiring a rescan.

I turned on debug logging and rescanned one of the smaller ones in the hope that the root cause of this apparently random "out of sync" flag setting can be avoided by turning off whatever is setting it.

I've had a look at the log file and can't see anything untoward but don't know what I'm looking for. The rescan didn't take very long but was certainly not very quick as nothing was found, debug logging could be a large part of that perhaps.

Any hints what may be setting the flag appreciated.

[Mario]

Usually you won't even notice this.
IMatch detects that the folder timestamp has changed or responds to a Windows message and then rescans the folder in the background. When no changes are found, this is a very quick process.
The "refresh" icon is only visible next to the folder name as long as the folder has not been processed. When multiple folders are updated, IMatch processes them in order.

A quick scan of the 7,000 lines in the log file you've provided does not show that a file has actually been added or updated. The scan took 3 seconds.

Do you have background scanning disabled (Edit > Preferences > Indexing)? This would mean that you have to manually rescan folders when IMatch detects a change in the "last modified" timestamp.

Maybe you run any tools or software which causes the folder timestamps to change?

I see a number of "Cache image outdated or missing" for RW2 logs. Do you use cache purging or do you have deleted / moved the cache folder?

[rolandgifford]

I haven't deliberately turned off background scanning and can't see an option to do that on the Indexing tab in Preferences.

I'm not aware of anything running on my IMatch PC which would change folder timestamps. I do run mirroring software on a different PC which synchronises the indexed images to another PC. Perhaps that updates the source timestamp when it finds an image to copy, I'll check on that. It is a potential culprit given the large number of GPS changes I made recently.

The scan process (with the updating message bottom right) took a lot longer than 3 seconds.

If this flag is supposed to clear in the background, it could be that the number needing rescanning has built up as I haven't run IMatch for about a week. I'll leave it running to clear the backlog.

Thanks for looking

[Mario]

Sorry.  I meant Edit > Preferences >Background Processing (should have "import new and updated files automatically" enabled).

The AddOrUpdateFolder operation in the log file clocked at 2844ms.

Of course the UI is not immediate, there is always dome delay before the status bar updates, due to the parallel nature of the processing. Also, folder rescans invalidate categories and collections, which need to recalc.

The database has ~230,000 files and ~12,000 categories.
Several missing cache images were updated  or created.

There seem to be some expensive (slow) filters enabled. The Filter Panel reports delays of over 20 seconds when accessing the database, while collections are updated (> 20s to recalc for Reject because the database was super busy doing lots of stuff). The File Properties filter took 20 seconds to apply. Either large scope or system not coping too well.

230K files is substantial, and keeping categories, collections and filters updated will take at least twice as long as with a 100K database. IMatch copes, but it basically depends on how fast the database can read and write data. And some operations need to be performed in a "transaction", temporarily preventing other background threads from accessing the database.

Maybe disable or close filters you don't need so IMatch does not have to keep them up-to-date.

[rolandgifford]

Sorry.  I meant Edit > Preferences >Background Processing (should have "import new and updated files automatically" enabled).

That one is turned off as I don't want to import new files automatically. If there were the option to update automatically but not add new ones, I'd turn that on.

The reason it is off is because of my standard workflow:

I start with a folder containing all images from a birding trip.
I import these to IMatch and cull, label, GPS tag, and so on.

When I have a set I want to keep I copy all of them to an "Affinity" sub-folder for editing outside IMatch. I don't import this folder to IMatch at this stage.

I then work through this sub-folder and edit images. I move the ones I have done to a sub-folder off Affinity.

When I have finished editing I move all of them back to the Affinity folder and at that stage manually import to IMatch, propagate Keywords etc based on relations and job done.

There seem to be some expensive (slow) filters enabled. The Filter Panel reports delays of over 20 seconds when accessing the database, while collections are updated (> 20s to recalc for Reject because the database was super busy doing lots of stuff). The File Properties filter took 20 seconds to apply. Either large scope or system not coping too well.

...

Maybe disable or close filters you don't need so IMatch does not have to keep them up-to-date.

Hide Rejected takes a long time on my system, you have investigated previously and not found a reason. I'm happy to accept the "cost" of that delay as it helps my workflow a lot.

[Mario]

That one is turned off as I don't want to import new files automatically. If there were the option to update automatically but not add new ones, I'd turn that on.

Well that's unusual and explains why all this even becomes an issue for you.
It is usually not wise to deliberately hide new files from IMatch that way. Unless you are very disciplined and know that there might be files IMatch does not know about and is also forbidden to scan the pending folders automatically. Your choice of course.

[rolandgifford]

That one is turned off as I don't want to import new files automatically. If there were the option to update automatically but not add new ones, I'd turn that on.

Well that's unusual and explains why all this even becomes an issue for you.
It is usually not wise to deliberately hide new files from IMatch that way. Unless you are very disciplined and know that there might be files IMatch does not know about and is also forbidden to scan the pending folders automatically. Your choice of course.

I am very disciplined about files I create. That is why IMatch saying "something has changed in this folder" is a concern.

Having a file added (or changed) automatically isn't good from my point of view. I can perhaps have files added/changed in the background assigned to specific categories so that I can review them and move my Affinity/editing folder outside the scope of IMatch rather than a sub-folder. I'll think about that.

I have checked my mirroring software and that isn't what is changing folder modified dates.

[rolandgifford]

I moved my working/editing folder out of sight of IMatch, added two new categories for images added automatically and modified automatically (both chosen on Edit/Preferences/Indexing) then turned on background indexing. It took IMatch a little under 2 hours to catch up. I have done a recent full manual rescan. All of the flagged folders haven't been changed since that rescan.

6 images were set as having been updated automatically. The Keywords for all look as I would expect. None of these images were recent
No images in the added category

Separate to these new categories 13575 images are in the Updated Today category, it was zero before turning on background processing. The images I have random checked have the Keywords I would expect

I have uploaded the non-debug log file which I suspect is no help

I'm confused/concerned

[rolandgifford]

All 13575 images Updated Today are raw files of one type or another which are versions of master JPG files (under my relationship rules)

[Mario]

The log shows no AddOrUpdateFolder or AddOrUpdateFile.

It shows that the telemetry database 'C:\ProgramData\photools.com\IMatch6\config\imatch.tdb' is not writable for some reason.
Please delete this file while IMatch is not running.

There is also a warning that you have not entered your email and license key under Edit > Preferences > Application. This basically disables the built-in updater for IMatch. You should get warnings on screen about this.

Then there are some warnings about unsupported TIFF files (search for W> to find them).

Then IMatch launches several instances of ExifTool (apparently in preparation of indexing files). But the log is not complete and stops right there.

Tip: When IMatch checks a file for changes, it logs an AddOrUpdateFile. It also logs if the file was found modified or current right after.

[rolandgifford]

The log shows no AddOrUpdateFolder or AddOrUpdateFile.

Which is what I would expect as nothing had been added or changed since the last rescan.

However, the indexing rule I have defined to add modified files to a category did add 6 images to that category. Also, 13565 raw files were added to the 'Updated Today' category (standard as supplied)

It shows that the telemetry database 'C:\ProgramData\photools.com\IMatch6\config\imatch.tdb' is not writable for some reason.
Please delete this file while IMatch is not running.

Done. It wasn't marked as read-only

I have had this before. The shm and wal files were also present even when IMatch wasn't running. I assume that there has been an untidy shutdown at some point which has left them and it is this which is causing the problem

There is also a warning that you have not entered your email and license key under Edit > Preferences > Application. This basically disables the built-in updater for IMatch. You should get warnings on screen about this.

I have entered them and I am prompted when a new version is available. No warnings that they aren't entered

Then there are some warnings about unsupported TIFF files (search for W> to find them).

Odd as Affinity can open them and Windows Explorer correctly shows an icon for them

Then IMatch launches several instances of ExifTool (apparently in preparation of indexing files). But the log is not complete and stops right there.

At the point I copied/saved the log file IMatch was failing to clear the 'check flag' on three folders. Two of those have been changed recently and one not. The flag kept popping on and off for the old folder, the recent ones were staying flagged. I decided to close dow, saving the log file first. When I immediately restarted no folders were flagged as needing a rescan.

Tip: When IMatch checks a file for changes, it logs an AddOrUpdateFile. It also logs if the file was found modified or current right after.

My indexing categories for new or modified files is intended to show the same thing without having to check the log file. It appears to be triggered by something else

[Mario]

Odd as Affinity can open them and Windows Explorer correctly shows an icon for them

There are more TIFF variants than flies under the sun. Not all 3rd party libraries IMatch uses or thew WIC codecs it uses as fallbacks support all variants. Try re-saving the files and see if IMatch then handles them. You can send me a copy of one of the failed files so I can have a look t support email address.

As I said, I see no AddOrUpdate* log entries, so at least in this small section of the log you've uploaded, no folders were scanned and no files were added. But if there were files in the background processing queue from the last session, AddorUpdate* would not be logged because this was already done before. and IMatch is now processing the queue entries in the background.

The only reason I see for the folder "last modified" timestamp to change (and thus IMatch marking them as modified) is something external. The dir command shows the last modified date of a folder. If a file is added, removed or modified, the timestamp of the containing folder is updated.

[rolandgifford]

As I said, I see no AddOrUpdate* log entries, so at least in this small section of the log you've uploaded, no folders were scanned and no files were added. But if there were files in the background processing queue from the last session, AddorUpdate* would not be logged because this was already done before. and IMatch is now processing the queue entries in the background.

Understood from a logging point of view. Not understood why IMatch flagged them but I'll have to let that go.

Do you have any suggestion of why I suddenly gained 13575 raw files flagged in the updated today category but bypassing the log file and my new category as part of this rescan catch up. The files themselves haven't been changed and they are not flagged for write back. I assume that being in that category relates to IMatch database changes rather than external to IMatch.

The contents of these folders hasn't changed for a long time. Some static for months, others for years. Something else must be triggering that which I will also have to let go, I was hoping to stop it.

[Mario]

What does your updated today category do? Where does it get it's information from?
What does the file history of these files show? (History Panel).

[rolandgifford]

What does your updated today category do? Where does it get it's information from?
What does the file history of these files show? (History Panel).

Your Updated Today category as supplied with released IMatch. I haven't changed it

The History Panel simply says Updated, no details

[Mario]

Do you mean the Recently Updated > Today collection?
This collection is calculated based on the file history. And when the file history has an updated entry, the file record in the database was added or updated today. For a reason I cannot determine from remote.
IMatch does not write file records without a reason.
Run IMatch in debug logging mode (Help menu > Support) to produce a more detailed logfile. When there are SetFile entries, the file record was modified for some reason. Maybe what is logged before gives a hint.

[rolandgifford]

Do you mean the Recently Updated > Today collection?

It is under IMatch Workflow Categories on the Categories panel and I therefore call it a Category. Perhaps it is a collection in a confusing (to me) spot.

I've turned Background Indexing off again so that I can see if untouched folders again flag as needing rescanning. If they do I can then turn on debug logging before activating it again to check the log file as you suggest and hopefully find out why some files are flagged as being changed without me changing them.

I'm sure that IMatch doesn't write history records without that being triggered in some way, this is what concerns me. These are images that I have finished updating (subject to my occasional across the board 'improvements') so I don't want to see any new change history for them at all. Having a history update for the raw file but not the paired jpg is a similar concern, these two should mirror each other.

[Mario]

Ah, this category. It shows the same results as the Collection I mentioned.  It refers to the collection via a formula.
Let' see if you can catch something in the log.

AddOrUpdateFile is relevant, and the result logged below.
But there are dozens of other operations that may cause a file record to be updated.
But it the folder is marked as modified, it's timestamp has changed (you can see it with the dir command in a command prompt window) and IMatch must rescan to check for new or modified files. Ín debug logging mode more info will be logged.