Folders marked for Rescan

[rolandgifford]

No significant improvement by removing all of the Filter Panel bars (apart from the stored filter). Perhaps a couple of seconds but no more than that.

Image of folder icons attached. I have Experimental turned on

[Mario]

The translucent icons with the "Update" blue circle are folders which are scheduled for a rescan since IMatch has noticed a change in the "last modified" timestamp of the folder on disk, or Windows sent a "modified" message.

[rolandgifford]

The translucent icons with the "Update" blue circle are folders which are scheduled for a rescan since IMatch has noticed a change in the "last modified" timestamp of the folder on disk, or Windows sent a "modified" message.

Odd, it is more than 90% of my folders, most of which haven't been touched (apart from backups) for a very long time. This is a change within the last week. I would have noticed when uploading my current trip photos a week ago.

[Mario]

Since we are now talking about a totally different thing than "slow load when filter is active", I have split your original thread for you into two threads.

I can only tell you what I see.
Did you experience a crash? Did you forcefully clear the processing queues?
Which other applications do you use?

If this is the state the database comes up with, the "last modified on disk" timestamps of the folders on disk don't match the timestamps recorded in the database. IMatch will enqueue the folders and process them in the background.

Did you make any change to the default settings under Edit > Preferences > Indexing (screen shot)?

[rolandgifford]

Did you experience a crash? Did you forcefully clear the processing queues?
Which other applications do you use?

Did you make any change to the default settings under Edit > Preferences > Indexing (screen shot)?

No crash
No clearing queues
The only other applications used on this PC are Firefox and Windows Media Player

I don't remember making any changes to settings, certainly not for a very long time if I did. Screenshots attached.

I will no doubt have to do a full database rescan to clear this but I'm holding off till you say I should do that

[rolandgifford]

And Macrium for backups. I have done a backup of all my images within the last week

[Mario]

Select the top-level folders showing the circle and press Shift+F5 to rescan it.
The log file will contain many AddOrUpdateFolder and AddOrUpdateFile entries, each telling you if there were changes or the folder / file is current.
If nothing was really changed and only the folder timestamp was modified, IMatch will finish quickly. If file timestamps have changed or there are new files, IMatch will update the database accordingly.

[rolandgifford]

The initial scan of my "Waiting Sorting" branch (116,000 images) took 22 minutes with several delays of around a minute where the counter stopped. At the end of that there was a message about 6 files followed by a Importing Metadata dialogue for 7439 files which IMatch says will take in excess of 6 hours.

There are 6 files named in the log file from the folder I'm currently working on.

I'm running normal logging and the snapshot I have just looked at doesn't contain any AddOrUpdate strings.

[rolandgifford]

The 3439 images having their metadata imported is every MP4 file (apart from 12 - which is odd). They weren't flagged as being out of sync before the rescan and I keep very up to date writing back all images requiring writeback.

Hopefully this doesn't damage the data in the database for them.

[rolandgifford]

The rescan process will hopefully be complete sometime overnight and leave me with a doubt.

The rescan part of the process was reasonably fast with nothing new found as I would expect.

However, all Metadata had been imported from all MP4 and RW2 files (so far as I can tell from displayed file names and counts). All files were in sync before the rescan. I write back regularly as a backup but the key/master data is within the IMatch database. I don't maintain RW2 metadata directly, it is propagated from the JPG master.

Why did the Rescan import metadata from images which were in sync? Has the import potentially changed anything (I don't update metadata anywhere other than within IMatch)? How do I check?

[Mario]

Why did the Rescan import metadata from images which were in sync?

IMatch will rescan files if the "last modified" time stamp in the Windows file system differs from the time stamp IMatch has in the database. Else it does not re-import the file.

Maybe something on your system causes file stamps to change.
The log file records which files were current and which files had to be rescanned.

[rolandgifford]

Why did the Rescan import metadata from images which were in sync?

Maybe something on your system causes file stamps to change.
The log file records which files were current and which files had to be rescanned.

Where is the log file? I closed IMatch without saving a snapshot first. I now have a small number of images that won't clear from needing writeback but haven't investigated fully.

Something which changes the timestamp on evey MP4 and raw file seems unlikely

[Mario]

Where is the log file?

See log file, especially TEMP.

Note that IMatch keeps only the last and prior log files. Backup them before restarting IMatch.

[rolandgifford]

I should have thought of looking at the help, sorry

The files are in %TMP% and not %TEMP% on my system and the BACKUP log file is from 4 days ago, so isn't the previous log file. No useful file names in the log.

I'm now in the position where I have no idea what changes have been made to either my images or my database and wondering if a full restore from before I started editing/culling my latest trip would be the best/safest option.

[rolandgifford]

Why did the Rescan import metadata from images which were in sync?

IMatch will rescan files if the "last modified" time stamp in the Windows file system differs from the time stamp IMatch has in the database. Else it does not re-import the file.

Maybe something on your system causes file stamps to change.
The log file records which files were current and which files had to be rescanned.

I have Googled and found a probable cause and have subsequently "fixed" this but I'm confused.

There are a number of posts around relating to Windows taking account of daylight savings and presenting modified dates adjusted for DST, therefore shifting by one hour on either side of that event. In my case, every image that I looked at comparing live files and backed up files had modified dates exactly an hour apart which would be consistent with this. Nothing has changed the files but the time of last change is presented shifted by an hour.

Changing Windows settings so that Timezone on the PC used for IMatch isn't handled automatically and back again appears to have removed this one hour time shift.

As the rescan flagged some images as requiring write-back, some of which won't, I'm wondering what to do with my IMatch data and my image files. I know which images won't write back but I don't know which ones did, or what had been changed for those images. There are only 20 or so of them so I may decide I don't care enough to restore a backup.

[Mario]

Windows file system timestamps are in UTC and IMatch manages file system timestamps in UTC (except for old FAT disks, which used local time). Timestamps you see in Windows Explorer or in IMatch are usually formatted using your local time zone and also account for DST, based on the user locale.

The CompareFileTime Windows function IMatch uses to compare the UTC timestamp stored the database with the UTC timestamp on disk reported by the Windows file system is not affected by time zone offsets or DST.

[rolandgifford]

Windows file system timestamps are in UTC and IMatch manages file system timestamps in UTC (except for old FAT disks, which used local time). Timestamps you see in Windows Explorer or in IMatch are usually formatted using your local time zone and also account for DST, based on the user locale.

This is occasionally broken in the latest patch version of Windows 11. It was neither caused or fixed by the patch this week as the problem I had happened before patching and I have seen a different linked problem since.

The different linked problem since was the current time on the taskbar showing one time and the clock on the lock screen showing one hour sooner. The problem which started this thread was the same sort of thing but different as all of my latest modified times suddenly flipped back an hour, making them out of sync with IMatch.

I have now recovered from this problem using backup restores and rescans, a process which has raised questions and is the reason for this post.

I assumed that a simple rescan against the new dates (without a restore) was probably the wrong solution as at some point the dates will correct themselves and I'll have to rescan again. Having correctly dated images made sense to me.

The first restore was from a Macrium backup taken before I started loading and editing my latest trip. This was before the date/times went wrong. I restored all images and the database. After the restore most files needed scanning/updating and at the end of the rescan process I was left with about 70,000 images (out of 200,000+) which needed metadata writing back. Hovering at random on the pen icon told me that most of these were GPS related (Dest and Direction rather than core GPS data for the image).

I decided this wasn't good so restored again from a more recent backup being a folder copy to a different drive on another PC. Again, all images and the database. Again the rescan appeared to check every image but at the end of the process there were only 10 images requiring write back. Some of these needed work with embedded/sidecar differences and the like.

There was an IMatch upgrade between these backups.

Why did IMatch think that all (or a majority of) images were out of sync and needed rescanning even though the images and database were backed-up/copied at the same time?

Why did so many images need metadata writing back after the first restore even though all metadata had been written back before the backup? I'm assuming this is a result of a different IMatch version after restore but am asking anyway. If it is an IMatch version issue, should I be writing back metatdata for all images after a new version to avoid this?

Should I be backing up/restoring something else to make the two match?

[Mario]

Why did IMatch think that all (or a majority of) images were out of sync and needed rescanning even though the images and database were backed-up/copied at the same time?

There is not much ot thinking going on here.
When a database is loaded, IMatch starts a background task that compares the "last modified on disk" timestamp stored for each folder with the "last modified on disk" timestamp reported by the Windows file system. If there is a difference, the folder is marked for a rescan and processed by another background task.

When a folder is rescanned, IMatch checks for new files. IMatch compares the "last modified on disk" timestamps for each file in the database with the "last modified on disk" timestamp reported by the Windows file system. If there is a difference, the file is marked for rescan and enqueued.

Apparently, you seem to be the only user affected by this particular problem. At least I don't recall any similar reports and I have not experienced something similar on Windows 11 when DST became active here at the end of March.

Why did so many images need metadata writing back after the first restore

When you point the cursor at the write-back pen in the File Window, IMatch lists the first 10 tags that are marked for write-back.
Which tags are listed? Some timestamps perhaps?`

[rolandgifford]

Why did IMatch think that all (or a majority of) images were out of sync and needed rescanning even though the images and database were backed-up/copied at the same time?

There is not much ot thinking going on here.

Or reading as you seem to have missed that I answered your question in my post and seem to have missed my questions altogether.

I shall try again more briefly.

I restored from a Macrium backup of all images and database which were fully in sync at the time of the backup.
After the restore IMatch said that all/most needed rescanning and after that rescan found a large proportion were actually different. Hovering at random on the pen icon told me that most of these were GPS related (Dest and Direction rather than core GPS data for the image). There were a small number of InstanceID. I didn't see any timestamp differences.

Why were the restored images/database out of sync? Why were they different? How do I avoid this in the future?

Rather than writing back I chose to restore a different backup taken as a folder mirror to a drive on a different PC on the network. Again these restored files were out of sync but this time not significantly different.

What do I need to do to make sure they are in sync when restored?

[Mario]

I restored from a Macrium backup of all images and database which were fully in sync at the time of the backup.

Did Macrium also restore the original timestamps?

after that rescan found a large proportion were actually different.

so IMatch was right to rescan them?

Why were the restored images/database out of sync?

Out of sync with what?

Why were they different?

How can I know. IMatch looks at the timestamps. Timestamps no match => rescan.
If the checksum calculated for the file does not match the stored checksum in the database, full reprocessing. All is logged to the IMatch log file. Did you read the log file?

Search for AddOrUpdateFolder and AddOrUpdateFile and go from there.
It shows which files were rescanned and if they were "current" (unchanged) or modified.

I don't have any of the information. I don't know what Macrium does with file system timestamps or how you made your backups (file-based versus image-based)? File based restore may very well change the last modified timetamps, image-based may not.

I recommend you contact Macrium support when you need information or advice regarding one of their products.

What do I need to do to make sure they are in sync when restored?

See above.
How can I know? I don't have your backup or your computer here in the lab. I haven't seen the IMatch log file from these sessions. I literally have nothing to work with. I have explained when IMatch rescans folders and under which conditions it rescans files It's very easy.  I have nothing to add because I have no data.
When comparing file system timestamps via the Windows function CompareFileTime, Windows is usually right and tells IMatch if or not the two time stamps (database, file system) match or not.

[rolandgifford]

How can I know. IMatch looks at the timestamps. Timestamps no match => rescan.
If the checksum calculated for the file does not match the stored checksum in the database, full reprocessing. All is logged to the IMatch log file. Did you read the log file?

Search for AddOrUpdateFolder and AddOrUpdateFile and go from there.
It shows which files were rescanned and if they were "current" (unchanged) or modified.

I did look at the log files, no AddOrUpdate entries at all. Normal, not debug, logging as I wasn't expecting a problem.

Fortunately (?) the folder that I'm currently working on has just had the folder icon changed to requiring rescan. No backups/folder copies or anything odd involved. I'm just deleting some images and removing a single Keyword from ones that I'm keeping. Regular metadata write-backs. I changed to debug logging, restarted and rescanned.

Every file in the folder is mentioned in the log file as having been scanned and that the File Is Current. There are no details telling me what difference prompted the rescan.

I have looked through my folders and there are two others (which I haven't been updating) which are also flagger for rescan.

I've uploaded the log file

[Mario]

This log file has 21,000 lines.
A quick glance shows that many, many files have been checked for changes, but all are reported as "current".
No rescans. Not sure in which session this log file was produced. Since folders were rescanned to check for new or updated files, I know that the file system timestamps of these folders did not match the timestamps in the database. Or Windows has sent "Folder modified" events to IMatch.
IMatch scanned the folders, apparently found no files that were modified or new, and that's about it.

Since the log has 21,000 rows and there are many, many AddOrUpdate file logs, I cannot analyze each and every one, sorry. My time is limited. From what I see, the log looks perfectly normal. IMatch found folders with modified time stamps or Windows sent "Folder modified" messages, IMatch reacted. All perfectly normal.