Slow after latest update

[theresamarie1]

I am experiencing the same behavior I experienced a couple of weeks ago after updating the software. It's now 1 hour past the time I updated the software, and I'm still unable to use the tool as it's doing something called 'reading metadata' and is unresponsive on a very fast machine with lots of SSD.  The reading metadata jumps between 31 hours and 63 hours and has been at 1% for 10 minutes.   I thought this might be a one time thing from my prior post, but it seems not.  It's not using much CPU, like 4%, or memory (20%), nor is it using all of the disk bandwidth, but is unresponsive and unusable.   I do have a large database, which is showing 230K images now, although it's treating thumbnails like images so there is some redundancy.    On the top bar is says IMATCH Databased.imd5 (not responding).  I haven't added more data to the database this is just after updating the software and the new software needed to update the database and whatever else it needs to do.
Helpful suggestions welcome.

[Mario]

Please always include the log file in ZIPped form when you report any problem with IMatch.
This gives us a minimum of information to work with. See log file

it's doing something called 'reading metadata' and is unresponsive on a very fast machine with lots of SSD. 

IMatch is ingesting images. 
If the time estimate varies that much, file sizes or formats vary.
What kind of files are you processing, where are they stored?

Are you a new user or an old user who has upgraded from IMatch 2023?
Did you make an exception for IMatch and the database folder in your virus checker?

I do have a large database, which is showing 230K images now, although it's treating thumbnails like images so there is some redundancy.   

Not sure what you mean by "thumbnails as images". IMatch manages one thumbnail per file in the database, and ten for each video.

I'll know more when you attach the log file.
You can make a copy of it while IMatch is running. Open the TEMP folder on your system and select the log file and press Ctrl+C, Ctrl+V. Then ZIP the copy and attach. See log file.

Ps.: 
When you refer to a "previous post", please link to it. My memory is good, but I cannot remember everything. Your previous post may be from weeks ago!

Also, please press <Enter> once in a while. Your post has no paragraphs and is a wall of text when read on a smart phone.

[theresamarie1]

There is no imatch log file in my temp directory, in fact my TEMP directory is empty.
I did copy the log file from the support tab, here is a link to the zipped file: https://drive.google.com/open?id=1VWlyqGbjoX2r8LcHny0BnMTcD0dIU6CT&usp=drive_fs

I have used Imatch for a long time.. not sure when I began, but it was more than a decade.

There are many file sizes and formats for cameras I have used over 25 years of images.  Many different raw formats, TIFs and JPGs.

The files are all on fast Samsung SSDs, multiple drives.    64GB of memory, Threadripper AMD 16 core processor.

Well, the .THMs are showing as individual files in the database in some cases.   I don't recall it doing this in the past but now seems to have expanded some folders to show .THMs as files in my 2002 and 2003 raw image folders.

My post a couple of weeks ago is on March 15th when I first upgraded to 2025.  I don't know how to reference the post in a link.

Terri

[Mario]

There is no imatch log file in my temp directory, in fact my TEMP directory is empty.

Most likely you looked in the wrong TEMP directory. IMatch does not run without a log file.
Open your (not the Windows!) TEMP folder by typing

%TEMP% into the Windows address bar.

Or open it from the Dashboard. The Info Panel has a button for this.

.THM files were used by older digital cameras, when I recall correctly. The files contain their own metadata, or, at least for older Canon models, they contain the metadata for the RAW file. If you don't want these files in the database, you can exclude the file format: File Formats

I don't know how to reference the post in a link.

Right-click the data above your post to copy the link into the clipboard, then paste into your post so others know which post you refer to:



You can also copy & paste the link shown in the address bar of your web browser. Whatever is more comfortable.

Note: You can attach files up to 5MB in size directly to your posts.
No need to use Google Drive for a 1MB file and force me to give Google my IP address and have to clear all the 400KB of cookies Google Drive has planted in my browser.

[Mario]

Now, having all of that out of the way, let's look at your log file.

The first thing I see is a very large number of folders which have been found to be externally modified and have been scheduled by IMatch for indexing.

I'm sure you have read the release notes, but here is the info why this can happen again for your convenience:

https://www.photools.com/release-notes?search=2661

As it happens, many folders on your system were detected and are now processed. Search your log file for FolderSweeper: Folder to find the names of all affected folders.

The log file contains 579 warnings in only 5K lines. A lot.

ExifTool reports issues like
"Error creating file: C:/Users/Terri/AppData/Local/Temp/imt_et_2540172904629358.xmp" which is strange and rare.
It also reports many issues with broken maker notes, invalid CanonSettingsData  etc. Not uncommon when many (older) images are indexed which have been processed by who-knows-what in the old days.

A larger number of "invalid file formats" or "unsupported" file format entries.

Many warnings from WIC about unknown file formats, unsupported RAW formats and stuff.

I have attached a text file with all warnings and errors below, which shows you which files were affected.

While processing some of your files, the memory consumption of IMatch rises to 25 GB, which is a lot. But your system has 64 GB and 59GB were available when IMatch started, so this is OK.
My guess would be that LibRaw or Windows WIC choked on some of your files and allocated tons of memory. Broken or unsupported files sometimes cause this effect.

Of course each file failing to load slows down IMatch, and sometimes, when files are failing to load, it takes Windows 10 or 30 seconds to figure out that it cannot load a file.

Your PC reports 32 processor cores, which means IMatch will index at least 32 files in parallel, depending on the performance profile you have enabled.

If the IMatch and/or the WIC subsystem become unresponsive while processing so many "problem" files in parallel, I suggest you reduce the performance profile (Edit > Preferences > Application) to the "Low" setting until IMatch has chewed all these files.

Also, again, make sure you have made an exception for IMatch in your virus checker.
TODO #1 on this list: Welcome to IMatch 2025!

[theresamarie1]

Mario, I appreciate the reply and showing me how you like your feedback and I have gone through the log without finding many solutions (maybe get rid of a dozen errors).    A question:  Will this happen every time I upgrade the software?   Imatch was unusable for about 2.5 hours.

On the proliferating .THM files.    You really haven't said what I can or should do with these?  Should I just delete them?       Are they no longer used at this version of software?   I notice them only in very old images > 20 years old.   None of the newer files seem to have them.

So back to the log files.   YOu mentioned I have a lot of errors:

The very first item that is listed as failing is something to do with not finding a dictionary for language 'en'.   Nothing for me to do here.

I found 3 things that said 'Warning (minor)' and took minimal time.   These had to do with Maker Notes or Camera Settings, but nothing for me to do.   

Most of the  errors on unsupported file types  were just on TIF files.    While I didn't go through them all, I went through a number of them and each file was opened fine in photoshop.   So not sure why Imatch doesn't recognize them.    And not all of these are large, some are fairly new.     

THere were about a dozen .CRW files that seem corrupted.   They are old from a Canon D60, and have no idea what's wrong as they don't open in photoshop either.   But based on the timestamp, that didn't take any time, less than a second overall to get through them.     These I can just ignore or remove.

So I'm not left with a lot I have any idea how to fix.    Maybe a dozen files that didn't seem to take much time. 

In addition to these things, I ran diagnostics numerous times and I get  warnings on 'Failed to load parent folder [1274] with error 10020 and a folder is listed.   There are 66 of these but for every one of these that I've tested the folder opens fine in MS Explorer.   Files seem fine.    Security access settings seem fine.

Terri

[Mario]

A question:  Will this happen every time I

Did you read the release notes I linked to? They explain that IMatch 2025 may find some (or, in your case, many folders) which are out-of-date and need to be rescanned. As always, once a folder is up-to-date, IMatch does not need to rescan it until it changes by external events.

This was a one-time event. If IMatch has finished processing all the pending folders, you're good.
Why the simple task of recxanning made your PC unusable, I don't know. From hardware throttling under prolonged load to a virus checker interfering to keeping the database and files on the same disk - we have discussed the potential reasons many times over the past weeks. Only a very small number of users experienced this.

The very first item that is listed as failing is something to do with not finding a dictionary for language 'en'.  Nothing for me to do here.

Of course. You can just install a spell checker dictionary for your language. IMatch will show prompts for this, unless you have already disabled them with "Don't show again". See The IMatch Spell Checker

Most of the  errors on unsupported file types  were just on TIF files.    While I didn't go through them all, I went through a number of them and each file was opened fine in photoshop.  So not sure why Imatch doesn't recognize them.    And not all of these are large, some are fairly new.   

There are literally dozens of TIFF format variants. None of the open source of commercial image libraries supports them all.
When you can open them in Photoshop, just re-save them in a standard TIFF variant. I have many TIFF files myself, and many small libraries which archive in TIFF use IMatch successfully.

THere were about a dozen .CRW files that seem corrupted.  They are old from a Canon D60, and have no idea what's wrong as they don't open in photoshop either.

I think this is what many RAW file users will experience in the future. Canon themselves have retired (abandoned) support for older CR* formats in their software. And Canon has produced maybe 50 different RAW variants over the years. Nobody knows how long Adobe, RAW processors or Windows will keep support RAW files 20 years old. RAW is not a file format allowed for long-term archival for good reasons.

'Failed to load parent folder [1274] with error 10020 and a folder is listed. 

Never have seen this before. 10020 means "Oid not found" which indicates that you have folders in your database which have a link to a folder that no longer exists. What did you do to produce this problem? Any idea?
That the folder exists in the file system is fine. But does it exist in the database? Did you relocate folders recently? Moved folders around?

[HG]

@theresamarie1, I had similar issues, maybe different root causes. First, what was really bringing benefit, was to let db, cache and images running in different SSDs. Now, my db is running on an NVMe on drive C. Images, on  an SSD 4T. Nothing else, just images. For the cache I bought a new SSD 250GB. Now cache is located on the new drive. There is nothing more, just cache. For me, this was the first step to fix my delays. In a second step Mario identified a time consuming script. But I think, this is different in your environment. DB and cache I excluded from defender.
Maybe this is a good starting point for you as well. Good luck! Regards HG

[theresamarie1]

A question:  Will this happen every time I

Did you read the release notes I linked to? They explain that IMatch 2025 may find some (or, in your case, many folders) which are out-of-date and need to be rescanned. As always, once a folder is up-to-date, IMatch does not need to rescan it until it changes by external events.

This was a one-time event. If IMatch has finished processing all the pending folders, you're good.

Yes, I did read the release notes.    But while my folders moved to a new drive, and I relocated things, I don't believe data changed.  Some of this data is just 20 year old groups of files, more or less archived.  So that's why I'm confused on the time.

At this moment, I have some number of folders which continue to have the little rescan icon over them, in spite of rescanning many times, and it seems each time I open IMATCH, it will allow me to clear one of more, but not all of these folder rescan icons.  Most of the time when I rescan, it's very fast and nothing really seems to happen. 

  But just now I did a rescan of one of these folders on one of my drives, where it took 15 minutes to read 3000 files, and now it says it's adding and updating those 3000 files! Why, I have no idea.  Data has not changed, folders have not changed, Meta data has not changed. 

I'm looking at the log and the status screen, and it's moving at a snails pace.  All this data is on fast SSD and a different SSD than the database.  Presumably, I had already done this the day it rescanned everything for the software update, and no changes since, but now it's doing it again. 

I get tons of geolocation errors on the initial reading of meta data complaining it doesn't have enough information (my original 1DS did not have GPS) and then a few messages on not being able to create a thumb.  Now it completed (maybe took another 5 minutes so 20 minutes overall) and the blue rescan icon is still over the folder!  And if I rescan again, something blinks on the screen but nothing happens and the rescan icon is still over the folder.

One of the things that's disappointing is that some of my catagory tagging seems to have gone away on some number of images.  In place of the actual catagory (all of my files were tagged with specific locations and or formal names) they now just have an 'edited' catagory checked.  That's really worrisome as it's hard to figure out 20 years later, exactly the location of this or that waterfall or covered bridge or other item that I had to hike to see.  It's what I use Imatch for.. to keep track of what images I took and where.

That the folder exists in the file system is fine. But does it exist in the database? Did you relocate folders recently? Moved folders around?

I did relocate data due to expanding storage and needed to reorganize it, but I did this and relocated files to new drives.  Somewhere along the line I seem to be losing data.

[Mario]

But while my folders moved to a new drive, and I relocated things, I don't believe data changed. 

IMatch has found and logged which of your folders contained at least one file with a "last modified" newer than the folder "last modified". That's all I can say. IMatch detected the folders and rescanned them. If you are unsure, check the time stamps of files in the folder to see what the newest files are.

Have you checked that IMatch in fact has processed all files in the queue?
When it has not finished, it will automatically continue when you start it again.
See Info & Activity Panel or Dashboard.

Most of the time when I rescan, it's very fast and nothing really seems to happen. 

This is strange.

The first entry is:

Folder [24] 'D:\Photos_1\2005\1DS\' 'last modified in db' (3/14/2025 8:21:27 PM) timestamp is older than the last modified file 'AA4W2146_on1.psd' (11/23/2009 8:31:32 PM). 

which seems wrong. Because the folder is newer than the psd file, but IMatch things otherwise.

Can you please check the properties of the folder and the psd file in Windows Explorer and provide the "last modified" timestamps as shown in your reply? IMatch should not consider the file newer as the folder. I have not seen this before.

Which Windows version do you use? Which time zone?
Is this a FAT-formatted drive or NTFS?

Insufficient information to determine geolocation

This seems to be an ExifTool issue. ExifTool has a limited copy of the GeoNames.org database no included and can perform reverse geo-coding automatically when instructed. But IMatch does not request this feature since it is much more limited than the reverse-geocoding IMatch can perform. I wonder why ExifTool is still issuing this warning.
I have posted a question in the ExifTool forum.

One of the things that's disappointing is that some of my catagory tagging seems to have gone awa

This is very unlikely. Category assignments are stored in the database. They only go away when you manually change them or when a file is removed from the database.
"Edited" is not a category IMatch creates or ships with.

You mentioned that you have relocated files. 
Moving files between disks is sometimes prone to user error, when a relocate is performed wrong or files are added multiple times to the same database, instead of relocating.
Check if you accidentally added the same folder(s) multiple times to your database. Which would create multiple copies of the files, some with the original categories, and some without. That's a typical tell tale for relocation done wrong or not done at all after moving files to other disks.

[theresamarie1]

Hi Mario, thanks again.  see below

This is strange.

The first entry is:

Folder [24] 'D:\Photos_1\2005\1DS\' 'last modified in db' (3/14/2025 8:21:27 PM) timestamp is older than the last modified file 'AA4W2146_on1.psd' (11/23/2009 8:31:32 PM).

which seems wrong. Because the folder is newer than the psd file, but IMatch things otherwise.

Can you please check the properties of the folder and the psd file in Windows Explorer and provide the "last modified" timestamps as shown in your reply? IMatch should not consider the file newer as the folder. I have not seen this before.

The folder is listed as created sept 18, 2022 and last modified on 11/27/2024.

The file you listed was last modified Nov 23, 2009 (which makes sense based on how old these files are).

Which Windows version do you use? Which time zone?
Is this a FAT-formatted drive or NTFS?

I use windows 11 professional but have uninstalled the 24H2 update due to issues. 
The drive is NFTS as are any of my internal disk drives (these are also all SSD).  I do have some spinny disks for backups. 

This is very unlikely. Category assignments are stored in the database. They only go away when you manually change them or when a file is removed from the database.
"Edited" is not a category IMatch creates or ships with.

You mentioned that you have relocated files.
Moving files between disks is sometimes prone to user error, when a relocate is performed wrong or files are added multiple times to the same database, instead of relocating.
Check if you accidentally added the same folder(s) multiple times to your database. Which would create multiple copies of the files, some with the original categories, and some without. That's a typical tell tale for relocation done wrong or not done at all after moving files to other disks.

I would never claim to be infallible.  Dealing with this much data is a chore, thus trying to use a tool. 
I do have some exported catagory .IMCS files I used back in 2019 when I upgraded Imatch after a lapse for some years and you gave me instructions for the category export and recombine with the data (I forget details now).  I kept those files.  I can look at it in a text editor but it has what looks like some hex words that are probably meaningful in some way that isn't readable.  Is there a way for me to look at this and determine the category I had a file in?  The translation I did worked, and I had very few (few thousand) uncatagorized files after that update but now I have something like 100,000 unassigned files after my drive updates and moving to 2025.  Whether it was me or something else, this is far too many for me to look up one at a time, plus after 25 years in some cases, I'll never or remember where the image was taken.  I took many nature photos in Vermont back in the woods seeking out obscure waterfalls or old and now destroyed covered bridges. 


Thanks for the help,
Terri

[Mario]

The folder is listed as created sept 18, 2022 and last modified on 11/27/2024.
The file you listed was last modified Nov 23, 2009 (which makes sense based on how old these files are).

Hm. IMatch has the folder as last modified on "3/14/2025 8:21:27 PM", and the file at the 2009 date. So it should not consider the file newer than the folder. IMatch simple gets the timestamps from Windows, then uses the Windows function CompareFileDateTime(folder,file) to check whether folder is older, and if so, schedules the folder.

I need to reproduce your time stamps and see what IMatch does.

The category export files contains everything, including category colors (your "hex words") and, if you have enabled this, the file names associated with each file. Search for the file name and then lookup the category.

but now I have something like 100,000 unassigned files after my drive updates and moving to 2025.

Did you check that you have not added the same folders multiple times (from the old drive and the new drive).
I just had a similar case in email support. A user had to replace his disk and then did not use relocate to tell IMatch that the files are on a new disk, but instead added all files again - giving him two versions of each file. One with categories and Attributes, the other without. Removing the extra copies from the database and relocating the folders to point to the new drive fixed the issue.

Update

The log file provided was not in debug logging mode, which means that some detailed data is missing.
The "timestamp is older than the last modified file" message can also be logged if IMatch found a file that is not yet in the database during the initial deep scan. In that case it does not matter that the file is older than the folder, the folder will be rescanned to ingest all files not yet in the database. Can this be the case?