Problems, perhaps WIC related, with rescans

[jonz]

I've been having a persistent problem with rescans where IMatch seems to hang and wait for a timeout when doing rescans (rescans drag on forever, it seems).
I am attaching both the log file entries for a rescan and the WIC diagnostics. I'm a bit frustrated with the Windows WIC installation. I had purchased and used the FastPictureViewer codecs but I read in Mario's documentation that those were no longer developed, so I tried uninstalling them. Some of them (including the TIF codec) still showed up in the WIC diagnostic. So I currently have the codecs from FastPictureViewer installed, since I don't seem to be able to get rid of them.
It may be pertinent to say that the TIFs are generated by Silkypix Developer Studio Pro. I certainly don't have any trouble opening them in other programs etc.
Any help much appreciated, thank you.

[Mario]

The log file is not complete so important info is missing.

I see that IMatch reports database locks (which would be unusual for a database 20K files, but might be OK for 200K files - I don't know more because the log file is incomplete). Always ZIP and upload the entire log.

I also see

Error loading F:\jonz ssd\photos\2019\2019-09\2019-09-27\2019-09-27__DSF1699_color.tif with error 1 'Unknown image type.'

which means that the reference TIFF implementation used by IMatch (TIFF files are not processed by WIC, only RAW files) does not know how to process the file. This would make this a very unusual TIFF file, because TIFF is pretty much standardized for 20 years.

Are these TIFF files with more than 32,000 pixels in either dimension? A special type of TIFF? Unusual color space or something?
I would need the full log file and a sample TIFF file which exhibits this problem.
Double-check your SilkyPix and make sure that these TIFF files are stored according to the TIFF standard.

[jonz]

I've been using the Japanese program (SilkyPix) for about three years, but it's not inconceivable that it writes weird TIFs. I haven't had trouble in the Adobe or Affinity programs with TIFs placed from Silkypix, but anything is possible. It seems to me that even though tifs are "standard", there's still a bit of funkiness in compression types etc.
I am uploading the full log file. If you send me instructions I will also forward a sample TIF. They are 4x6k so generally about 36-40 Mb.
Thank you Mario

[Mario]

Your database has about 250,000 files. So database operations may be slower at times. Is the C: disk an SSD or a normal spinning disk?

IMatch needs 0.5s to load a JPEG file from F:\... this seems a bit slow. Is this an external disk or a remote network disk?
But it also needs 0.5ms to load a cache JPEG image from your C:\ProgramData\photools.com\IMatch6\previewcache\ folder.
And that's not really fast either. Typical times for 36 MP files is about 0.2 seconds from a SSD.

You can send me a TIFF file to support email address
Please include a link back to this topic because I get many emails per day and I have always a backlog.
The IMatch 2020 Beta Test is running and I'm really busy currently.

[jonz]

Both drives are SSDs on a fast system. So nothing is on a hard drive.
I don't see why my load times are slow. On the other hand, problems may be the large database or the big cache (47Gb / 52k files).
Looking through the log file it appears to me that the problem with TIFs is intermittent, so maybe that is a locking problem. I may have been on the wrong track with the codecs, though I don't see why I'm getting such slow load times. The image files (jpgs with a few RAFs) are mostly Fuji variety plus some from a cell phone.
Thanks. Looking forward to 2020.

[Mario]

The number of files in the cache is usually not relevant for the load times. Unless something strange in Windows kicks in and somehow slows down the cache access.
IMatch automatically spreads cache files between many folders to avoid the issues Windows has with many files in a folder.

On my 5 year old PC typical load time for cache images (~ 16 to 48 MP) is 0.1 to 0.3 seconds. So a constant of 0.5 seconds is a bit odd. Nothing to worry, though. IMatch loads cache images in advance and in the background so the absolute time is irrelevant - unless you scroll faster than IMatch can load the images.

An error message like "invalid file format / unknown format" is usually not intermittent. Unless the files read by IMatch are still being written by another application (?)...but that would be odd because usually the files are locked for the time of writing and IMatch cannot touch them.

[jonz]

After thinking about this problem I thought the way to test would be to create a new vanilla database. So I did that and added a month's worth of files to it as a test, in this case about 1600 jpgs, rafs, and tifs.
Adding these files to the new database took less than two minutes (log attached). In my normal, working database the same operation would take 5-10 times longer. Where should I look? Obviously, the vanilla database had no relations defined or other user-data...but I don't know where to start looking for where this slowdown problem is. I of course run database diagnostics and optimisation regularly. I would like to get to the bottom of why my working database is comparatively so slow.

[Mario]

Adding 1500 files is nothing, even if the database has already 250,000 files.

You can repeat your test with your regular database and then ZIP attach the log:

1. Start IMatch fresh with your big database
2. Switch to debug logging via Help menu > Support
3. Add the f1500 files.
4. ZIP/attach.

Things to check:

Many data-driven categories which you don't need? Remove them.
Many expensive formula-based categories? Ditto.

Many complex or expensive file relations? This can bring down IMatch's performance a lot.
Check for relation rules which need to search many folders or deep hierarchies to find versions.
Sometimes users just use the "Search entire database" option and then wonder why IMatch becomes slow.

Immediate write-back enabled (this causes IMatch to ingest, write-back and re-ingest each file during folder scans, which means a lot less speed).

[jonz]

Ok so in my working database I don't have any data-driven or formula-based categories.
I have four relations active, but they only point to [master folder - specified folder], and not even sub-directories. I looked through the relations  carefully and I don't see an obvious problem, but something is certainly slowing things down.
The scan using the working database and adding the same 1600 files took about 8 minutes and a little more, versus about 2 minutes for just adding the same group of files to a virgin database.
Immediate write-back is not enabled.
I did do a full uninstall of the FastPicture codecs, for what that's worth.
I am attaching a debugging log that records adding the 1600 files (actually sent it to you by email, because it's too large).
Probably this is one of those problems that has several things going on at the same time, but I'm not having much luck figuring it out alone. I appreciate the help.

[Mario]

Ok so in my working database I don't have any data-driven or formula-based categories.

IMatch by default creates many data-driven categories and formula-based categories under "IMatch Standard Categories" and "IMatch Workflow Categories". But usually these don't interfere. IMatch is very fast at updating them.

No log attached.

[thrinn]

(actually sent it to you by email, because it's too large).

It is always best to ZIP the log. In most cases the reduction in size is enough to attach it here.