iMatch freezes constantly

[javiavid]

I don't know if I'm doing something wrong, but it's constantly happening to me.
I have tested on 2 computers, with and without connecting the library hard disks (nas gigabit), with and without cache, with small and large folders.

My library is about 35,000 images and the computers are powerful i9 32 GB, the database is on an SSD disk.

I attached the debug logging.
Logging has 2 minutes and there are 4 frozen.
-16: 26, 16:27, 16:28, displaying and activating some panels in a folder of 18 images with cache and offline.
-Other at 16:28 for longer (when selecting an offline folder of 292 images)

Thank you

[Mario]

Log file is missing.

I understand you are working with files on NAS? Which operations do you perform?
NAS are sometimes cause of trouble. Many are not designed for operating under stress for extended times, e.g. IMatch writing back hundreds or thousands of files.

See also this article in the IMatch knowledge-base: https://www.photools.com/3225/configuring-process-control-slow-media-cd-rom-dvd-nas/

[javiavid]

I forgot to attach the file 

[javiavid]

It happens to me with all kinds of operations, I thought it was because I didn't have the metadata written in the files, but it still happens to me without even having the NAS connected.
Simple things like opening a folder with few files, activating a menu, changing a panel ... it's something random, I can't see what happens. I have been monitoring the resources of the CPU, disks and network and they are not saturated.

[Mario]

Please provide more info, e.g. what you are doing when you encounter problems. Such problems are pretty unusual.

The log file reports that

it took a whopping 20 seconds (!) for IMatch to query information about which application is associated with the file

C:\Users\javia\Documents\previewcache\122826B6-B1ED-49A2-A72D-FB45FD1FBD74\38\38237.jpg

This is ridiculous. Usually this takes 20 Milliseconds or so. IMatch queries this information to display the proper "Open with..." icon and name.

When it takes a long time, the problem is usually something messed up badly in the Windows file associations management, or "shell extensions" installed which take a long time to load. This may block IMatch badly. I have also seen this a while ago with a software called BoxCryptor which installed some faulty handler in Windows Explorer.

Try to close all background applications, everything running in the taskbar. And, first of all, REBOOT Windows once. This often works.

[javiavid]

I tried to restart and it remains the same, in the last log I sent had several applications open.
As soon as the system is booted I have opened imatch and you will see that I still have frozen.
The last frozen before saving the log has been scrolling in thumbnails in the offline folder with cache of 292 images.
It also happens to me when I have not used the program for a while but without closing it, I have it open in the background. Pressing on the imatch window freezes for a few seconds.

I attached new log file to see if doing the scroll indicates something interesting.

Thank you

[Mario]

Let see:

1. A tiny database with only 35,00 files.
2. 35 seconds to load that database (from a SSD as you say). WAY to slow.
3. 21 seconds (100 times longer than normal) to query Windows for the application responsible for JPEG files

Either your virus checker is interfering badly with IMatch (did you make IMatch and the folder containing your database an exception as recommended in the IMatch help?).
But I think the problem is the 20 seconds Windows requires to answer the question "Which application is configured for JPEG files". Which normally takes 0.2 seconds. Not 20.

1. Try to associate another application with JPEG files in Windows Explorer ("Open with...") in case something is messed up.
2. If you have off-line folders which are on a NAS, please note that Windows may be blocked for prolonged times by your NAS when Windows asks Windows to check for the state of folders and files (to determine if folders are accessible). IMatch then caches this information for some time, but refreshed it occasionally. This would happen when IMatch was inactive for a while and then you switch back to it. Maybe bring all your folders on-line, for a test.

[javiavid]

I have changed the option to open with and added the exception to the antivirus and now .. it is amazing! it's going super fast
Thank you!

[Mario]

Can you attach a current log file, please? I would like to see how IMatch is performing now. "super-fast" is the expected performance for a database with less than 100,000 files   

[javiavid]

I tried with folders with many files and everything goes pretty smoothly. Only once did he stay a little slow, but the change has been enormous.
The disk is a SATA SSD, when I put an M2 I guess it will be even better.

I attached the log.

[Mario]

The function to get the associated icon for files now takes 0.03 to 0.15 seconds - instead of 20. BIG difference.
Especially since some of the files are on a network which may cause delays of several seconds occasionally.

The database load is still a bit on the slow side, for a database that small and a SSD...

You are processing MP4 files over a network. This is probably the worst case, performance-wise.
But once IMatch has the files processed and the cache images created, the location becomes irrelevant (until you write metadata back).

[javiavid]

Hi Again..

These days I have been working with the connected disk and now I am testing with the offline disk to be able to use it on a laptop. I have all the images with a cache.
The whole program slows down, it takes several seconds to open an image in quick view.

I attached the log.

In the log you will see that the disks are disconnected first, then I close the program, connect them and check again.
With the connected disks it is all fluid.

Thank you

[Mario]

With the connected disks it is all fluid.

I don't see any 'slow' operations in this log file. Except for loading the database, which takes about six seconds.

A disconnected drive may cause issues with the Windows file system. For example, when IMatch asks Windows for information about a folder (to determine if it is on-line), Windows may block for several seconds. I have seen this on some notebooks and also on some systems which use consumer-grade NAS systems. Windows just waits until some internal timeout expires or something. And this means IMatch is also blocked. This may depend on the firmware or drivers installed on your systems. Nothing IMatch can do. IMatch has to query the file system data at least once - and then keeps it cached in memory for some time or until the Windows system sends change modifications which cause IMatch to reload the data.

If IMatch is fast when your drive is connected but is slowed down when not - this is a typical result of this file system issue.

Also check your anti-virus and make sure it is not interfering. I have seen real strange things happening. From databases taking 20 minutes to load with virus checker on (and 15 seconds without) to Windows file system operations slowing down to a crawl.

You may want to hide off-line folders in the Media & Folders View (In the Filter panel below the tree) if your system is affected by this issue. This way IMatch does not need to poll the file system so often.

[javiavid]

I am not sure how log file generation works.
Maybe closing and reopening creates a new file instead of continuing the previous one.
I attached the previous file before connecting the disks, maybe here if you see the error.

I understand what you say you are looking for on the net.
Is there a way to tell imatch to stop trying to connect to the network folder?

I tried to disconnect the disks again and now it is perfect.
I don't know very well what it means, but before when I loaded an image in quick view, a green tick came out in each thumbnail and slowly loaded in each photo, now those green ticks load very fast in all the thumbnails.

What do green ticks mean?

[Mario]

I am not sure how log file generation works.
Maybe closing and reopening creates a new file instead of continuing the previous one.

Yes. IMatch always creates a new log file and keeps the previous one. See log file for all details.

The Viewer shows a green tick mark to indicate files which have been loaded into memory. The performance depends on your system, if the files are already in the cache, if IMatch has to process the RAW first etc.

Windows again seems to have problems retrieving the icon associated with some files in your database.
For example, to get the icon of the application associated with \\CH1\Fotos\ANTES\2003\...\nikor.zip  Windows needs almost 10 (!) seconds. This is bad.
It seems that your network is not performing well, making Windows stall and thus IMatch stall.
And this is the case for all files on \\CH1... IMatch queries the applications only once per unique extension (and only when you select the file in the File Window). And it uses a background thread to not block the UI. But if you move very fast between files, IMatch may have to wait for this to complete and then it stalls. Usually this operation takes 0.1 seconds, not 10...

[javiavid]

Hi Mario, here I am again with the same problem.
I have found a solution, but it's not a very good.

The problem is that if the main path of my images is a network folder (the one I usually use for the NAS) and this IP does not exist every few seconds iMatch slows down a lot, without being able to use it (freeze), then it comes back, but every few seconds the same thing happens.

If I connect the VPN to the NAS, the IP already exists, even if I don't enter the password iMatch already works better.
If the unit exists, the problem does not occur.

The solution I'm doing is when I want to offline, relocate to the C: drive and when I want online, I relocate the disk back to the network drive.

Does this happen because iMatch it tries to search for the drive every few seconds?

Is it possible to disable this so that it only searches for it once when you open iMatch?

Thanks

[Mario]

I'm not quite sure that I understand what you do with IP, VPN, NAS
Sounds like a worst case. So many additional layers, from the network to the SAMBA system running on the Linux on your NAS...

When you click a folder, IMatch asks the file system for information about the folder and the files it contains.
This is how IMatch learns if the folder and files still exist, if there are new and updated files, off-line files etc.
IMatch then caches this information for some time.
The file system monitoring is performed by background threads and does not block the UI.
It updates the cache occasionally, mostly when it receives messages from Windows about file system changes in monitored folders (folders in your database).

File system functions are known to block for a long time if a network path cannot be found or is slow to respond.
Another function IMatch uses to determine the associated application for a file ("Open with BLA") and the function which IMatch falls back when there is no thumbnail and no cache image (the icon of the application which has registered the file format) may also block when there is trouble in the file system or network.

It seems you are trying to use IMatch in a very rough environment, where one or more of the involved sub-systems don't work properly.
There is nothing I can do from the IMatch side. IMatch has no "turn this or that off because..." features to cater for unstable environments or network connections dropping of or respond slowly.
I bet there are many related warnings or #sl entries in the IMatch log file, complaining about timeouts and whatnot.

[javiavid]

No problem, I will still use the relocate option to a local disk when I want to use it offline.  Thanks

[Mario]

If you need to relocate often, there is a good chance that your NAS or the SAMBA version running on it changes the unique media serial number of the network resource randomly or occasionally.
Since IMatch uses that ID to identify media, a relocate is required to bring the files back online.
The media serial numbers for media are displayed in the property panel below the Media & Folders tree. Keep an eye on that. It should never change, unless you reformat a disk. But a NAS running Linux which runs SAMBA to simulate a Windows server for file system access can do strange things sometimes.

[javiavid]

Thanks, if I detect anything unusual I'll let you know.

[Tveloso]

The solution I'm doing is when I want to offline, relocate to the C: drive and when I want online, I relocate the disk back to the network drive.

Thank you for this work-around for the issue.  I have used it a number of times now when traveling (and the majority of my files - which are on a NAS - are offline as a result).

I actually just posted a feature request around this:

        https://www.photools.com/community/index.php?topic=12047.0

...but I see now that the suggestion you made:

Is it possible to disable this so that it only searches for it once when you open iMatch?

...might be an even better approach (and maybe easier for Mario to implement?)

This does seem to be something that other users should be seeing, since I imagine that it's typical for folks to have offline media when traveling with their IMatch Database...so maybe there's something that's in common between mine, and javiavid's PC, that manifests this issue, and most users don't see it? 

I'm now wondering of mapping a drive letter to the Network Location might make a difference...(I'm using a UNC Path to reference the folders on my NAS that are indexed in IMatch).  As I write this, I cannot test that, as I'm away from from my Home Network.  I'll be able to test it later this week, and will report back if I find that a mapped drive works differently (and does not cause the "freeze").

But if the issue persists even with a drive letter, this work-around (of relocating to a local disk) does address the "freezing problem", so thank you javiavid for suggesting it.

[Mario]

Does this happen because iMatch it tries to search for the drive every few seconds?
Is it possible to disable this so that it only searches for it once when you open iMatch?

IMatch retrieves file system data from Windows only once and caches it in memory. Only when Windows sents "changed" events, the data is refreshed.
Even for thousands of folders with thousands of files, updating the cache takes a second or two. Less on SDD.
Off-line NAS boxes are the worst case. Depending on Windows, asking something like "Is this folder online" can take a second or multiple seconds. That's just how the networking works

I work with off-line media all the time in IMatch, when working on the notebook or with one of the virtual test PCs running locally or on a cloud server.
Since IMatch does all the file system caching in the background, this does not interfere with working with IMatch at all. Unless something really obscure is going on on your PC. And, in my experience, such effects are usually caused by a virus checker or other 'security' software interfering.

I wonder what is actually causing the íssue here. Did your log file above reflect this re-occuring blocking? What is blocking?
Maybe it's the function IMatch is using to ask Windows for the icon for a specific file (to show it in the toolbar)? If the file is on an off-line media and Windows does not know that, it may be forced to wait for several seconds until your network stack realizes that your NAS is off-line. But why? Even if, IMatch queries the icon in a background thread to not stall the UI...

[Mario]

I have added some improvements for the "Get icon for the application associated with this file".
I could cook-up a case where Windows took 10 seconds (!) per file to determine that the file is off-line when IMatch asked for the icon of the application associated with a file.
When IMatch first checks if the file is online (directly or by looking up the info in the IMatch memory cache) and then avoids to call the icon function at all, the overall performance is improved a lot.
I wonder why Windows takes 0.2 seconds to tell IMatch that the file is off-line, but apparently does not do this simple check internally when trying to figure out the icon for a file.

[Tveloso]

Awesome!  Thank you Mario.

Hopefully this eliminates the need for my FR.