Pack&Go taking too much time

[neal]

My usual Pack&Go time takes 5-15 minutes. Since 3 days, even though I have not made any major changes to my database, it now takes 7 hours (1066 files, 310 folders, 1.72GB)!! What is happening?
Neal

[Mario]

It would help if you could provide a bit more info.
Like, the Pack & Go protocol, the Pack & Go log file etc.
We don't even know if your database has 2GB or 50GB or if you include multiple databases...

In general, if something suddenly takes longer than before, something on your system has changed.
Did you check that your virus checker is not impacting the Pack & Go performance? Maybe a recent update brought some changes?
Did you reboot your system once?

[neal]

Here are the requested files.
Neal

[Mario]

Pack & Go runs a full database diagnosis.
It appears that the routine that checks the on-disk cache is super slow. What this routine does is scanning the IMatch cache folder to look for orphaned files.
Just to check if deleting a file from the database or similar has left a superfluous cache file.

This is usually a very fast operation. IMatch fetches all file names in the cache folder and looks for files without a matching file record.
The slowest part is scanning the file system folders for the file names, but that's 100% running "in" Windows.
On your machine, Pack & Go reports that this step took 27136109ms, which is about 7 hours (!). So, there is the problem.

My first bet, also already mentioned in my first reply above, is that your virus checker or some other security facility is badly interfering with Pack & Go when it retrieves the file names from the cache folder.
Since this problem was never reported before, this diagnosis function is super-fast usually and you experienced this only recently, double-check your virus checker. Maybe make Pack & Go an exception, exclude the cache folder from on-access scans (C:\ProgramData\photools.com\IMatch6\previewcache\) etc.

[neal]

I will try excluding that folder and see how it goes.

[neal]

I prevented Windows Defender from accessing C:\ProgramData\photools.com\imatch6. I ran Pack&Go and waited for 16 hours but it still didn't finish! I finally aborted it. Attached are the logs.
Neal

[Mario]

The log starts at 19:04:52 starting up showing some dialogs etc.
Then nothing (!) happens until 23:32:31 - which looks like IMatch was showing a dialog but the user did not respond to it (e.g. clicking OK or Start).
Then diagnosis starts.

As before, checking the cache folder was painfully slow.

1. Remove your database from Pack & Go and do a test run.
This avoids diagnosis and cache folder checks.
Does this run fast?

2. Exclude the folder C:\ProgramData\photools.com\IMatch6\previewcache\ from on-access virus checks.
Add your database again to Pack & Go and repeat your test.

3. Check the log files your virus checker produces. Maybe it has logged something. Sometimes they do, mostly they don't want to confuse users with too much info and just silently do harm.

[neal]

Tried everything. Nothing made any change. No mention in virus log. Yes, when running Pack&Go without a DB, it is fast, 8 sec. After adding the DB, I waited 45 minutes, but no joy. What is next? Should I delete the cache folder(s)? If so, where are they?

[Mario]

I have listed the cache folder to exclude in two replies above. It is also shown in the Edit > Preferences > Cache dialog and in the Info & Activity panel.
Did you follow my advice to exclude it in your AV? Did this not change anything?
What if you disable the database diagnosis in Pack and Go?
What happens when you rename the folder C:\ProgramData\photools.com\IMatch6\previewcache\3A059548-B875-4C8C-A8E5-46969EE2ABDD\ before running Pack & Go?
Don't forget to rename it back.

Your database has only 40K files. Even if each file would be cached, a folder hierarchy with 40K files takes only a few seconds to scan. Then IMatch knows the file names and can quickly lookup the ids in the database.
This is very fast. If it stalls that long, this must be some file system issue or something external.

[neal]

I tried a database diagnostics from within the DB. It is stuck on the FILE CACHE item.
There are 4 folders in the C:\ProgramData\photools.com\imatch6\previewcache. Only 2 have files. Should I rename one at a time?

[thrinn]

Your DB file is on drive G:, right? Is this a regular harddisk or SSD? Just to be sure: The folder where your DB resides is also excluded from virus checker scans?
You could also try to disable your virus checker completely during the Pack&Go run, just to see if it makes a difference.

[Mario]

The cache folder for your database is listed in my last reply. This is the one you should rename (while IMatch is not running). Then test Pack & Go.
Did you follow me other tests and what are the results?

Since Pack & Go uses the same diagnostic routines as IMatch, it is expected that both behave the same.
Something seems to interfere when IMatch tries to scan that folder. My bet is still on the virus checker or maybe some weird file system issue.

[thrinn]

There are 4 folders in the C:\ProgramData\photools.com\imatch6\previewcache. Only 2 have files. Should I rename one at a time?

Each database has its own preview cache folder. It is sufficient to rename the one belonging to the DB you try to Pack (see Mario's post - or have a look at the Info & Activity panel where the Cache folder is also displayed).

Just one another thought: Did you by any chance copy the DB in question without changing the Database ID afterwards? It's a long shot, but if two different databases access the same settings and caches, this can give strange effects.

[neal]

There are 4 folders in the C:\ProgramData\photools.com\imatch6\previewcache. Only 2 have files. Should I rename one at a time?

Just one another thought: Did you by any chance copy the DB in question without changing the Database ID afterwards? It's a long shot, but if two different databases access the same settings and caches, this can give strange effects.

I don't recall copying the DB from or to  anywhere

[neal]

FYI, there is no folder like what you mentioned in C:\ProgramData\photools.com\IMatch6\previewcache\3A059548-B875-4C8C-A8E5-46969EE2ABDD\.
So, should I delete the cache, or restore an earlier version of the DB?

[Mario]

This is the folder Pack & Go reports as the cache folder.
What does IMatch show under Edit > Preferences > Cache?

[neal]

C:\ProgramData\photools.com\IMatch6\previewcache\  which opening file explorer to that folder where there are 4 folders, but none of them are what you mentioned.

[Mario]

IMatch creates one numerical sub-folder for each database you have used.
The folder used for a database is shown in the Info & Activity panel.

[neal]

Mario, is there a way to delete the cache and then regenerate it? Wouldn't this solve the problem?

[Mario]

That is exactly what happens when you rename the cache folder.
The next time IMatch starts it will re-create the folder (empty). And then re-create cache images on-demand.

That's why I said you should rename it for testing and which folder etc.

[neal]

That doesn't seem to solve the problem. Now I can get the P&G to backup even if the optimization box is ticked, but not if the database diagnosis is checked. Then it just hangs (memory usage is 330MB for P&G). 

[Mario]

If the cache folder is empty, the routine for checking the cache files should take less than one second.
Did you check the log file to see how long it takes now?
Sure that you have renamed the correct folder? If in doubt, rename the folder I have listed multiple times C:\ProgramData\photools.com\IMatch6\previewcache\

[neal]

Good news! After renaming the folder again, it became empty. Filled it in IMatch by cache command on root folder and then did Pack&Go successfully. P&G took only 2 minutes. Thanks so much!
Neal

[Mario]

Strange...