Pack & Go: Database Diagnostic runs slow

[CVKen3]

Does diagnostics do something different in the Pack & Go process, like a more thorough job? On my system, the database diagnostics portion of the Pack & Go procedure takes forever, about 30 minutes. If I run diagnostics from the menu directly, it takes 2 or 3 minutes. Pack & Go times are about 2 minutes without compacting or diagnostics, 4 minutes with compacting, 35 minutes with diagnostics and compacting.

No super rush on an answer, as I don't backup that often and I workaround by running diagnostics before the Pack & Go process, then uncheck this option in the Pack & Go dialog. Note: Windows 10 home, 64 bit using the current iMatch 2023.12.2

[Mario]

Please attach the Pack & Go log file, the diagnosis log and the protocol file. This will give us information to work with:

IMatchPackAndGo_diag.txt
IMatchPackAndGoLog.txt
IMatchPackAndGo.txt

You can find the files in the TEMP folder. The name and path of the protocol file is displayed at the top of the Pack & Go window.
I've made a quick text with two small (50K) databases and the diagnosis times vary only by a few seconds between Pack & Go and IMatch.

[CVKen3]

Tried it again this morning, still slow (3 min from the database tools menu, 30 min from the Pack & Go dialog). The log file shows lots of entries about the people manager and the family manager. I have organized people into families, fwiw. Not sure why Pack & Go is different, though.

Logs attached. The big one contains the 3 files from the Pack & Go run. (Sorry about the 73MB logfile!) The small file contains the logfile running database diagnostics from the menu.

[Mario]

I can see that the diagnosis from P&G takes 30 minutes instead of 3 minutes.
The diagnosis log file does not log timings for individual steps, just the total runtime.
The P&G log logs more info, but I don't have your IMatch log file to compare. Can you attach the IMatch log file from a session where you ran a diagnosis?

My first guess would be the virus checker interfering.
Maybe you have made an exception in the virus checker for the IMatch executable but not for the P&G executable?

For a database with only 5GB in size and 90K files, a diagnosis runtime of 3 or 4 minutes is OK. 30 is not.
Since Pack & Go and IMatch run the diagnosis in the same way (unless you are in the habit of holding CTRL pressed while starting the diagnosis in IMatch), there should be no real difference in runtime. Unless something external interferes...

[CVKen3]

I only use Windows Defender. I do have it set up to disallow file access to a bunch of places, but it triggers an alert when an unauthorized program does this. And, I have both IMatch2023x64.exe and IMPackAndGo.exe on the exclusion list. There's no indication that either program triggered this. Note: After I post this, I will disable the Windows Defender feature, reboot and retry. I previously disabled it mid-run of P&G, which did nothing.

I ran the diagnosis both from the menu and from P&G and watched the TaskManager / Resource Monitor information. The difference seems to be the amount of data logged. The P&G diagnosis call is constantly writing both to the P&G log and to the Volume log:




Is it possible that there's some odd debug switch or setting that I've enabled?

Attached is the Imatch log. The difference I see is the CheckEntityGraph section. The P&G log has 300,000+ lines of debug (log?) information between the start and end of the CheckEntityGraph call.

[CVKen3]

Disabling the windows "ransomeware protection" (turning off "controlled folder access") did nothing. I don't see anything obvious on my system that would make P&G diagnosis run slower than running via the IMatch menu. I looked through the windows security (defender?) settings and didn't see anything there, either.

Again, this isn't really an issue, especially if no one else is having the problem. I can turn off the diagnosis step in the P&G, and just remember to run it before exiting IMatch. Given that I normally leave the room after I start P&G, it was a total accident that I even noticed it was slow on my system.

[Mario]

Hah! Figured it out using a 100K files database.
The difference in diagnosis runtime is caused by the log file!

Pack and Go opened the log file with default settings, which means in debug logging mode.
When debug logging is enabled, the log file always writes directly to the disk, without caching. The reason is to have the log file up-to-date on disk when IMatch crashes or is no longer responding.

Writing always to the disk is much, much slower than the normal mode, where the log file caches log file entries in memory and only occasionally writes data to the disk.

The database diagnosis, especially for the IMatch Graph (events, persons, families, ...) performs many checks, and each creates log file entries. It the log file is in debug mode, the diagnosis logs a lot more data, too.
In my test, this caused 450,000 physical writes to the log file. Instead of a few hundred when debug logging is off.

I have no configured Pack & Go to use default logging and this brought the diagnosis time down from 11 minutes to 1.

Than you for bringing this to my attention. The bug fix is included in the next regular IMatch update.

[CVKen3]

Haha! It's always something unusual. Thanks for the update and thanks for all the work you do. As a user, I really appreciate it and love IMatch.

[Mario]

Haha! It's always something unusual. Thanks for the update and thanks for all the work you do. As a user, I really appreciate it and love IMatch.

You're welcome.
Another bug bites the dust 

If you like IMatch, it would be cool if you can let others know.
There is so much crappy sub-optimal software out there, but fitted with big marketing money...

[CVKen3]

I occasionally mention IMatch on Threads, when people are looking for Lightroom replacements, and the like.

[Mario]

I occasionally mention IMatch on Threads, when people are looking for Lightroom replacements, and the like.

Thanks. That's a very good form of "marketing". Actual users of a product liking it and telling others about it.

For a small software vendor it has become nearly impossible to get tracking since the big players like Adobe buy out all of the DAM -related keywords in Google ads, making it impossible expensive.
Their marketing and social media departments constantly churn out SEO-optimized "content" and post it on all social channels, from YouTube to X. "Endorser" deals with famous people in the field open up another layer of channels to praise their products.
It's hard to get heard or mentioned these days...

[Mees Dekker]

I actively promote it to the members of my fotoclub.

[Mario]

I actively promote it to the members of my fotoclub.

Thanks! Much appreciated