Questions on Process control

[Damit]

I was looking for a help section on this new feature but could not find it.  I think this is awesome.  I have had IMatch lock up my computer many times and always thought it would be great if we could save some processors for the rest of the computer so we can perform tasks while IMatch is working in the background. Then Process Control was introduced!      Valhalla! I can't wait to see how this works.

I have a question on processor count. I have a processor with 8 regular processing cores and 16 efficiency cores. So does that count as 24 cores, or is it only eight? I don't know what value to set, either 6 or 22, as I want to save a couple of processors.  If there is a help section, please point me to it.

[Mario]

Look at the top of the IMatch logfile. IMatch there reports the number of processors it "sees".
It does not know what performance or eco cores are, though.

I was looking for a help section on this new feature but could not find it.

The Process Control options exist since IMatch 5, when I'm, not mistaken.
Searching the help for process reveals the corresponding help page as the first hit:  Process Control (Advanced Setting)

Note: This is an advanced setting (not many users will ever have a need for this) and it only shows in the Edit > Preferences > Application page when the Show extended settings check box is ticked.

It is also very unlikely that IMatch maxes out your processor cores when indexing files or reading metadata. These operations are usually I/O bound, not processor bound.

What will intentionally max out your CPUs is face recognition, because it CPU-bound. Try to dial the corresponding down first to leave some processor time free for other purposes.

[Damit]

I guess it is the number of threads you can run because I am getting 32 processors, which corresponds to the thread count.  I have set it to 28 to see how it works.

Searching the help for process reveals the corresponding help page as the first hit:  Process Control (Advanced Setting)

Your browser must work differently than mine because that is the exact word I used to search and that is not the result I got (see attached screenshot). I was like, WTH? How can that be? Very strange. Thanks for the link!

It is also very unlikely that IMatch maxes out your processor cores when indexing files or reading metadata. These operations are usually I/O bound, not processor bound.

My experience has been that when importing, adding and updating files my CPU will get very taxed, if not at, almost to 100% and the system becomes unresponsive. The IMatch window will enlarge, covering up the task bar and forcing me to use the task managed to try to switch programs.

[mopperle]

Your browser must work differently than mine because that is the exact word I used to search and that is not the result I got (see attached screenshot). I was like, WTH? How can that be? Very strange. Thanks for the link!

The first match of search is typically the one that fits.

[Mario]

The first match in the search results looks OK to me.

[Damit]

Am I missing something, because I do not see in any matches on the screen shot for "Process Control (Advanced Setting), and the screen shot is scrolled all the way to the top? The first match I got using "process" as a search word is Configuration - Application Settings (Background Processing). The link does take you to the right place, but the Title is wrong, it should read "Process Control (Advanced Setting)," and I believe the link took me to background processing when I clicked it earlier, but I am not sure. Either way, I think the help section requires an edit because Background Processing in the preferences references to indexing options not process control, and, at the very least, switching the name of the link to correspond to the title of the subject may make things clearer.

 

[Mario]

The screen shot shows

Configuration - Application Settings (Background Processing Options)

does it not?

When you click it, the help opens Process Control (Advanced Setting)  or does it not?

My experience has been that when importing, adding and updating files my CPU will get very taxed, if not at, almost to 100% and the system becomes unresponsive.

Then Windows is not doing a good job at scheduling on your system.
Or you have automatic face recognition enabled, in which case my suggestion given above stays true.

[Damit]

The screen shot shows

Configuration - Application Settings (Background Processing Options)

does it not?

When you click it, the help opens Process Control (Advanced Setting)  or does it not?

Yes and Yes, but should the screen shot not show "Process Control (Advanced Setting)? Because that is what I am searching for not Background Processing Options, whose tab has nothing to do with Process Control on the Preferences.  I have certainly found it helpful when the help subject topic actually has a title that corresponds to the subject matter you are searching for.

My experience has been that when importing, adding and updating files my CPU will get very taxed, if not at, almost to 100% and the system becomes unresponsive.

Then Windows is not doing a good job at scheduling on your system.
Or you have automatic face recognition enabled, in which case my suggestion given above stays true.

Nope, I do not have automatic face detection on.  Believe me, I would know if I did and I am very careful not to enable it. It is funny how it is always some other process that is at fault. I have now run iMatch in 3 separate systems, 2 on Windows 10 (one completely fresh with ONLY IMatch running on it) and one on Windows 11 in a brand new install, and every time the processing power has been maxed out when importing files. So I am fairly certain it is your program.

I also sent you a list of all the programs I have on my computer and the windows exceptions I have created for this program in the emails I sent concerning the duplicate entries problem I am still waiting for you to address, so that precludes the usual blame being placed on antiviruses. That is not the issue either. I will send you a debugging log so hopefully you will address the issue, since apparently this is not normal behavior.

Below you will see the screen shot showing how much of my top extremely powerful CPU the i9 13900K.  IMatch has been importing 25K files for over an hour, all the while at 80+ cpu processing consumption.

It would be great if it did not take so long to import files.

Update: I was trying to figure out the reason things are taking long looking at the log file and it occurred to me, could it be that I have opted to have high resolution preview thumbnails?  I think I went with 600 or 800 since I have a very large 4K screen.

[Mario]

Importing is usually I/O bound, not CPU bound.
Do you index local files or files on a NAS/network?
The only thing that stresses CPU during import is face recognition.
Or, sometimes, WIC codecs when processing too many RAW files in parallel, for some rare RAW variants.
Dial down the number of parallel threads in process control if you want to free the CPU cycles for something else.
Also double-check that your virus checker is leaving IMatch and the database folder alone.
Note that the log file in debug logging mode show exactly what IMatch is doing and what takes how long.

[Damit]

I attached a screenshot of the exceptions in another thread, and also sent you a list of the programs on my computer to rule out the virus checker factor.  I do not have any other virus checker in my system other than defender and I set up the exclusion, so hopefully we can rule that out. All my files are housed in Nvme SSDs locally. I do NOT use any external drives or NAS, specifically to avoid problems such as these.

I sent you the log, please take a look.  Not that I know anything about what is going on in the log, but I see a lot of plugins being loaded. I also saw what I bolded and that is why I thought the resolution of my images may be the problem. Each file is taking almost half a second to load, which I believe is quite long:

04.05 14:24:56+    0 [1802C] 10  I> EUQH::Load(1) of N:\Temp Import\Personal\2023-03-12-Viscaya\DSC06995.JPG with 7008 x 4672 (O: 7008 x 4672) in 407ms
04.05 14:24:56+  78 [1802C] 10  I> EUQH: Time for file (277179) 500ms
04.05 14:24:57+  140 [1802C] 10  I> EUQH: Calculate CRC for file N:\Temp Import\Personal\2023-03-12-Viscaya\DSC06996.ARW in 15ms
04.05 14:24:57+  16 [1802C] 10  M> >  0 PTPlugInManager::GetPlugInForExtension  'V:\develop\IMatch5\src\ptpicore\PTPlugInManager.cpp(70)'
04.05 14:24:57+    0 [1802C] 10  M> >  0 PTPlugInManager::GetPlugIn  'V:\develop\IMatch5\src\ptpicore\PTPlugInManager.cpp(115)'
04.05 14:24:57+    0 [1802C] 10  M>  <  0 PTPlugInManager::GetPlugIn
04.05 14:24:57+    0 [1802C] 10  M>  <  0 PTPlugInManager::GetPlugInForExtension
04.05 14:24:57+    0 [1802C] 10  M> >  0 PTPIImage::LoadFile  'V:\develop\IMatch5\src\ptpicore\PlugIns\ptpipip\PTPIImage.cpp(287)'
04.05 14:24:57+  203 [1802C] 10  I>  Loaded image with PTD2D at 7008x4672
04.05 14:24:57+    0 [1802C] 10  I>    ImageInfo: 'Format: Bitmap from Handle'
04.05 14:24:57+    0 [1802C] 10  M>  <  0 [203ms] PTPIImage::LoadFile

[Damit]

Or, sometimes, WIC codecs when processing too many RAW files in parallel, for some rare RAW variants.

I gotta say that all these problems with slow upload started after I took the FastPictureViewer out, as requested here

Here is an excerpt from the log that is currently running:

04.05 16:44:34+    0 [1097C] 10  I> EUQH::Load(1) of N:\Temp Import\Personal\2023-03-12-Viscaya\DSC07112.ARW with 4672 x 7008 (O: 4688 x 7028) in 698313ms
04.05 16:44:34+  46 [1097C] 10  I> EUQH: Time for file (277412) 698390ms

That is a loooong time for one file, and there seems to be issues with .arw files, of which this import has plenty. Currently I have the windows WIC working.  Should I switch to the photools?

[Damit]

I also want to report that both times I changed the process control settings while I match was inputting files caused a crash. Also, after switching to photools processing things are moving MUCH faster, so it seems that the FastPictureViewer codec was performing better than the native Windows codecs, but it had other problems.  The photools codec is doing much better. It has processed 5k files in the last 20 minutes.

So, perhaps all the processing is the WIC because the task manager is still reporting very high usage but my memory and disc are barely being touched (though since my drives are so fast, that makes sense).

[Mario]

That is a loooong time for one file,

Yep. Contact Microsoft support and SONY for additional guidance. SONY should provide their customers with a working and super-fast and high-quality WIC codec. They decide not to and leave it to Microsoft.

1. Run a WIC Diagnosis on the same file to see performance differences between WIC and LibRaw. Maybe your ARW files are super-special and cause the Microsoft WIC codec to become very slow. If LibRaw is also that slow, it's the file format itself.

2. Copy the file from the N:\ driver to a new folder on your C: drive.
Run the WIC diagnosis again. Any differences in performance?

Maybe it's your virus checker? It often is, IMHO.