Is there something wrong with my IMatch?

[RobiWan]

Hi,

I have not used IMatch for a few days ~2 weeks or so. Now it wants to import data from ~7000 images. OK so far. Only it takes all the RAM the system has (64 GB) until it stops working and IMatch shuts down.
[url="https://www.photools.com/community/index.php?action=dlattach;attach=30837;type=preview;file"]imatch_log20230123.txt[/url]
Robert

[Mario]

The highest memory load IMatch recorded in the log was less than 500 MB.
This would suggest that the memory is used up by an external helper or driver.
IMatch does not need much memory for itself, except maybe for the Viewer, which uses up to 80% of the free memory to cache files in advance.

The log file shows IMatch indexing ORF files. Maybe the Windows WIC codec is not happy with the ORF variant you are using, allocating tons of memory trying to process it? Hard to tell.
IMatch itself is using only 0.5 GB. But if a WIC codec runs berserk with the proprietary ORF variant your camera produces, the memory will be shown as used by IMatch.

Contact Olympus and ask them to provide a WIC codec that handles the proprietary ORF format your camera uses. I mean, you've paid them money for your camera. And if they are forcing you to work with undocumented and proprietary RAW files, they should at least provide you with a WIC codec that is able to handle the RAW variant your camera is using, right?

Maybe ask Microsoft for an update of the Windows ORF codec with better support for the ORF variant your camera is producing. Send them some samples of your images so they can look into it.

It appears to me that this is not something in IMatch. But something caused by the proprietary RAW format you are using, which is not handled well by the WIC codec in Windows for ORF files. Always fun to use undocumented and proprietary RAW formats.

Maybe switch IMatch to use LibRaw instead of WIC codecs. Maybe this works better with the particular ORF RAW variant you are using. I don't know. Olympus does not document their RAW format and does not provide developer support.
See photools.com RAW Processing for more information.

Maybe the files are corrupted, crashing the Windows ORF RAW codec? I don't know, it's a black box. IMatch logs when a WIC codec returns error messages, but nothing is logged in your log file.

[Lord_Helmchen]

Hi,

I have not used IMatch for a few days ~2 weeks or so. Now it wants to import data from ~7000 images. OK so far. Only it takes all the RAM the system has (64 GB) until it stops working and IMatch shuts down.
imatch_log20230123.txt
Robert

I had a similar problem with my ORF files (OM-System OM-1) and Mario recommended to active IMatch's internal Raw conversion (in settings "photools.com RAW process"). That solved my problem. You may give it a try.

[RobiWan]

Hi again,

long time I have this issue and no sulution for this. If IMatch inserts new images into database, so I have always close them more than one time. In other case I will run into "out-of memory" issue.
I don't know what Imatch is doing but this is not correct.



Robert

[Mario]

40 GB RAM usage? Yikes!
Never seen that before.

I see you are using ORF files. Probably the WIC codec provided by Microsoft runs amok with the particular ORF variant you are using? The highest memory load IMatch has logged is about 1 GB. If TaskManager shows 40GB for IMatch, the memory was allocated by something IMatch is "using", like a WIC codec processing your RAW files.

Go to Edit > Preferences > Application and enable photools.com RAW processing.

Close and reopen IMatch and then retry whatever you did for that screen shot.
If the problem is the Windows WIC codec for your ORF file, this will solve the problem.

[RobiWan]

Thanks Mario for your quick reply.

Yes I have ORF files in my database too. ONLY - since 2 years I import only Canon .cr3 files. Why does IMatch bother with files that should not be imported at all?

[Mario]

Why does IMatch bother with files that should not be imported at all?

What do you mean by that.
IMatch only imports files in folders you have added to your database.
It imports all "known" file formats in these folders, unless you explicitly disable specific formats.

I don't see what you see. More details is always better.
All you have provided is a screen shot of a File Window with some OFF files, and a screen shot of the Task Manager.
Not much I can make of this.

IMatch does not "bother" for files. It indexes them, or not. Depending on your settings.
If you don't want to manage specific folders anymore, use the "Remove folder from database" command to remove the folder. For example, remove the folders containing the ORF files you don't want to manage anymore.

[RobiWan]

IMatch only imports files in folders you have added to your database.

Yes, and the new folders have only Canon CR3 files. ORF files from my desktopn-Screeshot are old files. They are 3 and more yers in my IM database.

Fromy my previous screenshot they are 2 folders that have changed (they have new images)

IMatch does not "bother" for files. It indexes them, or not. Depending on your settings.

OK, wrong word. I try again - Why should IM have a problem with files that already in my database (my old ORF files. Here has nothing changes since last 2-3 years.

Robert

[Mario]

We're not communicating right.

What kind of problem does IMatch have? The memory consumption?
What did you do when this happened? When does the problem occur?

You write:

If IMatch inserts new images into database, so I have always close them more than one time. In other case I will run into "out-of memory" issue.
I don't know what Imatch is doing but this is not correct.

which I don't understand.

The two log files you have attached show just IMatch starting and then closing normally again.
No files were indexed during these sessions, no features were used by the user, memory consumption minimal.

Where is the log file of the session you show in your screen shot?

If IMatch inserts new images into database, so I have always close them more than one time. 

When does IMatch index files? What do you mean by "close them more than one time"? What do you mean by "close"?

[RobiWan]

The memory consumption?

Yes

The two log files you have attached show just IMatch starting and then closing normally again.

Yes because I have closed Imatch manually before my system runs in out-of-memory

No files were indexed during these sessions,

Very interestng, because Imatch has imported 1916 new files.

Where is the log file of the session you show in your screen shot?

Both "IMATCH6_LOG" and "IMATCH6_LOG-2"
The first one is before I closed Imatch, and the -2 file is after restart and IMatch indexed I mean last 630 files.

What do you mean by "close"?

Exit IMatch before my system runs in out-of memory. My System has 64GM Ram and is running Windows 11. If the system has about 1% free memory I have exit IMatch before Windows will start to kill processes.

When does IMatch index files?

Yes if Imatch has more than ~1000 new Canon cr3 files to index.


Robert

[Mario]

Both "IMATCH6_LOG" and "IMATCH6_LOG-2"
The first one is before I closed Imatch, and the -2 file is after restart and IMatch indexed I mean last 630 files.

Nope, sorry.
Both log files show an IMatch start and close. No files were indexed.
Maybe you've attached the wrong files. Both files are less than 120KB, and that's way to small for even a single file being indexed.

You can search for AddOrUpdate file, which is logged when IMatch checks if a file needs to be added.

Switch IMatch to debug logging via Help menu > Support
Index (or force update via Shift+Ctrl+F5) 20 to 50 files.
Close IMatch and copy the log before restarting.
ZIP and attach.

[RobiWan]

OK,
I have restored my Database from yesterday backup and imatch is scanning the new images again.
First part or indexing - where many exftools instances are running was all OK, no bis RAM consumption. After this step was done, I don't know what happens then, but Imatch had been taking more and more RAM. Can it ew. have to do with the preview generation?

And what this means?
type .\IMATCH6_CEFLOG.TXT
[0516/074846.095:ERROR:gpu_init.cc(457)] Passthrough is not supported, GL is disabled, ANGLE is

[Mario]

IMatch reports a memory usage of about 600MB after loading the database.
Then it starts processing CR3 files using the installed WIC codec, and memory consumption goes over the roof. Very unusual, I don't think I have seen anything like this before.

1.  Which WIC codecs to you have installed?
Select a CR3 file in a File Window and then run Help menu > Support > WIC Diagnosis.
Attach the resulting file.

2.  Switch to photools.com RAW processing via Edit > Preferences > Application. Search for RAW and enable photools.com RAW processing.
Close and re-open IMatch.

Repeat your test. Does IMatch still use that much memory? If not, we know that the WIC codec processing the CR3 variant your camera is producing is the problem.

[RobiWan]

Thanks a lot for your support Mario.

This is output from WIC diagnose:
"WIC: Testing file 'C:\DSLR\DSLR-RAW\ExtHDD 2508\EOS R6 Mark II\2023\04\23\_rr62987.cr3'
Thumbnail: Codec 'Microsoft Raw Image Decoder'
() 1024x682 pixel in 62 ms.
Preview: Codec 'Microsoft Raw Image Decoder'
() 6000x4000 pixel in 313 ms.
Full resolution: Codec 'Microsoft Raw Image Decoder'
() 5999x3999 pixel in 1172 ms.

WIC Result: A codec for this file format is installed and it looks like it fully supports the format.


IMatch Integrated RAW Processing.

Analyzing image 'C:\DSLR\DSLR-RAW\ExtHDD 2508\EOS R6 Mark II\2023\04\23\_rr62987.cr3'
    Preview: (Format:1, Dimensions: 6000x4000) in 0ms. Result: 0
    Original Image: (6188x4120 6188x4120) in 3141ms. Result: 0"

With photools CODEC IMatch goes up to 5GB RAM but no more.

But photools RAW processing is not alternative for me, see attached screenshots

Robert

[Mario]

The first image (IMatch.jpg) looks like an unprocessed RAW file.

But IMatch always uses the embedded preview, for WIC and LibRaw if the preview is larger than the minimum size configured under Edit > Preferences > Cache (2000px by default). The previews in your CR3 files are 6000 pixels and IMatch will use them.

The result should hence be identical for either LibRaw or WIC since both use the same preview JPG.
Very strange. Send me this file (and maybe another) to support email address with a link back to this thread. Then I can tell you more.

[mopperle]

Just tested it with my RAW files (EOS R): I have no memory problem when importing and I do not see the strange colours when using the photools raw processing.
So just an idea: Do you have Dual Pixel RAW avtivated in your camera? I dont, but cant test DPR as I have my camaera not with me.

[Mario]

@mopperle Thanks for looking into this.
I've download some EOS R6 Mark II sample RAW files from two web sites and copied/pasted them to produce 100 files.
I've then added them to an IMatch database.

First test was with WIC enabled. The Peak memory usage of IMatch was less about 1.5 GB according to Windows Task Manager.
I noticed that WIC gets the orientation wrong for some files - sigh, how can this be so hard for Microsoft...?

Second test was with photools.com RAW processing enabled.
Same peak memory consumption, orientation correct, images looking good (for all files IMatch used the embedded preview).

In a third test I've forced IMatch to re-create cache images for all 100 files, once for WIC and once for LibRaw.
The peak memory consumption was still at about 1.5GB.

I cannot find any fault. Maybe your images are special in some way. I will know more after you've sent me some samples.

[RobiWan]

Hi guys,

thanks for testing and sharing information.

Now I foud this strange problem. I'm not sure at this time if I can change this setting (because in the past I have had problems with rotation in portrait mode pictures )


@Mario - I think it is no longer necessary to send you images

Cheers Robert

[Mario]

That's what I wanted to ask but then forgot.
You force IMatch to develop the full RAW image when this is set to Nein (default ist Ja) and then you get whatever LibRaw produces from the RAW file - which is something without all the advanced algorithms and AI modern RAW processor software employs. This explains why your results were different.

The orientation problem found in some ORF files and some CR2 files can be handling with the corresponding options under Edit > Preferences > Application (look for RAW).

I recommend you set this to "Ja" and enter a minimal size of 2,000 pixels.

[RobiWan]

Hi,
now I have changed this settings. Ist this now correct and recomended?

[Mario]

The cache settings look OK.
The "Vorschaubilder rotieren" should only be set to "Ja" if your images come out oriented wrong for either WIC or LibRaw.
This is a work-around for some unusual RAW files which have rotated previews but no indication for how the preview was rotated. Neither WIC nor LibRaw can then tell.

Your problem started because you were using WIC.
Now you are using WIC again instead of photools.com RAW processing. So your problem will be back, no?

If photools.com RAW processing works, keep it enabled.
Since you now allow IMatch to use the embedded preview, it will be much faster and the RAW files will look as the camera vendor wants them to look. Select the "flat looking" files from your test in a File Window, press Shift+Ctrl+F5 and choose "Force Update". IMatch then reloads the images, this time using the preview. The images then will look fine.

[RobiWan]

Thanks again for your Support Mario.

I have now changed it again and will see what happens on the next import.

[Mario]

If the problem was really the WIC codec consuming so much memory, this should solve it.

On the other hand, maybe it was not the WIC codec, but your request to develop the full RAW instead of using the embedded preview that caused this. Maybe the WIC codec now also works fine...
I've tried my test yesterday with both preview and full RAW with both WIC and LibRaw, and the memory footprint was never larger than 1.5GB. Which would indicate that this is something specific for your machine and WIC codecs installed.

[RobiWan]

but your request to develop the full RAW instead of using the embedded preview that caused this.

Correct. This was my problem.