First feedback and first questions

[Dominique R]

Hello everyone, hello Mario

First of all, let me introduce myself. My name is Dominique. I am a photographer and content creator on YouTube (in French). I create tutorials about photography.

For a few years now, I have been using Photomechanic as a DAM. Since it did not meet all of my workflow, I looked into alternative solutions. I tested many of them. Long live the free trial period. I must say that none of those tested met my expectations until I discovered IMatch.

I started by reading a lot of the excellent online documentation. Which for me is already a very good point for the software. I then noticed that there was a forum where people came to ask their questions and receive answers. Another very good point.

After spending a week learning and watching all the videos, I decided today to install the trial version.

Another good point that I did not expect is that the software interface can be put in French. Personally, English does not bother me but it may be important for people who follow me.

After this presentation and all these good points mentioned, here are my first remarks.

I specify that I am on Windows 10 22h2 build 19045.4894

I wanted to import some RAW+Jpeg files from my Panasonic S5 mk2 camera. The RAW files in RW2 format from this camera do not show me thumbnails. I started looking on the forum and found that I was not the only one having this kind of problem with the RW2 files of the panasonic (problem of integrated thumbnail size, problem of WIC codecs, LibRAW problem, etc.)
I tested raw files coming from some of my cameras like the canon 5D mk4, the canon 60D, the sony ZV1Mk2 and the lumix GX8. For these cameras, all the thumbnails are displayed perfectly.

On the forum I saw that we could use the WIC diagnostic utility. Another excellent point. I launch it and check the 5 different types of RAW files.
For all, the utility tells me that I have a list of installed microsoft raw codecs which correctly includes the 5 formats. Then, for all, I have this sentence:
WIC Result: It looks like no WIC codec is installed which can handle this file.
Then I have a list of all the formats supported by libraw v21.2. The Sony ZV1 mk2 and the Panasonic S5 mk2 are not there. On the other hand, I know that the sony mk2 has the same sensor as the sony ZV1 which is there. Maybe that's why it's not a problem for the sony?

In short, I think that the problem with my panasonic comes from the fact that libraw is in version 21.2. However, I see that since March 30, 2024, the Panasonic S5 mk2 is supported by Libraw. I also see that Libraw has been upgraded to version 21.3 since September 18, 2024.

For your information, I have read several times on the forum that the problem could come from embedded thumbnails smaller than 2000 pixels. Now, 3 of the cameras have such thumbnails (the Panasonic S5 II and GX8 as well as the Sony). And only the S5 II is problematic. I also tried to force LibRAW in the preferences but that doesn't change anything. That's why I really think the problem comes from the aging version of LibRaw.

After my remarks, my questions are therefore the following:

1. Why do all the DAM software I've tried display a thumbnail for my Panasonic RAW except IMatch? Do they work differently?

2. How often is the LibRaw library updated?

3. Shouldn't we expect a DAM to support new cameras within a reasonable time?

4. What is the use of WIC codecs if I am told every time that they are installed but do not support my RAW files?

Sorry for posting this long message. I'm trying to be as precise as possible. And I am at your disposal for additional information if necessary.
Thank you in advance for your support.
Thank you for reading me to the end.

Dominique

[Mario]

Welcome to the community.

IMatch 2023 currently ships with LibRaw 0.21.2. The 0.21.3 was officially released only 3 weeks ago, according to the official GitHub. Which was after the current version of IMatch 2023 was released.
Please understand that I don't include "developer", "experimental" or "nightly" builds of LibRaw with IMatch.

1. Why do all the DAM software I've tried display a thumbnail for my Panasonic RAW except IMatch? Do they work differently?

IMatch fully supports the Microsoft Windows WIC infrastructure.
The idea of WIC is that Microsoft provides the framework and applications like IMatch interface with WIC. The camera vendor (Panasonic in your case) provides a WIC codec to their customers which handles understands the proprietary RAW format and can load and provide it to WIC. No company secrets have to be shared, no parents violated etc.

Panasonic has unfortunately decided not to provide a WIC codec and neither do they (to my knowledge) provide any kind of developer support. Cost cutting and profit maximizing as usual, I suppose.
Same for Canon, unfortunately.

This means that support for the many different flavors of the Panasonic RAW format in circulation depends on Microsoft and their release schedule.

In addition to WIC, IMatch also supports the free LibRAW project, which is a one person effort.
I include new versions of LibRaw when they are released and I had time to collect new sample RAW files and test them here. Same with other 3rd-party components used in IMatch.

Software vendors like Adobe, C1, DxO have dozens if not hundreds of developers, which also reverse engineer new RAW formats and lens data in order to support them into their products. I cannot compete with that. I'm only me.

2. How often is the LibRaw library updated?

See introduction above. Earlier, LibRaw was updated about every six months (official stable releases). Igor may have sped up things a bit recently.

I always try to include LibRaw, ExifTool, about a dozen 3rd party components and tools used by IMatch in their latest stable and tested version when I ship IMatch updates.

3. Shouldn't we expect a DAM to support new cameras within a reasonable time?

Probably. But why do you use and rely on undocumented and propriety RAW file formats without any support from the camera vendor? All Panasonic needs to do is to provide a WIC codec for their formats, and IMatch, Windows Explorer and all other WIC-enabled applications would work immediately with the proprietary and undocumented Panasonic formats.
Should be easy to do for them. Microsoft provides the tools for free. I'm sure Panasonic has more developer resources than I have.

4. What is the use of WIC codecs if I am told every time that they are installed but do not support my RAW files?

Again. Panasonic decided not to provide a WIC codec (which is like a printer driver).
And If Panasonic does not provide a WIC codec, you depend on Microsoft deciding to add support for the variant of the Panasonic RAE format you use to the standard WIC codecs. And this means Microsoft needs to collect samples of the RAW format, reverse-engineer it and add support for new variants of the RAW format to the standard WIC codec. Which might happen at some point in time. Or not.

Undocumented RAW formats are evil and bad for everybody.

If Panasonic does not want to support the WIC interface, which is available on a billion PCs out there, then why don't they upgrade their firmware and create DNG files instead? WIC supports DNG and LibRaw too. DNG would ensure much better compatibility than Panasonic's proprietary RAW format.

DNG is the to-go format for all billions of Android phones, Leica and others. Should be good enough for Panasonic, I suppose. Unless they want to sell their own software or use the proprietary format as a form of customer retention.

I have upgraded LibRaw for the upcoming IMatch 2025 release and also bit the bullet and merged in the entire Adobe DNG software developer kit just in order to support the JPGXL format Adobe uses in some of their products and DNG.

Update 2024/10/7

I've downloaded some sample RAW files for the Panasonic S5 mk2 camera from dpreview.com.
Unfortunately, all I get with both WIC (Windows 11, latest version) and LibRaw-0.21.3 (latest release) is pinkish noise.
Adobe Camera RAW can read the files.



LibRaw returns error -1 when trying to process the RAW data. The embedded preview is to small (1920) and hence IMatch tries to develop the full RAW file. LibRaw returns the error code LIBRAW_REQUEST_FOR_NONEXISTENT_IMAGE and then Windows falls back to WIC for a second try. This is why I get the same result either way.
I shall investigate and ask in the LibRaw community.


TIP (for now): Go to Edit menu > Preferences > Cache and configure the minimal size to 1920 pixels (instead of 2000).
IMatch can then use the smallish embedded preview in the RW2 files, which works.

[Dominique R]

Hello Mario,

Thanks for the quick reply. I think we misunderstood each other about WIC codecs.
I was saying that on 5 different types of RAW, the WIC Diagnostics utility gives me a negative result. In fact, I was wrong, I have a positive result only for my canon 60D. I also see a thumbnail for this RAW file in my windows file explorer.

For all other cameras: canon 5Dmk4, panasonic S5 II and GX8, sony ZV1 II. The utility tells me:
WIC Result: It looks like no WIC codec is installed which can handle this file.
So I concluded that WIC codecs only support a small number of RAW formats and wondered what they were really used for. By the way, I later saw on the online help that you said:
Sadly, as far as we can determine, the only camera vendor who cares enough about their customers to provide an optimized WIC codec is Nikon.

Since the WIC result is negative for 4/5 RAW files, if I understood correctly, the 4 files with a negative result are processed by LibRaw. The thumbnails are displayed well in IMatch except for my S5 II, which is why I thought of an aging LibRaw version.
I can understand that you only update LibRaw for major updates even if I find the delays between versions quite long. Finally, if I know that the problem comes from there, I will wait for the update of IMatch with version 21.3 of LibRaw.

*****

While I was preparing my answer, I just saw your update to my post.
I have exactly this pink image. I tested the trick with the cache at 1920 but it doesn't work. I specify that I did a rescan.

On the other hand, if it can help you, I want to tell you that I use the FastRAWViewer software which I think is a product of the company that also takes care of LibRaw. And in this software, I have no problem with thumbnails or demosaicing of RAW files.
I also wanted to mention that DigiKam displays RAW and thumbnails perfectly. And since this software is based on free software, I assume it uses libraw as well.

Thank you for your investigations.

Dominique

[Mario]

The pink result comes from the WIC codec processing the file but not understanding it correctly.
There are now literally hundreds of RAW variants in use and while WIC codecs support many, they don't support all.

You say that you still use Windows 10, which is on it's last leg.
Maybe Microsoft does not update WIC codecs anymore for this old Windows version and hence you get so many faults.

Use the trick I mention. Reduce the minimum size IMatch requires for cache files to 1920. IMatch then uses the embedded previews in your files and they will show a decent visual. I don't know if 1920 is the largest preview size your camera supports, or if you can configure it to embed full-size previews, which is beneficial for many uses.

I have contacted the author of LibRaw and he is puzzled about the error code I get. I should get a different error, since the format is not yet supported by the .3 but only by nightly developer snapshots, when I understood him and another participant correctly.

This will require some discussion between the author of LibRaw and myself. Maybe we can find something out. May take a few days, since is only one of the many side shows for me.

[Dominique R]

You say that you still use Windows 10, which is on it's last leg.
Maybe Microsoft does not update WIC codecs anymore for this old Windows version and hence you get so many faults.

I stayed on Windows 10 because everything is stable and my software works. I know there are (were?) problems with some of the software I use on Windows 11. So as long as I don't need to change, I'm not changing. Well, I think I'll switch soon because Windows 10 support ends in 2025.

Use the trick I mention. Reduce the minimum size IMatch requires for cache files to 1920. IMatch then uses the embedded previews in your files and they will show a decent visual. I don't know if 1920 is the largest preview size your camera supports, or if you can configure it to embed full-size previews, which is beneficial for many uses.

I've already tried and it doesn't work. There is no setting in my camera to change the size of the embedded thumbnail.

I have contacted the author of LibRaw and he is puzzled about the error code I get. I should get a different error, since the format is not yet supported by the .3 but only by nightly developer snapshots, when I understood him and another participant correctly.

This will require some discussion between the author of LibRaw and myself. Maybe we can find something out. May take a few days, since is only one of the many side shows for me.

Strange. In release .3, it is clearly indicated that the camera is supported. For information, this camera was released in the first half of 2023.

Thank you

[Dominique R]

Use the trick I mention. Reduce the minimum size IMatch requires for cache files to 1920. IMatch then uses the embedded previews in your files and they will show a decent visual. I don't know if 1920 is the largest preview size your camera supports, or if you can configure it to embed full-size previews, which is beneficial for many uses.

I've already tried and it doesn't work. There is no setting in my camera to change the size of the embedded thumbnail.

I need to fix this message.

When I tried, I was rescanning the database and it didn't work.
Now, I just deleted the files from the database and by rescanning the folder, the thumbnails appear fine when importing with this option set to 1920 pixels.

Why doesn't rescanning on an existing file work? Strange right?

[Mario]

Why doesn't rescanning on an existing file work? Strange right?

No. That's the expected result. When you rescan a file (Shift+F5 or menu command), IMatch checks the "last modified" timestamp on disk. And when that has not changed, IMatch does nothing. IMatch rescans files every time Windows reports a change in the folder, so this has to be quick to check if work needs to be done.

To force IMatch to rescan a file and re-create thumbnail and cache images, you use the force update with Shift+Ctrl+F5 on a folder or selected files. See Advanced Rescan Options I should have mention that when giving you the top. Sorry, my bad.

The issue with your Panasonic RAW file was resolved.
Support for this particular format is not available in the official LibRaw release, only in snapshots. Not even the 0.21.3 version released 3 weeks ago has support for this format. It produces the same pink noise WIC does.

I don't include unofficial developer snapshots or nightly builds in IMatch. As I have been told, DarkTable uses the latest snapshots, so it might support RAW variants IMatch does not support yet. Probably FastRawViewer does so to, since it is used as a development platform for LibRaw.

As usual, I ship the latest official release of LibRaw with IMatch and as soon as support for this Panasonic format is rolled into the official LibRaw version, it will work in IMatch, And probably also with WIC, since Microsoft uses LibRaw in WIC now.

Ask Panasonic to provide a WIC codec. This would solve all these problems. And Panasonic could produce the best possible quality for their RAW formats in their WIC codec.

[Dominique R]

Thanks for the investigations and support.
Now that I see thumbnails, that's the main thing.

I'm going to continue to take advantage of my trial period to test Imatch and see if it suits me. There's a lot to discover.

Have a nice day.

[Mario]

There's a lot to discover.

Definitely. Just post when you need support or guidance.
IMatch is designed for demanding users. It definitely has a learning curve once you go beyond the basic stuff. It's more Photoshop than Windows Paint

Oh, I forgot! Feedback and questions from new users are especially important to me.

I'm spoiled, I cannot see IMatch in the way a new user sees it. The feedback I get from new users helps me to figure out where to simplify things, add more documentation, maybe sit down and film another video etc.

[ubacher]

If adobe (ACR) supports a format then you have the option of converting your raw format to dng using their
dngconverter.

Just don't convert to the latest version of dng (1.7) since it isn't (yet?) supported by Windows, but will be supported by Imatch 2025.

[Dominique R]

Why doesn't rescanning on an existing file work? Strange right?

No. That's the expected result. When you rescan a file (Shift+F5 or menu command), IMatch checks the "last modified" timestamp on disk. And when that has not changed, IMatch does nothing. IMatch rescans files every time Windows reports a change in the folder, so this has to be quick to check if work needs to be done.

To force IMatch to rescan a file and re-create thumbnail and cache images, you use the force update with Shift+Ctrl+F5 on a folder or selected files. See Advanced Rescan Options I should have mention that when giving you the top. Sorry, my bad.

For information, I was sure I had tried SHIFT CTRL F5 but I chose the update cache option which seemed appropriate to me. So I try again today and It doesn't work. Only the force rescan option shows me the thumbnails once the cache size is set to 1920 pixels

Definitely. Just post when you need support or guidance.
....

Oh, I forgot! Feedback and questions from new users are especially important to me.

I'm spoiled, I cannot see IMatch in the way a new user sees it. The feedback I get from new users helps me to figure out where to simplify things, add more documentation, maybe sit down and film another video etc.

Thank you.
I will not hesitate to give you feedback.

If adobe (ACR) supports a format then you have the option of converting your raw format to dng using their
dngconverter.

Just don't convert to the latest version of dng (1.7) since it isn't (yet?) supported by Windows, but will be supported by Imatch 2025.

I want to say that I do not want to convert my RAW files to DNG. You will find on Google many people who have had problems with their processing software. Often colorimetry problems.
Unfortunately, as Mario says, as long as RAW files are hidden proprietary files, we risk this kind of problem.

Dominique

[Mario]

Only the force rescan option shows me the thumbnails once the cache size is set to 1920 pixels

Yes. Force-updating only the cache image does not update the thumbnail stored in the database.
The Force Update Cache image is mostly designed for the situation where a use changes cache settings to smaller / larger or has purged the cache and wants to ensure that certain folders are fully cached while he's on-location or traveling.

DNG offers an option (at the expense of about twice the disk space) to embed the original RAW file. And the free Adobe DNG Converter has an extract option to extract the original RAW again. I don't know how often this is used, but since DNG is an "allowed" archival format (alongside TIFF) but RAW files are not (for obvious reasons), this might be used in libraries and archives perhaps.

Often colorimetry problems. 

Color management is always a bit tricky, especially when you work across devices, platforms and you deliver images to clients like printers, who actually really deeply care about color management.
Not sure why there should be a specific challenge with DNG files. They support embedded color profiles. Could be a workflow issue, depending on which application was used to create the DNG file.

[Dominique R]

A few years ago now, I know that capture one had trouble handling dng files generated by the Adobe converter. And it wasn't the only software that had problems with dng.

It's probably solved but it doesn't give me confidence.

The best solution would be (I think I read it in the online help) that camera manufacturers stop their proprietary format and save their file in dng format.

Dominique

[Mario]

The best solution would be (I think I read it in the online help) that camera manufacturers stop their proprietary format and save their file in dng format.

Yes. DNG is the standard for all Android phones, Leica and others. If it's good enough for them, it's good enough for other camera vendors too.

I guess the reason for the proprietary formats is control and user lock-in, like so often.

Or, if their RAW formats contain super-important business secrets and algorithms, they can just provide a WIC codec.
No need to reveal or document the proprietary format, just provide a WIC codec to read it and everybody is happy. They have the program code already.

I still recall when Nikon tried to make it illegal to reverse engineer their NEF/NRW RAW format by encrypting the white balance data (end some lens data, I think). Reverse-engineering is allowed by law in the US, breaking encryption is not. 

Adobe was really pissed about that at the time, so Nikon provided them (only Adobe!) with a library that was able to read the encrypted white balance and lens data. Which is really fishy. A camera vendor encrypting image file data and then give the key to only one selected company...

These days are over, I hope.