nubie question - Video file thumbnbail

[Christos]

Hi all,

I am really sorry of this has been asked (and answered) before. (I was not able to identify any similar threads).

I'd like to be able to see thumbnails of my video files. Instead I can only see the vlc icon.
I guess I am doing something wrong, but am not sure what.

Thanks

[Mario]

Hi. Welcome to the community.

IMatch produces thumbnails and short preview sequences automatically for video files using FFMPeg .
This seems to have failed on your system for some reason. This could have many reasons:

1. The video files are unsupported by FFMpeg. Which is unlikely.
2. The virus checker blocked IMatch from running FFMPeg.
3. There was some sort of overload or similar.

I assume you have not kept the IMatch log file (see log file) and so we'll never know what happened.

Select some of the problem files in a File Window, press Shift+Ctrl+F5 and use "Force Update". 
If this also fails, make a copy of the log file via Help menu > Support and ZIP and attach it. This will give us more info to work with.

If it works now, the most likely reason is that the virus checker does currently not interfere or that at the time the video files were initially processed, there was some sort of system overload or cascading problem. Without a log file, impossible to tell.

[Christos]

Dear Mario,

Thanks for your response.
I am not sure what is happening.
I used the Shift+Ctrl+F5 / "Force Update" option and it worked with some of the files but not with all. Not sure if there is any pattern in this. I guess the best at this early stage would be to send you the log file, in order to eliminate obvious problems / errors of mine.

Here it is.

Thanks a lot for your help.

Best

[Mario]

This log file shows no trace of files being added or rescanned?

Please do this:

Switch IMatch to debug logging via Help menu > Support.
Select two of the files which failed and do a Shift+Ctrl+F5 > Force Update.
If this fails again, we should have error messages and warnings in the log file that help figuring this out.

I see some warnings from the IMatch MediaPlayer: 'No suitable transform was found to encode or decode the content.' which means that the format of the video was not supported.

But that's independent from creating thumbnails or preview images for video files.

[Christos]

Thanks Mario,

I have now some more info.
It appears that some of the files can be updated when forced. But there are some other files that cause IMatch to freeze. It doesn't actually freeze. But it won't proceed with updating.
For example I see that the program reports "Adding and updating files (25%) 6 Minutes", but it has been like this for the last half an hour.

This is where I am now:

I prepared the log file while this is happening and I attach it here.
I tried to close the program. It stayed with the closing database dialog, reported as no responding.

Then the program closes.

When I opened it again, and it reports that it did not close properly.
I attach here the backup log file created.

Thanks a lot,

Christos

[Mario]

The log files are not in debug logging mode as I suggested, so the contained information is minimal.
Enable debug logging via Help menu > Support before doing your test so we get more data in the log.

This is a rather small database about 64K files.
The log shows IMatch starting, loading the database etc.
After loading everything, IMatch reports that it uses 839 MB of memory (RAM).

On 05.07 09:46:09, IMatch scans files (probably triggered by you).

ExifTool processes metadata for the files
"D:\Videos\Completed Work\Final Films\Compressed\Art\1984 Î'νασκόπηση (1080).mp4"
"D:\Videos\Completed Work\Final Films\Compressed\Art\1985 Ιωσήφ Î" (1080).mp4"

and reports the warning: "Warning: [Minor] Extracted only 1000 xmpMM:Pantry items. Ignore minor errors to extract all" which seems that the video contains a ton of XMP pantry metadata entries, for whatever reasons.

what's more important is that IMatch reports a massive memory peak of 21,829 MB (22 GB!) after ExifTool has processed the first file.  This drops to 6,909 MB after processing the second file.

Then IMatch produces some cache images, performs some processing and then the BACKUP log file ends. The other log file shows a similar scenario, also a massive memory peak.

It looks like ExifTool is not liking something of the metadata in these videos, allocating a massive amount of memory while trying to process these video files. It may even get stuck or continue to remain in memory.

Since IMatch usually processes many files in parallel (depending on the CPU core count, which is 12 in your case), ExifTool behaving this way can very well bring down IMatch to a crawl. IMatch monitors ExifTool and terminates "stuck" instances after two minutes, but if 20 or more GB are allocated, who knows what Windows is doing.

I have never seen this behavior very often so I have no idea what makes ExifTool do this. Can you provide a download link to the two files I mentioned above and send me this link to my support email address with a link back to this thread (support email address).

If I can reproduce this behavior here, I can forward the link to Phil Harvey from ExifTool so he can check what the problem is with these files. I remember a similar issue some years ago, where broken XMP metadata which caused infinite loops caused a similar behavior in ExifTool (which is why now default limits like the 1,000 entries) are in place.

Maybe just re-encoding the video files with Handbrake or removing / rewriting the metadata with ExifTool can fix this. Not sure.

[Christos]

I am sorry, I did not notice the "debug mode" suggestion. 

I am not near that computer now, but will be later this evening (GMT).
I will then try to prepare the debug log file.

It might well be a problem with those particular files, as you suggest. And of other similar files.
But if this is the case, how can we explain the fact that in the beginning I did not have any video thumbnails. None at all. (Please bear in mind that most of my video files are straight out of camera.)

Be it as it may, later this evening I will experiment a bit and will try to provide more useful log files.

Thanks a lot for your help.
 

[Mario]

But if this is the case, how can we explain the fact that in the beginning I did not have any video thumbnails.

Maybe the system was already thrashing (running low on memory) during that session. Hard to tell without a log.

Please do this: When processing the files, go to Edit > Preferences > Application and reduce the number of threads for import and metadata import to 2 (default is 0). This reduces memory consumption and load on your system.

See Process Control (Advanced Setting) for more info.

You can reset this back after processing these files.

Which camera produces these videos with so much XMP::xmpMM pantry entries? That's unusual.

[Christos]

Which camera produces these videos with so much XMP::xmpMM pantry entries? That's unusual.

No no, these two files (and quite a few more) were produced by Premiere. Perhaps there is something there.

The mail bulk of the rest of my video files, however, are straight out of camera. I was wondering about those.

So, anyway, we'll see. I am not near my computer at the moment. When I will be, later this evening, I will do as suggested and report back to you.

[Mario]

No no, these two files (and quite a few more) were produced by Premiere. Perhaps there is something there.

That was my thought.

I need to see some samples before I can give any advice.
Usually all this "just works". Premiere videos are not that uncommon among IMatch users, as is Resolve or Final Cut.

There seems to be something in your files IMatch / ExifTool / FFMpeg don't like.
Maybe Adobe added yet another "enhancement" that makes the file format not compatible. Like recent DNG versions with JPGXL which is not supported outside the Adobe world (yet) and breaks WIC codecs and LibRaw...

[Christos]

Hi Mario,

Here is the updated debug log.
I reduced the number of threads as suggested.

The program seems to remember from previously that I asked it to update thumbnails and starts "Adding and updating files" as soon as I open it.

Therefore I could not see it reacting to the Shift + Ctrl + F5 / Force update in any specific way.
It was already updating.

I can now see some thumbnails but I am not sure which ones have a problem and which not (on guessing that the system did not have the time yet to parse all of them).

[Mario]

The program seems to remember from previously that I asked it to update thumbnails and starts "Adding and updating files" as soon as I open it.

Yes. IMatch ads new and modified files to a processing queue and processes that queue in the background. If you close IMatch while there are still entries in the queue, it will continue processing these entries when you start it the next time.

This log file just contains a few warnings:

- spell checking in IMatch is disabled because you have not installed a dictionary for your language (see: Download Dictionaries for Common Languages)

- IMWS needed several attempts to find a free port since the default port configured under Edit > Preferences > Application is blocked by another application or service.

- You have not entered your email address and license key (License Information) so the automatic updater cannot work. It will remind you when it runs next time

That's it. No file was processed at the time you copied the log file. IMatch was basically just finished starting up.
Wait a bit longer next time. Tip: You can see in the Info & Activity panel if there are entries in the queue and how they are being processed.

[Christos]

Update:

I started the program again at 8:30am.

These have been my actions.

As soon as I started, in the Info & Activity panel it reported "Adding and Updating files", 0 of 36 files processed.
10 minutes later it was still at the same: 0 of 36 files processed.
I decided to clear the queue. It took 2 to 3 minutes to acknowledge this, but then the queue was emptied.

I asked to force update one file. Strangely it tells me that the media format is not supported, but still it created the thumbnail.
I then asked it to force update 5 more files. It took some time (~2 minutes) but it completed the task.

Then I asked to force update one of the files that had this memory peak yesterday. It has completed the task with no visible problems. (Haven't checked the debug log).

Then I asked to force update 2 files that I anticipated would have issues. They did not.

And finally I asked it to update 7 files which were straight out of camera. They were updated pretty fast (less than 2 mins, possibly less than one).

So here I am. It's about 9:10am now, i.e. 40 minutes after I started the program, and I will first collect the debug log and then close it.

Hope this all makes sense and is helpful in some way.

Please note, I will not be near this computer until much later today.

Thanks again.

:-)

[Mario]

it tells me that the media format is not supported,

Where does it do that? IMatch does not display such messages while indexing files?

Do you maybe have the QuickView Panel open with the Media Player loaded? Maybe Windows is locking the file because it runs into an error displaying it? Close the Quick Preview Panel to rule that out.

I see many MediaPlayer error messages about an unsupported format.
I see the same ExifTool warnings about more than 1000 pantry entries in the XMP record.
I see warnings that processing the DNG files (?) produced only a thumbnail or a preview that is smaller than 2,000 pixel, and IMatch tries again with LibRaw to see if it can produce a larger image.

Please run a WIC diagnosis for the file (D:\Pictures\Φωτογραφίες\Digital\2024\20240403-0430 April\20240419\20240419 133020 (_1001801).DNG) via Help menu > Support > WIC diagnosis and ZIP and attach the results.

I have still not seen a video sample or image sample yet, so I don't even know which device you use. Many cameras and smart phones emit one of many DNG variants these days...

Memory peak (max. IMatch memory usage) was about 8 GB this time, which is still way, way too much.
Either ExifTool allocates too much memory or you running Media Player in IMatch for files you are processing while you are processing them and the Windows Media Framework being unable to handle the video format variant you use plays a role in this.

Without samples for your DNG files and video files here in my lab, this is all I can do.

[Christos]

Good morning,

Here is the WIC Diagnosis for the file in question.

I am not sure what is supposed to be going wrong with this file.
It is shot with a relatively older camera (Panasonic Lumix TZ200) that has been already used for hundreds of other files in the database.

It might be just a random problem with this specific file. I cannot be sure. But I can open it and work on it with no problems.

Regarding the video files, I understand what you say, but I am afraid that I do not feel comfortable to start circulating them in the internet. If there are tools I could use locally to diagnose them, I am more than happy to help. Please let me know what I need to do.

But for your information, most of my recent video files (the "straight out of camera" files) are shot with Lumix GH2, Lumix G80, and Lumix G9II. That is, not with any strange or arcane mobile phone / camera.

Premiere files, on the other hand, are encoded as either MP4 or MXF (Avid). The ones that made IMatch complain have a slightly unconventional frame size (1520x1080). I cannot say if this is the reason for the odd behaviour observed.

As I said, if there are tools I could use locally to diagnose them, I am more than happy to help. Just let me know what to do.

I hope this all helps.

[Mario]

The Microsoft Windows WIC codec can extract an image with 1024x683 pixel as the "Full resolution".
Which is smaller than the minimal cache size (2000 pixel) and hence IMatch falls back to loading the file with LibRaw, which extracts a preview with 1024x683 but can also extract a full-size image with 5480x3664 from the file.

Seems like this is a DNG variant the Windows WIC codec does not handle well. There are so many DNG variants in use that this is rare but not unheard off.

This explains why IMatch had to fall-back using LibRaw and develop the full image data instead of using the embedded preview.

For the video file and massive memory consumption I cannot diagnose anything further without having one of the problem video files here in my lab. You can upload one to your cloud space and send a link (with a link back to this thread) to support email address.