IMatch Not Always Acting On "Marked For Update" Folders

[Darius1968]

I just wondered why sometimes if I for instance, create a new folder from within IMatch, then add a few hundred or so files to it from an external source, and then, after some time, the folder will get marked with the blue dot, indicating it needs updated, but IMatch doesn't act upon that, and hence, the only way to bring those files into the database is via a manual refresh, or to close and reopen the database. 

[Mario]

The log file from that very session would give us a minimum of information to work with.
Maybe just close and re-open IMatch, in case a Windows file system notification has been locked?

[Darius1968]

Okay, I've attached the latest log file that represents a file added to my "C:\Users\Darius\Downloads\" folder, 20 min. ago.  Since then, at the time of producing this log file, IMatch had marked that folder, but hasn't yet added the file to the database. 

[Mario]

I see a number of warnings from ExifTool.
I see a number of error messages from Windows when it tries to produce s thumbnail.
I see a number of warnings about issues when trying to process MP4 video files.
I see some warnings about non-existing files when IMatch tries to sort in the file window.

Run a database diagnosis, reboot your system (to clean up all Windows shell thumbnail handlers).
This usually solves such random errors and warnings.

[Darius1968]

Okay, so, I followed your suggestions, and it did seem to clear up the problem for a while, but after a few hours, it started again, taking 15 min. or so for IMatch to recognize and import an image into that "... downloads" folder.  I'm attaching the latest log file for troubleshooting. 

[Mario]

I see the same error messages from Windows when it tries to extract a thumbnail from MP4 files.
IMatch uses the Windows function ShellExtractThumbEx when the FFMPEG 3rd party tool fails to extract a thumbnail for video files. Which is pretty rare. Then Windows also fails. My guess is that after several unsuccessful calls to ShellExtractThumbEx, something in the Windows file system blocks or hangs. ShellExtractThumbEx is notorious for blocking or hanging and Microsoft seems to be unwilling or unable to do something about it. This often depends on other shell extensions installed, the file format used, maybe the virus checker or whatever. Impossible to track down from the outside. IMatch just calls ShellExtractThumbEx to get a thumbnail for a file, and whatever this Windows function does is outside of my control.

I would suggest you re-encode the MP4 files to produce a "cleaner" and more standardized format. Then the FFMPEG library used by IMatch should have no problem processing the file and the Windows function is no longer called. If you have downloaded the MP4 files from somewhere they may contain all kinds of unwanted 'payload', even malicious payload aiming at cracking or hacking your system. The effect that FFMPEG fails and Windows somehow hangs or crashes in the thumbnail handler would indicate that there is something fishy with these files.

Re-encoding them in a video software may strip the bad content and produce a file FFMPEG and/or Windows can process.

[Darius1968]

Okay, but what I'm wondering about now is how MP4 files are even in the equation because I did not even have those in my scope during this time.  Are there any specific file names that you could say that were causing trouble?  Also, none of the media files in my database are flagged by my antivirus, even when I use the antivirus (Bitdefender) to manually scan the files.  So, is there any way to check these files to verify them containing some unwanted 'code' that needs 'stripped'? 

[Mario]

Search your log files for lines containing W> and a file name.

[Darius1968]

I found a couple such files.  Do MP4 files prefaced with "I>" count as well?  And, again, if such files have the potential to be malignant, why isn't Bitdefender flashing them, and what can I use to check these files for malicious content? 

[Mario]

I> is just an information. When looking for problems, the lines with W> and E> are the interesting bits.

I don't know if your files contain something malicious or if they just have been created in a way that prevents the renowned FFMPEG library from extracting a thumbnail and cause potential locking issues with the thumbnail handler in Windows. Virus checkers are of course also not perfect, especially not when something targets specific software like video viewers in web browsers, VLC or something.

I would re-encode the problem MP4 files with something like HandBrake or your video software and then see if this solves the issue. When it comes to obscure effects like this, trial-and-error is usually the only way to deal with it. Video files are sometimes an even bigger mess than metadata. So many formats around, and container formats like MP4 can contain a free mix of video and audio data, metadata streams and whatnot. A nightmare when it comes to long-term archival.

[Darius1968]

In the interim, between the log file I'm attaching now and the last one, I've since removed all such MP4 files from the database that were associated with a warning/"w>".  Again, this temporarily seemed to have fixed the problem - only having to wait for at the most, a few min. for files to be automatically ingested.  However, as hours went by, the delay problem manifested again.  This time, I can't find any MP4 files that have a warning flag, in the log file.  Can you tell me what the problem could be now, by examining this log file?  Thanks. 

[Mario]

Still the "invalid file" warning when IMatch tries to sort. Did you run a diagnosis?
IMatch has also scanned hundreds of files for updates (search for AddOrUpdate to find the names). Looks normal to me.