Setting up DB for first time: video files take way long

[evanwieren]

Hello... I'm new to iMatch... or rather, I have finally committed to the software. I have just started to create the DB of all my images/video. There are over 100k stored on a NAS. I tried once to add the whole thing at once and... well, after seeing an estimated completion time of 200+ hours, I stopped. I am now adding the subfolder to result in smaller batches. 
I've noticed that during the add process, video files take WAY longer than image files. I'm curious what I can do to speed the whole process up.
I'm cool with adding them later (i.e. is there a way to add only image files and then video later?).
I'm cool with having some way to add them "quickly" and then to process occasionally at some later date.
Heck... maybe (but not likely), I'd be ok with the video files not even being in the DB!

Any help would be super helpful.
TIA
Ed

[Mario]

NAS storage is the "worst case" for indexing, of course.
A NAS responds up to 1,000 times slower than a SSD plugged into your PC.

To index (add to your database) IMatch has to read the file once to get a checksum, file information etc. ExifTool has to read the file to extract metadata. FFMPeg has to read the file once to produce thumbnails. If the video file is large and the NAS is slow, this will reduce performance considerably. A video file stored on a NAS may take 100 to 1,000 times as long to process than the same video stored on a disk on the computer running IMatch. It's just physics, nothing IMatch can do about it. The video file has to be transferred from the NAS to your PC so Windows can cache it and IMatch, ExifTool and FFMPeg can process it. How fast this is depends entirely on the speed of your NAS, network and the size of the video files.

You did not include the IMatch log file (see log file) so I don't have any information about how long it actually takes to process one of your video files.

Usually, it is a a lot faster to process video files locally (from your SSD or spinning disk) before moving the video to long-term storage on the NAS box.

[evanwieren]

Hi Mario... Thanks... I do recognize that going across the network to access is crazy slow as compared to local SSD. With so many files over the years accumulated, it would be a bear to copy/move the data locally, run the process, and then go forward. However, perhaps there is a workflow for this that I can locate a document/video to help me.

I see that there are different levels for the logging. Is one or another preferable for you? When I opened the log file and searched, I did not see the mp4 files in it.
I am just now looking into what the best approach to attach a log file is within this forum.

[evanwieren]

Evidently, I'm not as smart as I think I am! I cannot seem to attach a file... so, here is a URL for a log file.

[evanwieren]

I have been reviewing other forum posts and some of your previous articles (eg IMatch and NAS Systems) to see what I can do differently. I should mention that the imdb is stored locally on my C: drive which is a NVME.2 SSD drive.

I'll continue to review some other posts to learn more about how to automate moving files locally to process them and then remove the local version. i.e. create a local mirror. At this point, I am not editing or tagging. I am simply trying to get all the files into iMatch.

[evanwieren]

I think I found the issue. It turns out that my antivirus was the issue. When importing 16,000+ files, IMatch ran quickly until about 1200 files in... then everything ground to a halt. I checked the network activity both from my local computer and on the NAS. Network utilization was very low and intermittent. It didn't seem to be the NAS at issue. My first thought was... what could IMatch be doing to process all that? However, checking the IMatch process, as it was also not at full utilization.

Due to previous experiences when working with numerous files (open/close/copy/etc), a light bulb turned on and I thought to disable the antivirus. Immediately, the IMatch progress bar returned to full original performance. Simultaneously, the NAS showed network utilization that was 10x or more than it was with the antivirus enabled. Better still, the NAS was at consistent high utilization rather than the previous spiking.

I need to determine what exceptions need to be added and what I'm willing to risk. Perhaps disabling it during the initial setup will be acceptable.

[mopperle]

I do not know which AV software you are using, but some AV scanners are far too sensitive, causing all kind of problems. Thats why I only use The scanner included in Windows.
In your case, excluding the IMatch exe from being scanned should be enough.

[Mario]

I think I found the issue. It turns out that my antivirus was the issue.

That would have been my second question after looking at the log file.
Virus checkers misbehaving these days accounts for maybe 95% of all "IMatch is slow" support tickets.
See IMPORTANT: Virus Checkers for some more info.

Depending on which IMatch operation triggered your virus checker (did it not inform you? Maybe there is a log file you can  look at) you should

1. Exclude the folder (!) containing the database file
2. Exclude the IMatch executable 'C:\Program Files\photools.com\imatch6\IMatch2023x64.exe'
3. If your virus checker blocked ExifTool, exclude the 'C:\Program Files\photools.com\imatch6\exiftool.exe'

This should do the trick.

Looking at your log file, I see that IMatch used less than 1GB of RAM, but 85% of the available memory was used while IMatch was running. This is OK, plenty of RAM available.

If you notice that the memory utilization goes higher than 90% while IMatch is indexing files, you can reduce the number of files processed in parallel under Edit menu > Preferences > Application. See Process Control (Advanced Setting). Set the thread count for reading files and reading metadata to 4 or so. This means IMatch processes only 4 files in parallel, which reduces the amount of memory ExifTool, the WIC codecs etc. use.

I see a number of ExifTool warnings about to small FPXR segments and invalid maker notes in files. This can happen if you processed these files with software that did not care for metadata or or software that re-saved JPG files containing maker notes without relocating or deleting the maker notes, breaking them in the process.
You can search for W> in your log file to find the corresponding file names.

I see a number of warnings about unlabeled XMP face regions. Adding face regions to files but not adding a tag/label/name was a problem Apple devices had, for some time, when I recall correctly. IMatch imports the XMP region and create a face annotation, but it cannot link it to a person via the tag.

I see a number of warnings for "Only thumbnail or small preview extracted. Falling back to ptc RAW processing.".
This means that IMatch tried to read a RAW file via the standard WIC codes in Windows. but the codecs could not process the file. IMatch tried to compensate by falling back to LibRaw. The log file is not in debug logging mode so I cannot see the file names.

All this is normal when users index their file collection for the first time. A lot of "stuff" and problems have accumulated over time, with files produced by multiple devices and processed with a variety of software.

Tip: To attach a file when you use the Quick Reply box under a post, you need to press Preview once to see the Other Options below the post editor, including the attach file options.
If you click the Reply button for a post (not using the Quick Post editor) you see the options immediately.

[evanwieren]

Good day Mario,

Thank you for those extra tips and suggestions. I'll review those during the day. I did notice that after I had restarted iMatch, the full debug logging was enabled. I was unaware that I had to restart after changing that setting. I'll also keep those logs to review for issues in files at a later date.

And... yay! There's the attachment area for files. 

Thank you again,
Ed

[evanwieren]

Oh... lastly, in the case someone needs to know, I am use Avast anti-virus.

[Mario]

Since the free Windows Defender became so good, anti-virus companies have a hard time justifying their annual cost.
So they come up with all kinds of tangents, browser plug-ins and frequent feedback to users about how "good" they are.

Unfortunately, in order not to upset/confuse modern users, virus checkers work silent, meaning that they don't tell the user that they have interfered and blocked some software. Which of course is stupid, since the user has no idea why software A suddenly stopped working. If you're lucky, you'll find some info somewhere buried in the virus checker log file.

A simple "Hey user! I've just blocked this software called IMatch from reading files because I think it is a virus. What do you want do do?" like in old times would be so much more helpful. But, nope...