importing Videos: Why does it take that long?

[abgestumpft]

Hi,

I was importing 552 videos clips (.mp4 / H264 / 4k - maybe 1min runtime in average).
iMatch was importing them for several hours (5h if I interpret the log correctly: was running overnight and was still running after 2h when I checked the last time), during that time HDD read was at 100% and iMatch / ffmpeg processes running on files.

I just wonder why it takes that long to query the metadata and extract 5 thumbnail frames from each video.

Also when I open a video in the viewer for the first time, it takes around 20sec. until the video thumbnail is displayed (also HDD at 100% during that time).

[Mario]

It would help if you would provide at least minimum of information.
Like, are your files on a local SSD or a slow remote NAS box? Are they 200 MB or 4GB in size?
Virus Checker has an exclusion for IMatch and the database folder? In 95% of all "IMatch is slow" cases, this is the problem.
The IMatch log file from that session, which would tell us what takes how long (see log file).

[abgestumpft]

Windows 10
No dedicated Virus SW installed (using built in Windows Virus Scanner)
iMatch DB and cache are on a nvme SSD.
Videos stored on local classical SATA HDD (6TB) (Fotos are stored on a different SATA HDD drive -> import performance of fotos is good)

Sizes vary from several hundred MB to around 2GB, here some examples:

Code Select Expand

09.08.2020  14:39       980.297.728 P8098078.mp4
09.08.2020  17:34     1.739.125.760 P8098079.mp4
09.08.2020  17:47     2.147.572.224 P8098080.mp4
09.08.2020  17:48       451.295.232 P8098081.mp4
09.08.2020  17:50       701.160.448 P8098082.mp4
10.08.2020  12:01     1.876.869.632 P8108352.mp4
10.08.2020  12:02       254.859.264 P8108353.mp4
10.08.2020  12:04     1.425.069.568 P8108354.mp4
10.08.2020  12:06     1.055.043.584 P8108355.mp4
10.08.2020  12:07       205.578.240 P8108356.mp4
10.08.2020  12:07       201.835.520 P8108357.mp4
10.08.2020  17:34     2.172.019.200 P8108358.mp4
10.08.2020  17:36       185.118.720 P8108359.mp4
10.08.2020  17:37       458.017.280 P8108360.mp4
10.08.2020  17:39        42.584.576 P8108361.mp4
10.08.2020  17:40       690.142.208 P8108362.mp4
10.08.2020  17:43     1.828.627.456 P8108363.mp4
12.08.2020  16:43     1.640.658.944 P8128673.mp4

Logfile did not log very much:

Code Select Expand

10.08 22:30:42+  125 [2A00] 02  I>   Caches started in 125ms.
10.08 22:30:43+ 1313 [2A00] 00  I>   # Process Memory Info: WSC: 518MB, WSP: 558MB (NEW PEAK), PF: 1172057
10.08 22:30:43+    0 [2A00] 00  M>   <  2 [3922ms] CMainFrame::LoadDatabase
10.08 22:30:43+    0 [2A00] 00  M>  <  1 [6313ms #sl] CIMatchApp::InitInstance
10.08 22:30:45+ 1281 [2A00] 00  M>  >  1 CIMGeoLocationManager::GetLocations  'V:\develop\IMatch5\src\IMEngine\IMGeoLocation.cpp(228)'
10.08 22:30:45+    0 [2A00] 00  M>  <  1 CIMGeoLocationManager::GetLocations
  file://X:/Videos/2020/ Media-ID: VIDEOS, Serial 1119716875
  file://X:/Videos/2020/ (VIDEOS 1119716875)
...
<LIST OF FOLDERS>
...
10.09 00:13:16+6151609 [2A00] 01  W> Spelling: Cannot find a dictionary for language 'en' or failed to load.  'V:\develop\IMatch5\src\IMatchNG\IMatch.cpp(3842)'
10.09 03:32:35+11958547 [2A00] 00  M>  >  1 CIMGeoLocationManager::GetLocations  'V:\develop\IMatch5\src\IMEngine\IMGeoLocation.cpp(228)'
10.09 03:32:35+    0 [2A00] 00  M>  <  1 CIMGeoLocationManager::GetLocations
10.09 09:33:00+21625328 [2A00] 01  W> Spelling: Cannot find a dictionary for language 'en' or failed to load.


I'm just asking because me feeling is that during Import a hughe amount of data is transfered (disk with Videos at 100% all the time).
When I manually run exiftool and ffmpeg on that file, the commands are finished within one second each:

Code Select Expand


"C:\Program Files\photools.com\imatch6\ffmpeg.exe" -i P8098078.mp4 -vf "select=eq(n\,34)" -vframes 1 x:\out.png
"C:\Program Files\photools.com\imatch6\exiftool.exe" P8098078.mp4


Is IMatch doing more than reading metadata and extracting 5 frames during video import?

[Mario]

1. Please always include the full log file. There are important bits at the beginning and elsewhere.
2. Don't just copy & paste log file fragments. This does not help at all and is fills the community search engine with junk
3. Switch to debug logging (log file) when you try to analyze something. This produces much richer log files
4. ZIP and attach the log file.

IMatch does a lot when importing files, from calculating check sums to visual query data, a probably much more metadata import that you did, extracting frames etc.
For the checksum it of course has to read the entire video file. And it does that in PARALLEL for multiple files at once.

If this brings down your system so much, reduce the number of parallel threads IMatch uses.
See Process Control (Advanced Setting)
Set this to maybe 2 or 4, depending on how maxed out your system is.

[abgestumpft]

Did now a quick test:
Same folder with 18 video files (and one text file) one time on X: (video HDD) and one time on C: (nvme SSD)

HDD: 10.09 11:16:10+    0 [12B30] 10  I> EUQH: Processed 19 files in 275265ms
SSD: 10.09 11:21:15+    0 [11928] 10  I> EUQH: Processed 19 files in 165062ms

Is there some rescaling of the full video happening to 640x360 ?

Logfile contains:
Extracting video with: "C:\Program Files\photools.com\imatch6\ffmpeg.exe" -y -hide_banner -ss 00:00:02 -i "<filename>" -filter:v "fps=1/5,scale="min(640\,trunc(640*dar)):min(640\,trunc(640/dar))"" -vsync 0 -frames:v 10 "C:\Users\SEBAST~1\AppData\Local\Temp\582-%04d.jpg"

0 [11928] 10  I> EUQH::Load(1) of <filename> with 640 x 360 (O: 640 x 360) in 9437ms


When running import with one file only, it takes:
14sec on HDD (3,5sec checksum + 9.7sec for 640 x 360 rescale?) - see attached for part of log from that import with debug enabled.
11sec on SSD (0,5sec checksum + 9.9sec for 640 x 360 rescale?)


Cross-checking the logs from the 18 video files, depending on filesize we have for each video:
HDD: (3 - 13sec checksum) + (4 - 10sec  for 640 x 360 rescale)
SSD: (0,1 - 1sec checksum) + (4 - 10sec for 640 x 360 rescale)


When comparing HDD vs SSD time the SSD is faster in "Calculate CRC for file" (includes initial load of video from disk?)
For "Load(1) of <filename> with 640 x 360 (O: 640 x 360)" they are identical (-> should be CPU tasky only)

[Mario]

It takes about 10 seconds for FFMeg to extract and resize the thumbnails IMatch uses.
So, the other 1 or 2 seconds is taken by IMatch for all the rest.

Did you reduce processor load as I suggested?

[abgestumpft]

Yes, this runtime maily seems to be checksum + ffmpeg thumbnail extraction.
When I reduce the number of thumbnails from 10 to 1, then the ffmpeg-runtime is reduced from ca. 10sec to ca.3sec

I did not yet change the processor load, because so far (with Fotos) I'm running very good with my config.

Maybe for video import it's a little to high:
1 Video (1GB) from HDD: 14sec
18 Videos from HDD: 275sec (14sec x 18 = 252sec)
552 Videos from HDD: ~5h (14sec x 552 = 2,1h) - still have to verify if the 5h are correct though -> as said I checked after 2h and there it was still running. Will check with next import using debug logging.

Anyways I'm currently doing a step by step one time import of my Videos to iMatch now. In Future I would only add new Videos -> usually only like 1-10 files at once.
I just did not understand why it took that long. But checksum + ffmpeg thumbnail extraction now explains this.

Is there a reason iMatch uses 640px as fixed value for videos and ignores the Preferences->Database->Thumbnail size parameter?