57.217 files

[timur]

Hi,

i am trying to import a few of my files into imatch 2023. I started first with my first folder and then added another folder. It says it will take 120 hours to add and update the files (Import of Meta data).

This is unreasonable and I have still so many files which I want to import. I have rough 5 TB of pictures from the last 20 years.

My C drive is small and I use external HDDs. I changed the preview file making to the same HDD as were the pictures are. can this be the problem? What is the recommendation to speed things up. I don't believe at this rate that 120 hours will be enough for 57k files.

Any ideas?

[Mario]

IMatch adds between 50 (RAW files) and several hundred (JPG files) per minute. Videos take longer.
The time estimate shown varies and is based on the past 20 to 50 images processed.

IMatch processes many files in parallel and a storing the database on an SSD is HIGHLY recommended.
I understand that you have both regular disks for image and database storage?
Using a spinning disk for the IMatch database is much slower than using an SSD.
Using external spinning disks for the images is OK since IMatch only touches the images once, unless you write-back metadata later.

Note: Always include a ZIPped copy of the IMatch log file (see log file) when you report such issues.
The log will tell us what IMatch is doing, how it is performing and what may slow.

For example, it is not uncommon that the virus checker a user has installed terminates ExifTool (which is a software IMatch is using for reading and writing metadata), causing extreme delays and very sluggish behavior.
The log file will tell us. Please attach it to your reply.

Recommendations:

1. Edit menu > Preferences > Cache: Switch to "on-demand" for cache image generation.

2. If your PC is older or a notebook, it may be overloaded by IMatch processing too many files at the same time.
Go to Edit menu > Preferences > Application. Search for process to find Process Control. Set each setting, except face recognition, to the value 2.
See Process Control (Advanced Setting) for more information.

[timur]

Thank you Mario! I just ordered a 2 tb ssd and will try it again then.

[Mario]

A SSD is the greatest boost for Windows performance you can get (except from buying a new PC).
If you so far booted Windows from a regular spinning disk, and now swtich to SSD, you're in for a treat

[timur]

The 2TB SSD is in. I created a new database and started from scratch since I did not do much on the first one. Database and previews are done on SSD and Pictures are still on External HDD.

In the beginning it looked great. Then I went to bed and on the next day at around 2 pm I checked how long it is still going and it said 654 hours...

My CPU is between 60-70% I7-3770 @3.4 GHZ.
Ram at 13GB from 16GB

The SSD is low in usage but the HDD is at maximum.

Not sure if this information helps.

[Mario]

60K files is not much. I produce databases of this size frequently for testing.
You did not attach the log file (see log file) so we cannot tell what IMatch is doing, what is taking so long. Which kind of files you are processing etc.

HDD at max. means IMatch is processing images as fast as possible.
Maybe your virus checker is interfering?
Maybe ExifTool is choking on corrupted files.
Impossible to tell without the log file.
Maybe your PC becomes overloaded? 13 GB RAM usage seems quite high. But we don't know how much of it IMatch is using.
Microsoft's WIC codecs are known to sometimes using huge amounts of memory when certain RAW files are processed.

Maybe reduce the number of threads IMatch is using to 4 or 6. See Process Control (Advanced Setting)
If IMatch does too many things in parallel and your PC becomes 'stressed', reducing the threads may result in better performance.


Tip: Use Help menu > Support > ... to copy the log file, ZIP and attach.

[timur]

Here is the attached log file

[Mario]

Nearly every of your files has some metadata problems....
ExifTool reports warnings like

IPTCDigest is not current. XMP may be out of sync - G:\Fotos\D2x\2006-01-20\Schneemann.tif

This means some software has updated the metadata in your files, but did not update/write the required digest/checksum.
Sloppy metadata handling.

Invalid date/time (use YYYY:mm:dd HH:MM:SS[.ss][+/-HH:MM|Z]) for XMP-xmp:ModifyDate - G:\Fotos\D2x\2006-04-04_newSigma10_20\_D2X6436.NEF

Some software has written date and time in an invalid format.

Fixed incorrect URI for xmlns:MicrosoftPhoto - G:\Fotos\D2x\2006-06-12_17_55\_D2X7367.NEF

Some Microsoft (presumably) software has written proprietary Microsoft metadata, but did it wrong.
A known problem with many Microsoft products. Not even Microsoft cared much for their proprietary (and now abandoned) metadata.

Import for [123251] 'G:\Fotos\D850\20230622Japan2023\iphone Japan\IMG_9800.JPG' failed.

This looks like you are working with files from an iPhone and an IMatch version older than the current 2023.1.14 release.
ExifTool had a bug that caused the XML data produced for some iPhone versions to be invalid. This has been fixed in the mean time. Please update IMatch.

Then IMatch is processing many .VOB files. This is an old video container format used by DVDs, I believe.
IMatch tries to extract a preview both with its internal decoder and Windows WIC but fails.
Each attempt takes several seconds.

Then I see dozens of warnings

Failed to load image with PTD2D.

which is logged when IMatch tried to load a file via Windows WIC but failed. Each of the failed attempts takes between 1 and 3 seconds. Your log file is not in debug mode (see log file) so I don't see the file names.

All this mess will surely slow down the indexing process.

[timur]

Here is the Debug file. Computer is running now over 2 days constantly and over 1000 hours still to go. Preview mode changed, ssd changed, antivirus turned off. I think I need to give up...

[mopperle]

...I think I need to give up...

or solve all the problems being found. Or do not import all files together, e.g. do import the vob files later

[mopperle]

And your last logfile seems to be incomplete/not the one Mario wants.

[Mario]

Darn, it seems the long answer I wrote is somehow gone. Probably I had too many browser windows open again and lost track and closed it...I like my work so much, I do it twice.

First thing I see in this log is

Error loading G:\Fotos\d800\20121018_hafencity\_DSC2069_mond1.jpg with error 1 'File Sharing Error'

This means that the JPG was open in another application?
IMatch tries again and now gets:
Error loading G:\Fotos\d800\20121018_hafencity\_DSC2069_mond1.jpg with error 1 'Unknown image type.'
which is rare for JPEG files. Something is wrong here.

Then

Failed to open file G:\Fotos\d800\20121024_ausflugsophie\_DSC2113.NEF with -100009
88982F50 - Die Komponente wurde nicht gefunden.

This means that the Windows WIC codec installed cannot read the NEF file. Either a file access error again or an incompatible NEF version or a corrupted file.

All these errors need to be checked and solved the user. IMatch cannot do more.
Also all the other metadata warnings ExifTool logged in the previous log file. The files need to be checked, e.g. with the The Metadata Analyst app.

Note: the IPTC digest and the Microsoft namespace error can be ignored. They are fixed by ExifTool once you write back the file.

For the failed NEF file, run a WIC Diagnostics to see if the file is supported or which WIC codecs are installed.
For the failed JPEG, open it in another application to see if the file lock is gone.


What's way more interesting towards the original effect of this bug report:

00:21:03 IMatch finishes initializing.
    then nothing happens until
05:35:16
    when IMatch reports the file sharing error for G:\Fotos\d800\20121018_hafencity\_DSC2069_mond1.jpg

This is highly unusual unless the PC was sent to sleep.
There are several care taker processes in IMatch which run occasionally and log some data to the log file.

This happens again

07:43:26 Error loading G:\Fotos\d800\20121024_ausflugsophie\_DSC2113.NEF using ptcRAW with error -100009
    then nothing happens until
15:36:51 when the log level is changed to "Very High" and, apparently, the Dashboard is opened.

A 5 hour and over 2 hour pause in which IMatch sits idle and does nothing, not even background tasks?
Very unusual. And if IMatch is forced to do nothing for such long stretches, not much work gets done.

Are you sure your virus checker is not interfering and blocking IMatch?
Results like this in the log would be an indicator for something pausing/blocking IMatch for hours.
See IMPORTANT: Virus Checkers for more info.

[thrinn]

Just to rule out hardware or cable problems with you external HD, I would copy maybe one or so folder with some of the problematic files to the SSD and import them from there. If the same files can be imported, this might hint at some problems with you external HD.

A a wild guess: You don't have Immediate Metadata Writeback enabled, do you?

[timur]

Just to rule out hardware or cable problems with you external HD, I would copy maybe one or so folder with some of the problematic files to the SSD and import them from there. If the same files can be imported, this might hint at some problems with you external HD.

A a wild guess: You don't have Immediate Metadata Writeback enabled, do you?

What do you mean with metadata writeback enabled? I don't think so.

[timur]

Thank you so much for the long answer. I will start the process one more time and start with a small folder.

I have standard windows 10 antivirus activated. I turned it off but it did not change much. Yes, the pauses are very weird. I see that sometimes that it is working on one file for a very long time.

Darn, it seems the long answer I wrote is somehow gone. Probably I had too many browser windows open again and lost track and closed it...I like my work so much, I do it twice.

First thing I see in this log is

Error loading G:\Fotos\d800\20121018_hafencity\_DSC2069_mond1.jpg with error 1 'File Sharing Error'

This means that the JPG was open in another application?
IMatch tries again and now gets:
Error loading G:\Fotos\d800\20121018_hafencity\_DSC2069_mond1.jpg with error 1 'Unknown image type.'
which is rare for JPEG files. Something is wrong here.

Then

Failed to open file G:\Fotos\d800\20121024_ausflugsophie\_DSC2113.NEF with -100009
88982F50 - Die Komponente wurde nicht gefunden.

This means that the Windows WIC codec installed cannot read the NEF file. Either a file access error again or an incompatible NEF version or a corrupted file.

All these errors need to be checked and solved the user. IMatch cannot do more.
Also all the other metadata warnings ExifTool logged in the previous log file. The files need to be checked, e.g. with the The Metadata Analyst app.

Note: the IPTC digest and the Microsoft namespace error can be ignored. They are fixed by ExifTool once you write back the file.

For the failed NEF file, run a WIC Diagnostics to see if the file is supported or which WIC codecs are installed.
For the failed JPEG, open it in another application to see if the file lock is gone.


What's way more interesting towards the original effect of this bug report:

00:21:03 IMatch finishes initializing.
    then nothing happens until
05:35:16
    when IMatch reports the file sharing error for G:\Fotos\d800\20121018_hafencity\_DSC2069_mond1.jpg

This is highly unusual unless the PC was sent to sleep.
There are several care taker processes in IMatch which run occasionally and log some data to the log file.

This happens again

07:43:26 Error loading G:\Fotos\d800\20121024_ausflugsophie\_DSC2113.NEF using ptcRAW with error -100009
    then nothing happens until
15:36:51 when the log level is changed to "Very High" and, apparently, the Dashboard is opened.

A 5 hour and over 2 hour pause in which IMatch sits idle and does nothing, not even background tasks?
Very unusual. And if IMatch is forced to do nothing for such long stretches, not much work gets done.

Are you sure your virus checker is not interfering and blocking IMatch?
Results like this in the log would be an indicator for something pausing/blocking IMatch for hours.
See IMPORTANT: Virus Checkers for more info.

[Mario]

1. Copy the folder G:\Fotos\d800\20121018_hafencity\_  to your SSD C: .
2. Create a new database in IMatch
3. Add the hafencity folder to the new database.

Any problems?
Performance good?

Just to rule out that the G: drive is the problem.

[timur]

Hi Mario,

G is not the problem. I could import my D70, D700, D2x folders. Then it came to the D800 folder and it got stuck again. For 3 days the computer was running now and between 50 hours and 600 hours was shown. The progress is minimal. The D800 files are pretty big compared to the D700 files for example.

I turned off the preview making and used the new SSD as Data Base folder and also as preview folder. All my pics are on G.

I created a Debug File which is pretty huge unpacked (600mb) As Zip File it was 11 mb and too big. As a rar I could get it under 5mb

I can not upload a rar file so I made a Gdrive Folder with the zip, rar and original file:
https://drive.google.com/drive/folders/1XEah_GOxN4DU6SLGNRtN1PORUh-jgK4e?usp=drive_link

Final try and too bad that this is so difficult. Not sure if may be capture 1 is the issue which I normally use to edit files and I believe I started using Capture 1 with D800. So may be Capture 1 is doing sth with the Raw Files and their Exif Data. Otherwise I have no explaination. The VOB files were no problem by the way and it was just a few files since I produced one video a long time ago and the files were in the same folder.

Best
Timur

[timur]

PS: Computer went not to sleep and worked tirelessly for days :-(

[Mario]

The files behind your link are not accessible without a Google account. Which I don't have.
This is not a public link. If you have privacy issues, you can send the files to support email address.

Did you run a WIC diagnosis as suggested?
What does it show?

Which WIC codec for NEF files is installed? The standard codec in Windows or did you install the one from Nikon?
The WIC diagnosis will tell us. Attach the result.

If this affects only certain RAW variants, the most likely culprit is the WIC codec that crashes or runs amok with your files. One of the side effects on using proprietary RAW formats.

Switch IMatch to photools.com RAW processing.
Go to Edit > Preferences > Application and search for photools.
Close and re-open IMatch.
Force-Rescan a folder with D-800 files to test (select the folder in the Media & Folders View and press <Shift>+<Ctrl>+<F5> to force rescan it.
If this works, the problem is the installed WIC codec and/or your RAW variant.

[timur]

I have not done the WIC diagnoses and will check. Nikon has changed sth. with the D800 if I remember correctly. I did not install anything as far as I remember. What could I install as a codec to change this behaviour?

I will do the WIC test today.

[Mario]

D800 files usually cause no issues. Not with the built-in WIC codec and neither with the one provided by Nikon.
See WIC (Windows Interface Components) and photools.com RAW Processing for more info.

[timur]

If this does not work?
Switch IMatch to photools.com RAW processing.
Go to Edit > Preferences > Application and search for photools.
Close and re-open IMatch.
Force-Rescan a folder with D-800 files to test (select the folder in the Media & Folders View and press <Shift>+<Ctrl>+<F5> to force rescan it.
If this works, the problem is the installed WIC codec and/or your RAW variant.

I did the change and basically not much is happening after selecting a single D800 Folder in a new database as import

[timur]

List of installed WIC codecs:
   Codec 'BMP Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .bmp,.dib,.rle
   Codec 'GIF Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .gif
   Codec 'ICO Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .ico,.icon
   Codec 'CUR Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .cur
   Codec 'JPEG Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .jpeg,.jpe,.jpg,.jfif,.exif
   Codec 'PNG Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .png
   Codec 'TIFF Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .tiff,.tif
   Codec 'DNG Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .dng
   Codec 'WMPhoto Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .wdp,.jxr
   Codec 'DDS Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .dds
   Codec 'Microsoft HEIF Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .heic,.heif,.hif,.avci,.heics,.heifs,.avcs,.avif,.avifs
   Codec 'Microsoft Webp Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .webp
   Codec 'Microsoft Raw Image Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .3FR,.ARI,.ARW,.BAY,.CAP,.CR2,.CR3,.CRW,.DCS,.DCR,.DRF,.EIP,.ERF,.FFF,.IIQ,.K25,.KDC,.MEF,.MOS,.MRW,.NEF,.NRW,.ORF,.ORI,.PEF,.PTX,.PXN,.RAF,.RAW,.RW2,.RWL,.SR2,.SRF,.SRW,.X3F,.DNG
   Codec 'Microsoft Camera Raw Decoder' (Vendor: F0E749CA-EDEF-4589-A73A-EE0E626A2A2B) for extensions .ARW,.CR2,.CRW,.ERF,.KDC,.MRW,.NEF,.NRW,.ORF,.PEF,.RAF,.RAW,.RW2,.RWL,.SR2,.SRW,.DNG
   Codec 'Nikon .NEF Raw File Decoder' (Vendor: 00000000-0000-0000-0000-000000000000) for extensions .nef

[mopperle]

Would be usefull to post the complete output of the WIC Diagnostic. And 3 RAW decoder is IMHO a bit strange. The "Microsoft Camera Raw Decoder" is afaik outdated and maybe this causes the troubles. On my system (and as a Canon user) only the "Microsoft Raw Image Decoder" is installed.

[Mario]

I did the change and basically not much is happening after selecting a single D800 Folder in a new database as import

Please ZIP and attach the IMatch log file from this session. Looking at a screen shot of the progress dialog does not reveal much of that is going on. See log file.

[Mario]

Thanks for sending the log file.
I assume the D: disk that stores the database is your SSD?

First thing I noticed was that ExifTool needs 10 (!) seconds to extract metadata for only 5 JPEG files from the G: drive.
This is unusually slow. More typical for 5 high-res JPG files is something around 3 seconds.
Ingesting the data into the database takes 30 to 50 milliseconds, which is fine.

IMatch is processing JPG files in about 150 to 200ms, some MOV files in 3000 ms.
But then come D800 NEF files, where IMatch suddenly needs to wait up 40 (!) seconds before WIC returns a rendition of the file.
But only for some of the files, e.g. in the "20120408_Rathaus by night" folder. JPG files in the same folder take only 150 to 200 ms to process....

In the last 3rd of the log, IMatch waits for 47 seconds or more before one of the NEF files is processed, e.g. in the "20120421_random\" folder.

Near the end of the log file I see that IMatch waits for 1593 seconds (!) to process some files, e.g. "20130127_liechtenstein\_DSC3358.NEF

This looks like a complete break-down of performance.

Is this a notebook PC or something?

Which virus checker do you use?

Try this to reduce the load on your PC:
Open Edit > Preferences > Application.
Scroll down to Process Control (search for process) and set
threads for file import / threads for metadata import / threads for metadata write-back to 8.
Then retry your test.-