Importing Metadata Never Stops?

[richard49]

I'm new to iMatch and on the trial basis.

I'm using version 2018.8.7 in Windows 10

I managed to tell iMatch where all of my images (36,000) are stored and it found them easily.

I then asked iMatch to refresh the images and now it's taking (what seems like) forever to finish.

I started the process about 24 hours ago and it's still running!

I've had to give up using iMatch as the 'Importing Metadata' isn't (as far as I can tell) doing anything and appears to be going around in circles.

Yesterday I left iMatch running and left it going when I went to bed; in the morning nothing had changed; the Importing Metadata was still running but there was no change in the amount of files it had found!

[Mario]

Hi, welcome to IMatch

Processing 30,000 files is not that much. Even if these are large RAW files IMatch should finish within a couple of hours.
In rare cases badly corrupted metadata can cause ExifTool to crash and then IMatch restarts it to continue with the next file. This can take up to 30 second, which is usually not a problem because IMatch processes many files in parallel and one ExifTool instance crashing because of a corrupted file does nothing.
But if you have many damaged or corrupted files, and all IMatch is doing is recovering from crashed or hanging ExifTool instances, things can come down to a crawl.

What we need to diagnose this further:

Details about your files (which file formats? Where are the files stored? Local Disk? Remote server)
Some details about your computer

The IMatch Logfile (see log file):
You can access the log file for the current session (while IMatch is running) from the Help menu. Or anytime directly in the TEMP folder on your system. See the link above for details.
The log file will tell us what IMatch is doing, what is taking so long, which errors it has encountered in your files etc.

[richard49]

Mario,
Thank you for a very quick response, it's very much appreciated.

The files in question are stored on a separate hard drive within my desktop computer, my D: drive

Nearly all my files are RAW (.nef) files from my Nikon camera.

I found three imatch log files which I hope I have attached correctly.

[Mario]

You have attached one log file from an IMatch session from yesterday starting at 18:06.
First JPEG files are processed, from D:\Sync\Master Catalogue\2009

This log file contains 1080 warnings. Normal would be zero warnings.


Nearly all files have problems. ExifTool complains about invalid date and time formats in files. This means that the EXIF date and time recorded in the file are not in the proper format.
This can be corrected by writing proper date and time information to the files with ExifTool (IMatch can do that).

18:10:32

For files in D:\Sync\Master Catalogue\2010\ ExifTool reports invalid CanonCameraSettings data.
For other files ExifTool reports that it has to fix/rebuild broken maker notes.

I don't know if these files come directly from the camera or if the metadata has been damaged by another application.
All these problems of course slow down ExifTool. There are apparently several minute breaks while IMatch is doing nothing or waiting for ExifTool to make sense of the broken data in your files.
These files are in real bad shape. I don't remember seeing this amount of problem reports before.
What kind of software did you use to create these files? I doubt that these are straight from the camera, because Canon usually outputs clean JPEG files.

No NEF files were processed in this session.

Before you repeat your test, do this:

1. Enable debug logging in IMatch : Help menu > Support > Toggle Debug Mode.
2. Go to Edit > Preferences > Background Processing and make sure that "Write-back changes to metadata immediately" is off.

This ensures the smoothest possible workflow. And detailed debug info.

Now:

3. Add one folder of your JPEG or NEF files. Not 30,000 at once. While this is usually not a problem, looking at the bad shape the metadata in your files is in, working small batches may work better.
This also produces a smaller log file.

[richard49]

Mario,
I've been using Capture One Pro V11 to edit all of my images and Adobe Bridge for a catalogue.

I'm surprised that there is so many problems as neither of the afore mentioned applications have shown any problem.

Where you state...

Before you repeat your test, do this:

1. Enable debug logging in IMatch : Help menu > Support > Toggle Debug Mode.
2. Go to Edit > Preferences > Background Processing and make sure that "Write-back changes to metadata immediately" is off.

Toggle Debug Mode isn't showing?

How best should I go about correcting the (as you state) the incorrect date format?

Your help is appreciated...

[richard49]

Mario,
I've just looked under Tools > Modify EXIF Date And Time and it seems to state that iMatch wants the date format to be Day/Month/Year whereas the format I (and I believe most people use) is Year/Month/Day

I don't seem to be able to tell iMatch that this is the format I use?

Regards,
Richard

[Mario]

I've been using Capture One Pro V11 to edit all of my images and Adobe Bridge for a catalogue.
I'm surprised that there is so many problems as neither of the afore mentioned applications have shown any problem.

This may explain that. Both applications had various 'issues' over the years.

You usually don't notice metadata problems if you only access your files with the applications which created the problem.
Only when you switch to another application or platform or you send files to clients or upload them to a web service you notice that the metadata is incomplete, invalid or damaged.
This is especially true for maker notes, which have never been designed to be copied to other files.

ExifTool is used by IMatch for exactly this purpose. It is the de-facto reference when it comes to reading and writing metadata. When ExifTool says that the data in your files is broken or incorrectly formatted, you can trust it.

I don't seem to be able to tell iMatch that this is the format I use?

The date and time format used by EXIF and XMP is an ISO standard (YYYY-MM-DD{T|Z}HH:SS:SS.S100{{+|-}HH:MM}) and not your locale time format. IMatch automatically takes care for that internally.

The Modify Date & Time dialog uses the standard Windows date and time input controls to accept a date.
The format in which these controls format your local date depends on the date and time format configured in Windows for your user account. IMatch does not make the format up.
This is all automatic. Check the Windows control panel if this is unexpected.

How to enable debug logging is described in the log file help topic (log file) I linked to in my initial reply.
Here is a screen shot for your convenience:

[richard49]

Mario,
I checked in my Windows 10 time setting and it was set to be DD-MM-YYYY so that's been changed.

I'm now adding my catalogues one at a time and so far everything seems to be going well.

Thanks,
Richard

[Mario]

Excellent. You should still look at the metadata in your files really carefully. And also which software you allow to tinker with it.
I have never seen so many problem reports. The badly formatted EXIF timestamp ExifTool can probably handle by guessing, but the corrupt maker notes can be a problem for applications which need to access it.

[loweskid]

When I first upgraded to IMatch 5 I had problems with 'hanging' while trying to rescan folders (NEF files).  I eventually traced it to corrupted Bridge cache files - I had Bridge set to Automatically Export Caches to Folder When Possible which added files to every folder, such as  .BridgeCacheT  and/or  .BridgeSort.  I searched in Windows for anything that had 'bridge' in the file name and deleted them.  Also deleted the main Bridge cache.  This cured the problem.

I know this isn't related to your metadata problems but might be worth looking at.

[Mario]

Yes, Bridge can spread tons of files all over the harddisk if not properly configured, and this can double or triple the files to process in IMatch.
The cache files are JPEG and hence processed by IMatch when it scans the folders.

You can either configure a cache folder for Bridge outside the folders you manage in IMatch or disable the "export when possible" option.
Usually you don't need Bridge anyway when you have a real DAM.