IM5 - updating files - process needs to be reviewed!

[Gerd]

Hi,

yes, I made a fault ... I want to add some pics (24) from 2014. I tried to add a new folder 2014, but IM5 said, in the view "friendly names" it is not possible, use rescan ... and I did it and that was my fault! Because I had to rescan from master folder (EOS7D) with the years 2010, 2011, 2012, 2013 and that are 50.000 pics!

It took some hours 1) to add pics to the database, because all 50.000 pics are scanned ...
The next step was "Adding Metadata", also all 50.000 pics, that process needs 1,5 days without switching off the pc. It was ready today ca. 12:00
Now the third step: "Adding and updating files", calculated time: 12h !!!

... and all that time the pc is very slow for other actions ...

All data are local on the internal harddisk (SSHD), Intel i5-430M (2,2 GHz), 8 GB memory, 1 TB SSHD.

If I do a rescan (synchronize) in LR, then I get the message, that there are 25 new files and it is done in 2 min.

Something needs to be changed in IM .... to speed up!

[joel23]

IM says "revert to normal folder names" not rescan ;-)
You should have done that or drag and drop an existing folder.

Also I guess a sync LR does is much different from a full rescan (generating thumbs, reading metadata etc. pp.) IM does. 

[Mario]

You cannot create new folders while you are in friendly folder name mode - because in this mode you don't necessarily see the folder names in the file system, but some custom/fantasy names you have entered. This is why IMatch shows the dialog and allows you to switch back to normal folder name display.

From your description I understand that you have rescanned your entire 50,000 files database! And this takes a while.
To solve your mistake without waiting so long you could have:

1. Close IMatch
2. Restore the last daily backup of your database.
3. Restart IMatch
4. Add a new folder via the commands provided in the Media & Folders View.

[Gerd]

Dear Mario,

what is thedifference between first step "Adding files to the database" and third step "Adding and updating files"?

And why does IM do not skip allready known files?

[Mario]

IMatch performs three steps:

1. Scanning the folder for new and updated files.  During this step IMatch is blocked and you see the dialog.
New files are added to the database and put into the background processing queue.
Non-existing files are removed.
Updated files (with a last modified timestamp newer than what's in the database) are put into the queue.
Unmodified files are automatically skipped.
Unless you force a rescan. Which is what you apparently did.

2. In the background, IMatch now reads the metadata. Because metadata may influence how thumbnails/cache images are rendered etc.

3. Also in the background, IMatch extracts thumbnails from files, file system data, calculates check-sums and visual query info, creates cache files.

2. and 3. run in parallel, where 2. hands over files to 3. as soon as metadata has been read and processed.

[Gerd]

Hi Mario,

what I do not understand is now the 3. step "Adding and updating files". It started roundabout 12:00 and now it's 17:30, 5,5 h => 330 min. for 6.600 files, means 3 sec. per file ... is that a normal needed time?. If I read pics (ca. 8 MB per pic)from my CF-card via USB 2.0, it takes also ca. 2,5 to 3 sec., but this is a slow process. Is this update-process in IM5 so slow or is there a chance later to optimize (speed up) this process?

[Mario]

95% of the time spent in that step is caused by image processing, for RAW files, mostly the WIC codec.
If you generate cache files during ingest, the time will be twice or three times longer for each image. Extracting a preview from a RAW file can take 0.5 seconds to 2 seconds, depending on the format If the RAW does not contain a suitable preview and IMatch needs to extract the full RAW, times can go up to 3 to 10 seconds. I've seen some RAW codecs taking 10 seconds for each file in that mode!

You did not include a log file so I cannot say more. Shut-down IMatch, and copy the log file. This will give us performance info.

[Gerd]

Hi Mario,

I saved the log-file and closed IM, here tyhe download-link, the zip-file is 12 MB:

https://skydrive.live.com/redir?resid=986345DDF78B81F1!986&authkey=!AKUQbyhbO3QJhq0&ithint=file%2c.zip

[joel23]

Hi Gerd,

you should do a database diagnostic. Your log shows that there is a problem since it was opened last time (yesterday)
01.04 13:17:27+    0 [0694] 01  W>       database disk image is malformed  'IMSQLite.cpp(2274)'

[Mario]

database disk image is malformed

Uh, Oh. That's bad.
This is a panic message from the database core system which indicates physical damage to the database file. Such errors cannot be recovered. The database has to be replaced by a backup. Do you recall when this happened?

[Gerd]

Hi,

After closing I have started again my database, disabled the cache and now it's going much (!) better. Since 18:20 till now 21:30 50% from the rest of 40.000 pics are read. I'll wait till process is finished and then I will see, what the status is.
Then I try to check and optimize.

[joel23]

Gerd,
you were telling about an internal SSHDD. Is this a solid state disk (SSD) or a normal HDD on SATA?
Anyway, from your log I saw almost everything is stored / done on one disk.

You might consider to spread things around several disks if possible.
The DB on a SSD.
IM cache maybe on a fast SATA (external USB 3 or eSATA)
OS temp and pagefile on different internal disk(s) - a normal SATA 2 disk would be okay here if there is enough RAM.
Images on separated disks.

If everything is on one disk (C:) and it is failing or compromised you'd have a huge problem.
Even when the OSes nowadays are much reliable than they were in the past, I never would store private data (documents, images, music, movies) on C: if this is to avoid.

[Gerd]

Hello Jörg,

I do backup my data via a 1by1-copy to an external HD.

My internal hd is an SSHD (Solid State Hybrid Disk) with 1 TB.
The only thing at moment is, that I'm midden in my move and I have no access to my backup's ...