File Update takes forever

[samisbp]

Hello

I have recently upgraded from IMatch3.6 to 5 and I'm having quite a few troubles with the file updates.

Right now for exemple, I have two tiff files in a folder, each file is 220MB and 37MB respectivelly in size. If I update them in photoshop, IMatch automatically picks that up and then starts updating them. It shows the small blue icon over them and "Updating..." is showing over the Metadata panel. But this never ends..... it keeps going and going and going.....for ever.
Once I get tired of waiting, if I manually select those two files, right click and ask for a normal rescan, suddenly the update ends and the two files are ok.

In IMatch3.6 the update of such files was only taking a few seconds at the most.

Could someone tell me if this is a bug or if I am doing something wrong?

Thanks.

Sam

[Mario]

1. Please attach a log file. I cannot tell anything from your description alone.
2. See How to report bugs for more info.
3. Try closing Photoshop. It may keep files locked for a longer time, thus preventing ExifTool and IMatch from accessing them.

I will be able to tell more when I see a log file. Make sure you enable Debug logging, then repeat your scenario and then ZIP and upload the log file. This will tells us precisely what IMatch is doing, how long everything takes etc.

[samisbp]

I am once again facing some rather frustrating problems with file updates.

This morning, somehow things seemed to works fine. I was adding new files (from an external application) into some folders and IMatch would pick them up straight away, add them, apply any file relations correctly and propagate the metadata also correctly.

Then, for some reasons, I am now working from another folder and new files are no longer being picked up.
I have tried to run a normal rescan and although it did pick up the new files, the thumbnails and their metadata are not being loaded correctly. In fact fails to show the thumbnails of the new images (they are tiff files) as they are crossed by a red bar, and the blue double arrow keeps showing up on them so indicate that the metadata is still updating.
Note that I have tried to force a rescan and as the job was taking long to complete, I decided to click on the "Dismiss" button of the Load Progress Overlay window which shows up when we ask to rescan a folder.
The strange thing is that when I click Dismiss, only a few seconds after the Info & Activity window shows that IMatch has finished working. But the updates haven't been done. According to the message on that window, hitting the Dismiss button is not supposed to stop the updating and IMatch is supposed to carry on in the background isn't it?

Is it a bug, or am I missing something here.

I attach the log file.

[attachment deleted by admin]

[Mario]

IMatch is happily ingesting TIFF and JPEG files from your

T:\SBP-DB2\ and  U:\SBP-DB3\

drives (which are NAS drives or?)

Your database is on drive K:

IMatch and ExifTool are installed in

"J:\Program Files",

which is what kind of drive? Extern? Mapped?

Each file delivers about 220 metadata tags with information, and IMatch processes the metadata of 20 files in about 7 seconds.
But it seems that one of the background processing threads takes a very long time, blocking the database so other background processes cannot longer access the database and time out.

From what I can tell, the last files processed by that thread were from

U:\SBP-DB3\Travel\Co..., T:\SBP-DB2\Urban\... and T:\SBP-DB2\Nature\...

anything special about these files? Maybe they take very long to load? Or maybe IMatch being blocked for long periods because the U and T drives don't respond?

If you can reproduce this behavior, please try to lower the stress on your system by configuring IMatch to use less parallel threads.

When this problem happened, IMatch, which is installed on drive J: uses a database on drive K: while it processes files from drive T: and drive J: on your system. This means that four drives are in parallel use, and your system is probably maxed out.

Go to Edit > Preferences > Application: Process Control and set both values to 2.

Press <F1> while in this dialog to open the corresponding help topic which explains this in details.

This IMatch Knowledge Base article has further info:

http://www.photools.com/3225/configuring-process-control-slow-media-cd-rom-dvd-nas/

[samisbp]

Thank Mario for such a fast reply.
I'm gonna try what you suggested and report.

I'll try to explain to you my setup so you can maybe have a better feel of the problem.
IMatch taking long to perform some tasks is one thing but being stuck is another so let's hope we can find out.

Because I travel a lot, last year I decided to only work with a laptop from now on so I have tried come up with setup which is small enough to travel but still allows me to always take all my photos with me.
So I've got one laptop (HP Zbook 15), with 32GB RAM and one 1TB SSD drive on it which is partitioned like so:
C: System (Windows) - I can't remember why IMatch is installed on it though as normally I install all my applications on the dedicated partition J:
J: Applications - Where I normally install all my applications
K: Data -  where all my general data is stored, including IMatch database (my actual pictures are all stored on external drives - see below)

Then all my pictures are (currently) stored on three external MyPassport Slim drives (S: T: and U:) connected via USB3. There are 2TB each.

When I first import images, I do work on them on my internal SSD drive and then when the post processing is done I move them to the external drives.

Note that I am considering investing into two DroboMini connected via Thunderbolt, as an alternative to my external MyPassport Slim drives. Would that really improve things?
Any other thoughts as to how I could already improve things with my current configuration?

Thanks again for your help.

[ubacher]

As far as I am concerned there are still problems with automatic indexing.
When I tried automatic indexing recently again I started to run into problems just like you describe.
I reverted to manually initiate my indexing - and I had no problems with it. (Unfortunately it is rather
inconvenient to initiate a manual update - else I would be happy to live with it.)

PS: I do have only 1 thread set up for background processing - so that did not help.

[Mario]

@ubacher

You seem to have many issues that only ever show up on your PC, with your database and images.
Feel free to comment on every bug report, but I would really much prefer if you would open your own bug reports. Your issues may be totally unrelated to the issues reported by the OP, and it is much easier for me to deal with one issue at a time.

I'm sure that if there would be general problems with indexing, the community would be full of reports.
IMatch hanging or not processing files is not something that goes unnoticed.

If you have to run one thread to make your system stable, there are most likely severe problems with the hardware or a general system instability under stress. Not good.  Usually IMatch works just awesome with one thread per CPU, with the notable exception of users who work with massively large files (several hundred megabytes per file or more) over slow NAS networks or similar.

Experience tells me that if ExifTool fails to process files or stalls as in the case of the OP, the reasons are most often corrupted files, temporary locks caused by other software or virus checkers, non-responding NAS boxes, sluggish SAMBA implementations. Or, very rarely, a bug in ExifTool or IMatch.

[samisbp]

I can already report that setting the Process Control down to 2 unlocked IMatch straigh away.
Then taking it down to 1 seemed to improve things even more.

I'll keep an eye on it but as far as I am concerned, my problem really seems to be related to the fact that my image files are split onto several external drives accessed via USB3 and also that among all the RAW files in my database I am also working with very large files (panoramics) which are often above 2GB and going up to 4 GB at times.

[ubacher]

Mario responded to me:

Feel free to comment on every bug report,

I did this to indicate to you that the OP is not the only one running into this problem. So you won't
think it is only his machine/setup/db which is affected by this.

And I indicated that I run only 1 thread because I expected you to answer that I should reduce the number of threads to solve the
problem.

My layman assessment of these problems is that there are conflicts in the  interactions between background and foreground processes
which occasionally result in thumbnails w/o image, unfinished indexing or endless looping of the background process.
How to nail this problem down is, of course, difficult. I am hoping some user can provide a helpfull log/dump.

[Mario]

I am also working with very large files (panoramics) which are often above 2GB and going up to 4 GB at times.

That's an important info. ExifTool may have problems extracting metadata from such large files, or take a rather long time. I did not see any related warnings in the log file you had attached so I was not aware that you process files with several gigabytes. Please understand that a general purpose DAM like IMatch does not necessarily have the special codes and routines included in dedicated  panorama software, which has been designed from ground up to handle files in the multi-GB range. Such files are not really common.

Combining the long extraction times of ExifTool with too many parallel threads, multiple active drives etc. probably creates a worst-case scenario, which in the end causes IMatch to time out while waiting for the disks, ExifTool and the database to get the job done. By lowering the number of parallel threads you reduce the stress on your system, ExifTool can process the large files faster (because only 2 and not 4 files are processed at a time) and this solves the problem. This is a fringe case which is best handled that way - by doing less at the same time.

Keep an eye on your log file. If you find lines with PTWrapper hangs in ProcessRun this means that ExifTool really hang for 3 minutes, without doing any read/write operations or using CPU time. In this case, IMatch stops the ExifTool process and re-creates a new process. The file being processed at the time is then processed again. Usually you should not see such errors in the log, unless the files are damaged, or they are huge and the disk/network is slow and the PC is too busy to shuffle all the data around in time.