Speed improvements on "relocate" function

[KimAbel]

Hello Mario

Not quite a feature request, but rather a speed improvement request.

I have been testing my database on two different computers (one stationary with files on a internal harddrive and one laptop with the files on a external harddrive). The relocate function is extremely slow on my laptop.  I use a express card with USB3, but I guess I dont get the same speed as a normal USB3 port. This is quicker on my stationary pc, but still slow (perhaps a few minutes on my stationary pc and 20-30 minutes on my laptop)(around 60.000 photos). IM3 is much faster at this and Lightroom is way ahead of both IM3 and IM5. Lightroom does this almost immediately.

My hope is that you could fine tune the speed of the relocate function when you find a little spare time? Perhaps you need something else to do in between all the crash log debugging ;-)

I know that this isnt way up on the priority list, but it would be nice if you could have it on the "to do" list.

Regards Kim Abel

[Mario]

Relocate involves checking each folder and file against the file system. This can take a while, depending on the Windows version and device type.
IMatch has to change the internal link in the database, which is fast. But IMatch also has to change a lot of metadata in that process (all metadata tags which refer to the folder) plus update the full-text search index completely, for all files. IMatch 3 did not have to do that, which is why it can be faster than IMatch 5 in that respect. All the luxury of what IMatch 5 can do comes at a price.

Relocating files is an exceptional process, not done on a regular basis. Usually only to correct folders or files moved/renamed outside of IMatch, or after moving your entire image collection to a new disk. Nothing you do on a regular basis.

Maybe we can improve it when we check for a special case, .e.g. only the media serial number changed but not the path or file name (e.g. relocating from c:\images to c:\images). Then we maybe could skip the metadata updates and search index update. I don't know yet if I already do that, haven't touch relocation for quite a while.

Attach the log file from your slow relocate attempt because this may tell us what's slow, exactly.

[BenAW]

Relocating files is an exceptional process, not done on a regular basis. Usually only to correct folders or files moved/renamed outside of IMatch, or after moving your entire image collection to a new disk. Nothing you do on a regular basis.

Relocate may be used more than you think.
I personally use it quite often. I have a rather small image set (20.000 images), so the slow relocate speed doesn't bother me much.
I use IM on 2 computers (PC and laptop). I keep the images, dbase and .PTS file in synch.
I mostly work on the PC (2 monitors), but when running IM on my laptop I first have to do a relocate.
In principle the images and dbase are always identical on the two machines. This could indeed be a place to look for a speed increase.

[KimAbel]

I will attach a log file the next time I will relocate. Probably tonight :-)

I would think that the relocate function is a much used feature. Taking your database with you on travel for instance, or just to be able to work on the database from a laptop. I for instance use my database at work to. I take a lot of pictures at work and need to implement these pictures in my database. In order to do that I have to use my laptop and a external harddrive.

Using the same disc letter "c" wont help in this example. An external harddrive will have another drive letter than c. Is another possibility to use relative pathnames? As long as the folder structure with the photos stays the same? (all photos are contained inside a folder named for instance Pictures/.

Perhaps another solution could be for IM5 to store two versions of the filepaths (if needed and asked for). In most cases the relocate function is between two different discs.

My comparison with Lightroom may be wrong, but I would think Lightroom also have to do some of the same tasks as IM5 when relocating, and Lightroom is extremely fast.


Kim Abel

[KimAbel]

Log file attached. I realized that I still had my database open with a log from the relocate task :-)

https://dl.dropboxusercontent.com/u/12694628/IMATCH5_LOG_relocate.zip

[Mario]

IMatch checks for the path and the volumne serial number (to differentiate different removable drives from each other).

You can mount your external hard disk under the same drive letter on all your PC's and you will never have to relocate.
If you work against a network server or a NAS, use UNC paths like \\server\share\folder, which are identical for all machines in your network.

For drives which mount as removable (USB sticks and suchlike) IMatch automatically relocates when the path is the same, just the volume serial number is different. IMatch does not do this for "fixed" disks, for security reasons.  So you only need to relocate when you move your images between different disks. IMatch considers (and has to) these disks as different.

My comparison with Lightroom may be wrong, but I would think Lightroom also have to do some of the same tasks as IM5 when relocating, and Lightroom is extremely fast.

This made me laugh, thank you. LR does almost nothing with metadata, honestly. If I temporarily disable a lot of functionality, IMatch can relocate 1000 folders with 50,000 files in about 5 - 10 seconds. But then the metadata of these files will no longer match the physical disk structure, variables will return the wrong values  etc. If all you need is LR "DAM" functionality, things are much simpler. Just go ahead.

The problem with relocation are the file system checks which IMatch performs to ensure you are not doing any harm to your database (and which can be painfully slow), and the resulting metadata and search engine updates. Relocating 20,000 files can cause 100,000 updates to the database (about 5 metadata values per file, plus) and a search engine index update for these 100,000 tags. And that is what takes so long. No way around this, if your database should match the physical file layout, I'm sorry.

I will look into the improvement idea I mentioned above. This should bring down the relocation time considerably, if you just relocate from the same drive/folder to the same drive/folder (just on another disk). If the folder names are identical (including case!) IMatch can skip all metadata updates because the components of the file names don't change. And relocate will just fly 

[KimAbel]

I will look into the improvement idea I mentioned above. This should bring down the relocation time considerably, if you just relocate from the same drive/folder to the same drive/folder (just on another disk). If the folder names are identical (including case!) IMatch can skip all metadata updates because the components of the file names don't change. And relocate will just fly 

Maybe its me that dont understand your idea, but isnt it so that on one machine you cant have the same drive letter on two separate drives? This will be the case when I need to sync all my folders on the two discs (one internal harddrive on my stationary and one external on my laptop). Perhaps I could set a new drive letter on my external disc when I attatch it to my laptop, but I havent tried that before. Is that easy?

This made me laugh, thank you. LR does almost nothing with metadata, honestly.

Well, Lightroom is not totally "empty" when it comes to handling metadata and collections, but since IM5 has a lot more power in this area I will stick with IM :-)

Kim Abel

[Mario]

Maybe its me that dont understand your idea, but isnt it so that on one machine you cant have the same drive letter on two separate drives?

That's correct. The shortcut solution is designed for users who use relocate because they move their database/image collection between two computers, but have the same drive letter on both. E.g. "c:\images" on the desktop and "c:\images" on the laptop. The new solution in this case just can update the database with the media serial number of the c: disk on the laptop or desktop. Very fast.

If you relocate from "c:\images" to, say, "d:\images", IMatch cannot use the shortcut. Because several of the metadata tags in the database use the folder name, so IMatch has to update them all. And also the search engine index afterwards. These tags are used for purposes like sorting, the search bar in the file window, to fill variables etc. They must be in synch with the file system.

Well, Lightroom is not totally "empty" when it comes to handling metadata and collections,

Like I always say, if all a user needs is in LR, he does not need a real DAM like IMatch. Which is OK of course.
But it is rather more likely that the user does not even know what he is missing - because he never used a real DAM. Unfortunately, Adobe has the money and marketing power to make their products look so bright that all other (sometimes even better) products are cast in shadow. That's always the case when you have a monopoly...