Write-back metadata very slow on network share

[blackhead2]

Hello all,

I have all my photos stored on a NAS and facing an issue that the write-back of metadata is extremly slow for files located on it. I have 10 jpeg files (randomly chosen) which take around 7 seconds to copy from my laptop to the NAS via WLAN. When I write-back metadata to these files (adding or removing a single keyword) it takes around 10 seconds per file!

Is that the expected duration for metadata write-back? Any suggestions how to speed up the this? Feels pretty long...

Regards,

Jens

[Jingo]

Yes.. that does seem on the slow side.. my images (all JPG's) are also on a NAS (an older non-speedy one too) and my images write back in about 1-3 seconds each... and I question that speed!  Of course, the metadata writeback is only part of the story.. IMatch also reads metadata back in from the files to update the database... I typically blame my NAS for such speeds..

[blackhead2]

Yes.. that does seem on the slow side.. my images (all JPG's) are also on a NAS (an older non-speedy one too) and my images write back in about 1-3 seconds each...

In my case at least the NAS is not a slow one (https://www.qnap.com/de-de/product/ts-253%20pro/specs/hardware) so I think the wireless connection is the bottle neck in this particular setup. But nevertheless the write and read speed from my laptop to the NAS and vice versa is around 9MB/s for these jpg files. So either IMatch does a lot read/write operations which slows down the transfer rate extremely or it transfers a huge amount of data as it takes around 100s for the write back of all files --> 900MB@9MB/s?! But as I only add/remove a single keyword I wouldn't expect to much write/read operations, isn't it?

[Mario]

ExifTool uses a very secure and reliable way to update metadata. It never just changes some bits here and there, because this may damage existing maker notes in the file. ExifTool always rewrites the file, splicing it and inserting the metadata where needed. After this has been done, IMatch must re-read the file to update the database. Writing back a file can change a lot more metadata than initially written, e.g., timestamps, IPTC and XMP digest blocks and stuff.

Doing that over a (wireless) network on a device which is designed for streaming file access (copying files, streaming videos) is the worst-case scenario for performance. This can be more than 100 times slower than doing the same thing on a local hard disk or even an SSD.

I recommend you process your files locally, and when finished, move them to your NAS for long-term storage.

See also this article in the IMatch knowledge-base: IMatch and NAS Systems (Network-Attached Storage)

[blackhead2]

Thanks for your reply. I wasn't aware that Exiftool does SO many read write operations that it slows down the metadata write-back by this factor. I read the linked article before but as it manly addresses the database location I didn't pay too much attention on it in this case.

I recommend you process your files locally, and when finished, move them to your NAS for long-term storage.

For backup reasons I currently transfer my pictures to the NAS after culling and then do the metadata stuff. But I will do some measurements with the NAS connected via a GBit connection and additional with pictures located on my SSD. Maybe I will change my workflow. At least for the write-back of the metadata...

[Jingo]

Funny - I was thinking of redoing my own workflow for the same speed reasons... after looking at my own metrics and performance - I am going to do just that!  Store all images on local drives for the database to access - and only backup the items to the NAS for other purposes...  Why didn't I do this months ago?   

[sinus]

Funny - I was thinking of redoing my own workflow for the same speed reasons... after looking at my own metrics and performance - I am going to do just that!  Store all images on local drives for the database to access - and only backup the items to the NAS for other purposes...  Why didn't I do this months ago?   

The answer, my friend, is blowing in the wind! 

[Jingo]

It's blowing in the wind alright... and its a HURRICANE!

I just did a test and have my answer... bye bye NAS.. Hello local drives!

So, my NAS was taking about 3 seconds per image to update metadata for.. pretty darn slow when you are updating hundreds of geolocations at a time.  I moved a few folder to a local drive and just did some geolocation writebacks...  0.3 seconds per image... 312 ms...  so.. much... FASTER!

So.... I will be moving my images to a local drive, repointing the database to the new structure and then just using the NAS as backup... and secondary access as needed.

Hope this info helps you Jens.... it certainly helped me!

[blackhead2]

I also did some testing with my files and the results are similar to yours:

Writing a keyword to 10 files took...
... 4383ms with the files located on a local SSD
... 22963ms with the files located on my NAS and a GBit connection
... 100605ms with the files located on my NAS and WLAN connection (300Mbit)

So in my case working via WLAN is 4.4 times slower than using a Gbit connection and 23 times slower than the local SSD 

At least good to know...

[Jingo]

Yes.. makes sense in the scheme of things.. and someday we'll look back when LAN and wireless speeds then are what SSD write speeds are today and shake our heads in wonder.  Until then - my files are going quickly to my local drive now!