Performance issue on NAS

[mosdubindi1]

See https://www.photools.com/community/index.php?msg=100455 for the original thread.

Quote:
What's disturbing is that IMatch logged extremely long execution times for CIMEngine5::UpdateFileAttributes.
This function fetches information for a file from the file system (e.g. file size and last modified date) and stores it in the database.

Hello Mario,

I did some test for import photos, taken by different cameras, from NAS and SSD. Also I did it with different working mode.

Number of imported photos = 100 per each camera.

Result for import from NAS for different cameras:

Camera	Raw File	Average Size, MB	Working Mode	Import Time, min	Average cache file size, MB
Sony α7R II	ARW	83	Default	4:04	3,5
Sony α7R II	ARW	83	On-demand	1:54	3,5
Canon 1Ds MarkIII	CR2	23	Default	0:59	0,4
Canon 1Ds MarkIII	CR2	23	On-demand	0:53	0,4
Canon EOS 5DS R	CR2	63	Default	1:36	4,6
Canon EOS 5DS R	CR2	63	On-demand	1:28	4,6
Fujifilm GFX100S	RAF	203	Default	3:26	0,7
Fujifilm GFX100S	RAF	203	On-demand	1:25	0,7

Import time depends on camera, most problematic are Sony and Fujifilm. For these cameras also big difference between Default and On-demand working modes.

Result for Sony for import from NAS and SSD:

Camera	Test folder	Working Mode	Import Time, min	Log file
Sony_a7RII	NAS	Default	4:04	IMATCH6_LOG_SONY_default_NAS_cacheE.TXT
Sony_a7RII	NAS	On-demand	1:54	IMATCH6_LOG_SONY_on-demand_NAS_cacheE.TXT
Sony_a7RII	SSD	Default	2:42	IMATCH6_LOG_SONY_default_SSD_cacheE.TXT
Sony_a7RII	SSD	On-demand	0:19	IMATCH6_LOG_SONY_on-demand_SSD_cacheE.TXT

Attached pls. find the logs in debug mode. There is no  CIMEngine5::UpdateFileAttributes in the log files, but given a big difference between import from NAS and SSD, something is not good. I cannot reproduce a log with such function.

I think there are some problems with network or NAS settings somewhere. Maybe looking at the logs you can advise, what could be the problem?

Thanks in advance,
Dmitry

[Mario]

The majority of the time processing a RAW (90%+) is spend in the WIC codec (or LibRaw, depending on what you use).
Different RAW formats, even with the same file extension, have different execution times. It's all up to the WIC codec provided by Windows. I'm sure SONY could provide a better and faster WIC codec than Microsoft, but SONY does not give a sh*t.

NAS is the worst case. The WIC codec has to pull the file over the network, ExifTool and IMatch too. If you have a pro-grade NAS connected with a 5 or 10 GB cable network, the difference compared to files indexed from a local SSD or spinning disk are usually not that noticeable. But if the NAS is a lower-end product, maybe even connected via Wi-Fi, thinks look very different.

As I said, try to reduce of parallel threads if too many files processed in parallel bog down your NAS box:  Process Control for Slow Media (DVD, NAS ...) Things may be faster if the NAS / network connection is not maxed out.
Well worth a try. There are many variables when it comes to network resources.

[mosdubindi1]

Thank you Mario,

I've performed additional tests, playing with different number of threads and codecs.

Changing number of threads in Process Control does not help, even doing it worse. Changing codecs has almost no impact of import time.

So finally the main impact on import time is SSD or NAS. Depending on different RAW format and cache Working mode,  NAS is 2-10 times slower for import then SSD.

For metadata write-back into XMP the situation is even more dramatic – NAS is 13-15 times slower then SSD. I used same 100 files for metadata update.

Time for NAS metadata write-back = around 14 min, which is around 8,4 sec per XMP.
Time for SSD metadata write-back = around 1 min, which is around 0,6 sec per XMP.

Test has been done on NAS Synology DS1821+, which is not low-end product I think. So looks like something is wrong in network or NAS settings. I'll do further investigation.

In meantime attached pls. find 2 logs for metadata write-back to NAS (here for one folder there is update of existing XMP and another folder for creating new XMP) and SSD (update of existing XMP). If you can give me any idea where could be the problem, it would be very helpful for me.

[Mario]

I have nothing else to say. NAS boxes are usually designed for archival purposes or "streaming" content like videos or backups.
Highly interactive applications like DAM which read entire files or access segments of files will be a lot slower when the processed files are on a NAS.

The typical workflow is to "finish" (cull, edit, export, add metadata etc.) on the local SSD or spinning disk.
Then move them for long-term archival to the NAS system or offsite storage / cloud.

Reading and writing data from a local SSD is 100 to 1,000 times faster than doing the same over a network. Caching effects in the Windows file system or things like SSD caching in the NAS itself can greatly impact performance. It all depends. File access patterns, "streaming" entire files like ExifTool does for safety, how WIC codecs or LibRaw access files or sections of files etc. all plays into this.

Time for NAS metadata write-back = around 14 min, which is around 8,4 sec per XMP.

IMatch/ExifTool don't just write the XMP data. They also synch native metadata embedded in the RAW (EXIF, GPS, legacy IPTC) to not let the copies of this data in the XMP file become out-of-sync.

If you still actively modify the files you keep on your NAS, maybe consider first finishing them while they are still on your local disk. And check if there are ways to improve network throughput (switch to 5G or 10G network, add SSDs to the NAS for caching etc.).

[mosdubindi1]

Thank you Mario for the reply. Trying SSD cashing on NAS is a good idea. 

As for split workflow between local disk and NAS for far does not look efficient for me as it requires to separate metadata changes and pure IMatch changes like categories etc., and following synchronization. However I'll be still having it in mind.

[RobiWan]

What is 'efficient' needs to be clarified. For me, it means that I can do what I want to do as quickly as possible. I'm not interested in whether there are things running in the background that I'm not aware of (Syncovery synchronisation between local disks and NAS).