Exiftool Failure?

[Ian Pollen]

I am having problems writing metadata out to a 760MB 16bit tif file which is stored on a NAS drive.

The same file writes perfectly when stored locally or even on external USB connected HD.

The only difference I can see is the Exif tool runtime difference of 24sec local, 1min USB and 3min on the NAS.

Would this long runtime be causing it to fail. There is no error message in the Exiftool output?
The process just stops and leaves the Exiftool tmp file in the directory.

Ian

[Ian Pollen]

I forgot to say the TIF file ends up corrupted when it fails :-(

[Mario]

IMatch logs all warnings and errors to the log file. This is the first thing I would check. See log file in the IMatch help for details, and also check the ExifTool output panel.

ExifTool uses a streaming file access pattern which makes it usable and compatible over a wide range of platforms. This means it needs to read the entire file, update the received data and write-back the file. This means it transfers about 1,5 GB over your network in order to do this, and this means plenty of chances for problems with the file transport, network, NAS etc. Pretty much a worst-case what you are trying to do and a real stress test for all involved components and the NAS box.

If the file becomes damaged, the most possible reason is that the NAS/network just stopped writing data 'in the middle'. And this probably also caused ExifTool to hang (or be stopped by IMatch after 3 minutes). Without the log file I cannot tell more.

But, frankly, shuffling 800 MB files over a network from and to a NAS box is really not a good idea.

[Ian Pollen]

Thank you for the quick reply Mario.

I will do some more tests and check the logs.
The NAS is a Netgear ReadyNAS Pro Pioneer RAID running over Gigabit Ethernet so should be man enough for the job :-)
The same hardware config works fine with metadata writes from Lightroom and Capture One.

Ian

[Mario]

I use pro-grade NAS boxes from Synology and QNap and I know that IMatch works great if all components behave.

LR and the other software may do in-line updates, writing only parts of the file, patching it. This is not how ExifTool works, for a number of reasons Phil can explain better than I. This means that LR does not put that much stress on your network and NAS as ExifTool. And since ET can write the file when on the local disk, it does not seem to be something caused by the file, but rather by some problem caused when moving the file over the network or back.

I can't tell how long this takes, if IMatch considers ET to having stopped working (because it is not responding anymore after 3 minutes) without a log file.
Maybe open the file in Photoshop and do some changes and save it a few times, do more changes, save it again etc. This will also pull the file from the NAS into local memory, and then write it back. Maybe this reveals something.

LR only updates the cache file (local) and the XMP data with your changes so it won't pull the full file, causing a lot less stress.

[Ian Pollen]

This hardware setup does get pushed very hard on a daily basis reaching anything up to 80 MB a sec throughput and never fails in any other use.
I noticed that ET barely gets above 15% network usage.
I actually took the same file and added another layer to it to increase the size further and did repeated edits and saves via Photoshop and other apps. No problems worked perfectly.

I had assumed LR wrote the whole file when saving metadata to a TIF file.

What level logging is are best for you to look at?

Ian

[Mario]

Enable Debug logging under Help > Support. Then repeat your test.
Without a log file, I cannot say anything.

[Ian Pollen]

OK here is the log file during a failed write.

I have repeatedly tried writing the same metadata to this file using the ExifToolGUI and it works OK.

[attachment deleted by admin]

[Mario]

I seems that writing back to the file takes more than 3 minutes. IMatch abandons ExifTool after 180 seconds if there is no feedback, assuming it hangs or is no longer responding. 3 minutes so far was always long enough and I'm reluctant to increase this timeout value because it will affect all other users.

PS.: ExifTool GUI does not apply any of the advanced Metadata Working Group compliance argument files, it just writes whatever you configure. This means it will be probably faster, but it will also not take care that EXIF, IPTC, GPS and XMP are properly synchronized. Keep that in mind.

If you consider this exceptional slow performance a problem and you would like to configure IMatch to use a longer timeout than 180 seconds in order to deal with your environment and workflow, please add a feature request to the feature request forum. This allows other users to comment and prioritize. I don't recall many situations where updating a file took that much time so maybe this is nothing that will affect many users.

[Ian Pollen]

Yes I would like the timeout increased please. I will make the request now.
I don't understand the internal workings of IMatch but how does increasing the timeout effect other users if their run times are shorter?

I get the impression that you think my ET runtimes times for this file are excessive... What sort of ET runtimes do you experience on your NAS with a TIF file of this size?

I am amazed that no one else has experienced this problem. Most photographers working with high-res DSLRs or medium format backs and carrying out post-production on 16bit TIFs with just a few layers will have files of this size.

[sinus]

I am amazed that no one else has experienced this problem. Most photographers working with high-res DSLRs or medium format backs and carrying out post-production on 16bit TIFs with just a few layers will have files of this size.

I am not amazed, because I do not know one collegue of me (professionals photographers), who does use files with such a lot of MBs (except some rare occasions).

If you would say, some photograhers, I could agree. But I doubt, that most, as you write, photographers work with such tifs.

[RalfC]

I don't understand the internal workings of IMatch but how does increasing the timeout effect other users if their run times are shorter?

IMatch waits the timeout duration before cancelling Exiftool: The longer the duration, the longer everybody has to wait if Exiftool hangs. (And I am not sure if this waiting time makes IMatch look like "hanging" or unresponsive)

Btw., what kind of network are you using (LAN cable(s), WLAN, Powerline? Speed? Switch or router involved?) Those different network types may have a huge impact on data transfer performance.

Regards,
Ralf

[Ian Pollen]

Hi Ralf,

I wasn't aware of any ExifTool hanging problems. To date it has always worked for me.

The hardware involved is all wired and every device is Gigabit Ethernet capabale running through a Netgear ProSafe Switch to a Netgear ReadyNAS Pro Pioneer 6 disk RAID.

Ian

[RalfC]

Hi Ian,

the possibility that Exiftool hangs (or otherwise does not respond anymore) is the reason why Mario has implemented the timeout.
Doing so, is a rather common way to ensure that sudden interruptions (SW hangs, etc.) do not bring the whole system down.

IMHO, your network setup should be ok. Slower networks would certainly not be able to to handle this data amount in reasonable times, but handling such hugh files still might bring almost any setup to the edge, especially if more than one file (of such a size) would be handled at the same time.

Regards,
Ralf

[Ian Pollen]

I agree Ralf.

I wonder how hard it would be to make the ExifTool timeout a user configurable setting?
We could all choose our own desired and acceptable time based on our individual hardware setup.

Ian

[RalfC]

Ian, I took the freedom to comment your feature request mentioning the idea of the user setting, but I am not sure if many users would be able to set the time correctly (or if it is set wrongly identify the problem)

What still came to my mind is that IMatch might create many threads/sub-processes depending on your processor (the amount of cores).
There are settings in 'Preferences > Application > Process control' with default '0' (= let IMatch decide). [Although the help and the naming suggest that it would be for reading /ingesting data, I have the impression that it also affects the amount of Exiftool instances when writing data.]

Did you try to set the values to '1', i.e. limiting Imatch to one thread, which in turn limits the amount of resources/bandwith needed?

Regards,
Ralf

[Ian Pollen]

Thank you for adding the idea to the Feature Request Ralf.

I will try reducing the number Process Control to a lower figure and run some tests.
The PC involved is an Intel i7 with 8 Gig RAM.

Ian

[Ian Pollen]

Tried various process settings and no change. But I did try some other things which may be pointing us in the right direction...

See post in feature request thread:
https://www.photools.com/community/index.php?topic=4169.msg28100#msg28100

[Ian Pollen]

Quick update on my ExifTool 3 minute timeout problem when writing to a 760Mb TIFF file stored on a NAS accessed via CIFS.

Last night I created an iSCSI target on the NAS and accessed this on my Win7 PC. Copied the big TIFF files to it and ran the metadata save tests again.

Well what a difference... 37sec to complete the metadata save as opposed to never completing and hitting the 3min timeout. There is something seriously wrong with ExifTool operating over CIFS.

Just thought the info may be useful to others.

[Mario]

ExifTool uses file-system routines provided by the Perl runtime system. Perl uses the 'C' library of the given platform (in this case Windows). ExifTool does not interface directly with the file system or network protocol, and neither does Perl. They use services provided by Windows.

I'm not aware of any issues regarding CIFS. I have a CIFS share here and I get standard performance from the NAS. I use it to stream HD video material. If you see such dramatic differences, maybe the CIFS implementation is old or the SAMBA version is old or not properly configured.

I doubt that Phil can say much about the CIFS or NFS or whatever network-protocol is used in a specific NAS setup. But you can always ask a question on the ExifTool forum.

[Ian Pollen]

I have absolutely no problem is believing Windows is the culprit here ;-)

Thanks for your help with this Mario.

Ian

[Ian Pollen]

I know this is a VERY old thread but it is a continuation of the same topic / issue.

In the nearly 4 years that have passed I have on and off been trying to find a way to get decent speed over a CIFS / SAMBA network share when performing metadata writes. When using an iSCSI target the speed is good but VERY slow when not. In an ideal world I would like to share the data with multiple PCs so iSCSI is not ideal.

Is there a way to control where exiftool creates it temp files when performing an update? If there was a way to force temp files creation to a local drive it may improve the update speed when writing to network CIFS shares. My logic is it would half the traffic.

Ian

[Mario]

ExifTool creates the temporary files produced during merge/split tasks in he same folder containing the image file.

If your network is slow and causes issues, why don't you 'finalize' the files on your local disk and then move them to NAS for long-term storage?
This is a proven workflow used by many users, including myself. NAS can be 1000 times slower than a local SSD and it's not worth the wait still working on files. Once the files are finished, moving them to slow NAS systems is no problem at all.

[Ian Pollen]

As always Mario, thank you for the fast reply.

I understand that using the NAS purely as archive storage would be a workaround but I wondered if there was a quick fix by redirecting the TEMP file creation.
As my iSCSI targets from the same NAS work very fast, I can only assume the speed problem is caused by CIFS file access overhead for the multiple read / writes when updating metadata.

Time to look at some Thunderbolt 3 DAS devices ;-)

Ian

[Mario]

ExifTool uses a very safe streaming pattern when modifying files. Clearly a decision for safety and reliably instead of speed made by Phil.
This that when modifying metadata in a file, ExifTool transfers the file over the network to a temp file (also on the network). There is no separate TEMP storage location to configure. Phil has explained his reasons for that a number of times in the ET forum.

In addition, since IMatch implements MWG compliance and maps between XMP and EXIF, IPTC, GPS on on write back, this requires additional time and file operations. Since IMatch performs multiple write-backs in parallel since a few months, this can easily saturate a high latency network. Updating metadata on files already stored on long-term archival media like NAS boxes is not that common I guess so this is probably not a problem many users will ever have. And the work-around is simple: finish the files before moving them to long-term archival systems.

I once played with the though to, when a network drive is detected when attempting to write-back a file, make IMatch copy the file from the network to a local folder, perform the modifications there, and then copy it back to the original location. But this would a) complicate things a lot and b) would cause some of the metadata tags ET creates from the source file name and folder to be invalid. And it would be complicated. And networks and NAS systems get faster all the time.

[Ian Pollen]

I agree, given a few years and once 10GbE is the norm for LANs like 1GbE is now, then these problems will be less likely... assuming we are all not working with 100mpix files by then ;-)

Ian

[Mario]

IMatch Anywhere will solve this earlier. If IMWS runs on the same computer hosting the files (often) write-backs are super-fast. And, due to the asynchronous never-block IMatch WebViewer, it is irrelevant for the user editing metadata how long a write-back takes. He just moves on to the next file and lets IMWS sort out the rest.