Imatch 2017 crashes when trying to write back metadata to tiff files

[muggost]

Hi,
I've been looking in the forums but cant find any answers so a try with a new topic.

I have a database with TIFF-files (slides scanned with an epson perfection V800 photo). I use Lightroom to edit the pictures and just started using Imatch to organize them. I've used Imatch for many years so I'm familiar with the basic functions of the program. When I try to write back metadata to these TIFF files, Imatch hangs and crashes every time. Anyone else encountered this problem?

Please have a look at the attached logfile.

[Mario]

Hi,

this log file unfortunately does not show information about writing back metadata.
Please do this:

- Switch IMatch to debug logging (Help menu > Support).
- Select one TIF file and click the pen icon to trigger the write-back.
- When IMatch crashes again, secure all IMATCH_*.* files in your TEMP folder before you start IMatch again.
- Attach to your reply.
- Switch debug mode back to normal.

IMatch writes metadata back via ExifTool. Maybe ExifTool has a problem writing to your files. Or your files are very large and on a very slow medium and ExifTool takes too long. The lo file will tell me the details.

Note: When IMatch crashes it produces a dialog with details and a DUMP file. Did you see this message?

[muggost]

- Switch IMatch to debug logging (Help menu > Support): DONE
- Select one TIF file and click the pen icon to trigger the write-back. DONE
- When IMatch crashes again, secure all IMATCH_*.* files in your TEMP folder before you start IMatch again. : Which TEMP folder? I cant find the files or folders you're looking for.

IMatch writes metadata back via ExifTool. Maybe ExifTool has a problem writing to your files. Or your files are very large: About 35MB
and on a very slow medium: QNAP Turbo NAS 459. Should be ok.

Note: When IMatch crashes it produces a dialog with details and a DUMP file. Did you see this message? Nope, it only hangs and the windows tries to shut it down.

Heres a new log file though...

[muggost]

Interesting....
In the folder where the file is located, a new file has appeared:

FILENAME.TIF_EXIFTOOL_TMP

It doesnt go away when I reopen and close Imatch again.

[Mario]

hich TEMP folder? I cant find the files or folders you're looking for.

Please use the IMatch help. Search for log file in the Help Index on the left to find the help topic on log files. It explains about IMatch log files, where to find them etc.



So, IMatch does not really crash? It just hangs for a long time and you then shut it down?

When ExifTool 'hangs' or 'freeues'while processing a file, IMatch destroys the process after 120 seconds and then continues.
ExifTool is very, very stable and this is ia last resort, basically to protect IMatch.

Are you processing very large files over network? Maybe ExifTool is working normally, just taking more than 2 minutes to shuffle your files over the network and back?

You can also test-write again on the NAS, but this time wait for at least three minutes before you consider IMatch as crashed. I think we see then messages about ExifTool timeouts in the log file.
Then download the file to your local disk and retry. If this works, we know what the problem is and how to avoid it.

Large files + NAS + read7write => potential problems.

In the folder where the file is located, a new file has appeared:
FILENAME.TIF_EXIFTOOL_TMP

This would also indicate that ExifTool has crashed when processing the file.
ExifTool uses temporary files for certain operations, and when it crashes it cannot clean up these temporary files.
You can delete this file if the original file exists and is OK.

[muggost]

Ok, will test out these things tonight and see if I can get any wiser

Just thought I'd mention that when I write the same data to the same files on the same NAS with Adobe Lightroom, the write process completes in less than a second. So why don't I use LR to write these data then? Well - LR is an absolute nightmare when it comes to organizing large amounts of files. Imatch is much much better, so I'd prefer to use Imatch if I can.

[Mario]

Lr uses the risky way by updating individual bytes in the file. ExifToo does the safe way, using a streaming approach, ExifTool processes the whole file, ensuring that all metadata, including proprietary Exifdata and maker notes, is processed correctly before rewriting the file. This behavior is built into ExifTool and not under the control of IMatch. If you prefer the way Lr updates your metadata, use Lr to update your metadata.

[muggost]

Ok, here are the log files from the first try:
Changed metadata in "headline" field.
Clicked Writeback metadata
Waited about ten seconds: Windows notifies me that the program has hung. I click "wait for program to finish"
Waited five more minutes: Aborted process myself.

I will move the file to local machine and see if that helps now.

[muggost]

Ah...
I moved one file to C:\TEMP\ and tried to writeback the same metadata and it completed successfully in 1,5 seconds. Do you need to logfiles from this try?

Anyway... This poses a problem for me - I cannot have all these files on my local storage. The database will grow to about 33.000 TIFFs in the coming years. I depend on having it on a NAS, so I hope we will manage to find a solution to this problem. Going back to LR to organize these pictures isnt really feasible at all...

[Mario]

At the time you forcefully shut down IMatch, IMatch was still waiting for ExifTool to complete the write process.

IMatch is intelligent. It does not just wait for 120 seconds for ExfiTool to finish (which is a very long time) but it also monitors the ExifTool process. If the ExifTool process still consumes CPU and I/O resources (reads/writes data from/to disk) IMatch considers it "busy and working" and does not shut it down. If ExifTool needs longer than 2 minutes to write one of your files, something is truly wrong. Even if the file has several hundred MB your NAS should be able to deliver it that fast over the network.

If the same file takes 1.5 seconds to write on your local disk but more than several minutes when on your NAS, either the network or the NAS have an issue. ExifTool uses "streaming" I/O, which is usually very fast even over slow networks...

[muggost]

Ok. I'm at a loss of what to do then. The NAS does its self checks every night and reports nothing wrong. I cant say I've noticed any issues with any of my other applications either, and LR writes the metadata successfully to the same files on the same NAS.

I will try to test other applications in combinations with the NAS and see if I can find anything to enlighten this issue. If any of you other people on the forum has any ideas, I'd appreciate it if you wrote something here

- M.

[Mario]

I use a NAS myself for archival storage. Gigabyte cable link.

If you have these issues, maybe use a standard workflow where you "finish" the files locally (including writing back metadata) and then move them to the NAS for archival.

[muggost]

Just an additional note: I didn't have this issue with Imatch 5.4.18. I didn't test it on TIF's but JPGs were written very fast then. I only had version 5.8.4 for 30 minutes today, so I didn't get to test that version. I'm on 2017.6.6 and it seems like this is the first time for me with this issue.

I have rescanned the Raid 5 volume on the NAS, and updated the firmware but the same problem exists. Lightroom and all other applications works fine in conjunction with the NAS. I'm no expert, but to me, this seems like an issue with Imatch/Exiftool, not the NAS...

[Mario]

No changes in metadata handling between recent 5 versions and the current versions. IMatch 5.4 is more than a year out of date so I cannot comment.
No similar problem reports in my Inbox or here in the community.
Maybe this is a problem with ExifTool and your files. Upload a sample file somewhere so I can download and test it.
How large are your files?

But if the file writes OK in 1.5 seconds locally, and fails on the NAS, the problem must be caused by how your NAS transfers the file, or rather, the speed.
How many MB/sec can your NAS transfer? Cable or Wi-Fi connection?
If you look at the Performance tab in the Windows task manager while ExifTool is writing the file, is there network and disk activity?

[muggost]

Glad you can help
The files are from 30-60MB in size. I've put on of them on google drive so you can download via this link:
https://drive.google.com/file/d/0BxCLR0Fz4BWOcGNLdHNOUWl2R3M/view?usp=sharing

The NAS is a QNAP TS-459 which I bought specifically for good transfer speeds back in 2009. Its served me absolutely flawlessly since then. I'm using a cabled network connection and it gives me about 100MB/sec when transferring large files and about 70MB/sec when transferring large amounts of small files. I'm due to replace it with a new one before the end of 2017, so this issue poses a bit of a conondrum for me. I don't want to spend €1500 on a new one if I can't use it with Imatch anymore

When I transfer files theres plenty activity in bort windows task manager and the NAS's internal resource monitor, but almost nothing when writing metadata to the files (<400KB/sec). I also discovered yesterday that the same problem persists when trying to write to my older JPGs. I have some pictures thats less than 1 MB in size, and they also take a long time writing metadata to. Not the same time as with the large TIFF's but still, 10-12 seconds for a 900KB file is not normal...

[Mario]

60 MB is nothing. And if ExifTool also fails to write-back fast to files with only 1 MB on your NAS, there must be some external reason. Virus Checker?
I've downloaded the file. Write-back locally takes 1 second, write-back when the file is on my Synology NAS takes 7 seconds. Write-back when stored on a Windows server takes 3.6 seconds.

[muggost]

I only use windows defender on the computer. I'll see if the NAS has some internal virus checker and if it can be disabled.

[muggost]

This message was waiting for me just now...
https://drive.google.com/file/d/0BxCLR0Fz4BWOa3FZdUFVVWRMWkU/view?usp=sharing

[muggost]

Disabling the internal virus checker didnt help. It took about 3 minutes to writeback metadata on a 34MB file.
I will try to reboot the switch and the router thats between my computer and the NAS and see if that helps. I've had issues with them in the past, but nothing like this.

[muggost]

nope, no joy :-/