Writing back metadata taking forever....

[Jingo]

Hi Mario - curious.. I just finished adding 3 keywords to 29 images... did a Ctrl-Alt-S to write pending back to the image files... and it took almost 15 minutes!  Here is part of the log... why is there a
"PTWrapper hangs in ProcessRun for 120000 ms"  after each image?  Seems like 4 images write back at a time but this still takes forever.   Files stored on V drive (7200 rpm brand new drive partition) database is on C drive (840 EVO SSD).  I have no virus scanner on the system running (only Windows Defender) so the system can't be blocking Exiftool.... thoughts?  Also, the cancel button doesn't seem to respond.... I clicked in a number of times and it still continues to process the selected files.

Thx!

Code Select Expand

05.03 17:20:23+600609 [2658] 00  E>     PTWrapper hangs in ProcessRun for 120000 ms.  'v:\develop\imatch5\src\imengine\ptetwrapper.cpp(3746)'
05.03 17:20:23+    0 [2658] 00  E> -overwrite_original_in_place
-charset
FILENAME=UTF8
-m
-P
-q
-q
-use
MWG
-charset
ExifTool={PTETCHARSET}
-ex
-tagsfromfile
V:\Family_Photos\2019\2019-01\D500_01-21_2373.jpg
-@
C:\Program Files\photools.com\imatch6\arg_files\exif2xmp.args
--Exif:rating
-@
C:\Program Files\photools.com\imatch6\arg_files\iptc2xmp.args
-@
C:\Program Files\photools.com\imatch6\arg_files\gps2xmp.args
-sep


-XMP-lr:HierarchicalSubject=
-XMP-dc:Subject=
-XMP-lr:HierarchicalSubject=Events|snow
-XMP-lr:HierarchicalSubject=Others|Posing
-XMP-lr:HierarchicalSubject=People|Family|Lauren
-XMP-dc:Subject=Family
-XMP-dc:Subject=People
-XMP-dc:Subject=Posing
-XMP-dc:Subject=snow
-XMP-dc:Subject=Lauren
-XMP:CreatorTool=photools.com IMatch 19.4.0.2 (Windows)

-xmp:InstanceID=xmp.iid:d0f981a6-ae80-4cf7-892a-b13cc4ee1979

-XMP:MetadataDate=now
-XMP:ModifyDate=now
V:\Family_Photos\2019\2019-01\D500_01-21_2373.jpg
-execute
-overwrite_original_in_place
-charset
FILENAME=UTF8
-P
-m
-q
-q
-EXIF:ImageDescription=
-EXIF:Software=
-EXIF:Copyright=
-EXIF:Artist=
-EXIF:UserComment=
-IFD0:ModifyDate=
-IFD0:Rating=
-IFD0:ImageDescription=
-IFD0:Software=
-IFD0:Copyright=
-IFD0:Artist=
-IFD0:XPTitle=
-IFD0:XPComment=
-IFD0:XPAuthor=
-IFD0:XPKeywords=
-IFD0:XPSubject=
-GPS:all=
-tagsfromfile
V:\Family_Photos\2019\2019-01\D500_01-21_2373.jpg
-@
C:\Program Files\photools.com\imatch6\arg_files\xmp2exif.args
-@
C:\Program Files\photools.com\imatch6\arg_files\xmp2gps.args

V:\Family_Photos\2019\2019-01\D500_01-21_2373.jpg
-execute9999
  'v:\develop\imatch5\src\imengine\ptetwrapper.cpp(3747)'
05.03 17:20:23+    0 [376C] 00  E>     PTWrapper hangs in ProcessRun for 120000 ms.  'v:\develop\imatch5\src\imengine\ptetwrapper.cpp(3746)'
05.03 17:20:23+    0 [376C] 00  E> -overwrite_original_in_place
-charset
FILENAME=UTF8
-m
-P
-q
-q
-use
MWG
-charset
ExifTool={PTETCHARSET}
-ex
-tagsfromfile
V:\Family_Photos\2019\2019-01\D500_01-21_2372.jpg
-@
C:\Program Files\photools.com\imatch6\arg_files\exif2xmp.args
--Exif:rating
-@
C:\Program Files\photools.com\imatch6\arg_files\iptc2xmp.args
-@
C:\Program Files\photools.com\imatch6\arg_files\gps2xmp.args
-sep


-XMP-lr:HierarchicalSubject=
-XMP-dc:Subject=
-XMP-lr:HierarchicalSubject=Events|snow
-XMP-lr:HierarchicalSubject=Others|Posing
-XMP-lr:HierarchicalSubject=People|Family|Lauren
-XMP-dc:Subject=Family
-XMP-dc:Subject=People
-XMP-dc:Subject=Posing
-XMP-dc:Subject=snow
-XMP-dc:Subject=Lauren
-XMP:CreatorTool=photools.com IMatch 19.4.0.2 (Windows)

-xmp:InstanceID=xmp.iid:6985503b-5f9d-44a5-bd16-fedfd053fc3b

-XMP:MetadataDate=now
-XMP:ModifyDate=now
V:\Family_Photos\2019\2019-01\D500_01-21_2372.jpg
-execute
-overwrite_original_in_place
-charset
FILENAME=UTF8
-P
-m
-q
-q
-EXIF:ImageDescription=
-EXIF:Software=
-EXIF:Copyright=
-EXIF:Artist=
-EXIF:UserComment=
-IFD0:ModifyDate=
-IFD0:Rating=
-IFD0:ImageDescription=
-IFD0:Software=
-IFD0:Copyright=
-IFD0:Artist=
-IFD0:XPTitle=
-IFD0:XPComment=
-IFD0:XPAuthor=
-IFD0:XPKeywords=
-IFD0:XPSubject=
-GPS:all=
-tagsfromfile
V:\Family_Photos\2019\2019-01\D500_01-21_2372.jpg
-@
C:\Program Files\photools.com\imatch6\arg_files\xmp2exif.args
-@
C:\Program Files\photools.com\imatch6\arg_files\xmp2gps.args

V:\Family_Photos\2019\2019-01\D500_01-21_2372.jpg
-execute9999
  'v:\develop\imatch5\src\imengine\ptetwrapper.cpp(3747)'
05.03 17:20:23+    0 [3564] 00  E>     PTWrapper hangs in ProcessRun for 120000 ms.  'v:\develop\imatch5\src\imengine\ptetwrapper.cpp(3746)'
05.03 17:20:23+    0 [3564] 00  E> -overwrite_original_in_place

[Mario]

This means that ExifTool never completed and the IMatch watch dog automatically killed the process after 120 seconds.
Typical causes: Badly damaged file or corrupted metadata. Virus checker blocking ExifTool.

Reboot your system. Retry. If the problem persists, check your virus checker.

[Jingo]

This means that ExifTool never completed and the IMatch watch dog automatically killed the process after 120 seconds.
Typical causes: Badly damaged file or corrupted metadata. Virus checker blocking ExifTool.

Reboot your system. Retry. If the problem persists, check your virus checker.

Thx Mario - no virus scanner was running on the system and the files are fresh JPG's just generated from Dxo RAW files.  I'll try the shutdown/restart to see if this helps.... never had this happen before but then again, this is a new system build.  thx!

[ubacher]

You say Windows Defender is running: you need to put exceptions for Imatch, exiftool......

[lbo]

Here is part of the log... why is there a "PTWrapper hangs in ProcessRun for 120000 ms"  after each image?

I also experienced "PTWrapper hangs", see details in thread "Hunting intermittent write back hangs" https://www.photools.com/community/index.php?topic=8529

I couldn't reproduce it on my old Windows 7 Notebook after setting logging to "debug" and ExifTool output to verbose (but I rarely used IMatch after that).

This year, I got a new desktop PC with Windows 10, and during my first tests, it happened again! I had no verbose logging active because it was a new PC and new IMatch so I didn't expect problems. Will continue with extended logging now.

Since it happened on two very different machines (Win 7 <-> 10, Desktop <-> Notebook, fresh install), it is rather unlikely that "my system" caused the problem.

We'll see how this develops.

[lbo]

You say Windows Defender is running: you need to put exceptions for Imatch, exiftool......

Many widespread recommendations to fix problems (in a Windows environment) are not necessarily the right ones, e.g. "run as administrator", "disable virus scanner", "reinstall" etc.

I've been using exiftool for years without setting AV exceptions (and as non-admin).

But the way exiftool resp. the PAR packer works (unpack and run executables in %temp%) is somewhat suspicious from an antivirus software point of view.

For security reasons, it's not advisable to run executables in the public writable temp folder. It would be a perfect place to put persistent malware in.

And why does IMatch need an AV exception? No other software on my system needs such an exception!

[Mario]

IMatch does not need an AV exception in general. Only when your AV makes problems.

IMatch launches ExifTool as a process to read/write metadata. This might trigger your AV. IMatch also launches other helper applications that way, which may trigger your AV.

In 90% of all "IMatch is slow" reports, the AV was the culprit. Blocking IMatch. Permanent on-access scans on the database file and temp files created in the database folder, etc.

I just recall a "The IMatch database takes 20 minutes to load" report from a couple of weeks ago. Adding IMatch and the database folder to the AV exception list resulted in the database loading in 20 seconds instead 20 minutes. Such things just happen, even if they don't happened to you yet.

Don't judge tips given by other volunteer users based on your own experiences alone. What works for you may not work for others.

IMatch uses the per-user TEMP folder, and so does ExifTool to store files required by its Perl runtime system. Neither I nor Phil (most likely) can change how the Perl runtime system works. Feel free to ask them or the Perl authors.

[Jingo]

You say Windows Defender is running: you need to put exceptions for Imatch, exiftool......

Sorry - I should have been more clear - only the Firewall portion of Windows Defender is running... the antivirus piece is off. 

Just to note, after closing and restarting IMatch - the problem no longer appears....  who knows...  could be anything really but so long as it is does not happen often, I'll just chalk it up to PC Gremlins!

[Mario]

Sometimes WIC blocks files for prolonged time. If this happens (e.g. while you have the Quick View panel open) and you also attempt to write to the file, ExifTool may stall for some time and then return with an error.
But IMatch having to shutting down multiple instances of ET after 120 without any action (IMatch monitors both ET CPU usage and file system usage to determine if it is dead) indicates that there was something else going on.

I use 3 different AV products on different machines (for extra safety) and I had never problems so far. But as I said above, things just happen...

[lbo]

Don't judge tips given by other volunteer users based on your own experiences alone. What works for you may not work for others.

if that comment is directed to me: My posting should be understood in the same manner you wrote above: An AV exception may "work for you may not work for others".

I just disagree with ubacher's general statement "you need to put exceptions for Imatch, exiftool".

If I understand correctly, you also write that an AV exception is not needed generally but may be tried as a solution.

IMatch uses the per-user TEMP folder, and so does ExifTool to store files required by its Perl runtime system. Neither I nor Phil (most likely) can change how the Perl runtime system works.

Well, there are several approaches to avoid the runtimes being executed from the temp (sub-) folder.

Likely the simplest is to set the PAR_GLOBAL_TEMP environment variable. Phil writes that this should work also if you are running more than one PAR packed application.

[lbo]

Just to note, after closing and restarting IMatch - the problem no longer appears....  who knows...  could be anything really but so long as it is does not happen often, I'll just chalk it up to PC Gremlins!

Same here. Even without restart, the problem vanished one time on my old PC while I made some experiments to track it down.

[Mario]

Well, there are several approaches to avoid the runtimes being executed from the temp (sub-) folder.

Likely the simplest is to set the PAR_GLOBAL_TEMP environment variable. Phil writes that this should work also if you are running more than one PAR packed application.

That's a great tip for when your AV solution reports in its log files that it has blocked the Perl runtime in the TEMP folder.
Unless this happens, I would not change this, in case this breaks other applications which use the Perl runtime (if you have any).

I'm not aware of any support case where this was a problem, though. Usually the exiftool.exe itself is blocked (or IMatch).

[DaveO]

I just experienced the same issue yesterday.  The images were imported a few days ago into Adobe Lightroom and one was a JPG file and the rest raw files from a Sony A77, so ARW.   I got LR to generate the XMP files before importing the folder into iMatch for the raw files but iMatch hangs on processing both JPG and ARW files. 

I tried to add the GPS location using OPenStreetMap as the map (I have an account set up) and it took forever.  I added the exceptions for C:\Program Files\photools.com\imatch6\IMatch2019x64.exe and C:\Program Files\photools.com\imatch6\exiftool.exe in the anti-virus program (Avast) which did not fix it.

Interestingly after it "failed" a little yellow triangle with an exclamation mark in the middle appeared on each image and when I hovered over it, a message appeared saying the operation had failed and this indicator would disappear in I retried the operation successfully or if I rescanned the image.

On rescanning the image the yellow triangle disappeared and the globe symbol indicating the image has GPS data now appears on each image.  Clicking on the globe shows the correct location on the map.  So it appears to have worked, not failed which suggests maybe iMatch is not detecting exiftool has completed the operation?

Back in LR importing the settings from disc works and the images are flagged in LR as having GPS data.

I have attached the iMatch log.  It isn't very big.

[Mario]

The yellow warning triangle is displayed when ExifTool returned an error or there was another error writing back metadata or reading metadata.
If you see this icon, something went wrong and IMatch is using the warning icon to inform you about this.

Your log shows:

PTWrapper hangs in ProcessRun for 120000 ms

which means that IMatch has waited for 120 seconds for ExifTool to complete processing (usually this takes 1 second) and then aborted the process.
A typical result of a file being locked by another application, a virus checker blocking ExifTool, ExifTool running into problems with a file.

Did you have Lr running at the time? Adobe is notorious for locking files for long times so maybe try again next time without Lr running (or Lr using a different set of files).

[lbo]

although this thread is a bit old, I need to add an update:

Well, there are several approaches to avoid the runtimes being executed from the temp (sub-) folder.

Likely the simplest is to set the PAR_GLOBAL_TEMP environment variable. Phil writes that this should work also if you are running more than one PAR packed application.

That's a great tip for when your AV solution reports in its log files that it has blocked the Perl runtime in the TEMP folder.
Unless this happens, I would not change this, in case this breaks other applications which use the Perl runtime (if you have any).

I'm not aware of any support case where this was a problem, though. Usually the exiftool.exe itself is blocked (or IMatch).

setting PAR_GLOBAL_TEMP does not work for multiple applications. It even blocks updates, so do not use PAR_GLOBAL_TEMP! Roderich Schupp (the current maintainer of PAR) wrote "Use of PAR_GLOBAL_TEMP is discouraged".

This only as a warning not to follow my previous recommendation, more details in a new thread.

Oliver

[muggost]

I also have this problem and been having it for some time. I have tried removing and reinstalling imatch, i've tried to make exceptions for Imatch and exiftool in windows defender and I've tried to move my images (Canon CR2-raw files) from a QNAP NAS to a local drive. Still, metadata takes forever to write and in most cases, it fails

There's a little triangle icon on the thumbnails saying that metadata write back failed. On rescanning the file, all metadata is reverted to the original. Also, exiftool leaves broken temp-files in the folders after trying to write back metadata.

Everything worked so fine before (I think the problem stared with Imatch 2018). Although my 1GB network connection to my QNAP NAS is a little slow, everything still worked. Now it fails from NAS, from Local Hardrive C (SSD) and Hardrive D (HDD). Sometimes I can try to write back metadata 3,4 or 5 times before it suddenly works.

Where do I start fixing this?

[jch2103]

A log file would be useful (debug logging: Help/Support).

[muggost]

Will supply that tomorrow.

[Mario]

A warning indicator means that ExifTool has failed to update your files.
Which is rare and usually indicates badly corrupted data in the file, a broken or shaky network connection etc.

The IMatch log file and the ExifTool output panel will show you detailed information.

Move the file from your NAS to your local disk and retry. If this works, the NAS, SAMBA implementation running on your NAS or your network connection is the source of the problem.

[muggost]

The problem is exactly the same, no matter which drive I use. I cant attach the log file. Its bigger than maximum size allowed.

[Mario]

1. When you ZIP your log file beforer attaching you can attach even very large logs. They usually zip down to 20% or so.
2. When you hover the mouse over the yellow warning icon in the file window panel it displays a tooltip with the error message returned by ExifTool.
3. The error message returned by ExifTool is also available in the special "Error" tag available in the Metadata Panel, Switch to Browser mode.

ExifTool needs 15 seconds (!) to extract metadata from 20 of your CR2 files, via your network and from your NAS box. This is quite slow.
About 200 tags are extracted from each file.

No errors are reported in this log and no write-back operations are performed.

What I see are tons of messages like "Only thumbnail or small preview extracted. Falling back to ptc RAW processing.".
This means that the WIC codec installed on your system cannot handle the RAW files you are processing, and that IMatch has to try with the built-in photools.com RAW processing or even the Windows thumbnail handler. This will reduce performance massively, because your files have to be pulled over your network multiple times, unless Windows has them in the cache.

Please make sure that you have a WIC codec installed which can handle the RAW format produced by your camera.
Your camera vendor is responsible for providing you with a working WIC codec.

See WIC (Windows Interface Components) in the Help System.
And WIC Diagnostics for how to find out which codecs are installed in your system and if they are able to process your files.
It seems you are processing files with the CR2 extension. Canon has produced about 50 incompatible variants of that format over the years.

[muggost]

Ok, I will check out the WICs first then, and get back to you if that fails too.

[muggost]

Ok, maybe WIC is the problem then:

In the attached file it says "WIC Result: It looks like no WIC codec is installed which can handle this file."

But it also says:
Version: 0.19.0-Beta1
List of supported cameras: 1063
Canon EOS 6D Mark II (this is the camera I'm using).

[Mario]

You have a WIC codec for CR2 installed (the standard CR2 codec included in Windows 10).
But it seems not to handle your particular CR2 format - which is not uncommon because Canon has created so many different and incompatible CR2 formats over the years (a real archive mess lurking there).

But since a codec is installed fr CR2, IMatch will use it.
And the codec can neither extract a preview or a full RAW, IMatch will fall back to using LibRaw.
But that means that each of your files need to be read 3 times. Usually not that much of a problem, but with your slowish NAS and network, this will bring down performance considerably.

I suggest you contact Canon and ask them for a WIC codec supporting your files.
Just kidding. Canon has stopped providing a WIC codec years ago, for cost-cutting reason. A shame, really. They let their customers down by not supporting the official way to work with RAW files in Windows...

Unless you find a WIC codec which supports your CR2 variant, you can speed IMatch up by forcing it to ignore WIC and only use LibRaw (Edit > Preferences > Application, Search for RAW and enable the "Prefer photools.com RAW processing). Now IMatch will not using WIC codecs anymore. Which could cause other problems, if you have also other RAW files which would benefit from a WIC codec supplied by the camera vendor (they know their cameras and file formats best, after all).

All this "Our RAW format is undocumented and secret, we don't provide developer support and we change it with every new camera model or firmware update"  is utter crap. It's bad for the user and only plays into the hands of the big companies like Adobe which maintain entire teams of (cheap) developers in India to reverse-engineer RAW formats so they can support them in ACR and Lr.

Smaller companies cannot do this. Microsoft created WIC over 10 years ago to exactly handle that case. Windows provides the infrastructure and the camera vendors provide the WIC codec (similar to a printer driver). The camera vendor can produce superior outputs in their WIC codec without revealing anything about their secret file formats or algorithms. But Canon stopped supplying a WIC codec years ago, in favor of their "Canon RAW Codec Software" which apparently nobody is using...

Microsoft does a fairly good job supporting many cameras out-of-the-box with the standard Windows 10 WIC codecs. But they don't support all camera models. Some vendors have produced dizens of format variants, all with the same file extension. Users are usually unaware of this and let the camera vendors get away with it. But you have paid for your hardware and you should demand that Canon supports it properly via an up-to-date WIC codec. Even if they had to hire a full-time programmer to do this, it should costs them 50K a year or so. They spend more on TV commercials in a week...

[muggost]

Thanks for the extensive answer.
I let Imatch prefer photools processing. Estimated time to complete updating metadata in 83 files: 35 minutes! That is from my local D:\ drive (Western Digital Green 1TB disk).

Why is it that Imatch / Exiftool sometimes can update the xmp-metadata file correctly and other times it fails? Why doesnt it either work 100% or fail 100% ?

[Mario]

35 minutes is ridiculous.
Maybe your virus checker is interfering with IMatch?
Make sure you make IMatch an exception and also exclude the folder containing the database (!) from on-access virus scan. See IMatch Help for details.

All your log files show IMatch reading metadata (20 files in 14 seconds).
I did not see any trace of writing metadata. What does make you think writing is the problem.

IMatch by default does not write-back metadata automatically. It waits for you to explicitly trigger the write-back via the Pen icon or the Commands menu.
Or do you have enabled the write-back immediately option in the preferences? In that case the slow write-back may be caused by infinite loops created by the metadata in your files in combination with your settings and import options. But then the log file would show write-back operations, which it does not.

If you have a file which produces this behavior:

1. Upload the file and the associated XMP file (if any) to your cloud space and send me a link to support email address
2. Include a description of your metadata settings (Edit > Preferences > Metadata and Metadata 2) if different than default.

[ColinIM]

Ah! I see Mario replied to you muggost while I was compiling my long blurb below, and of course I wrote this as a suggestion for a quick & easy 'test' - without having seen your LOG files.

Why is it that Imatch / Exiftool sometimes can update the xmp-metadata file correctly and other times it fails? Why doesnt it either work 100% or fail 100% ?

As a quick and easily reversible test, consider changing (or specifying) the number of CPU threads that IMatch uses to process the metadata in your files.  (I relate my own experience at the end of this post with the changes I made to these CPU thread settings.)

The 'number of threads' setting is here:

IMatch Menu | Preferences | Application (tab) | Process Control

There are three sub-settings in that Process Control menu section:

1. Threads for file import
2. Threads for metadata import
3. Threads for metadata write-back

As an easy 'first step' you might try setting all three of these initially to '1' (one thread each), or to '2' (two threads each), then test whether you see any benefit.

If you find that your experience is unchanged with different CPU thread settings then I'd urge you to return them all back to their default value of '0' - then continue to look elsewhere for the cause(s) of your hiccups.

If you do see an improvement then - depending on your CPU type - you could then try other small adjustments to one or more of the three 'threads' sub-settings.  In my case I have set all three sub-settings to '2' (two threads each).

I don't pretend that '2' is a magic number for any of these sub-settings nor for any other circumstances.  You must run your own tests and make your own conclusions with your own computer and disc-storage or network-storage scheme.

TL;DR

Here's my own experience with these 'CPU thread' settings.

It was more than a year ago that I saw my IMatch take a long long time to update (write-back) metadata, even on a small number of files, and I saw that my hard drive was permanently active during these write-back sessions.

I took an informed guess (as a long-in-the-tooth old techie) that my computer was 'thrashing' - or running in inefficient circles - during the disk input/output while it was updating the metadata.

When I changed the number of CPU threads - as above - from '0' (where '0' allows IMatch to optimize its use of our CPU's threads) to just 2 on all three of the sub-categories under that Process Control menu option, the thrashing stopped.  My metadata processing speed returned to what I remembered from an earlier IMatch era as approximately 'normal', and with only occasional hard drive activity during metadata write-backs on those same ordinary .TIF and .JPG files.

Naturally this doesn't indicate that there is or was a "problem", either with IMatch or specifically with my computer system.  Rather it's an example of how IMatch was able to interact with my specific computer and its hard drives.  We can applaud Mario for including these CPU thread adjustments for the rare occasions (such as mine) when they could be fruitfully tweaked.

Your mileage may vary ....

[Mario]

Interesting. Thanks for sharing.

Usually keeping disks at 100% utilization (totally busy) is what IMatch aims at - because writing back metadata is a purely I/O-bound task (not much to do for the CPUs in that situation).
IMatch uses multiple threads, based on the CPU count and some other measures. This usually works  just great, but may need some tweaking in some cases - especially for very slow disks or NAS storage.

There is a knowledge-base article for that: Configuring Process Control for Slow Media (CD-ROM, DVD, NAS, ...)
Some NAS boxes show really strange behavior when under heavy load, depending on the board, SAMBA version running, throttling and power consumption settings and whatnot.
The same is true for Notebook systems and similar.

Hence I provided these settings as a way for the user to take charge and optimize things for her/his system.

As we all have learned with the recent "Mozilla forgets to renew a certificate and all Firefox plug-ins world-wide no longer work" disaster, programmers should make good judgments but should allow their users to override these in case the poo hits the fan. Mozilla did not and we all had to deal with the "Your plug-ins no longer work until we figured out how to renew our certificate, if you want or not"  disaster.

I more and more get the impression that we, as users, are incapacitated  by software companies. Which is not a good thing...

[lbo]

There is a knowledge-base article for that: Configuring Process Control for Slow Media (CD-ROM, DVD, NAS, ...)

you might want to update this article to reflect the current options (third entry for writeback threads).

Oliver