IMATCH5_LOG.TX_ has gotten so large it has filled the boot drive

[Beach Photog]

What is "IMATCH5_LOG.TX_" used for and can it be located somewhere else? The installation program quite some time ago placed it on the boot drive and now it has grown to over 60 GB. The drive has very little space left and the computer has slowed to a crawl. I have plenty of space on other drives if need be. The actual iMatch 5 data file seems to be much smaller but it would be nice to put all the files elsewhere.

[Mario]

IMatch always creates the log files in the TEMP folder configured for Windows. You cannot change this.

On every start, IMatch renames the existing log file to IMATCH6D_LOG.TX_ and creates a new log file under the name IMATCH6D_LOG.TXT.
This means you have always the current log file and the previous log file on disk. This is very helpful for problem analysis.

Unless you switch IMatch to debug logging (Help menu > Support) or you let IMatch run for many days in a row, the log files remain quite small.
You did not write how large the log files are, but even several days or IMatch running produces barely more than a few gigabyte.

[sinus]

Mario is working on IMatch2017.
Hence he is a bit ahead from us, I think. 

He means, I think, not IMATCH6D_LOG.TX(T), but IMATCH5_LOG.TX(T).

I think, if one of your log is very big, I would temporarly store it (copy) to another drive, just in case.
Then I would delete this big file on C and start IMatch again.
If all is ok, I would then later, in some days, delete also the copied big one.

My log-files are for example
IMATCH5_LOG.TXT 143 KB
IMATCH5_LOG.TX_ 119 KB

You can also check the date of such a log.

[Beach Photog]

'sinus' is (more) correct here. My files were at the time I posted:
IMATCH5_LOG.TX_  about 62 GB and,
IMATCH5_LOG.TXT about 2.5 GB,
and I do mean GB  GigaBytes
This morning, having shut everything down and restarted, I have only  92.6 KB for  IMATCH5_LOG.TXT  and no sign yet of   IMATCH5_LOG.TX_ . It is correct that IMatch 5 had been running for about 36 hours because it has been so slow to extract data from files. Apparently there is something majorly wrong here, and that was going to be my next question out of 3 planned. The image file size on my machine is about 500 GB on a separate data drive, and I am operating Windows 10

[Mario]

Do you run the log file in debug mode?
I have never seen a log file that large. Probably something went seriously wrong, forcing IMatch (or probably ExifTool) into an infinite loop.
Without the log file, we can not tell. The problem would have been reported in the log file.

500 GB of file data is not much.

How many files do you try to process?
Do you have image files or video files? RAW files? PDF or Office documents?
You are not giving us much to work with. The more info you provide, the better we can help you.

I suggest you keep an eye on IMatch and stop it when you have the impression it is not making any progress,
The loader overlay will show you how fast IMatch is processing files etc.

The log file will tell us which files you are processing, how long IMatch needs for each file, if there are file format errors or invalid metadata etc.

[Beach Photog]

I do appreciate the information given here, and in fact the offending file was deleted when I closed iMatch so I can't send that as an example, but it did give me some headroom on the boot drive. I will experiment a bit with a small dummy database and try to find out what is going wrong. I did find that there are more corrupted or otherwise problematic files than I would have expected, mostly recent Photoshop files. I usually save with layers, in compatibility mode, and don't expect that would be a problem, but we will see. I have several other priorities at the moment (including finding out what has cluttered up the boot drive) so this experimention may take a while.

[sinus]

Good luck!

[Beach Photog]

OK, a little more time to study the problem. Looking at the image files I can see that many have been corrupted. Somehow a good number have been turned into .psd_exiftool_tmp files and empty new ones created in their place. The affected files have been mostly .nef, .raf, and .psd. Fortunately I could recover the data by deleting the empty files and renaming the temporary files to the original. The files I am working with are these types plus a few .dng and very small .mov plus sidecar files that seem unaffected. An example is attached as "damaged files screenshot.jpg".  I then started with a new iMatch database using only about 1000 images and did not find a repetition of the file corruption, but iMatch would only process a few hundred files before it obviously got into a loop, spending hours trying to get past one file. This time there is no file corruption, but the timestamp on the file is updated every few minutes. A log of the last attempt at running this is attached.

[Mario]

Go to Edit > Preferences > Background Processing and make sure that the Metadata Write Back is off (this is the default).
IMatch then no longer automatically writes back your files every time you make changes. Or directly after the import, which usually produces "new" metadata that needs to be written back. Which is what causes the loop you are seeing.

Your files are damaged. If ExifTool crashes or gets stuck (both highly unusual) during reading, IMatch will stop processing files after a while. IMatch detects hanging ExifTool instances and tries to close them. It then starts a new instance to continue. But IMatch does this only so often.

If you see temporary files produced by ExifTool, something catastrophic happens during write-back. ExifTool may use a temporary file during write-back to protect the original file. If it crashes, the temporary file may remain. I suggest you send some of your files to Phil from ExifTool for a deeper analysis.

[Beach Photog]

OK, setting Metadata Write-Back OFF cured that problem, but leaves numerous icons suggesting that write-back is pending. When I click the icon, there is some activity but the icon returns, still telling me that "XMP::dc\Subject" write-back is still pending. What exactly does this mean?

Otherwise, the system seems to be doing most of what I want except in the area of Categories display, but that will be the subject of a different thread.

I have found a lot more corrupted files which appear to have been created in the past by Exiftool. Where should I send samples of these?

Also, I use Photo Mechanic for file import operations and setting keywords and categories. This was originally optimized for the old version of iPhoto, and never had a problem. Is there a recommended menu of settings for the current version?

[Mario]

1. Open the ExifTool Output Panel (View menu > Panels) to see which error messages and warnings ExifTool reports when it is writing back to your files.

If keywords (dc:subject) is returning all the time, IMatch cannot synchronize your keywords between IPTC and XMP. This usually means that either your files contain non-synchronized keywords in IPTC and XMP (bad, usually caused by incapable applications) or that your keyword mapping settings, your thesaurus etc. cause infinite loops with your keywords. Please give details about your settings under Edit > Preferences > Metadata and ... Metadata 2. Also details about your IMatch thesaurus, if you use one.

To see whether or not it is useful to send your files to Phil, we need the IMatch log file and the output of the ExifTool output panel to judge. ExifTool producing problems or even crashing is so rare, I don't have much experience with that. In the past, badly damaged files or massively corrupted metadata was usually the cause of the problem. Since you mentioned PM and iPhoto, this is probably also the reason for your files.

Attach a sample file that makes problems so I can see what metadata your files actually contain. If the file is too large to attach, send it to my support email address (link back to this topic since I handle 30 to 50 emails per day).

[Beach Photog]

I've been busy generally re-organizing my file structure for simplicity and back-ups; and cleaning up files where the EXIF tool seems to have created the TMP versions cited above, and have not had a chance to pursue this particular issue much further except to establish that setting automatic write-back to OFF stopped the problem from growing. We will see and report back if the problem continues. However, there are other issues that I have spent a lot of time on such as Fuji raw and jpg files not being imported the same as Nikon files, and not being able to get the IPTC Category field to transfer into a hierarchical keyword similar to the supplemental categories. This is critical for my older files, but I will start separate questions since since they aren't appropriate for this title.

[Mario]

IMatch handles Fuji files just fine. I have maybe 50 from different Fuji models in my test set.
I don't know if they can contain IPTC data, though. XMP in a sidecar, yes. Embedded IPTC, dunno.
I will need at least a sample file to check. Keep that in mind when you make your other post.