Issue after relocating

[Carlo Didier]

Due to disk conbfiguration changes, I have re-organized a number of my folders which are indexed by iMatch.

I thought I'd simply use the relocate function, but that goes completely bonkers ... First, I didn't realize it was iMatch, but I finally found out that it drowned my disk with .tmp files in the C:\User\username\appdata\local\temp folder.
When I only re-locate some small folders, it works, but still, iMatch doesn't cleanup the temp folder (see screenshot).  When I tried to relocate all, it filled up all the free space on C: (over 160GB in >4000 .tmp files !!!). I don't understand what it does and why, after the re-locate, it re-scans all the files as if they had changed (and yet, it doesn't show the "refresh needed" icon on any folder!).

Attached a screenshot and a debug log where I just re-located a few folders.

[Mario]

IMatch does not use temporary files occasionally, e.g. while working with LibRaw in "cannot use embedded preview mode", but these temporary files use a unique number and don't start with "pli".
Temporary files created by IMatch always use the .imt extension (.imt = IMatch Temp). The files in the screen shot all start with "pli" and use ".tmp" as their extension.

IMatch may create a larger number of temp files when you force a temporary cache (for cache images). But these also use unique numbers as the file name, and the .jpg extension. They are deleted at program end or when there are too many of them. But using a temp-only cache is really rare. The temporary cache folder on your system is C:\ProgramCaches\iMatch\958695B0-6164-45EF-AA43-0F0CF83CF6FA\temp\Carlo\

Relocating does not create temporary files.
I see that IMatch is indexing a large number of files. And logs warnings about missing cache images (have you moved the cache folder?)
I don't see how relocating a folder or a disk can cause this. Relocating is just checking the file system to ensure that the relocation target exists and then updates the database with the new path. This does not cause any files to be created or anything.

I did not find any place in IMatch that produces temporary files with a "pli" prefix or not using the .imt extension.

You could use the SysInternals process Explorer (free, from Microsoft) to figure out which application or process is producing these files.
Maybe it's LibRaw or a WIC codec, I don't know. But that would not be triggered by relocating files, but by IMatch re-indexing files and restoring no longer existing cache images. But that would also indicate that something in your setup has changed, because relocating does not cause files to become re-indexed or missing cache images or similar.

[Carlo Didier]

I couldn't understand it either.

I went back to a backup of the database, then relocated folder by folder and finally got it sorted without issues.

But it was definetely iMatch (or maybe Exiftool?) who created those temp files, because they always started appearing as soon as iMatch started re-indexing and they stopped being created as soon as I quit iMatch (but they weren't deleted).
It's working correctly now, so I can't reproduce it with Process Explorer running. If the issue re-appears (hopefully not), I'll try to see more with Process Explorer.

[Mario]

I've never seen a "pli * .tmp" file in my temporary folders, so I have really no idea. Hence the Process Monitor, which would provide additional info. If it is ExifTool, process monitor shows exiftool.exe as the creator.
When you force rescan a folder, do the files also show up?

[Carlo Didier]

It just happened again on a re-locate!

I got Process Explorer running just before it finished but had no time to see which process was creating the files. But they are back in the temp folder, appearing while iMatch was re-indexing. The only thing I managed to see was that iMatch was calling ffmpeg a few times, but not as often as there are .tmp files. Log and screenshots (first one when iMatch was running and the second one after iMatch was closed!) attached. 59 such files were created within 6 minutes, from 17:44 to 17:50.

[Carlo Didier]

I just renamed one of the .tmp files to .jpg and could open it in a viewer. Definetely one of my images from the database!

[Carlo Didier]

I have now checked with Process Monitor and it clearly is iMatch who creates those files!

See attachment

[Mario]

The lgo file shows no images being processed by IMatch (nothing added)?
Which files did you process? RAW files? If so, was the WIC codec used or did IMatch have to fall back to LibRaw?

What happens when you close IMatch? Are these temporary files deleted?
Again, the "pli" name is used and ".tmp". IMatch does not create temporary files with the ".tmp" extension. It always uses .imt, so this must be caused by some external component, driver, library or whatever.
Since this was never reported before and relocating does not update any files on disk, I have no idea what could cause this.

[Carlo Didier]

As I already said, the files are not removed when I close iMatch (see last screenshot of the temp folder).
Also, Process Monitor clearly shows that iMatch2020x64.exe creates those files. Example:

Code Select Expand

"18:18:36,0617443","IMatch2020x64.exe","23496","IRP_MJ_CLOSE","C:\Users\Carlo\AppData\Local\Temp\pliB014.tmp","SUCCESS",""
[b]"18:18:36,0621911","IMatch2020x64.exe","23496","CreateFile","C:\Users\Carlo\AppData\Local\Temp\pliB014.tmp"[/b],"SUCCESS","Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened"
"18:18:36,0622767","IMatch2020x64.exe","23496","QueryStandardInformationFile","C:\Users\Carlo\AppData\Local\Temp\pliB014.tmp","SUCCESS","AllocationSize: 72 998 912, EndOfFile: 36 500 022, NumberOfLinks: 1, DeletePending: False, Directory: False"


There are also "buffer overflows" from MsMpEng.exe and from exiftool.exe:

Code Select Expand

"18:18:36,0674803","MsMpEng.exe","6564","CreateFile","C:\Users\Carlo\AppData\Local\Temp\pliB014.tmp","SUCCESS","Desired Access: Read Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Open For Backup, Open No Recall, Disallow Exclusive, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"18:18:36,0675443","MsMpEng.exe","6564","QueryInformationVolume","C:\Users\Carlo\AppData\Local\Temp\pliB014.tmp","BUFFER OVERFLOW","VolumeCreationTime: 06.12.18 21:10:09, VolumeSerialNumber: 72B2-83CF, SupportsObjects: True, VolumeLabel: SysŔ"
"18:18:36,0675656","MsMpEng.exe","6564","QueryAllInformationFile","C:\Users\Carlo\AppData\Local\Temp\pliB014.tmp","BUFFER OVERFLOW","CreationTime: 29.10.20 18:18:35, LastAccessTime: 29.10.20 18:18:36, LastWriteTime: 29.10.20 18:18:36, ChangeTime: 29.10.20 18:18:36, FileAttributes: AT, AllocationSize: 72 998 912, EndOfFile: 36 500 022, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x9400000000b6f8, EaSize: 0, Access: Read Attributes, Synchronize, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Word"
"18:18:36,0675973","MsMpEng.exe","6564","QueryInformationVolume","C:\Users\Carlo\AppData\Local\Temp\pliB014.tmp","BUFFER OVERFLOW","VolumeCreationTime: 06.12.18 21:10:09, VolumeSerialNumber: 72B2-83CF, SupportsObjects: True, VolumeLabel: SysŔ"
"18:18:36,0676125","MsMpEng.exe","6564","QueryAllInformationFile","C:\Users\Carlo\AppData\Local\Temp\pliB014.tmp","BUFFER OVERFLOW","CreationTime: 29.10.20 18:18:35, LastAccessTime: 29.10.20 18:18:36, LastWriteTime: 29.10.20 18:18:36, ChangeTime: 29.10.20 18:18:36, FileAttributes: AT, AllocationSize: 72 998 912, EndOfFile: 36 500 022, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x9400000000b6f8, EaSize: 0, Access: Read Attributes, Synchronize, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Word"


Code Select Expand

"18:18:18,9750896","exiftool.exe","29584","QueryInformationVolume","C:\Users\Carlo\AppData\Local\Temp\par-4361726c6f\cache-f9e591c36f8d569e675d681b21bf6c7ead9498fc\6b9bcbc1.dll","BUFFER OVERFLOW","VolumeCreationTime: 06.12.18 21:10:09, VolumeSerialNumber: 72B2-83CF, SupportsObjects: True, VolumeLabel: Sysã"
"18:18:18,9751028","exiftool.exe","29584","QueryAllInformationFile","C:\Users\Carlo\AppData\Local\Temp\par-4361726c6f\cache-f9e591c36f8d569e675d681b21bf6c7ead9498fc\6b9bcbc1.dll","BUFFER OVERFLOW","CreationTime: 27.10.20 13:18:18, LastAccessTime: 27.10.20 13:18:18, LastWriteTime: 27.10.20 13:18:18, ChangeTime: 27.10.20 13:18:18, FileAttributes: A, AllocationSize: 49 152, EndOfFile: 48 640, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x3e000000000929, EaSize: 0, Access: Read Attributes, Synchronize, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Word"


[Mario]

The MsMpEng.exe (Microsoft Malware Protection Engine) is part of Windows Defender, not IMatch, Buffer overflows in this context are normal.

This may be an important line:

"18:18:36,0674803","MsMpEng.exe","6564","CreateFile","C:\Users\Carlo\AppData\Local\Temp\pliB014.tmp"

CreateFile can mean both create file and open file. Maybe Windows Defender is kicking in and prevent whatever in IMatch is creating these temporary files from deleting them? No idea.
What did you do, exactly, when this was logged? Which kind of file were you processing? What are your settings (See my questions above). Maybe LibRaw is creating these files. Or a WIC codec.
IMatch does not use CreateTempFile anywhere without the .imt extension. And "pli" does not ring a bell, either.

[Carlo Didier]

Could be that Defender tries to scan the files, but it is definitely iMatch that's either creating or at least accessing them, and that folder isn't indexed by iMatch, so why would it access them if it doesn't create them?

"18:18:36,0621911","IMatch2020x64.exe","23496","CreateFile","C:\Users\Carlo\AppData\Local\Temp\pliB014.tmp","SUCCESS","Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened"

[Carlo Didier]

As to what type of files, I renamed one again to .jpg to look at it and it is either a .arw from Sony A700 DSLR or a converted DNG from the same image.

[Mario]

In that call, IMatch is only opening the file for reading. It does not create the file ("Desired Access: Generic Read).
Not sure why. IMatch explicitly excludes the Windows TEMP folder from monitoring, but maybe you have indexed files from a folder above and this causes the shell event handler for file system notifications to kick in or something. As asked above, if you process RAW files, do you use the WIC codecs or did you switch to LibRaw? Or maybe the codecs failed and LibRaw was used as a fallback...?

When IMatch uses LibRaw and LR cannot deliver a preview, IMatch makes it produce a temporary TIFF file from the developed RAW. These files are stored in the TEMP folder, but they use a numerical file name and the .imt extension. Not "pli" or similar.

Looking at a JPEG will never create a temp file, IMatch does not use or create cache files for JPEG.

[Carlo Didier]

See my last reply. One of the temp files is a jpg generated from either a .arw or .dng file.
Nothing indexed on C:\Users by iMatch.
How do I know what iMatch uses for any given file?

[Carlo Didier]

If I rename to .tif, it also displays in FastPictureViewer, so it could effectively be a tif file, which would explain the file size.

[Mario]

Most image viewer don't look at the extension, but identify the file format from data in the actual file. Hence renaming it to TIFF, PNG or JPEG fill make no difference.
If LibRaw fails to extract a preview, IMatch lets it produce a temporary tiff from the RAW as the last try. The naming scheme used is <number>.imt. Not "pli....tmp" something. This would be visible in the IMatch log file.

And this happens only when you relocate? Why does IMatch processing images after a relocate?
Or does this now happen also when you just process images? And has never been happened before? Only after you switch the hard disk?
Did you reboot the system once, just in case?

[Carlo Didier]

And this happens only when you relocate? Why does IMatch processing images after a relocate?
Or does this now happen also when you just process images? And has never been happened before? Only after you switch the hard disk?

No processing of images at all, just re-location. Those are old images which have been in the DB for many years.
Reboot did change nothing (first thing I did when it had filled up the disk the first time, plus several times afterwards).
It's very, very strange.

[Mario]

I have relocated a number of folders, and an entire disk. Not a single "pli*.tmp" file was created. Even when I setup a relocate to a partial folder, where IMatch had do rescan/add new RAW, JPEG and other files afterwards.

[afengler]

Just as a test I searched for pli*tmp files on my system and I found 951 objects, most of them with a size of 0 (see partial list attached). I am not sure which application created these files but most were created quite recently. The oldest was dated 8/21/2020. I have never relocated my image files. I am running Win 10 which of course has been updated many times since that date. Is this perhaps a Windows Defender issue?

Alf

[Mario]

I have searched all disks on two computers, one of them my main development workstation for IMatch. Not a single pli*.tmp file. But I clean the TEMP folder occasionally.

[sinus]

Also here:

no single pli in my temp-folder.
Hm, curious.

[Mario]

Thanks for checking.

This is a rather mysterious case. I guess this is caused by something external...not sure.