IMatch Consuming Inordinate Amounts of Hard Drive Space

Wow · October 22, 2021, 06:17:15 AM

My setup is as follows:
256 GB SSD (C:)
1 TB Hard Drive (D:)
IMatch is installed on C, but the databases are on D. When I perform a scan of a large number of folders, huge amounts of space start to get consumed on the C drive. Today for example it kept consuming so much space I'd have no space left on C. So I'd go and delete some huge programs like Adobe Acrobat (around 3GB) until that got consumed. I'd guess perhaps 15 GB got consumed total. But to put into perspective the IMatch database is only just over 1 GB. It's possible something else was causing the issues, but it sure seems directly correlated to whenever I add large batches of folders to IMatch. Any suggestions?

In case it's relevant the folders I'm scanning are local Google Drive folders.
Thanks

Wow · October 22, 2021, 06:56:15 AM

Follow-Up. So I just started another scan.

Here are screenshot showing the amount of free space I had when I just started a scan, and the amount of free space I have about 10 minutes later with the scan not even close to being done.

1. Why is this happening and where would I find all this space being consumed on C:?
2. Why is there only a dismiss button but not a cancel button? Do I have to force close IMATCH when this happens?

Thanks

ubacher · October 22, 2021, 07:23:29 AM

Just a few thoughts:

Your database should be on C (the fast disk) and your files on D:

Are you generating cache images - could they fill your disk?

What files are consuming the disk?
( A program to display the disk space visually will help to find the offending files.)

Mario will ask for a log file for sure.

mstief · October 22, 2021, 08:08:53 AM

You can use a program like treesize https://www.jam-software.com/treesize_free to get more information about the used data amount.

Mario · October 22, 2021, 08:56:53 AM

You have only 5 GB space left on the C: disk when you start. Which is very, very little.
This then goes down to 300 MB, so IMatch has consumed maybe 4 GB of disk space for the cache. Which is not much.

The only things IMatch consumes disk space for are:

1. The Database. Which is usually a few GB
2. Settings and Stuff, a few MB
3. The Cache, which can become huge.

The cache is by default created on the disk configured in Windows as the Common App Data folder (you can see it by tying in %APPDATA% in the Windows Explorer address bar).

I recommend that you change the cache folder to a folder on your D: disk and move the cache files over from C: in Windows Explorer (while IMatch is not running).
Tip: This dialog shows the name of the cache folder at the bottom.

Then move the database from D: to C: to improve performance. See also the notes here: Database Storage

Note: A system disk with only 5GB free is stretching things a lot. The SSD will have a hard time reallocating blocks when only this little space is left. I recommend you move some data from the SSD to your other disk.
The free Treesize application will show you the folders occupying the most disk space.

sinus · October 22, 2021, 09:24:05 AM

Quote from: Mario on October 22, 2021, 08:56:53 AM
You have only 5 GB space left on the C: disk when you start. Which is very, very to little.

This is true for sure.
I have actually 290 GB free on C.

In the past this was less than 30 GB, and I had often troubles, also with Photoshop and other programs.
Now I have a 500 GB SSD as harddisk C and that is great, never had problems since then.

Jingo · October 22, 2021, 02:14:16 PM

Best investment I made was a 1TB M.2 drive for boot, programs and databases... the thing is super fast, and I always seem to have about 60% in reserve. Highly worth it.. and a bonus - you get a fast 256GB SSD drive for your cache or other frequently accessed files.

thrinn · October 22, 2021, 03:06:27 PM

Quote from: Wow on October 22, 2021, 06:17:15 AM
In case it's relevant the folders I'm scanning are local Google Drive folders.

It may be worthwhile to check if Google Drive downloads your files first into your TEMP directory (e.g. while the download is in progress). If the TEMP directory is located on C:, this might also fill up the drive. For IMatch to scan a file, it must be available locally, so it is possible that Google Drive downloads it in the background.

Mario · October 22, 2021, 03:50:20 PM

I recommend using the free TreeSize (see my link above) to analyze the drive and find out which folders hold the bulk of data.

jch2103 · October 22, 2021, 07:28:43 PM

Thanks for the reminder about TreeSize Free - I'd installed it quite a while ago but forgotten it was there. Very informative app!

Wow · October 22, 2021, 07:45:32 PM

Thanks everyone, I am going to follow everyone's advice. To clarify, I had at least 25 GB on C: when I started, but it all got eaten up. I will move the databases to C: for better performance, and possibly the cache to D.

Mario, the IMatch cache folder (program data/photools) is currently only 5 GB. The IMatch folder in app data is next to nothing. What I am noticing is that the User\AppData\Local\Temp folder is the one that is exploding. It's crazy, I just watched it in real time go from 5GB to 20GB in about 2 minutes! It appears to be throwing 300MB file after 300 MB file in there. However this time it does not seem to be filling up my hard drive as when the folder hits 25 GB or so a ton of files get automatically deleted. So I am guessing that before with only 25GB free I didn't have enough space for it to finish the process and delete the files and to free up space. This time I had about 35 GB free and that appears to have solved the problem by letting the folder build up to whatever it's max .limit is and then it automatically deletes files to free up space again.

A naïve question - I assume I should be backing up the cache too if I move to another machine? Otherwise the database could take a day to recreate the cache, correct?

With regard to peoples comments about getting a larger SSD I usually go with 1TB or more, I got this specific laptop for limited use and made the mistake of thinking I'd rarely use it. Instead it ended up replacing my main laptop. I intend to get a newer machine with serious storage so I've resisted the hassle of trying to upgrade the SSD. This thread is motivating me to check out how much work is required these days to change hard drives without doing a clean install. I just don't want to go through installing from scratch, it's too painful, especially for a machine I won't be using much longer.

Thanks again everyone.

Mario · October 22, 2021, 07:55:24 PM

What are the names of the files in TEMP?

IMatch uses the TEMP file only for short-lived very small files (a few KB) and for temporary cache files (e.g. when caching is set to on-demand or off and IMatch performs face recognition).
But the temporary cache files are deleted after they have been used (face recognition done), and these are JPG files anyway, which will never be 300 MB in size.

Wow · October 22, 2021, 08:00:04 PM

Quote from: Mario on October 22, 2021, 07:55:24 PM
What are the names of the files in TEMP?

IMatch uses the TEMP file only for short-lived very small files (a few KB) and for temporary cache files (e.g. when caching is set to on-demand or off and IMatch performs face recognition).
But the temporary cache files are deleted after they have been used (face recognition done), and these are JPG files anyway, which will never be 300 MB in size.

It only happens when I am adding a folder to IMatch, so if it's not IMatch than I am guessing I am seeing interaction with some other program, and perhaps the other program starts throwing those 300 MB files in there. My first guess would be Google Drive (the desktop program) is somehow interacting with Imatch and is creating those 300 MB files, but that's just a guess.

Mario · October 22, 2021, 08:03:17 PM

If you index files on a Google Drive folder, Google Drive may download the files temporarily and place the bulk data into TEMP?
The temporary cache files IMatch creates (only when caching is set to off or on-demand and the cache files are needed for face recog) have a number as the file name and the extension .jpg.

Wow · October 22, 2021, 08:41:29 PM

I do see some Imatch files like that, if you look at this screenshot you'll also see what I am talking about with regard to the huge files. Strangely they started filling the SSD again instead of auto-deleting, but now that I know where they are I am playing whack a mole deleting them so this scan can finish.

thrinn · October 22, 2021, 08:57:36 PM

I remember an old topic related to pli*.tmp file - see here. We could not find out what these files are.

Wow · October 22, 2021, 09:04:32 PM

Quote from: thrinn on October 22, 2021, 08:57:36 PM
I remember an old topic related to pli*.tmp file - see here. We could not find out what these files are.

Thanks! Well that sure is it. I wonder what that poster and I have in common.

Wow · October 23, 2021, 06:17:17 PM

Thanks mstief and others who recommended TreeSize. It's really useful and just helped me to free up over 10 GB I'd have otherwise never found and I'm just getting started. Something like this should be built into the OS IMO.

Mario · October 23, 2021, 09:31:13 PM

10 GB does not sound much, on a disk with 256 GB.
Not sure how many applications you have installed, but some applications, especially games, can use 20 to 50 to 100 GB storage.

Carlo Didier · October 25, 2021, 08:42:54 AM

Quote from: Wow on October 22, 2021, 09:04:32 PM
Quote from: thrinn on October 22, 2021, 08:57:36 PM
I remember an old topic related to pli*.tmp file - see here. We could not find out what these files are.
Thanks! Well that sure is it. I wonder what that poster and I have in common.

That would be me ...
Good question. One thing, I do not use Google Drive on my PC (only online via the web).
I never found out what happened, and as far as I know, it may happen anytime again.
I have excluded Windows Defender as the culprit, but that's all I could find out.