Is my import overloading iMatch

[philburton]

I seem to have overloaded iMatch, because it seems stalled out on an import.  This import started at 1500 hours local time, and it is now 2140 hours local time (GMT-7)  I plan to leave my system on all night.

I'm running the trial version of iMatch, which I downloaded and installed 3 days ago.  I'm still "experimenting."  Following the advice I got in some of my recent posts, I set up a Database called "AllDrives_C-F." I have four partitions on my system, plus disk-to-disk backup. Except for E, all partitions are on a very fast NVMe drive.  E is on a spinnng HDD, but right now, I'm backing up the C drive a few folders at a time, per a suggestion in a post.

Right now, I'm backing up just c:\Program Files.  According to a Treesize utility I use, Program Files is 29.8 GB, with 113K files spread out over 12K folders.  Is this import operation too much?

The Progress Window started out with import taking about 1 hour, 15 minutes.  Last time I could bring up this window, perhaps 25-30% of the import was done but now remaining time is up to about 5 and 1/2 hours, and apparently stalled.

The logfile is 227 KB.  Should I upload that file?  Are there any entries that I should look for that would indicate performance issues?

Should I just kill the process, with Process Manager?  What happens to the database entries already imported.  The attachment shows a snapshot of part of Task Manager.



My desktop is home-built, (forgive the hardware geek stuff) based on a very fast ASUS X670E chipset motherboard, AMD 7900x CPU, 64 GB of RAM and an NVidia 3060 Ti GPU.

[mopperle]

Are you really importing ALL files from ALL drives of your system???

[Mario]

Are you really importing ALL files from ALL drives of your system???

That would be my question. How many files do you have on your four disks?

The typical way to use a DAM is to be selective. Like, adding c:\images\  and other folders containing the files you want to manage in IMatch. Don't add entire drives, unless the driver only contains files you want to manage in IMatch.

You don't let your DAM index the entire C: disk. This would be real bad.
C: usually contains the Windows installation and other folders which can make up several hundred thousand files (without any use for DAM). Adding all these files to IMatch will do nothing useful, but burning energy for a very long time.

Also, since IMatch monitors the folders you add to your database and Windows changes things in folders on the C: disk all the time. IMatch will have to resacan folders constantly, several times per second. Which is bad for performance.

In which phase is IMatch? Still adding folders and filers to the database? How many already?
Or has it already added everything and is now processing data?

Always include the ZIPped copy of the log file (see log file) with your posts in such cases.

For proper operation of IMatch, see Indexing Folders and Files which contains detailed information and usage tips.

it also recommends not to add all files in one go, but in batches of, say, 20,000 files or so.
For better control over the process. It is perfectly normal that IMatch will detect many files with problems during the initial ingest, like fails failing to load, files with corrupted metadata and so on. IMatch cares for such things, and will report them. If you tell it to process 500,000 files in one go, prepare for a long wait.

Learning by doing is a good approach.
Start with a database and index some of your images, say 20,000 files. Work with that for a day to get a feel for how IMatch works. Check for problems IMatch has detected in your files (The IMatch Dashboard), deal with them. Then add the next batch of files.

[philburton]

Are you really importing ALL files from ALL drives of your system???

Yes.  This is an experiment.  The takeaway is that for my C: drive I really need to import only the \users directory, at least for now.

Right now I use an Access DB file to track all my software installs, by system, date, install notes, licenses, and serial numbers.  Works pretty well, except for all the software tha auto-updates.  Access is not really designed for casual end-users like me.  For a FUTURE application I will want to replace that Access DB with an iMatch database, and also link installed programs to my F drive, which contains downloads of software installs.  But again, this is a future idea, probably for sometime in 2025.

[David_H]

Are you really importing ALL files from ALL drives of your system???

Yes.  This is an experiment.  The takeaway is that for my C: drive I really need to import only the \users directory, at least for now.

Right now I use an Access DB file to track all my software installs, by system, date, install notes, licenses, and serial numbers.  Works pretty well, except for all the software tha auto-updates.  Access is not really designed for casual end-users like me.  For a FUTURE application I will want to replace that Access DB with an iMatch database, and also link installed programs to my F drive, which contains downloads of software installs.  But again, this is a future idea, probably for sometime in 2025.

Erm. IMatch is designed for managing media assets - image files mostly, mp3, movies, pdfs, etc, all of which are usually self-contained in one or two locations (c:\images). Not software assets. Adding the entire \users directory will not end well.

[mopperle]

@philburton I agree with David and it seems you totally misunderstood the purpose of IMatch.
It is not an IT system management tool. It is not a tool to manage you installed software.
Trying to import a whole drive or even the user directory only is useless. It makes zero sense to have some hundred thousand files (binaries, DLLs, cfg log, dmp,......) in a database. There is nothing to manage. Those files are not Digital Assets. A DAM solution like IMatch is meant to manage pictures, videos, and similar assets. Read this:
https://en.wikipedia.org/wiki/Digital_asset
So best is when you stick to your Access database.
But anyway, I am out here now.

[thrinn]

Erm. IMatch is designed for managing media assets - image files mostly, mp3, movies, pdfs, etc, all of which are usually self-contained in one or two locations (c:\images). Not software assets. Adding the entire \users directory will not end well.

I agree.
@philburton: If you want you manage your software installations, IMatch is not the right tool. You would need some kind of specialized Software Inventory program. Just two reasons:

• A "Software" is not one file on disk. Take IMatch itself as example: You might see IMatch as one "Software Asset" - but just the program directory contains about 180 files. IMatch would interpret each file as a separate "asset". Which kind of information do you want to link to each of these files?
• "Software installation" on Windows is much more than copying some files into some directory. Just think about registry entries, shared libraries etc. IMatch will have no knowledge about these, so what would be the benefit of having the files in the Program Files directory listed in your DB?

[philburton]

@philburton I agree with David and it seems you totally misunderstood the purpose of IMatch.
It is not an IT system management tool. It is not a tool to manage you installed software.
Trying to import a whole drive or even the user directory only is useless. It makes zero sense to have some hundred thousand files (binaries, DLLs, cfg log, dmp,......) in a database. There is nothing to manage. Those files are not Digital Assets. A DAM solution like IMatch is meant to manage pictures, videos, and similar assets. Read this:
https://en.wikipedia.org/wiki/Digital_asset
So best is when you stick to your Access database.
But anyway, I am out here now.

Thanks for the reply.  Consider that I am striving to learn about the best uses for iMatch, its strengths and weaknesses, and how I can use iMatch to augment my overall workflow based on Lightroom (and sometimes Photoshop).  So my post was a sort of "thought expriment" and I got very useful feedback.  I'm no longer going to try for force this software management application into iMatch.

[philburton]

Erm. IMatch is designed for managing media assets - image files mostly, mp3, movies, pdfs, etc, all of which are usually self-contained in one or two locations (c:\images). Not software assets. Adding the entire \users directory will not end well.

I agree.
@philburton: If you want you manage your software installations, IMatch is not the right tool. You would need some kind of specialized Software Inventory program.

The reason I am using Access is that the typical software inventory program is designed for enterprises, and would be totally inappropriate for me, not to ignore the pricing. The free version of LANSweeper seems too limited for me.  (However, I am NOT trying to "scope creep" this thread by discussing software management tools. 

[6b6561]

This thread should probably be moved to off topic and maybe subject changed.

I'm not really 100% sure of what you are trying to achieve, but I guess you want to keep an eye on what's installed on your PC and the consistency of it. I myself couldn't care less of the consistency of the installed software nor the operating system, what I care about is the consistency of my install media and data files. Therefore I run regular checksum verification on all installation files and data files. I run these checks using AIDE on Linux.

The closest thing to AIDE on windows that I could quickly find is https://github.com/TechieGuy12/FileVerification. The idea with this program is that you run it in a directory such as "C:\Program Files" and it creates a simple text file with all the checksums for the files in the directory and it's subdirectories, subsequent runs should list and differences.

Software inventory is rarely done on the level of the individual files, most inventory tools are just scanning for .exe files and what's registered as installed by windows.

If I would want a list of installed software, then I would probably run something like the PowerShell list from https://www.makeuseof.com/tag/list-installed-programs-windows/

It's a bit hard to provide better suggestions as it's not really clear what your real need is.

[Mario]

IMatch is, like all DAM software, designed to manage digital assets like images, videos, audio files, PDF and Office documents and similar. It does a lot more DAM than Lr. But it is not a software inventory system and should not be used as such.

Since IMatch only manages "known" files (see Edit > Preferences > File Formats) and File Formats in the help, it does not index .exe or .dll or other executable and arbitrary files which are part of a software installation.