impossible to reopen database after interrupted metadata writeback

[kkiel]

Hi everybody,

I didn't find an answer for this case in the forum:

I managed to crash IMatch5 running a metadata writeback (6000 files, on 'DB 1'), but had to interrupt it when it was almost ready. When trying to reopen the database I got the message: "loading thesaurus data and tag metadata", I waited for several hours with no success.
When I tried to open another db ('DB 2', not affected from the crash or loop) i got the same message for more than an hour.
I then deinstalled and reinstalled IMatch.
After trying to create a completely new database ('DB 3') I got the message "creating metabase, generating and importing tag lists. This will take 2 minutes", which I interrupted after an hour. So I have not been able to reopen the program and look at the medadata settings again (hierarchical keywords- or exiftool-preferences)
When I try to open the first corrupted database I (DB 1), i again get the message "loading thesaurus data and tag metadata" for several hours.
So at the moment even with a freshly installed Imatch I can't open any database.
What can I do?
I attach a log file (of course ...)

kkiel


[attachment deleted by admin]

[Mario]

How did you "interrupt" the write-back process? By closing IMatch normally or by shutting down the process in Task Manager?

The IMATCH5_LOG-1.TXT shows a normal open and close of IMatch. No database is loaded.

The IMATCH5_LOG_BACKUP-2.TXT starts with a panic message because almost 90% of the system memory are already used.
This log file covers only 27 seconds of runtime. A database is loaded, looks normal.
IMatch then tries to launch ExifTool, and this is where the log file ends.

Did you try to reboot your system?

[kkiel]

I interrupted the process by klicking the 'cancel'-button, when nothing happened for half an hour I shut the system down. I needed to take the computer with me and I was accustomed to IMatch reacting kindly to this kind of treatment ...

I now rebooted (Warmstart), tried to create a new database again, only 60 percent of RAM is used, and it still hangs trying to "creating database, generating and importing tag lists. This process takes about 2 minutes". Taskmanager shows inactivity, but thats not always true.

Should I let it work over night?

Or I could try to install IMatch on another computer (all 4 RAM, unfortunately), create a new database, get the preferences right, and try to save my database at least.

Thanks for your efforts!

no new logfile yet.

[Mario]

The problem seems to be related to ExifTool. ExifTool seems to always hang on your system, or to be unable to run for other reasons.
This is uncommon and I don't recall a similar case.

Do you have perhaps some security software running which may block ExifTool?
Did you install software recently (except IMatch)?
Did IMatch work before that?

When ExifTool runs for the first time after installing a new ExifTool version, it creates a bunch of files in the Windows TEMP folder. These files are basically the runtime environment ExifTool needs. It unpacks them to the TEMP folder, in a folder starting with "par" followed by your user name. Below that are folders named like "cache-exiftool-9.62".

Perhaps some software on your system prevents ExifTool from creating or using these folders. Try this:

While IMatch is not running, delete the "par-USERNAME" folders.
Then run IMatch again. When IMatch first runs ExifTool (which happens during loading a database) ExifTool re-creates these folders. Maybe this helps. Other things to check are overambitious virus checkers, behavioral analysis systems and similar candidates which may block executables.

[kkiel]

Hi Mario,

thank you very much, that finally helped!

Regards,

kkiel

[Mario]

That's good news 

Never heard about such a problem before. Must be something real specific on your computer, or just a weird one-time problem.

[kkiel]

I hope so! I didn't install anything new. But my c: partition was very full, maybe that was a reason. I cleared it, bought 4 GB more RAM and think that will do!

Schönen Abend,

kkiel

[whitefern]

Hi ~
I can't get my database to load either. I had added more folders to it, categorised them, ran an update of metadata and then optimised. From memory (happened yesterday) it completed that but said not everything was done, and I'm not sure what happened next. I think it just froze. Sorry, I know nothing about how these things work so I don't have the language for this. I found this thread about deleting the Par-Username folder which I did, but that made no difference.  Attached, hopefully, is the log I got yesterday. There was no crash report. When attempting to open the database, it hangs and then says there's a problem and has to close.

I haven't installed anything new, no changes, am not running a competing programme, have 165GB on my C: drive

I just tried opening, offline, with my anti-virus closed down. I got a bit of orange in the progress bar, which wasn't happening before, but then it hung again.

Any help will be very much appreciated. Thank you. Please let me know if there's anything else I could look for that might help.
~ Robin

[attachment deleted by admin]

[Mario]

Hi,

the log file contains no error messages. It ends when IMatch starts loading the catalog from the database. Since the log file was not produced in debug mode the information is only minimal. We need to switch IMatch to debug logging to get more detailed info. Proceed as follows:

1. Start IMatch while holding down the <Ctrl> key on your keyboard. Double-click the shortcut on the desktop or taskbar and immediately press <Ctrl> and hold it.
2. In the dialog box, chose the option to start IMatch 5 without loading a database.
3. Go to Help > Support  and enable the Debug Logging mode.
4. From the Database menu, load your database.
5. Attach the log file resulting from that (you may need to ZIP it because it will be much larger than before).

A debug log file contains more info which may help us to see what's wrong.

[whitefern]

Thanks, Mario. I think I did that. The file isn't too much bigger though. ?? It's attached.

[attachment deleted by admin]

[Mario]

The last entry in the log file shows that IMatch has started loading categories from the database. Then the log file ends.
Everything else up to that point looks perfectly normal, just the performance is a bit slow for a small 1 GB size database.

How long did you wait for IMatch to complete the operation?
When you look at Windows Task Manager, how much CPU utilization is caused by IMatch? Does it use the CPU at all? Or maybe always something close to 25%?
In this phase IMatch is not accessing metadata or anything. Just loading the categories into an internal cache.

Can you upload the database to my FTP server or your cloud space?
Contact me via email (see below) for access details.

[kkiel]

Hi everybody,

I'm opening this thread again because this problem happened to me again when writing metadata back. I found out with a tool named windirstat that IMatch produces preview image files in the following folder:

C:\ProgramData\photools.com\IMatch5\previewcache\D3AD931D-7A9D-429C-ABA7-5E8D6B8F7F50\37\37279.jpg

Each folder with a number like "37" as in this example needs several GB space which leads to a full c: partition and later to a crash of IMatch.

Can I simply delete those jpg-Files? What are they needed for? And can I avoid that they are stored?

Thanks for your help!

kkiel

[Mario]

There are the standard cache files IMatch creates for your database.

Type Caching in the IMatch help index. This leads you to a) The explanation what the cache is, why it is used etc. and b) explains where you can configure the Cache (Edit > Preferences > Cache).

[Richard]

Can I simply delete those jpg-Files?

Pardon me but I am going to be blunt. The question you should be asking is: should I delete those files. If you are running out of storage capacity it is time to make some choices. You can free up a lot of disk space by simply removing all of your image files but that makes little sense. The best alternative is to invest some bucks in more storage capacity. The first hard drive I considered buying cost $850 for 5 MB and would have been the equivalent of 30 floppies . Storage is no longer expensive and for a small  fraction of what that HD would have cost we can buy Terabytes of storage.

[jch2103]

Just to chime in, the log files you posted at the beginning of this thread show a warning [06.16 15:39:01+   63 [1638] 00  W>  # Memory Info: 87% (TP:3363, AP: 0)  (TPF: 404, AVP: 0)  (TV: 7363, AV: 0)]. I'm not an expert in reading IMatch logs, but it looks like you had a severe memory shortage when you started IMatch. Did you have other programs running at the time (e.g., Photoshop)?

Also, the log file appears to show there's about 2 GB free on your temp folder drive [06.16 15:39:01+    0 [1638] 00  I>  Temporary folder (TEMP/TMP): 'C:\Users\Quagga4\AppData\Local\Temp\' with 2.473.345.024 free bytes]. That's not much free space. You could be running into hard disk thrashing and/or, if your temp file is on an SSD, you could be running into additional related lack-of-space issues.

As Richard said, storage is no longer expensive.

John

[Mario]

Yes, in the first log posted the memory utilization was already 85% (85% of the available memory used. Windows reports 3GB in total and 0 MB free memory. And only about 2 GB left on the C: drive.

[jack@vankinsbergen.com]

I am having an issue on my main development system - Win 8.1.  iMatch5 won't convert (hangs for hours) from older DB, won't load already converted databases or create new.   

I have installed iMatch5 successfully on two older, slower,  systems, one win 7 the other win 8.1. They will do all three functions.


I have done the normal - rebooted, uninstall, re-install, etc. See attached sample log files of two of the failures.  It also hangs trying to create a database - see 2nd  log file.

Jack

[attachment deleted by admin]

[Mario]

Both log files stop after IMatch has found the database file in the file system and retrieved it' size.
The next step is to open the file via the built-in database system. There is no trace of this in the log so I assume that this call is blocked.
The call may be block for 30 seconds or so when the database file is already open in another application. But then the call returns and IMatch logs a "access denied". Or another error. Or the file is opened successfully.

If you can start IMatch without a database (hold down <Ctrl> immediately after you double-click the IMatchg shortcut and then choose the corresponding option in the Advanced Startup Menu). Switch to Debug Logging in Help > Support > Debug Logging.

Then repeat your test, maybe we get more info in the log file.

What is D: for a drive type? Did you try to open the database from another disk?

Any virus-checker, behavior monitoring software installed? Or anything which may block IMatch from opening the database file in writable mode? If the database system is blocked in such a way (never seen that before) the chances are that this is caused by an external influence.

[jack@vankinsbergen.com]

Mario:

   Attached is a log with debugging turned on for the failure on my development system.

   This database loads properly on two other systems via the network via mapping to this same D: drive. I copied to another drive just to try it and it failed on my development system just like the D: drive failure.

   I have already tried deleting the PAR-Jack folders with no effect on the problem.

   I have already tried with Norton virus check and firewall disabled.

   I hope this log helps.

Best Regards,

Jack

[attachment deleted by admin]

[Mario]

IMatch fails to launch ExifTool on this system. The last line in the log is

ExifTool: Launching from 'E:\Win 8 - Graphics\iMatch 5\exiftool.exe', using IMatch config file 'E:\Win 8 - Graphics\iMatch 5\imatch_et.config' (E:\Win 8 - Graphics\iMatch 5\imatch_et.config)

but this function call never returns. It seems that either ExifTool is not launching or blocks, or is blocked by some external event.

When a new version of ExifTool is launched for the first time, the Perl runtime environment used by ExifTool produces a folder in the TEMP folder, and extracts all Perl runtime files into it. This runtime is then used for all future calls of the same ExifTool version, until the user clears his TEMP folder. In this case, the Perl folder is re-created automatically.

If any of these is blocked by whatever security software you run, ExifTool will not run. The same is true when some security software conisders IMatch launching the exiftool.exe process as a potential security risk and blocks it. This is what I would check for.

[jack@vankinsbergen.com]

Mario:

     I have been screwing around with the version of Exiftool that was in Par-Jack.  I suddenly realized from the log that it was executing a version of Exiftool that was in E:/Win8 - graphics/iMatch5 instead.  I went in and changed properties to "run as administrator" and it loaded properly.  I was never asked to approve administrator privilege by the way.

    I have done nothing to the installation of iMatch except install it and Exiftool required Administrator privilege and it wasn't set?   

    I expect this is something you probably need to look at in your installation process.

    Thanks for your help.

Best Regards,

Jack

[Mario]

Neither IMatch nor ExifTool need Admin privileges to run or process files. IMatch needs Admin privs during installation, in order to install properly and to register the scripting libraries. After that, IMatch runs without access privileges.

Maybe your non-standard installation location is the cause for the trouble because it may trigger all kinds of Windows security mechanisms. Or your system did not allow the ExifTool process to write the par folder in TEMP for whatever reason.

There is nothing I could change in the installation routine. Firstly, it's all done by Windows installer anyway. And if the user installs IMatch not in the default Windows program files folder (which is treated specially by Windows) but installs software in other folders, he takes over responsibility to ensure that access privileges and other settings are set properly. I cannot control this, and IMatch has no way to prevent the user from installing software outside the standard program files folder.

[jack@vankinsbergen.com]

Mario.

        The Par folder is fine.

        You could be correct on the non standard location, but I never install anything (that gives me the choice) in the default location. I have been doing that ever since Win 3.1. I have never run into such a problem before that I recall. iMatch5 also is installed in non standard folders on the other two systems that ran without any EXIFtool.exe issue.

        My posts have been part of a thread with similar problems (at least externally). Maybe, you might keep my situation in mind if this issue pops up again.  That is - Exiftool won't execute and "non standard" folder install.  Giving Exiftool.exe admin privileges doesn't bother me at all.

       Anyway, I am happy as I am past the problem.

Best Regards,

Jack

   

[jack@vankinsbergen.com]

Mario:

   As You know I have been having EXIFTool problems.  Let me review:
     1. iMatch5 runs fine on a Win7 and a Win8 system that I have for other purposes.
     2. iMatch5 would hang during load of EXIFTool on my main development system.
     3. I Gave EXIFTool Admin Privilege and it loaded and seemed to function.
     4. I discovered that new images were not displaying EXIF data, but those converted from 3.6 did display EXIF data.
     5. I am interested in finding a script that will write Categories into IPTC keywords, and found one from Ferdinand.
     6. It hangs trying to execute EXIFTool.
     7. I copied EXIFTool to "c:\Program Files" and pointed  the script at the copy and it worked for my test data.
     8. I can find no way to point iMatch5 at the copied EXIFTool.
     9. So of necessity (and unhappily) I uninstalled iMatch5 from "E:\Win 8\Graphics\iMatch5".
    10. I installed in "C:\Program Files" and did not alter Admin privilege.
    11. I now have EXIF data for new images after reloading Metadata for my test files.
    11. The script seems to run as well.

  You did suggest early on that installing in  "E:\Win 8\Graphics\iMatch5"  could be a problem.  I just didn't want to believe that in this day and age. It clearly is a selective problem with EXIFTool itself not iMatch5, I suspect.  Unless there is some error in passed parms.

  I really like the idea of keeping my applications on a separate SSD from the system and in an organized set of folders. 

  Also, the idea that it works fine on two other systems.  I might point out, by the way, that it coexists on the C: drive albeit in a separate directory on the alternate Win8 system, but is in the Program files directory on the alternate Win8 system.

  Is there any way to point iMatch5 to an EXIFTool installed in "C:\Program Files" without having to install all of iMatch5 there? Or maybe fix EXIFTool in the first place?

  I think that we now have the core of the problem.  I will see if more comprehensive testing surfaces anything else with EXIFTool.

  Thanks for your help.

Best Regards,

Jack

[Mario]

See my reply to your similar comment in the other thread.