Heavy problems with performance, do not know, what IMatch works all the time

[sinus]

Hi
In the last time I have massive problems with the performance from IMatch.
I think, it was after the last (or second last) update.
I have attached a log. I have only startet IMatch, searched some pics and then opend them and worked in Photoshop.

I do this since years, always the same. But IMatch begins suddenly to work, I hear the harddisk and if I look into the log, IMatch does something with very old pictures, what I have not touched a long time (pictures from 2003, 2004 ...2014, 2015 ...)
Then I can almost not work, IMatch does respond that slow.

Of course I have a lot of files (300'000) but it was alway find with the speed. And now it is not a bit slower, it is a really lot slower.
And I wonder, what the heck IMatch does.

I wonder, if it has something to do with the relations, what have change lately (see attachement).
Because if I turn off the automatic the automatic relation (see attachement) then IMatch seems to work normal again, but then I have no more automatic detection of versions.
And the relation - preferences I use also since a long time, only enabeld nefs. (see attachement).

Diagnosis says, the DB is ok and healthy.

Hmm, curious, such a behiaviour is really new for me with IMatch.
I checked the virus-checker (what I have not changes), the DB is set to not check, I had this also since a long time.

Maybe the log give a hint.

[Mario]

But IMatch begins suddenly to work, I hear the harddisk and if I look into the log, IMatch does something with very old pictures,

Please define "something". What is IMatch doing, exactly?
Re-indexing the files? Rescanning the files because their "last modified" timestamp has actually changed?
Writing metadata back?

Do you have automatic write-back enabled?

Updating buddy relations requires disk access, because buddy files may not be in the database.
Version relations are processed only in the database, no scanning files or anything.

It would help to see a log file from an IMatch session where this happened. So we can see the something IMatch is doing.

[sinus]

But IMatch begins suddenly to work, I hear the harddisk and if I look into the log, IMatch does something with very old pictures,

Please define "something". What is IMatch doing, exactly?
Re-indexing the files? Rescanning the files because their "last modified" timestamp has actually changed?
Writing metadata back?

Do you have automatic write-back enabled?

Updating buddy relations requires disk access, because buddy files may not be in the database.
Version relations are processed only in the database, no scanning files or anything.

It would help to see a log file from an IMatch session where this happened. So we can see the something IMatch is doing.

Thanks Mario,
I have attached a log (a big one) in the post above, even with debug modus 

Automatic Write back is off, I do this manual.
I do not no, what IMatch does, I hope, you can see this in the log.
In the info-panel there is mostly written "Reading Metadata, 0 of 10 files" ... or maybe 0 of 15 files.
What is normaly done in seconds, not this take minutes and minute, 5, 10 longer ..

I hope, you can see the log in my first post.

[Mario]

Sorry. Missed the log file attachment between all the images.

The log covers 8 minutes.
The only "slow" operations reported are two write-backs to 10 files each, which take 10 seconds each (1s per file).

IMatch is checking tons of files because folders like "F:\Archiv-SINUS-AELTER-2015\1900-1999\" have either been modified and hence their "last modified" timetamp is newer than stored in your database or Windows is sending "folder updated" events to IMatch for these folders.
IMatch has to check each file in these folders to determine if the file has changed, if files have been added or removed.

IMatch had to check 270,000 files (!) for updates. Which is of course a massive amount of work, especially since Windows's "give the the base data for this file" call is quite slow. But IMatch needs the "last modified" date and some file system attributes to know what happened to the file.

If this repeats over and over, something is modifying the folders, causing the timestamp to change. Or causing Windows to send "folder modified" messages to IMatch.
It seems to start when IMatch notices that F:\ has changed (date different). Do you have the root folder on drive F: included in your database? If so, IMatch has no chance as to check each sub-folder and file "below" F:\ for changes.

[sinus]

Hey, Mario

You made me less nervous!!! 
Thanks a lot for looking into this so quickly! Really thanks.

Hmm, what you write makes sense.
If IMatch does not some harm, then it is good. After all the diagnosis is also fine.

Hmmm, lately I added a folder from C (what is very seldom) and the remove it again.
This could be a hint for me.

Mario, I think, I have to look tomorrow carefully, what you have written.

Maybe this is really a very good hint of you!
Puhh, it made me simply nervous, but this makes sense for me.

I will check this tomorrow (hmm, if I look longer now, not me are nervous but my wife  )

Thanks for now!

[sinus]

It seems to start when IMatch notices that F:\ has changed (date different). Do you have the root folder on drive F: included in your database? If so, IMatch has no chance as to check each sub-folder and file "below" F:\ for changes.

Maybe this could be the problem, but I am not sure, how this looked before.

What I am sure, I did not do anything with drive F (my main drive), nor inside IMatch nor outside.
What I did, I added lately for testing some folders from drive C and removed them lately again.

Could this be the reason, that IMatch, for some reason, changed the folders on F?

I made a test with some folders with a new DB, and there I cannot see the Folder F:\ in the DB, only the line "ZUKI_HOTSWAP_BARRACUDA (F:)

But now, with the slow DB I have the Folder-entry like in my attachement here.
I wonder, if the line F:\ is wrong, how can I get rid of it?

Finally I think, your idea with this (root folder included) could be really the the right idea.

Maybe "remove" the folder F:\ without subdirectories?

BTW, on my folder F are only files, what are in the DB, see attachement.

[Mario]

When you index a root folder in your database, every change on that disk will cause IMatch to rescan all folders indexed.

Make a backup copy of the database.
Reopen, select F:\ and use the "Remove from Database" command. IMatch will ask if you want to keep the sub-folders of F:\. Answer with Yes.

[sinus]

When you index a root folder in your database, every change on that disk will cause IMatch to rescan all folders indexed.

Make a backup copy of the database.
Reopen, select F:\ and use the "Remove from Database" command. IMatch will ask if you want to keep the sub-folders of F:\. Answer with Yes.

Mario, vielen Dank!
I think, this did it!
I seems, that IMatch works again normal, like before - yep, and diagnostics shows also no error!

Puhhh, super. I wonder, why this happend, but as mostly, has been almost for sure a user - error. 

You have simply an outstanding support, really! 

[sinus]

Mario, just for your info:

After successfully removed the F:\ like you told

Reopen, select F:\ and use the "Remove from Database" command. IMatch will ask if you want to keep the sub-folders of F:\. Answer with Yes.

it was away, no problem at all ... and now, don't ask me why, this is again here.

But at the moment I have no more problems, all seems to be fine though the F:\ - folder is again here (see in the attachements)

But nevertheless, just now seems to be all fine. 

[Mario]

How did you add the root of F: again to the database?
Which operations did you perform?

IMatch usually works "down" from the folders you index. It usually does not add other folders or parent folders of the folders you have indexed. At least I would not know which command or operation would do that.
Since root folders can only hold a limited number of files/folders (they are special) it is usually wise to create a folder like "images" or "data" in the root folder and then put all other folders into that folder. This way the root stays clean and manageable. And you index the "data" / "images" / "sinus" folder in IMatch. Not the root.

[sinus]

How did you add the root of F: again to the database?
Which operations did you perform?

IMatch usually works "down" from the folders you index. It usually does not add other folders or parent folders of the folders you have indexed. At least I would not know which command or operation would do that.
Since root folders can only hold a limited number of files/folders (they are special) it is usually wise to create a folder like "images" or "data" in the root folder and then put all other folders into that folder. This way the root stays clean and manageable. And you index the "data" / "images" / "sinus" folder in IMatch. Not the root.

I cannot remember, that I did some special operations except copy images.

I have now removed F:\ again and I will look more carefully, if this F:\ comes back again.

But what I do not understand, what you have written about root.
I have 4 folders in the drive F, nothing else.
And these 4 folders I have indexed in IMatch (attachement).

You mean, better create a new folder on F (inside IMatch) and move these 4 folder into this new folder, that I have at the end only 1 folder with a lot of subfolders?

[Mario]

4 folders in the root is perfectly OK.
Let me know what you did when IMatch re-indexed F:\ again (KEEP THE LOG!!!! file and keep IMatch running in Debug Logging while you test).

[sinus]

4 folders in the root is perfectly OK.
Let me know what you did when IMatch re-indexed F:\ again (KEEP THE LOG!!!! file and keep IMatch running in Debug Logging while you test).

Thanks, Mario
I will do so.

[sinus]

4 folders in the root is perfectly OK.
Let me know what you did when IMatch re-indexed F:\ again (KEEP THE LOG!!!! file and keep IMatch running in Debug Logging while you test).

Mario,
IMatch did it again, but this time I know, I think, why, and I have the debug-log.
Basically I opend IMatch, went to a fresh important file.

At this time all looks fine. Attachement 1.

Then I did rename this file with the renamer, like I did this the whole year.
I use this since many years, I do only at the beginning of a year change the pathes for Move/Copy.
But this here works no without changes since January. See attachement 2.
And if you are interested, the line in the MOVE-section is

Code Select Expand

F:\Archiv-SINUS-AKTUELL\00_timeline-aktuell\2019-2021\20_timeline-2020\1_archiv-bild-2020\b-{File.DateTime|format:YYYY-MM-MMM}

Then I went to the file-pane.
It was now closed, what is very unusual, see attachement 3.

And when I clicked now on the + to open the path, it F:\ was again here, see Attachement 4.

But as you can see in attachement 5, the file has correct changed the name and moved to the wished folder.

I attach also the debug-log.
It is this time not long, because I have only started IMatch new and did this "operation".

Hopefully you can find the problem.
Thanks for looking into it --- hmmm, I can send only 5 attachement, I will send the log just with a new post in a minute.

[sinus]

Here is the debug-log for the post just above.

[Mario]

Strange. This log file does not contain any trace of IMatch adding or updating a folder.
The file ...21\20_timeline-2020\1_archiv-bild is only logged once, when IMatch tries to find the icon of the associated application.

So, in short what you did is:

You used the Renamer to rename a file from one of the folders you have indexed from F: and moved it to another folder on F:
Did that folder exist before or was it created by the Renamer?

[sinus]

So, in short what you did is:

You used the Renamer to rename a file from one of the folders you have indexed from F: and moved it to another folder on F:
Did that folder exist before or was it created by the Renamer?

This is correct.
I used the Renamer to rename and moved it.
No folder was new created, the folder, where the file was moved, have already existed.

[sinus]

Mario,
MAYBE I have found the reason - I tried and looked into this, because it is not nice to work with this.

I have several different renamer-templates since years, all with almost the same preferences.
And they worked and worked ...since lately (for sure the last or the second last version).

My renamers does have a move-preset, see in the attachement.
in the entry "add created folders to database" I have "Yes, all" , because the folders are already in the DB.
Made sense for me and as I told, it worked for years.

Now I have the error (the new folder F:\) like reported.

If I now change this entry to "bottom level only", it seems to work again.
No F:\ does appear.

I tried not the entry "No", because that is too dangerous for me, not that IMatch would remove the already existing folder. 

Makes this sense for you, do you think, this could be the only reason?
If yes, I would change all my renamer-preferences to this.

Hope, this helps.

[sinus]

And here, after the update I rechecked this.

I am quite sure, that this is the source of my problem. I do not know, if this is a problem for others, but I will change the explained preferences in all of my renamer-presets, where move is involved.

Because I did not change this for ages, it must be a changing somewhere else during the last versions.

But for me I think (and hope) the problem is solved. 

[Mario]

"Bottom" means "only add the folder which is used as the target".
"All" means to add the entire path to the database, starting at the root folder.
This explains why F:\ was added. Not sure if this is correct...need to check that. Usually only the entire folder hierarchy should be added as "one" folder, not each segment...Strange. Need to try that.

The default for this option is "no", because usually you use the BP to export files "out" of the database and you don't want the Batch processed files become part of your database.

[Mario]

I've tried to reproduce this using a dedicated drive with this layout:

F:\
|- images
|- images2
   |- output

Only images2 (and sub-folders) was indexed, the folder images in the root was not.
From another folder I did a batch process and creates 5 new images in the images2\output folder. I used the "All levels" option.
The new files were added to the database. No other folders and neither the root folder were added. OK.

Now I repeated this with other images, this time creating a new folder in the output, under "images".
The new folders are added to the database. No other folders were added. OK.

Repeat again, this time outputting to the new folder F:\output. This folder was added, no other folders or the root were added. OK.

Forced a rescan of all top-level folders. Root or other folders not in the database not added. OK.

Closed and reopened the database. No change. OK

In Windows Explorer, copied an image to F:\. No change to the database (F:\ is not indexed). OK.

In Windows Explorer, copied an image to F:\images. No change to the database (F:\images is not indexed). OK.

I did this twice, once with advanced folder monitoring on and once when this option was off. Same result.

Looks OK to me. If you have steps to reproduce this behavior also on my computer, let me know.

[sinus]

Thanks, Mario, to look into this again.

Hmmm, then maybe I did something wrong.
If I see this again, I would report it again.

I would say: solved ... still in the old year!