IMatch is busy

Started by sinus, January 13, 2017, 12:21:21 PM

Previous topic - Next topic

sinus

Hi Mario
Yesterday I updated to the newest version 5.8.2

Since then IM is working very hard. The weel is running and I can almost not click on a thumb, because, it flickers and goes back to the old.
First I thought, ok, IM does some stuff updating, but until now, it is again the same.


I tried then to check, if I have the log correct (debugg logging), but even this I cannot do, because IM flackers after 0.5 Sec or so and the box is vanished.

The same, if I try to do a right click on a thumb, no chance, because the box is away again, before I can choose an entry.

Has this to do with the new ExifTool?

In the info is Idle time 0 and DB time 0 ... also no entries in the queue.

hm, I have never had this, I hope, it stops after some minutes, hours...

I cannot send a lot, because I cannot go to the entry. And I must do some photo-work now ;)


Hm, mein post muss geholfen haben, IM ist jetzt wieder normal.
Wirklich komisch, hatte gestern schon mehrmals gestartet, aber IM war sehr, sehr beschäftigt, mit was auch immer.

Na ja, hoffe, alles ist nun wieder ok, wirklich, puh, das Rad ist verschwunden, scheint wieder alles ok zu sein.
Ansonsten würde ich mich wieder melden ;)

So, nun noch Äpfel im Studio fotografieren .... ;)
Best wishes from Switzerland! :-)
Markus

sinus

Hm, 5 minutes after my update above IMatch was again busy, no clue, what it does.

Now I had a real crash, but, yes, with a dump.

Hope, this helps you ... hm, maybe I should really think about divide my DB into 2 pieces.

http://www.sinusbild.com/holen/im5dbg_5_8_2_d3cdd2ff-13f8-4868-b9c6-a9e7e0931dbc.zip
Best wishes from Switzerland! :-)
Markus

Mario

Your first post did not include a log file so there is no way to tell anything.
The new ExifTool has no impact on that. I have tested it for over a week before shipping.

I will download the DUMP file and maybe I can see something (30% of all cases).
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Mario

Your system ran out of memory.

Your database has 240,000 files and almost 13,000 categories.
I think you should consider reducing the number of categories. 13,000 categories needing to manage almost 250,000 files means stressing things quite a lot.
Have you considered removing data-driven categories you don't need? Or setting them to manual?

IMatch reports that your categories need almost 900 MB of memory. That is a lot.
Together with the size of your database, IMatch needs almost 1.7 GB just to load the database and categories.
After re-calculating thousands of file relations and updating all collections, IMatch needs 2 GB or the 3.5 GB it can use maximal.
Now you start viewing files via DirectX (Quick View Panel, Viewer) and this allocates memory in chunks of 500 MB or more.

IMatch is also rescanning folders with apparently hundreds of files (most files are current).  It finds some files with outdated or missing cache files and re-create the caches.
Suddenly while processing files in the folder \1_archiv-bild-2016\b-2016-10-Okt\ or thereabouts, the memory peaks at 3.5 GB.

Then you are viewing many files in the Quick View Panel or Viewer and more and more memory is allocated until IMatch has exceeded the ~ 3.5 GB it can use (it's a 32 bit application after all).
I see hundreds (?) of "Not enough memory" from WIC / DirectX in the log file...

Finally, the Microsoft Runtime environment aborts IMatch because the memory is full.

Are there maybe some broken files in that folder? Something that causes WIC / IMatch or DirectX to allocate a massive amount of memory?


-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

sinus

Quote from: Mario on January 13, 2017, 03:03:59 PM
Your system ran out of memory.

Your database has 240,000 files and almost 13,000 categories.
I think you should consider reducing the number of categories. 13,000 categories needing to manage almost 250,000 files means stressing things quite a lot.
Have you considered removing data-driven categories you don't need? Or setting them to manual?

Thanks, Mario, for looking into this so fast.
I have enbled that hte data-driven categories will only be updated manually. Finally, I have not that much.

The problem is, most of the categories are keywords.
And if I remove them, then the keys in the images will be removed ... or do I think here wrong?

Quote from: Mario on January 13, 2017, 03:03:59 PM
Are there maybe some broken files in that folder? Something that causes WIC / IMatch or DirectX to allocate a massive amount of memory?

Broken files in the folder shoulden't be there.

I guess, it is the best to split the DB into two or even three.
It would solve such things, I believe, and after all, it should also speed up things.
Hm, I have to think about it.

Thanks for your really fast answer, great support as always!
Best wishes from Switzerland! :-)
Markus

Mario

13,000 different keywords is quite a lot. You cannot reduce the number of categories then.
I have databases with 350,000 and more files and they work great. It's just that I never needed more than 3000 different keywords to describe all 350,000 files.
The combination of more than 10 thousand categories with 250,000 files is that needs 1 GB RAM to handle - just to keep the categories in memory. Categories need to be super-fast so IMatch has to keep them in memory. And the more images you have in your database, the more memory will be needed.

Even then, IMatch uses 1.6 GB when the database is loaded. So there is almost 2 GB of extra RAM it can use. I wonder why processing files in the background and using the Viewer consumes such a lot of memory...
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Mario

PS.: I have plans to reduce the category memory usage for IMatch 6. This should help a lot in such cases. And allow even 20,000 or 30,000 categories for large databases.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

sinus

Quote from: Mario on January 13, 2017, 04:32:31 PM
PS.: I have plans to reduce the category memory usage for IMatch 6. This should help a lot in such cases. And allow even 20,000 or 30,000 categories for large databases.

That would be great of course.
I think, there are some people out there, what does also have a lot of categories.
And I have the impression, that more people does give keywords for files, maybe because it is nowadays simpler and more in the heads of peoples.
Best wishes from Switzerland! :-)
Markus

Mario

I doubt that there are many people out there who have 13,000 different keywords. Usually it's much better to use a limited, controlled vocabulary. This makes finding this much easier, especially if you work for stock pr photo agencies.

I have seen quite a number of 'exploding' keyword thesaurus - but in most cases keywords were abused to store unique per-image data (basically producing one new keyword for each image or a few images). This kind of data is of course much better stored in descriptions, headlines or titles. It helps when you assign keywords from a controlled vocabulary instead of making them up as you go. The keyword workflow is usually one of the most tightly controlled DAM aspects with commercial users.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

sinus

If you have keywords in 3 languages for example, like it is possible in IMatch, then you have very soon a lot of categories.

But of course I agree: not too many keywords is very good.
We have here since ages a controlled vocabulary, but with sometime too many words.

I will look, what I can do to reduce this.
Best wishes from Switzerland! :-)
Markus

Mario

Would you upload your database for me? I would like to try out the improvements I have made for IMatch 6 on as many "live" databases I can get.
There is a trade-off between category performance and how much memory they use. I need to find a way to balance this. I don't want to loose speed but I also want to reduce memory consumption as much as possible.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Mario

I've made a check with the new IMatch 6 category implementation, using your case as an example.,

For a 150,000 files database with 20,000 categories I ran one of my test scripts to make sure that each file has at least 10 keywords (out of about 15,000 different hierarchical keywords with up to 7 levels).
Plus at least 5 of the other categories for each file. Plus data-driven categories.

The results are very encouraging. I don't 'feel' any slow down for category-based features (need to do some profiling to get exact numbers) and the memory consumption is down to maybe 30-50 MB in stead of 500-700 MB!

And this will hold. The larger the database becomes and the more categories (keywords) you have, the more efficient the new implementation will be. Even for my largest database (~400,000 files) the memory consumption is less than 100 MB for the categories. This reduced the overall memory footprint of this database by a whooping 50%. Yeah!

One of the cool new things I have for IMatch 6.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Menace

Quote from: Mario on January 14, 2017, 10:12:14 AM
One of the cool new things I have for IMatch 6.

8) I already can't wait for the beta.

Mario

I have this stress tested up to almost 60,000 (!) categories.
Memory consumption very low. Basically zero compared to the old implementation.

The problem may be the Windows tree control which has not been designed to handle tens of thousands of items.
But that is only the case when you have, say, 20,000 keywords all on one level.
In that case you may need to enable the "grouping" option so @Keyword produces top-level elements based on the first letter of the keyword. This partitions even very large flat keyword sets nicely.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

sinus

Quote from: Menace on January 14, 2017, 11:30:01 AM
Quote from: Mario on January 14, 2017, 10:12:14 AM
One of the cool new things I have for IMatch 6.

8) I already can't wait for the beta.

Nor do I.
At the moment I sit here since 11 minutes and cannot work. IMatch (5.8.4) does work and work and work.
I cannot go into the preferences.
And I cannot go into the log-file, because as soon as I go into the menus, it flickers and the menus are vanished again ... because IMatch works (I am in the Folder-view with 2000 images).

No chance at the moment. Since I know this behaviour (that is why I created this post once), I simply have to wait for maybe 15 minutes, 30 or 60 minutes roughly, than it is ok again.

That is why I wait on version 6  ;D ... though it was curious anyway, because I had never this behaviour before I created this post. Except there is somewhere "a border" what I had then overjumped with my files in combination with the categories.

I wanted only report this again, but Mario, you can and must of course nothing do, I will see it then once with IM - version 6, if this behaviour is gone. (btw, IMatch is still working, loading always the same folder, the image in the viewer shows up, flickers again, goes away and shows the message like in the attachement, then show again the pic and so on).

Mario, nothing to do for you. I wanted only say it, since 5.8.2 I have this behaviour. And this I find a bit curious, I had this never bevore, it was introduced since 5.8.2, but as I said, maybe simply a conincidence.

(wenn Du in 5.82. nichts geändert hast, das ein solches Verhalten zeigen könnte, dann ist es schlicht ein Zufall und es ist einfach zuviel für meinen Computer und ich warte schlicht mal version 6 ab. Denn schlussendlich kann ich irgendwann wieder arbeiten, ewig dauert der Zustand ja nicht).

Now I have to go away for 2-3 days, have fun and Mario, the weekend does come, maybe simply ignore all users postings  ;D 8) ;D and relax the weekend without IMatch.  :D



Best wishes from Switzerland! :-)
Markus

Mario

IMatch is only using 1.6 GB RAM. There is a lot RAM left.
It also uses only 10% CPU, which is nothing.

If you would have attached the log file, we would have been able to see what IMatch is doing.

It is really strange that IMatch works one day and then suddenly displays such a strange behavior. Unless something is broken or you have created something (category formula, data-driven category, variables in your file window etc.) that drives IMatch nuts and makes it run into circles.

You did not say what you did before IMatch was busy for 11 minutes...
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

sinus

Thanks, Mario

I closed IMatch, hence no log.
But I opened again, went to a folder and had the same behaviour.
IMatch is busy.

After some minutes I changed the folder and could get the log, what I do attach.
Hope this helps and I hope, it is simply something stupid on my machine.

Thanks in advance, low prio.  :D
Best wishes from Switzerland! :-)
Markus

ubacher

I do get the behaviour you describe quite often.
I just close (forced if need be)  IM and open it again and this fixes the problem.
When I look at the log it does not give me any clue as to what is happening. I assume some
sub-processes lock each other and the issue does not get resolved - or so slowly that it might as well be dead.

Mario

Quote from: ubacher on January 27, 2017, 12:32:39 PM
I do get the behaviour you describe quite often.
I just close (forced if need be)  IM and open it again and this fixes the problem.
When I look at the log it does not give me any clue as to what is happening. I assume some
sub-processes lock each other and the issue does not get resolved - or so slowly that it might as well be dead.
When IMatch encounters a database lock that takes more than 1000ms it writes an entry into the log file. IMatch also records all operations which are unusually slow. Potential dead-locks or race conditions. Naturally there is always a chance that a user somehow manages to create  "new" problem that is not properly logged, but after a couple of years these are rare occasions. I work with IMatch many hours a data and my largest database has 380,000 files now. With more than 30,000 categories.

The typical cause for IMatch becoming unresponsive for a short (!) time is when the current "display" (file window or a panel) requires results from one or more data-driven categories and that these categories are invalid and need to be re-calculated. Depending on the categories, the size of the database etc. this may take several seconds. But then the log file will tell us more.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Mario

Quote from: sinus on January 27, 2017, 10:29:08 AM
I closed IMatch, hence no log.
But I opened again, went to a folder and had the same behaviour.
IMatch is busy.

10:06:32  :  The log file starts
10:06:46  : Database loaded, UI ready.
10:06:47 :  File window starts loading files and data (344 files).
                  Collections are calculated in the background.
10:06:48 : App Panel loaded. Quick View Panel loading file.
10:06:51 : File Window load complete. Lots of stacks and file relations to check.

Question: Do you use a custom file window layout or one of the standard layouts?

10:06:58 : File window starts loading another set of files.
10:07:09 : 1352 files loaded.

This repeats many times. The file window is loaded (and lots of data), cleared, loaded again. IMatch is super-busy clearing and re-loading the file window. All the time. The entire log is full of the file window loading the data for about 1500 files, over and over.

The file window is loaded 780 times (!) in this log file.

No errors or warnings. No locks. No unusually slow operations.
Just IMatch loading the file window for minutes. And loading images into the Quick View Panel.

Do you perhaps have accidentally added the cache folder  D:\IMatch-PREVIEWS\previewcache\ to the database?
Because it seems that every time IMatch touches a preview image, the file window begins to reload...and this may be caused by a file system event...

Also, just in case, try this:

Close the App Panel.
Switch the file window to the Default layout (shipped with IMatch).
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

sinus

Quote from: Mario on January 27, 2017, 03:14:50 PM
The file window is loaded 780 times (!) in this log file.

No errors or warnings. No locks. No unusually slow operations.
Just IMatch loading the file window for minutes. And loading images into the Quick View Panel.

Do you perhaps have accidentally added the cache folder  D:\IMatch-PREVIEWS\previewcache\ to the database?
Because it seems that every time IMatch touches a preview image, the file window begins to reload...and this may be caused by a file system event...

Also, just in case, try this:

Close the App Panel.
Switch the file window to the Default layout (shipped with IMatch).

Thanks, Mario.
I know, that my file window has quite a lot of stuff (icons and so on), hence in such cases I do always switch to the default layout, to prevent, that the fault is in my file window.
But finally, although my personal file window means a lot of calculations for IMatch, it does always work like a charm!!

But I found the source of the problem, yeah!:

You created in one of the last updates new variables.
To not forget these variables I putted them all together into the VarToy-App.
And this was the problem.

If I copy them into the VarToy, IMatch count and count or whatever it does.
If I remove them from the VarToy, all is ok again.

So I think, this is the problem. I must simply not copy them into VarToy, then my problem is solved!  :D
If you see the variables, I am sure, you see quickly the source, maybe I copied them not correct or it is simply stupid  ::) to put them into the VarToy.  :-[


Problem solved!

Here are the variable, like I copied them into VarToy:

neue variablen:
relation.isMaster: {File.Relations.IsMaster}
relation.isVersion: {File.Relations.IsVersion}
Relations.Color: {File.Relations.Color}

Relations.Stack.Count: {File.Relations.Stack.Count}
Relations.Stack.State: {File.Relations.Stack.State}

stack.isTop: {File.Stack.IsTop}
stack.isElement: {File.Stack.IsElement}
stack.color: {File.Stack.Color}
stack.count: {File.Stack.Count}
stack.state: {File.Stack.State}


OT-BTW: VarToy is really cool, but it is a pity, that I cannot move the horizontal line between the code (oben) and the result (below). If I could move it, I could give sometimes the code/text above more room. Mabye I will do a feature request for this one day.






Best wishes from Switzerland! :-)
Markus

sinus

#21
Hm.... my last post was too quick.  :-\


I think, the variables was the problem.
As I pointed out, I made theses test with the default file layout.

So, since the problem was gone with the default layout, I thought, it must be these variables in the VarToy.
What are, I am quite sure, are the problem.

But now I switched back to my file window (what i did NOT changed since several versions), the problem is back.

Hence, could it be, Mario, when you added these variables in my post above, that some older stack-versions-variables are also affected?

(heisst, ich habe viele Variablen in meinen File Layout, diese aber echt lange nicht mehr geändert. Und die Problem liegen fast sicher in den neuen stack-variablen. Könnten ältere stack-variablen, die bis jetzt funktioniert hatten, mit der Einführung der neuen etwas geändert haben?)

If I have time, unfortuntely not now, I simply will revome variable after variable in my file layouts and hopefully dectect the bad one! And hopefully I am no the right track for this problem, but I am quite sure.

So, Mario, I will further look into this and maybe find the real problem.


update:
The problem occurs only, if I am in the Media-View. In Categories there is no problem.
Best wishes from Switzerland! :-)
Markus

Mario

When I remember correctly you use a very complicates file window layout, with icons and stuff.
Please export it so I can see why your layout causes IMatch to reload the file window hundreds of times. This must be a really strange side effect that affects only you.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

sinus

Quote from: Mario on January 30, 2017, 11:06:10 AM
When I remember correctly you use a very complicates file window layout, with icons and stuff.
Please export it so I can see why your layout causes IMatch to reload the file window hundreds of times. This must be a really strange side effect that affects only you.

I append the layout.

What I found out so far (hope this is correct):

In the first footer left I have a complex text:

{File.BuddyFiles|contains:xmp,y,,false;pereplace:y==<Image Source='file://c:\sinus-icons\xmp-20.png'/>}
{File.MD.XMP::dc\relation\Relation\0|default:;hasvalue:ST;pereplace:ST==<Image Source='file://c:\sinus-icons\stack18.png'/>}
{File.AT.Anmerkungen.Et|cast:int; pereplace:0==;pereplace:1==<Image Source='file://c:\sinus-icons\bulb16.png'/>}
{File.AT.Anmerkungen.Ze|cast:int;pereplace:0==;pereplace:1==<Image Source='file://c:\sinus-icons\doit24.png'/>}
{File.AT.Anmerkungen.OK|cast:int;pereplace:0==;pereplace:1==<Image Source='file://c:\sinus-icons\ok20.png'/>}
{File.AT.Anmerkungen.Zv|cast:int;pereplace:0==;pereplace:1==<Image Source='file://c:\sinus-icons\zv18.png'/>}
{File.AT.Anmerkungen.RG-norm|cast:int;pereplace:0==;pereplace:1==<Image Source='file://c:\sinus-icons\verr24.png'/>}
{File.AT.Anmerkungen.RG-Arc|cast:int;pereplace:0==;pereplace:1==<Image Source='file://c:\sinus-icons\archiv-24.png'/>}
{File.AT.Anmerkungen.Gratis|cast:int;pereplace:0==;pereplace:1==<Image Source='file://c:\sinus-icons\gratis-24.png'/>}
{File.MD.XMP::photoshop\TransmissionReference\TransmissionReference\0|default:;hasvalue:change;pereplace:change==<Image Source='file://c:\sinus-icons\Name-change-20.png'/>} {File.MD.XMP::photoshop\Category\Category\0|pereplace:ALLES OK==<Image Source='file://c:\sinus-icons\erledigt-metas20.png'/>;pereplace:ok==<Image Source='file://c:\sinus-icons\meta20.png'/>}


This text with variables I changed quite a long time not, at least  not since january 2016.
It worked all the time.
But not more now, it reloads always, as reported.

BUT: if I remove only the first text line, with the buddy-variable, this here:
{File.BuddyFiles|contains:xmp,y,,false;pereplace:y==<Image Source='file://c:\sinus-icons\xmp-20.png'/>}

then all is ok again, no reloding, all works perfect.
Maybe this buddy-text-line is wrong, although it worked all the time before?

And finally, I know, that this is a complex layout, but it worked all the time like a charm and without this buddy-line also  :D

Maybe somehow is my first line wrong.
Or there is a change in the variable.
Or I am completely wrong, and the problem is elsewhere in my layout.

Since I can now work with another layout, Mario, this has a very low priority!
[/b]



Best wishes from Switzerland! :-)
Markus

Mario

QuoteBUT: if I remove only the first text line, with the buddy-variable, this here:
{File.BuddyFiles|contains:xmp,y,,false;pereplace:y==<Image Source='file://c:\sinus-icons\xmp-20.png'/>}

This is a super-slow and super-heavy variable.
To find all buddy files, IMatch has to search the file system - because buddy files are often not in the database.
This means that for your file window layout, IMatch has to apply all your buddy file rules, search the file system one or multiple times for each single file in your file window. Geez, that's a lot of stuff going on just to display a page of files in the file window. I would never have thought that a user needs such complicated things. This will also make your file window scroll very, very slow.

I don't see yet why this could cause the file window to reload. Except maybe scanning a folder somehow triggers something in Windows (e.g. the thumbnail index) which then causes a "folder changed" event to be sent. And this event propagates into IMatch and from there into the file window, which reloads the folder. And that triggers again all your variables...
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

sinus

Quote from: Mario on January 30, 2017, 03:09:04 PM
QuoteBUT: if I remove only the first text line, with the buddy-variable, this here:
{File.BuddyFiles|contains:xmp,y,,false;pereplace:y==<Image Source='file://c:\sinus-icons\xmp-20.png'/>}

This is a super-slow and super-heavy variable.
To find all buddy files, IMatch has to search the file system - because buddy files are often not in the database.
This means that for your file window layout, IMatch has to apply all your buddy file rules, search the file system one or multiple times for each single file in your file window. Geez, that's a lot of stuff going on just to display a page of files in the file window. I would never have thought that a user needs such complicated things. This will also make your file window scroll very, very slow.

I don't see yet why this could cause the file window to reload. Except maybe scanning a folder somehow triggers something in Windows (e.g. the thumbnail index) which then causes a "folder changed" event to be sent. And this event propagates into IMatch and from there into the file window, which reloads the folder. And that triggers again all your variables...

Thanks, Mario, for looking into this.
So let it be, I have other possibilities to do, what I want.

The only thing, what I wanted stress (again), this variable and the whole window layout worked very quick and smooth for a long time.
No problems, all was fine and this was true with a lot of images, what I have, despite you say, this variable is super-slow and super-heavy.
So you can see, that IMatch does sometimes works better, then even you would imagine.  ;D 8) ;D

Then came version 5.8.2 and this slowness was suddenly here.
That was the reason, why I made this post (Yesterday I updated to the newest version 5.8.2. Since then IM is working very hard ...)

[b]So, you can forget this post, since I will find another way, because now I have found out, that this variable is the problem.
So it I can easy find a good way, really![/b]

Best wishes from Switzerland! :-)
Markus

sinus

I took the variable simply out.

Reason:
Since roughly 3 monthes I do add an xmp-sidecar file for all nefs.
Hence all nefs have a sidecar.

Therefore it is not more necessary to show this in the window layout.
Before I had only for some nefs xmp-sidecars and it was convenient to see this very quickly.

So at least this was good for, one variable is not more necessary. Fine.
Best wishes from Switzerland! :-)
Markus

Mario

QuoteSince roughly 3 monthes I do add an xmp-sidecar file for all nefs.

I'm not aware of any change in IMatch that would cause the file window to reload when a user uses the buddy files variable.
Maybe this is a side effect of something. I will have to investigate when I have some time.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Tallpics

This has been an interesting post to read.

It shows the depth of time and effort that Mario will make to try and understand and solve even the most obscure problem!

I think it also proves that if Mario wants to give IMatch a thorough stress test Sinus must be the man ;)

It also proves what a knowledgable 'power user' Sinus is!

Thank you.... both of you. All of us gain from following your experience  :)

sinus

Quote from: Tallpics on January 30, 2017, 06:46:04 PM
This has been an interesting post to read.

It shows the depth of time and effort that Mario will make to try and understand and solve even the most obscure problem!

I think it also proves that if Mario wants to give IMatch a thorough stress test Sinus must be the man ;)

It also proves what a knowledgable 'power user' Sinus is!

Thank you.... both of you. All of us gain from following your experience  :)

Well, thanks Tallpics,

It is correct for sure, that Mario puts a lot of time and effort into his support.
But not only for me of course   8) but for all users.

About my knowledge: I think, I do know IMatch quite good. Some parts of it special well, others not that good, depending on how often I use it.
But since I use IMatch for a long time and Mario knows his users, specialy if they have been longer in the forum, he can better (I think) decide, well, this could be a real problem or if a user simply does not have enough knowledge or not read the helpfile.  ;D

I think, I spotted several glitches and bugs and Mario solved them, or I gave some useful hints, but I asked of course also a ton of silly questions, that is for sure.  :-\
And, BTW, as I wrote just before on another posting, for example from IMatch Anywhere I do know very very little - at the moment.

I know over the years, that Mario does really takes care for all problems from all users.
Since 2001 I can remember a very few amount of users here, who had problems, what Mario could not solve ... because mostly IMatch was simply not the right program for them.
And Mario does even, if someone asks for huge tasks and so on, wrote clearly someting like "For this situation IMatch is not designed, you should look for another DAM".

Such things does not a lot of programmers, I think.
Therefore a lot  of users does like the great support from Mario.

Thanks for your nice words and have fun with IMatch. :D
Best wishes from Switzerland! :-)
Markus

Mario

I have looked a bit more into this, and I can reproduce the behavior, of sorts...

When IMatch processes variables, it tries to do as little work as possible, and to delay 'expensive' operations which cost a lot of CPU and database cycles.

For example, it does not 'fill' metadata variables for a file unless they are really needed. Or relation variables like "buddy" files. It divides the variables into groups ("metadata", "file", "annotations", "relations") and when the first variable of a group is requested, it fills all variables in that group. This is a performance feature because it means that, for example, metadata is only loaded once, and then all variables are filled from that and kept cached in memory.

Your file window layout requests buddy files. This is part of the "relations" group and so IMatch loads the data for all relations variables. This requires it to update all relation collections and then to figure out in which collections the file is. Is it a master? A version?

Is this usually a quick operation. But in the case of the file window redrawing, a lot is going on. To display even a single file in the file window, collections need to be loaded, categories, thumbnail, rating and label and so on.

And while all this is done, your variable causes another collection update (in addition to the slow file system operations required to find buddy files not in the database).
And this is where the problem happens. The file window may be using the database (most likely) while redrawing, keeping it locked.
Now, your variable needs to scan collections to find information. And if one or more of these collections are not up-to-date, the IMatch engine will try to update them. For that, it needs the database, but that is currently locked by the file window which is redrawing the file window, using the database all the time.

The "update collection" routine tries to get access to the database, but fails. This can happen and so the routine retries this a few times, always waiting a few milliseconds. After maybe a second or two it gives up and just uses the collection as is. And this happens for all relation collections in your database. This adds up to several seconds. And that for each file in the file window. This makes things really slow.

If the update collection routine is successful, however, it ill update one or more categories. Which in turn causes a reload of the file window, because it needs to make sure that it displays all collection icons and information right. And this causes again an update of your variable and everything starts over again...

The file window update is a very complex and fine-tuned dance. And it works great with all file window layouts and attributes. But there is always the chance that a user breaks the carefully tuned mechanism by adding the odd variable and hell breaks loose.

In short: This is a won't fix problem. Just don't use this variable in your layouts. I could spend a week trying to find a way to solve this, and fail. And since only you are affected and only one of your file window layouts, and your file window layouts are way out of the normal anyway, I won't spend any time with this. If this becomes a problem for several users, I will look again into this.

Until then: This behavior is by design.


-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

sinus

Quote from: Mario on January 31, 2017, 10:16:31 AM

In short: This is a won't fix problem. Just don't use this variable in your layouts. I could spend a week trying to find a way to solve this, and fail. And since only you are affected and only one of your file window layouts, and your file window layouts are way out of the normal anyway, I won't spend any time with this. If this becomes a problem for several users, I will look again into this.

Until then: This behavior is by design.

Thanks, Mario, this is fine for me. I am not (more) keen on this variable. The only thing, what puzzled me, was, that it worked for a long time before without any problems.

But all this is not more interesting, I do, like you wrote:

I simply do not use this variable
(and in my case it is also not more necessary; because all my raws has not xmp-files).

Thanks again, Mario, put it ad acta.
Best wishes from Switzerland! :-)
Markus

sinus

Update, not important:

The variable

<td align="center">{File.BuddyFiles|contains:xmp,y,,false;pereplace:y==<Image src="file://c:\sinus-icons\xmp-30.png"/>} </td>diese variable oben macht IMatch superlangsam, arbeiten geht nicht mehr, also nicht mehr nehmen, auch wenn es vorher bestens ging -->

with BuddyFiles makes IMatch almost useless, at least in my case.
I had this variable before in my Window Layout, then with a new version started the problems.
But with removing the buddy-variable all was and is again fine!

Today I switched to a htmp-app, where the variable is in also.
The same problems, luckily I remembered about the variable, searched, found, deleted it and voila: all if fine again.

And also here (simply to say again), this html-app I used very often before.

So at least in my case this buddy-variable makes IMatch very, very busy. Hence I will not use it.

Easy for me, remove a (now not more necessary) variable, and the problem is solved. Fine. ;D
And as Mario pointed out, this variable makes IMatch anyway slow, hence better not using it.
Best wishes from Switzerland! :-)
Markus

Mario

As far as I can tell, nothing has changed with how this variable is implemented in a long while.
But IMatch scans the file system (has to!) when you use this variable. Maybe this triggers your virus scanner, which then goes nuts and blocks IMatch somehow?
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

sinus

Quote from: Mario on February 07, 2017, 12:47:35 PM
As far as I can tell, nothing has changed with how this variable is implemented in a long while.
But IMatch scans the file system (has to!) when you use this variable. Maybe this triggers your virus scanner, which then goes nuts and blocks IMatch somehow?

Yes, can be, but to be honest, I think, not worth, to look into this for you and me  8) (with Virus-scanner I do not know a lot anyway).

Thanks for your answer.
Best wishes from Switzerland! :-)
Markus