Metadata-Panel and selecting several hundrets of images

[sinus]

Hi Mario

At first: I THINK (hope), my problems with the collections are gone!  Of course I am very carefully with setting collections and labels/ratings, but I had not any problems until now!

Phew, MAYBE you found the nasty goblin/s!?

Now not a bug, but maybe to think about it?!

I am in the Media/Folder -view.
No panels open exept the Metadata-panel.
No special windows-layout, only the default one.

Now, I want to select several hundrets images at once, say about 800 or so.
If I use a pane in the metadata-panel with no or very few data in it, then this is no problem, I can even select all images (2'187), no problem with the speed.

But if I turn to the default - panel with Core-data where some of my metadata appears (like copyright, keywords and so on), then it takes minutes, if I choose several hundrets of images.

In this case I selected 564 images and had to wait about 3 minutes, until I could work further.

I can reproduce this all the time, I guess, you too. The same of course with some individual metadata-panels with a lot of information in it.

It is not a problem with, say 100 or 200 images, but somewhere with over 500 images it takes a long time, before IMatch again responses.

I do not now, how to solve this, maybe it is not possible.
But I could imagine, that I am not the only one with this problem. But it is not very important, only annoying. Because I simply close (or hide) quickly the metadata-panel, before I select 800 images, but sometimes I must of course also write info in the metadata-fields, and in such cases it is not very convenient.

I wanted only ask, are you aware of this? (first I thought, it is gone with this version, but I think, when I tried it, I had simply luck).

Do you see any solution for this or will it stay there like it is? If yes, I am thinking about another way to write info in the metadata-fields, IMatch gives us often more then one way! 

Attached a log with at least selected 564 images and I had to wait over 3 minutes.
And an image of the surrounding for this work.

Thanks for thinking about this ... but as I wrote: it is not a thing to hurry!  Take your time!











[attachment deleted by admin]

[Mario]

When you select multiple files, IMatch has to load the metadata of all selected files in order to 'merge' the metadata for the Metadata Panel and/or Keyword panel. This allows IMatch to show you which data is the same in all files, and where the data differs. Or, for the Keyword Panel, which keywords are only in one file, some files, or all files.

IMatch optimizes this by loading only the data needed (the tags used in the Metadata Panel Layout). And if you load more than 1000 files (2000 for the Keywords panel) IMatch skips the merge step entirely and displays a message. When developing this I have done some timings and decided that these thresholds are OK for average/mid-large databases.

If you have all common XMP tags filled with data, and you display the MD panel and Keyword panel, selecting 1000 files requires:

1. Load 1000 * <number of tags in the MD panel>
2. Load 1000 * all keywords

Depending on how your metadata is filled, this may mean that IMatch has to load something between 20,000 and 40,000 records from the database. Quite a lot and the performance here depends to about 90% on how fast your disk can deliver the data.

The method name used in the log file is CIMControlContainerWnd::InnerSetValues. I see execution times between < 10ms and maybe 0.5 seconds in the log. There is one call to this method which requires a whooping 194s, I assume this is where you selected the large number of files. This is of course much too slow.

I'm not sure why this is so slow on your machine, or if this is a more general problem and I need to lower the thresholds. So much plays into this, from versioning to stacking to the amount and distribution of metadata in your database, the size of your database (180,000+ files), the speed of your hard disk, the Metadata Panel layout used etc.

I need to run some tests for this. I suggest you file a bug report ("Metadata Panel load too slow when 1000 files are selected") with a link back to this post. I can then allocate a slot for one of the next releases.

[sinus]

Thanks Mario, for your answer.
In the same!   time, when you wrote your answer, I made some further tests (in the media/folder-view, custom metadata-panel, custom file-window-layout):

Selecting
300 images: 0.5 sec or so
498 images: 0.5 sec or so

504 images: 4.5 minutes (not seconds)
744 images: 5.3 minutes
965 images: 7.5 minutes

1003 images: 0.5 sec or so (not multiple show)
1818 images: 05. sec or so (not multiple show)

So it seems to me, that until 500 images, there is no problem (at least here).
But beween 500 - 1000 images I have a very long time to wait.

Over 1000 images no problems.

I will write a bug-report, as you proposed, thanks, Mario, no need to answer.

[attachment deleted by admin]

[Mario]

IMatch stops merging > 1000 files for the MD panel, and > 2000 files for the KW panel.
There should be not much of difference between loading 500 or 600 files, except the data for the additional 100 files takes much longer to load, for whatever reason.

Can you switch IMatch to Debug logging and do this test again (400 files, 600 files, 1050 files) and attach the log to your bug report? Maybe I see something.

[sinus]

IMatch stops merging > 1000 files for the MD panel, and > 2000 files for the KW panel.
There should be not much of difference between loading 500 or 600 files, except the data for the additional 100 files takes much longer to load, for whatever reason.

Can you switch IMatch to Debug logging and do this test again (400 files, 600 files, 1050 files) and attach the log to your bug report? Maybe I see something.

Of course, I will do soon ... a bit later, because just now I started a "pack & go"  and this will take about 1 hour.