too many elements to display

[StanRohrer]

"There are too many elements to display in the value filter for this tag."
This shows in my Filter Manager / Value Filter: "Keywords" box.

Is this because my @Keywords category shows 123,716 entries? The @Keywords category is flat - no multilevel A, B, C,...Z structure? Can you point me to Help or instructions to fix this? Thanks!

IM 5.2.14.

Stan Rohrer- 20 Nov 2014 - See entry below. Edit to add:
11,406 Keywords may be more correct. Sorry about my confusion here.

[jch2103]

Is this because my @Keywords category shows 123,716 entries? The @Keywords category is flat - no multilevel A, B, C,...Z structure? Can you point me to Help or instructions to fix this? Thanks!

Probably; that's a lot of entries. I know there's a limit in the Value Filter but I don't recall what it is.

Try using the Category Filter (bottom left window) Search or Filter.

[Mario]

The list boxes and tree controls included in Windows have limits. Trying to load 120,000 entries in the check list box used in the filter panel would be a really bad idea, because it would a) require a lot of memory, b) would take a very long time, c) will fail in the end because Windows runs out of resources.

The Metadata Value filters protect you from that by restricting the number of elements to a sensible value of 10,000 elements. That's basically as security setting if a user accidentally chooses a metadata tag which produces a massive amount of elements, e.g. a date and time value with seconds.

Having 123,000 (!) keywords flat, on a single level will also stress out other parts of IMatch, e.g. the @Keywords tree in the Category View, the Category Filter and the Category Panel. The Windows tree control is also not really designed to display 120K elements under a single node... IMatch goes a long way to deal with large hierarchies (taxonomies!) by loading only the elements under expanded nodes, loading things dynamically on a as-needed bases etc. But 120,000 flat keywords, all on the same level, is so unusual (and probably very confusing) that it's plainly out of specification.

[StanRohrer]

So how do I structure @Keywords and/or the filter panel to work within the limits? Is there a tool or setting?

[Mario]

Group your 120,000 flat keywords into several levels. Really.
At least enable the auto grouping by the first n characters for the @Keyword category hierarchy (see help for details). This allows you to group your keywords into two levels, with a first level auto generated from the first n characters of your keywords.

The limits of the Windows list box controls cannot be overcome. But using a category filter on @Keywords will work. I don't test IMatch with flat lists of 120,000 keywords, that's just nothing that happens often.

[StanRohrer]

I've tried some combinations. Including:
Keep Empty Categories : NO
Add Auto-Group: YES
and Start and Length settings of 1.1, 1.2, 1.3, 1.4

I saw lots of Out Of memory boxes along the way during conversions but never did see entries appear in the Filter Manager / Value Filter: "Keywords" box.

[Mario]

This is a feature that's only available for @Keyword categories. When you use value filters, you work with the metadata contained in your files and cached in the database. So a value filter will never show the group levels you have created for a data-driven category or @Keywords.

Use a Category Filter instead of a value filter to work with your keywords.

When you get out of memory errors, did you keep the log file so I can have a look?
Why do you have 120,000 keywords on one level? How do you find anything?

[StanRohrer]

I'm learning something new about IM5 every day. I'm still pretty beginner at IM5. Thanks for the description of differences between @Keyword and the value filters.

I'm not sure the difference between a Category Filter and a value filter at this point. More scanning the help sections to figure it out I guess.

I have just sent an Email to support with dump and log of the first Out Of Memory error related to the Auto-Group conversion.

How do I find anything with 120,000 keywords? Easy. Your programming works well! Just click in the box and start typing the first few letters of a word and the box contents will scroll to the area I need. I did this in IM36 and it seems to have carried over to IM5.

[StanRohrer]

So I'm back to the basics of my problem. I tried to go back to IM5.2.12 because my failure became evident about the same time I upgraded. Unfortunately IM5.2.12 won't run due to database changes with the IM5.2.14 update.

Attached is a screen shot of my difficulty where I tried to get this down to a very small scale. I have one image in the active folder. It has 16 entries shown as keywords. Yet the Value Filter: Keywords thinks there are too many elements to display.

Value Filters work for selections of: File Extension, File Format, Copyright, Exposure Mode, Model, Shutter Speed, Description.  All of these only show one entry (since only 1 file).

Is the Value Filter: Keywords failure still related to my 123,000 keywords in the @Keywords system list? I think not since I'm looking at only one file.

Should the filter be only processing the 16 keywords from this one image? I think yes.

Have I screwed up some configuration setting somewhere that affects this filter? I hope not.

[attachment deleted by admin]

[sinus]

Hi Stan

I do not know really.

I made the same.
It seems to be, the middle field, will be filled with all the keywords, what are in the scope! (in your case only 1).

MAYBE IMatch goes anyway through all the keywords in your system??

Then, we will end with such a list like mine (attachement) and then we can enable, what we want looking for.

But, to be honest, I have not worked really with the filters. And your enabled "Files with values", I could get nothing really, what I had suspected.
But, when I changed to "Files with one ore more checked values" or the second entry (attachement), then I get the expected results.

I do not know, if this has something to do with your huge list of keywords, I have ROUGHLY about 7'000.
I had a quick look at a wellknown vocabulary ( http://controlledvocabulary.com/ ) what we can buy, they speak from 11'000 terms.

If you have really over 100'000, I cannot imagine, that this really help you. But of course, you have them and apparently works with it. But I can imagine, that this is for some windows or fields or scripts simply too much.

Maybe you try once with the changed value, choose the first (Files with one ore more...).

More I cannot say, sorry.

Ahem, an idea would be, to try the same with a new test-db with only a few images?
If you work with the thesaurus, you could also store your current one and add only a few keywords?

Ps: in my example I had 5'000 images in the scope

[attachment deleted by admin]

[Mario]

Is the Value Filter: Keywords failure still related to my 123,000 keywords in the @Keywords system list? I think not since I'm looking at only one file.

The Metadata Value filter works as follows: It retrieves all unique elements for the selected value (in your case keywords) and displays them in the list. It then analyzes the current scope (what's in the file window) and puts a check in front of each value that is used by all files, and an intermediate check in front of each value that is used by at least one file.

This works all well with the typical values used, e.g. lens, ISO, artist. But it will not work when you try to use it with a metadata value that returns 120,000 unique values - as in your very particular case.

Please don't use the Value Filter. Use a Category filter and there the @Keywords hierarchy.

[StanRohrer]

Mario,
I find using Categories and @Keywords on the left panel and then selecting Folders in the Value Filter a bit less intuitive in my mental workflow. My mental workflow is to select a day or event and then select or work down from there. Your alternate method does seem to work with a few noted limitations.

One of the limitations is selecting multiple keywords (which does work), at the end of some task, like writing a new keyword to all files, the display returns to only one keyword selected. So if I have another task on those same keywords I have to go find them all again. For example, today I selected keywords E102; E17; E323; E37; E66; E686; E8, which are the registration identifiers painted on race boats. I selected multiple folders of various events this summer. Then added keyword "2.5 Litre Modified" to all 897 files as the boat race classification. On return from the job only keyword E8 was selected. With the previous method of keywords in the Value Filter, the Keywords selection would remain for the next task. In this task my keywords happen to be close together as the "E" is the boat classification which relates to the new keyword phrase being added. However, having the multiple keywords this close together will not always be the case.

Now that I have @Keywords as Auto-Group 1.1, my click in the box and start typing a keyword does not work nearly as well. This parsing apparently doesn't look into the sublevels when the groups are folded closed. It looks like it might still work if I leave all @Keywords groups unfolded open.

Another bit of a picky. With the Value Filter active, in this case selecting 4 folders, sometimes when I pick a new selection of keywords the files fail to display (refresh based on new settings). Sometimes the files do show. In cases where they don't I have to uncheck the Value Filter main box and recheck to get the desired files to show.

I built a fast 4 core computer in anticipation of the multi-threaded IM5 coming. The speed of keywording in IM5 is AWESOME compared to my many overnight runs in IM3.6.  Thanks Mario!!!!

[StanRohrer]

Mario,
Now back to the Value Filter and 120,000 Keywords. I certainly don't have your programming knowledge nor the knowledge of the tools at your disposal. It seems to me that loading in @Keywords and then doing the file scans to match is using a lot of memory just for @Keywords. Certainly I understand a limitation here. I just updated this computer from 8GB to 24GB of RAM so feel free to allow tables to expand if the host computer has memory.   ;-)    Conversely, is there a way to scan the files for keywords first and then parse through @Keywords at the source location without loading all inclusive into a memory block? Display only the related and not the inclusive keyword list. Or to scan the files for keywords and then work through smaller chunks of @Keywords to parse? Since I'm not going to view and hunt through 120,000 keywords, I will use the search tool. Perhaps there is a way to search and display only the keywords or a group around such keyword (to see plurals and varied spellings). Just thinking out loud here to see if there is a possible IM solution available for my keywords situation.

I'll work with the @Keywords selections for a bit and see if I can get used to the method. In the meantime I will be considering how to break my database in to 2 databases, or something, so I can do keywording in my original method.

[StanRohrer]

Sinus,
So. how did I get to 120,000 keywords. I poked around a bit to see if I could understand my own system. I think there are a number of factors that contribute to this condition.

First, I started using IM back in year 2002 or 2003 or close. I was submitting these new digital photos to stock photo sales web sites. Most of them could not handle phrases as keywords. Likewise they didn't handle punctuation. I keyworded in IM and then uploaded files to the sites so IM had the same keywords as I expected to have used online. But the limitation made for "Perry's Cave Family Fun Center" to be keyworded as "Perry's", "Perrys", "Family", "Fun", "Center". As sites began to take phrases I also added them, e.g. " Perry's Cave Family Fun Center" and " Perrys Cave Family Fun Center". So now this name in the IM database was taking 7 keywords.

Early in my IM use I was shooting a lot of classic cars and hotrods. Again, on the eventual upload to stock photo sites I keyworded once early in my work flow. Then the keywords would be available to any site or location I desired. Hence keywording was aimed at potential search people, not just my use. A car, with background removed in Photoshop, might be keyworded as such: 1957; 57; Air; American; Bel; BelAir; Chevrolet; Chevy; USA; United States; antique; auto; automobile; car; classic; clip; clipping; clippingpath; custom; customized; cut; cutout; historic; isolate; isolated; modified; motorcar; motoring; nostalgia; out; path; purple; restoration; restored; transportation; vehicle; vintage

With cars and boats, it is useful to add the license plate number or boat registration number. So these can be found across multiple events across many days or years.

Boats and cars might also be tied to an event name or event location which would be added to the keywords. Here is a recent race boat entry: America; American Power Boat Association; APBA; boat; boat race; boat racing; CE52; Celina; compete; competition; fast; Grand Lake Saint Marys; hydroplane; inboard; lake; Lake Saint Marys; Lake St Marys; Ohio; Ohio Governor's Cup Regatta; race; races; racing; regatta; speed; speed boat; speed boats; The Ohio Governor's Cup Regatta; transport; transportation; United States; USA; water; watercraft; watersport; CE-52; vintage; old; antique

I shoot some events. You might guess this is a 5K Run/Walk:  5K; America; American; Beavercreek; Beavercreek Popcorn Festival; Dayton; Ohio; States; USA; United; United States; athlete; athletic; challenge; contest; fair; festival; fun; game; race; road; run; running; sport; sports; street; talent; walk

I also do some travel and tourism photography: America; American; Columbus; Ohio; USA; antique; architecture; building; capital; capitol; city; classic; column; design; door; doors; doorway; entrance; entry; exit; greek; historic; historical; history; nostalgic; old; pillar; post; property; restoration; restored; revival; rotunda; structure; town; travel; urban; vintage

For my personal use I wouldn't put nearly this number of keywords into a file. But one never knows what words a potential customer will use in his search so the bases have to be covered.

Not that all this is message is directly about IM5. But I hope a little understanding of work flows and IM use is helpful to the product design. I've hit 120,000 keywords in 130,000 files in the database.  In addition to the writing of file descriptions and keywording, the database also has captured file status: junk, at a given sales outlet, related to specific customer jobs. My backlog of photo work is 1 year and 15,000 photos. Yes, I shot a bunch of racing at 8 frames per second that need culled and worked. Photography is not my day job. So getting every bit of speed and efficiency from IM5 is very useful to me.

[ColinIM]

I can offer another example of keywords being embedded in 'stock' images, although on a much smaller scale than yours Stan. (Maybe someday I'll have as many stock images as yourself    )

In my modest but lovingly crafted collection of 300+ stock images, my list of unique keywords for just these 300 images is somewhere over 3000 keywords.

However ... this cannot easily be translated into a count of "average keywords-per-image" because (of course) any keyword or set of related keywords might be assigned to any single image or sub-set of images.  Many of my stock images hold over 30 keywords.  But perhaps these figures are useful as a ball-park comparison with Stan's.

These 300 images are only my 'stock' images, and they're still "in the care" of my old but familiar IM3.6 database. My full hoard of over 66,000 images - and their eventual keywords - will soon be poured into a new IM5 database.  My workflow will continue to be keyword-centric, so although I also look forward to building a good and logical category tree, I'm really itching to feed many many new keywords into this larger set of images.

(Iincidentally and off-topic, I've chosen to re-create this "new" IM5 database afresh, more than once, as I've become progressively more 'comfy' with IM5's almost limitless flexibility.)

Colin P.

[sinus]

Stan, thanks for your detailed answer.
Hm, crazy, like the number of keywords rises up.

If I look at keywords like these here:
might be keyworded as such: 1957; 57; Air; American; Bel; BelAir; Chevrolet; Chevy; USA; United States; antique; auto; automobile; car; classic; clip; clipping; clippingpath; custom; customized; cut; cutout; historic; isolate; isolated; modified; motorcar; motoring; nostalgia; out; path; purple; restoration; restored; transportation; vehicle; vintage

I would say, in my "system" I would think that some keywords are already there, because they are common, like 1957, Air, American, Chevrolet, USA, United States, antique, auto, automobile, car, classic, historic, motorcar, purple, restoration, restored, transportation, vehicle, vintage.

And other keywords I would not use: 57, clip, clipping, clippingpath, path, out

Of course, if you take "nostalgia", you could also add "nostalgie" or for "transportation" also "transport" and so on.
Never ending.

But it is like it is, and when you have such an amount of keywords, than you have them
Think about this, if you would add to your 120'000 keywords a second language!? 

In my own system, wait ... I have hmmm, 6339 categories, means about 6'000 keywords. (how do you count your keywords?).

I guess, I will never come on your number of keywords ... except I do a lot taking photos from cars, animals, sports ...

BTW, I am happy, IM5 makes it easy to change or edit keywords.
I do not have a hierarchical keyword - system, and I have read in the last time about it, I have decided, I will not use them. A lot of people preferes a "simple" keywords-system, and if we have added a clever synonym-system, so we can have a fast and good keyword-system, I think.
My own system has keywords like

01-Alle-Jahreszeiten
...
03-mäännlich-02-06-Jahre

and so on. This came, because in the early days of IMatch I had not another possibility to sort the order of keywords quickly, hence we made a system with numbers (01-).

Now, with IM5, I can quite easy replace these keywords throug a shorter keyword without numbers.

Good luck with your keywords!

And, Colins, thank you also for your adding remarks!

[StanRohrer]

120,000 Keywords IS A BAD NUMBER.

11,406 Keywords may be more correct. Sorry about my confusion here.

My bad. When working in the @Keywords category, made the bad quick assumption that the number at the end was a tally of keywords. Apparently it is a tally of files counted that contain keywords. So my @All category is 130,252 and my @Keywords is 123,708 today. But those must be file counts.

So I opened all collapsed keyword entries in the category view, counted the visible, and then counted the number of pages (using the Page Down keyboard key). After a few subtractions of the top level categories I arrived at a very different estimated number of keywords in my system.

I now think I have 11,406 Keywords. Leaving in the groups top levels I have about 11,475 entries under @Keywords.

Mario had earlier mentioned the Value Filter being limited to 10,000 elements. I caught that but it didn't register that my counting may be way wrong. Then Sinus made comments that my number seemed excessively out of range and that got me to take another look and reconsider. Indeed it appears to be my mistake. So I must have crossed the Value Filter boundary somewhere around or above Mario's 10,000 count at the point my filter stopped working.  Likely that was within hours of upgrading from IM 5.2.12 to 5.2.14.

So the "too many elements" problem still exists, as does the discussion of how it works - albeit at a bit of different magnitude of keywords counted.

Now I have to figure out, without keywords in the Value Filter and using the invert function therein, how to find 6,544 files without keywords. (@All 130,252 - @Keywords 123,708 = 6,544).

[Mario]

To find files without a value in any given metadata tag (e.g. hierarchical Keywords), I would give the Metadata Search a try.
Use settings like



The filter has been set to search via regular expression in hierarchicalKeywords. The regular expression .* finds anything, but we used the not regular expression, which inverts the result => find all files without a value in the selected tag.

[attachment deleted by admin]

[StanRohrer]

Thanks Mario.

It took a bit to end up where you were headed but I got a "Keywords Missing" filter created and running. Is there a way to find this filter as a file and share it for other IM5 newbies (like me) on the forum?

[Mario]

Since it only takes maybe 5 seconds to setup, no need to export anything. Just a 1,2,3 FAQ would be sufficient, if you want to write one of those.