Updating changes in the Attributes Panel

[JohnZeman]

Yesterday I began working with the attributes panel, mostly adding text to the attribute sets I'd previously created, and I noticed then (and have confirmed it again today), that depending upon the number of files selected, there seems to be quite a difference in the time it takes IMatch to update itself for the attribute changes.

For example if I select 100 images and change the text for 4 sets of attributes, then click elsewhere so IMatch 5 updates itself, it takes just a few seconds.

If I select 300 files the results are similar, a little longer but less than a minute usually.

But if I select 1200 files and do the same thing, it can take a good hour for IMatch to update itself.  So it's been much faster for me to update my attributes in batches of 500 images or less than it is to update over 1000 images at once.

This doesn't seem right to me, but if it is, I would find it beneficial if a progress status indicator could be added to the status bar so we would at least have an idea how long it's going to take.

[Mario]

The time should grow about linear, which means it takes twice as long for twice the number of updates.
I currently have no release built and my version of IMatch is spread open because I'm deep in the metadata plumbing again. Do you have a log file from your attempt (because it contains timing info)?

[Richard]

Hi John,

If you have the log file or if it happens again, I suggest that you look at how long it took for each file. I had a file that drove ExifTool nuts and updates took a long time due to that file.

[Mario]

The Attributes Panel is IMatch-only (no ExifTool involved).
Unless John was running other tasks which utilized the database, Attribute updates are very fast.
As soon as I have a release build again I'll run some tests.

[Richard]

The Attributes Panel is IMatch-only (no ExifTool involved).

Okay. About the time that happened to me I was also playing with Attibutes and I thought there might be a connection.

[Mario]

I've made some quick tests.

I selected a couple of folders for a file window, about 2400 files.
I then added a new Attribute record (note and comment) with about 100 bytes of text each. This took about 12 seconds on my machine.
Adding an Attribute record with about 100 characters 300 files takes about 5 seconds.
So it works with the expected performance.

Perhaps John can give us some more details, e.g. which attributes were used, database size, what else was running etc.

[JohnZeman]

Ok more details...

This afternoon I rebooted the computer then launched IMatch 5 (no other programs).  Selected 1621 images, made some text changes to 3 sets that I wanted each one of those 1621 images to have, and finally I clicked outside of the selected images to begin the update process.  I don't know exactly how long it took but when I checked at 65 minutes it was still updating.  20 minutes later when I came back it had finished.

A zipped log file and screen shots of the attributes are available here.

[Mario]

Hi, John

I don't see anything unusual.

Can you send me your Attribute schema? You can export it under Edit > Preferences > Attributes. I only need the schema so you don't need to export the data.

[JohnZeman]

I've attached it to this Mario.



[attachment deleted by admin]

[Mario]

Thanks, John

I used your Attribute setup with a 50,000 files sample database. About the same size as yours.

I then selected 2,500 files in a folder and entered data in each of your three Attribute sets. This resulted in about 17,500 update operations to the database, plus  the same number of search engine updates.

There was indeed a bad performance problem with this scenario. After doing some careful analysis I was able to speed up the operations involved (not only storing but also loading of that amount of data afterwards).

I stopped my first test after it went on for over 10 minutes (!). This was before my optimizations.
I now have the same operation down to between 20 and 35 seconds (!), depending on how busy my machine is. This is really fast now.

My standard "Update a 100 byte string attribute for 300 files" is now down to less than 1 second 

[JohnZeman]

Excellent!! 

[JohnZeman]

Just wanted to confirm the attributes panel updating is now MUCH faster than it was in build 106.  What used to take over and hour to do in 106 now finishes in less than 1 minute in 108.

[Mario]

60 times faster. Not bad 

Still, Attribute updates taking a minute are still slow-ish. I guess you are re-doing your tests with the 2000+ files and the multiple sets updates? This can easily generate 10,000 database or more records to update. Not a typical use case so I think it's OK for this to take a bit longer.