A File Finder-type app for Attributes; Export/Import performance

[dkorman]

Mario,

The more I spend time with new iMatch the more impressed I am.

The File Finder App is great, especially as it can display the search results in a separate window - something that I thought would be a great feature for iMatch - a "tabbed" file window. This is a wonderful addition.

(A) A File Finder-type app for Attributes

As you might be aware, I am a heavy user of Attributes - being able to conduct an Attribute search similar to File Finder would be very useful for me. It would also be nice if it could be "updatable," kind of a saved search in its own tabbed-like window!  And, having the search be able to use "Advanced Search" (AND OR NOT) would be nice.

(B) Export/Import Performance

I did a complete export and import of an Attribute Set, and I was wondering if the performance is reasonable and as expected.  I know that in some responses to users, you (knowing what's going on "under the hood") have written how a seemingly simple process requires more than X millions of database operations.

The Attribute Set that I exported has approx. 405,000 records, 20 fields, all type "text."  The maximum character count for any record is probably about 600, with the average probably about 350 (this is just a guess).  So, perhaps 140Meg of actual data. The iMatch export and import files are 405Gig.  (Machine is 3.4GHz, 16Gig ram, iMatch database on 7200rpm drive, system drive an SSD, paging file on a separate hard drive).

1) Export - can take over two hours. 
2) I process the exported file against a database (flat file, fixed record length) that I have historically maintained, and write a file for reimport into iMatch - a process that takes less than 10 minutes, which includes parsing the iMatch *imas file, conducting an indexed search for each record to see if the attribute data has changed, and writing out the new *.imas file. 
3) Import of the Attribute Set back into iMatch takes approximately 40 minutes. 

Does this iMatch performance make sense to you (I know that this is not a typical operation for your users.)  Are there settings that that would give iMatch more CPU or memory that might enhance performance (iMatch has, in my opinion, always been impressively reasonable in its operational "footprint").

(As an aside, when I select "Import" , iMatch immediately reads through the *.imas file, which at 405Gig takes a couple of minutes, not allowing me to first "repoint" to another directory first - this is not typical Windows behavior.  Could the reading of the file be postponed until I actually select the desired import file?  This is just a little annoyance.)

As always, thanks,
David

[Mario]

IMatch uses XML to export and import attributes. This is a very 'wordy' file format but the best option for optimal interoperability.
If you want to export 400,000 attribute records to import them into a database, use the Text Exporter. It should be faster and the resulting CSV file should be much smaller.

I think that any time spent with optimizing the performance of Attribute export/import would be wasted. Not many users use Attributes and not many of these have 400,000+ Attribute records with ~ 600 bytes each and not many of these will ever have a need to import / export them.

[dkorman]

I understand on the Attribute Export/Import.  I'll look into the Text Exporter. 

Any thoughts on an Attribute Find App?  In playing with the File Finder App, does it reexecute a search when the "tab" is reselected?

Any idea for tweaking iMatch for performance?

By the way, thank you for updating iMatch so quickly  You are the best!
David

[Mario]

I understand on the Attribute Export/Import.  I'll look into the Text Exporter. 

No. Let's let this feature request sit for a while and see if other users find it interesting.
The File Finder app has been designed for very special purposes, like matching files in a database with a set of files in a list. Or matching files by their first n characters.
Most user will be more than good with the file window search bar instead.
The File Finder App does not update when you click on whatever "tab". It runs and updates the search result when you click on one of the "Run" buttons in the App.

Tip: Mixing many topics in one feature request (like you did) makes things a lot more complicated. Better to make one request for one topic.
Like, not mixing "I would like an Attribute Finder App" with "Exporting of 400,000 Attribute records is slow". Which is not a feature request but maybe something more suited for General Discussion.

I don't know how often you plan to import / export 450,000 Attribute records, but I would say that this is a not-problem for the majority of users.

Any idea for tweaking iMatch for performance?

Tweaking the performance of IMatch for which feature? Is this a feature request or do you rather want to discuss something? In that case, the General Discussion board may get you more answers and tips from other users.