Announcing a major change in metadata handling (...creepy background music...)

[Mario]

Hi, Testers

As I mentioned in passing in some recent posts, I've spend a considerable amount of time since the last release (two weeks ago) to re-think many of the metadata options under Edit > Preferences > Metadata 2.

How IMatch handles metadata and the options it offers in this area have gone a loooong way since the initial Alpha version. As a result from discussions with early testers and attempting to support their workflows I've added options, removed options, added options for options, options for exceptions etc. There were some (partially heated) discussions about how to best handle all the different workflows in IMatch and ExifTool, and all the specialties required by the variety of file formats and applications in use today. Needless to say that I was still learning (and still am) about ExifTool at that time.

The internal handling of metadata and attempts to cover all angles required to code to become more and more complicated over time. Which is never good. The more complex a software becomes, the easier it is to introduce errors in place A when making changes in place B. And I had sometimes a hell of a time to debug problems reported by users, because the code was so complex.

This is no good omen for a brand-new product like IMatch 5.

So I decided to sit back and re-think things. Maybe I can make the code simpler again, without forcing my users into a specific workflow. I'm not in the lucky position of Adobe or Apple who just tell their users what's the right workflow...something that can make the life of a software developer much easier

To make a long story short, here is the new Metadata 2 panel:



Where are all the options gone?

Simple: IMatch now has per file extension metadata options. This allows me to configure metadata options centrally in a configuration file, and it also allows you to override these options for specific file formats. Read-only RAW files: Check. Don't want to synchronize IPTC/EXIF? Check. Don't want MWG? Check. Never change RAW files? Check. Apply crop records only to format XYZ? Check.

All options related to IPTC/EXIF/GPS/PDF metadata are now handled on a per file extension basis. This allows you to control precisely to which file formats these metadata formats are written (JPEG, TIFF, DNG, PSD by default) and to which not (RAW files). The defaults IMatch ships with from build 124 are the same as before (MWG compliance) but you can now easier specify the exceptions you may need to support your specific workflow.

This is how it looks:





[attachment deleted by admin]

[jch2103]

I like the concepts (greater clarity/simplification) even though I'm one of those who has just used the default metadata settings (for NEF).

[dcb]

 

[cytochrome]

If it simplifies the code and offer the same flexibility as the actual metadata2 panel it is very good.

There is one point that you should also address because it is central in this metadata handling: it is how IM displays diacritic characters, which is all wrong at present.

As an example: a series of JPGs with "journée" in both Headline and Description. Wehn I display these JPGs in Irfanview, FastPictureViewer, Xnview etc... they all show "journée" in all xmp and iptc fields concerned with these tags.

Only IM displays "journ?e". But internally IM stores it correctly as "journée" because a search for "journée" finds all these JPGs as seen in the screen capture!! while a search for journ?e finds nothing (all black);

At present, if a file is in Latin charset (many european files still are, utf-8 is a recent standard), with the default metadata preferences it is read but at write back im writes it back as Latin and tries to display it as utf-8 because (I suppose) exiftool produces utf-_ output by default. This can not work, after two write back gibberish is creeping all over the metadata.

Francis

[attachment deleted by admin]

[Mario]

Please don't hijack this thread to re-iterate an open bug report.

If you want to comment on a bug report, please do so in the corresponding thread.
If you have a file which only displays wrong in IMatch, please attach to your bug report or send it to me - if you not already did so.
Keep track of everything is hard enough, and spreading information related to one open bug over several boards and threads does not make things easier.

[Ferdinand]

I will need to see how this works in practice, but my initial view is that this is a very good idea.  It occurred to me some time ago that the way that Geosetter manages these settings was easier to understand and configure.  It's possible that I may have said so, although I doubt it.  But I had thought that you were too committed to the existing approach, and after all, it is (hopefully) getting late in the beta.  So this is a pleasant surprise.

[sinus]

This idea seems to me very good, and as you pointed out, easier also for the user.
I have a big hope, that this works fine.
I do not know Canon CRW, I guess, you have a good reason to let this format in the Metapane 2, because I thought, why not put this (special?) format simply also in the options like other formats too?

I use NEFs, hence I do really think, that your new approach will work for me.

Sometimes it is better, lean back and think, instead work and work ... Good done! 

[cytochrome]

Please don't hijack this thread to re-iterate an open bug report.

..

OK, sorry for that. I am working on metadata in my files and it appears to me as a whole : how metadata is moved and how it is displayed, because what is displayed is what we see.

I"ll wait for 124...

Francis

[Buster]

:-) :-)

[Mario]

I do not know Canon CRW, I guess,...

The screen shot was a bit older. I've dealt with the .THM files issues too with the new approach.

[BenAW]

Once the dust settles over this new approach I hope we can focus more on the core business of Imatch: managing images.
Writing and copying of metadata is but a small (although for some users apparently important) part of Imatch.

[Ferdinand]

Once the dust settles over this new approach I hope we can focus more on the core business of Imatch: managing images.
Writing and copying of metadata is but a small (although for some users apparently important) part of Imatch.

For some users, writing and copying metadata is a core part of managing images.
I guess it's only natural that we all see the parts of IMatch that we use as core and the parts that we don't as non-core.

[Mario]

Once the dust settles over this new approach I hope we can focus more on the core business of Imatch: managing images.
Writing and copying of metadata is but a small (although for some users apparently important) part of Imatch.

For quite a large number of users, managing and working with metadata is the priority argument for purchasing and using IMatch. They need more control and better adaption to their workflow that software like LR offers.

Other users don't care for metadata at all. They use the defaults and stay away from rather complicated things like IPTC, EXIF, XMP entirely. They just want to store a description and maybe some keywords for their files. Where this information is stored is not important for them - well, at least until they upload their files to their favorite web site and the description and keywords don't show up. Bugger 

Corporate users, scientists, doctors and archivers who deal with many non-image formats often go for properties (IMatch 5: Attributes) only. Because if you stick to these, you are not affected by all these nasty effects of metadata, in-file or external storage, migration between metadata standards and the like. Your data is safely stored in the IMatch database and that's it. These users also usually don't participate in Beta Tests or the community so usually only I know what these folks are doing 

And some use IMatch solely because of it's unique category system 

As Ferdinand pointed out, and you are of course aware of, every user uses IMatch in different ways, and prioritizes different aspects of it differently.  The old 80/20 rule. Metadata is maybe 5% to 10% of the application, but it is unfortunately very complicated and time-consuming to implement and maintain.  That there is no real standard and we have to deal with files with metadata added over the past 20 or so years just adds to the overall complexity



I'm not ashamed to say that I very much like to work on new and exciting features, Apps, scripts, other cool things on my to-do list.
This metadata handling re-engineering was rather major, brain-tumbling and sometimes even rather unpleasant. But I know that the new concept is much cleaner and easier. Better for the time to come, after the initial release.

And I could roll up some long-standing feature requests in as well. As a bonus.

[DigPeter]

Once the dust settles over this new approach I hope we can focus more on the core business of Imatch: managing images.
Writing and copying of metadata is but a small (although for some users apparently important) part of Imatch.

To my way of thinking,  metadata (e.g. keywords & categories) are part of managing images.

[DigPeter]

Once the dust settles over this new approach I hope we can focus more on the core business of Imatch: managing images.
Writing and copying of metadata is but a small (although for some users apparently important) part of Imatch.

For some users, writing and copying metadata is a core part of managing images.
I guess it's only natural that we all see the parts of IMatch that we use as core and the parts that we don't as non-core.

+1. 

[cytochrome]

I don't "manage" images since I don't understand what it is. What I want is simply to find back images on a series of criteria, and/or find out where/when/why an image was taken, who's on it and what were my personal thoughts at that time.

For this I rely totally on metadata writen to the image and on the IMatch categories. The rest (pins, labels, dots, fancy colors etc) I don't care for at all.

So yes, the way metadata is written and read back is 50% of IMatch use for me. And I look forward to the new metadata handling that Mario prepares for us.

Francis

[BenAW]

Once the dust settles over this new approach I hope we can focus more on the core business of Imatch: managing images.
Writing and copying of metadata is but a small (although for some users apparently important) part of Imatch.

For quite a large number of users, managing and working with metadata is the priority argument for purchasing and using IMatch. They need more control and better adaption to their workflow that software like LR offers.

Exactly. Simple things like "oldfashioned" IPTC worked for years without much problems. IM5 is going much further than this, and this may be useful for some, but I expect the majority of users to only use categories, write (multilevel) keywords and some other IPTC values like country, location, copyright  etc. Managing these is imo what IM5 is all about.
@keywords, data driven categories, filters etc. etc  are the core business for IM5 imo.

[BenAW]

Once the dust settles over this new approach I hope we can focus more on the core business of Imatch: managing images.
Writing and copying of metadata is but a small (although for some users apparently important) part of Imatch.

To my way of thinking,  metadata (e.g. keywords & categories) are part of managing images.

Exactly my point, and I feel that a lot of time is diverted away from these basic activities.

[Photon]

A very big thank you to Mario for this major change and the spending of valuable time to sit back an think.
This concentration on users and possible workflows and software transparency really differentiates IM from most other tools.
PhaseOne and Adobe and the whole DAM world can learn something from Mario.

I learned so much from IM v3.6, the beta test and the discussions in this forum for my private and business life,
that my grateful purchase of coming IM v5.x is practially fixed.

Martin

[ColinIM]

Sorry I'm late in responding to this, but yes, I look forward to this more 'granular' level of control of metadata 

It's always tough to juggle 'more granular program options' versus 'ease of initial use at the start of a learning curve' but this change is IMO a superb one, without doubt.

continuing applause .... 

Colin P.

[cnrgeorge]

I am probably an illiterate imatch user in that I use a very limited part of its potential. However, I have been happily using imatch 3.6 for years. We have over 160,000 images (not all of which are, alas, fully catalogued) but our habit has been to use iptc fields to label and search for our files. I could easily batch label images using 3.6's IPTC editor and then conveniently find them using search/for text in properties... While somewhat slow on our server, this worked. Test driving the new imatch I haven't figured out how to do either of those key steps yet. I have managed to upload my files and metadata and confirm that my iptc tags do appear when I browse the metadata in imatch 5.

So what I haven't figured out is the following:

1) how to do something equivalent to the text properties search (the search on the metadata toolbar doesn't seem to do the trick) across all files in my database.

2) how to batch label the iptc fields of multiple files.

As a comment on running the program, I attempted to upload all 160,000 images to a new imatch 5 database, but was unsuccessful... the loading would proceed but various hours into the process, it would freeze up. I finally tried just loading the past year (some 23,000 photos) and was successful with that. As I am sure others have said, it seems a pity that we can't just upload an existing imatch 3 database.

Some leads would be appreciated. I apologize if the answers are staring me in the face and I just haven't 'seen' them.

Thanks.

[DigPeter]

Some leads would be appreciated. I apologize if the answers are staring me in the face and I just haven't 'seen' them.

Mario states that IM5 is a test B version and recommends it should not be used as your primary DAM at this stage, but that you should use only copies of your images.  He says he will provide a means of migrating IM3 to IM5 when he is happy that the program is satisfactorily robust.

It appears you are trying to run before you can walk    I would recomment that you start small with only a few hundred images and get to know the system.  Have you looked at Help at the top of the opening page and read and follow the quick start guide?  Having got that under your belt, go to 'Open System Help" and go down the list of subjects which appear below 'IMatch activities and features" .

Others are better qualified than I to give detailed answer to your questions. IM5 is based around keywords and categories, not on IM3 type properties.  This should become clear after reading the opening help pages.

[Mario]

This is a Beta Test. Do not use IMatch 5 with your original files. See the Beta Tester Guide in the help for details and potential risks involved using Beta software on production systems.

Your questions:

1. Did you read about Attributes in the help? This topic covers how to create Attributes (which replace Properties), how to import your properties from IMatch 3 if you want, how to edit attributes, search for attribute and all the like. If you have read this topic and still need info, please let us know.

2. All metadata editing is done via the Metadata Panel and the Keyword Panel. With "Batch" I assume that you want to change metadata for multiple files at once? If this is the case then just select the files you want to change in a file window, and then edit the metadata in the metadata panel. IMatch then updates the modified tags in all selected files. This works for one image as it works for 1000.

See "The Metadata Panel" in the help for all details.

[cnrgeorge]

Thanks Peter and Mario,

I will go back and do my homework; I think part of my problem is that I never worried much about "properties" in imatch3, just went straight to iptc. Will review and ask questions if I'm still baffled.

I'm currently working with image copies. My impetus for trying the larger number of files now in imatch5 is that imatch3 has sometimes been painfully slow on our server. We're thinking about trying to find a volunteer to help catalogue photos (we're a field biology outfit), but I need to have an efficient process and so wanted to see if imatch5 seemed noticeably quicker. The slow speed may be completely a hardware issue, just trying different potential solutions.

Conrad.

[DigPeter]

(we're a field biology outfit), but I need to have an efficient process and so wanted to see if imatch5 seemed noticeably quicker. The slow speed may be completely a hardware issue, just trying different potential solutions.

@Conrad
I do not think that most people have found either IM3 or IM5 painfully slow.  I think the jury is out on whether IM5 will be a lot faster, but I stand to be corrected (shot down in flames) by Mario.

Your use is interesting.  I have far fewer images than you - some 40000 processed, plus another 30000 raw images if I decided to incorporate them (at present I do not include the latter in IM3).  But my usage is similar to yours, as a large number of my images are of plants.   To manage these in IM3, I have a all the species images listed as hierarchical categories.  This arrangement I am proposing to carry over to IM5.   The structure is as follows:

Taxa flowering plants | family | species (scientific name)
Taxa ferns | family | species (scientific name)
etc

These are in the IM5 Thesaurus.  In the metadata of each image, the full hierarchical species name is written to MP::Lightroom\ hierarchical subject. Once the images have been ingested, their hierarchical names are written automatically to IM5 database in @keywords.  So if I have 12 images of Ophrys apifera, for instance, I just click on this in @keywords and they will all be displayed in the file window.  I also put the scientific name in XMP::dc\title and some other info in MP::dc\description.

My metadata panel has a preset that displays all this info.  The attachment is an example.  I also attach a sample of the hierarchical categories from @keywords.

I hope this helps.


[attachment deleted by admin]

[cnrgeorge]

Thanks Peter,

I appreciate the detailed example. I have done something somewhat similar with IPTC, having location fields and then general (e.g., flower, frog, fern) and detailed (e.g. species' scientific name, recognized common name) fields. I can then search in text properties for any term. I started out with IPTC (and so will probably stick with IPTC) because I was unsure what organizing programs I would use in the long-term and so wanted to have the info. written to the file itself. Archaic perhaps.

I have spent some more time with imatch5 including reading the Attributes entries (thanks Mario, my confusion comes from my low starting point, not the lack of instructions). I am still grappling with how to set up imatch5 to fit my use but am making headway. I partially figured out how to set up a custom metadata panel that shows my iptc fields. As I understand it, rather than editing them directly as I used to do in imatch3, I edit them as XMP fields (which have mapped the IPTC data). I may well have further questions, but am still working on it. I periodically do have freeze ups where imatch5 just stops responding, if I get a dump file, I'll submit it.

For speed, I think I need to experiment with hardware. Imatch3 works much faster when we have the photo files we're working with on our hard drives rather than on our server. However, space and the fact we share images requires a more portable solution. Perhaps a dedicated external harddrive with original images kept on our periodically backed up server...

Again, thanks for the input.

Conrad.

[Mario]

I may well have further questions, but am still working on it.

This is the right place to ask questions.

I partially figured out how to set up a custom metadata panel that shows my iptc fields.

Since classic (IIM) IPTC has been phased out for good, IMatch uses a XMP-centered workflow. You should configure your custom metadata panel using the "Standard" tags (which are all XMP) or by selecting XMP tags only. At write-back time IMatch maps XMP metadata to classic IPTC data, if so required. This depends on your workflow and can be configured. If all your software, clients, services are XMP-aware, XMP is the best way to go.

I periodically do have freeze ups where imatch5 just stops responding

Windows considers an application as "not responding" when it does not react for a few seconds to user input- This can happen easily, when IMatch is currently processing data in the background and you force it to load data, e.g. switching folders or categories. In such situations IMatch may been to compete a background task before it can process your input.

You did not write under which conditions the not responding happens, if IMatch was still busy processing data in the background (check Info & Activity Panel) etc. Please see How to report bugs for more information about what to include in a bug report. I need much more detail to give you tips or to find out why IMatch stalls on your system.

Imatch3 works much faster when we have the photo files we're working with on our hard drives rather than on our server.

IMatch 5 is usually much faster than IMatch 3 for almost all features I measured. IMatch has to do a lot more work when ingesting files and writing back metadata, depending on the metadata options you set, if mapping has to be done between metadata standards and so on. But IMatch 5 does most of these operations in the background, without blocking the user interface. But the work has to be done nevertheless, and depending on your computer speed and all that performance will vary.

I would need to know which operations are slower. Adding files? Writing data? Looking at files? Sorting files? Running scripts?
Enable the debug mode for your log file (see the Beta Tester Guide in the Help) and always supply a log file. The log file contains timing information for all major commands and operations, plus general statistic information gathered when IMatch closes about the maximum and average performance of certain critical tasks. This not only tells us what is slow on your machine, but allows me to concentrate my performance tuning efforts on things which really make a difference.

Without detailed information, I can do nothing.

[BenAW]

For speed, I think I need to experiment with hardware.

The single best investment you can make here is a SSD drive.
If you just use it for the dbase, the speed gain is already enormous.
Letting IM cache your images (default) on this SSD drive lets IM5 fly.

[DigPeter]

For speed, I think I need to experiment with hardware.

The single best investment you can make here is a SSD drive.
If you just use it for the dbase, the speed gain is already enormous.
Letting IM cache your images (default) on this SSD drive lets IM5 fly.

Is this an external SSD, or does this mean a computer with integral SSD?

[BenAW]

I have in both my PC (128 G) and laptop (256 G) an internal SSD disk as disk C:
In the PC I have a second normal HD that holds my images and other data.

I don't know about external SSD disks. Assume the connection will limit the speed (USB2 or USB 3).

[DigPeter]

Thanks Ben

[Richard]

Assume the connection will limit the speed (USB2 or USB 3).

Below are specks for a flash drive. Since the Read speed increases with capacity, I would guess that an external SSD using USB 3 would be even faster.

Hi Speed USB 3.0  compatible (backwards compatible to USB 2.0)
8GB and 16GB: Up to 90MB/sec Read; Up to 30MB/sec Write
32GB and 64GB: Up to 150MB/sec Read; Up to 30MB/sec Write

[DigPeter]

Thanks Richard

[cnrgeorge]

Thanks, I also appreciate the drive details. Conrad.

[BenAW]

To put things in perspective:

the most popular SSD at the moment is 250 GB and when connected to a SATA 600 internal bus the speeds are:
read  540 MB/s
write 530 MB/s

Price here ~ E 145,-

[Richard]

I don't think giving specs for an internal SSD put things in perspective.

Perhaps a dedicated external harddrive with original images kept on our periodically backed up server...

If someone has specs for a 250 GB external SSD, I would like to see them.

[JohnZeman]

Richard B&H has a 240 GB SSD on Black Friday special right now so you can see the specs there:

http://www.bhphotovideo.com/c/product/1000580-REG/sandisk_sdssdxp_240g_g25_extreme_ii_240gb_ssd.html

I personally have no experience with an SSD yet, and thus no input in this subject, but I'd like to get one in the not too distant future to replace my C: drive.  My 3GB D: drive has so much data on it (and growing) that I'll stick with a standard hard drive there for a few more years yet.

[Richard]

Thanks John, Your link was for an internal SSD but I found external SSD as well as external HHD to make comparisons. It appears to me that USB 3 is the weak link and cuts the read time to less than half of what an internal drive can achieve. However, USB 3 HDD are down around 100 MB/s while SSD are around 250 MB/s. I found it interesting that HHD were often listed in Mb/s while SSD were listed in MB/s.