Using Lightroom

[samisbp]

Hi

I have a couple of questions regarding using LR and IMatch in parallel.

What is the best solution to access my IMatch categories in LR ?

Many thanks

[samisbp]

Just one more question, what is the best solution to access my IMatch File Attributes in LR ?

[Mario]

I also recommend the knowledge base article Using Adobe® Lightroom and IMatch together.

1.  Categories are a unique IMatch concept. Categories allow you to automatically categorize, cluster and organize your files, to combine categories with formulas or to use Alias categories to link the same category into different sections of your category hierarchy.

Adobe Lightroom implements none of these concepts.

1.1  The special @Keywords category hierarchy mirrors the hierarchical keywords in your files. If you work with keywords in IMatch, and the @Keywords category hierarchy, you can see this information in Lr in the Metadata Panel.

IMatch categories are stored as part of your IMatch database. This makes them both very safe and very quick. If you need to use categories other than @Keywords in another application, you can use the category export module in the Import & Export panel to expect the categories in XML format.  Or you use the Text Export module to export per-file category information.

2.  IMatch Attributes enable you to create your own 'database in database' inside the IMatch database. Attributes can be used to store all kinds of information, from cross-file format data to billing or submission tracking information. Attributes are also ideal if you want to link your IMatch database with information stored in other database systems, e.g., corporate back-office databases or library systems.

Adobe Lightroom implements none of these concepts.

If you want to use data you manage in IMatch Attributes in Lightroom, I would recommend to not using Attributes at all. Use XMP metadata to store descriptions and stuff, because XMP metadata is designed to exchange information between applications.

Other ways to use Attribute data in other applications are:

+ Export Attribute data via the Attribute Export feature (Edit > Preferences > Edit Attribtues)
+ Export Attributes via the Text Export module in the Import & Export panel
+ Export Attributes via a custom app

[samisbp]

Excellent.
Many thanks Mario for this prompt reply. I'll explore all this in details.

[samisbp]

I would need a bit of extra help.
As I have used extensively categories within IMatch, I do need to see them in LR.
I have looked at what you suggest Mario, the category export module and I have done a test on one category to export it into an .imcsx file. I made sure to tick the "include file assignement".
Then I have tried to understand in LR how to import that .imcsx file but I can't work out how to make this work so that this information is picked up and applied.

Any help will be very welcome.

Thanks

[Mario]

You cannot import an IMatch category import file into Lightroom. Lightroom has no concept of categories.
Even if you know how to write script for Lightroom and write a script which can read the category file, where would you expect Lightroom to import the categories into?

[samisbp]

Ok so I'm not sure I understand what you suggested by "If you need to use categories other than @Keywords in another application, you can use the category export module in the Import & Export panel to expect the categories in XML format.  Or you use the Text Export module to export per-file category information."

Would one option be to write a script that, for each file, writes its categories as keywords in its IPTC metadata ? Then in LR I could simply use those keywords which were categories to help search for my files.

[samisbp]

Just one more piece of information:
I used categories in IMatch, mainly as a means to organise my images and easily search for them. So if I can't export that piece of information into LR, than I loose quite a lot of information required to search for images.
Of course I'm also using keywords, but in the early years I didn't use them much and mainly used categories (stupid I know).

[jch2103]

Just one more piece of information:
I used categories in IMatch, mainly as a means to organise my images and easily search for them. So if I can't export that piece of information into LR, than I loose quite a lot of information required to search for images.
Of course I'm also using keywords, but in the early years I didn't use them much and mainly used categories (stupid I know).

The important difference between categories and keywords is that IMatch categories exist only in the IMatch database. This is an advantage or a disadvantage, depending on your needs and workflow. For example, some folks use categories for information they don't want or need to share with 'outside' users or programs. So, using categories isn't 'stupid' at all, it's just that apparently they don't (currently?) meet your needs. As Mario mentioned, there's more than one way to transform categories to keywords, so that option is certainly available.

[Mario]

Would one option be to write a script that, for each file, writes its categories as keywords in its IPTC metadata ? Then in LR I could simply use those keywords which were categories to help search for my files.

1. Please always use hierarchical keywords and regular keywords in XMP metadata. If you use the Keyword Panel in IMatch and other keyword features, IMatch will do that for you. IMatch will also 'flatten' the hierarchical keywords to provide them in the flat XMP and even legacy IPTC keywords, as needed.

2. Categories are a unique IMatch concept. IMatch can export them in many ways, but it depends on your other application if they are flexible enough to import them.
Lightroom has no concept for categories and no way to import anything.

Categories are a very cool IMatch feature. Use them for organizing your files, for working with formulas, data-driven categories and more.
Don't use them if you want to exchange information with other applications. There is no application out there which has anything that comes close to IMatch categories.

For exchange of metadata between applications use XMP metadata and keywords.

If you have organized your files using categories and you now want to use keywords instead:

1. See the IMatch help on categories, the Keyword Panel and especially the features provided by the special @Keywords category. This is a very powerful concept and you should know what it does so you can utilize it. Lr is pretty simple when it comes to working with keywords and metadata and when you look at IMatch through the Lr lens, you will miss a lot.

2. If you have organized your files using categories but now want to organize them with hierarchical keywords, you need to copy your categories into keywords and then remove the categories from your database - else you would have duplicate organization structures which is never good.

You can also just copy some of your categories into keywords and use categories for everything else. This is up to you. Usually a mix works best.

How to copy categories into hierarchical keywords has been discussed many times here in the community. See, for example:

https://www.photools.com/community/index.php?topic=6319.msg43670#msg43670

https://www.photools.com/community/index.php?topic=3428.msg22482#msg22482

https://www.photools.com/community/index.php?topic=7236.msg50227#msg50227

These topics discuss how to quickly assign keywords manually and how to use a Metadata Template to copy categories into keywords.
This all depends a bit on how you have structured your categories, which categories you want to copy into keywords etc.
Best to make some tests, try the variable out you will use for the metadata template (in the VarToy App).

Let us know if you need more help with that, but make sure to include details about what you have tried already and your category structure.
Note: If you have many categories and you convert them all into keywords, you might run into performance problems with Lr - it can bog down when you manage many files with many keywords. IMatch can easily manage 20,000 categories and keyword hierarchies as large, even for databases with half a million files.

[samisbp]

Many thanks Mario for all this information.

I'm going to try a few things and see what can work for.

Just on a side note Mario, for your interest, my main reason for switching to LR is that ever since I have been using IMatch I have always had serious performance issues regarding reading and writing back metadata to files. You may recall me contacting you on various occasions regarding that matter.
I have just imported all my photo library into LR during this past day. I have over 56'000 files in it, split onto 4 external HHD drives connected via USB3.0. Note that before my images are stored on those external drives, I always do my main work on them (organizing, post-processing, etc) on my local SSD drive which is quite fast.
I always thought that my performance issue for writing metadata back to my files in IMatch was due to this external setup. To my surprise, since I have been playing with LR these past few days, I have been very surprised to find out that LR is able to write back metadata to my files in an instant (and I really mean write back to the file, not just write metadata in the LR database). I've been able to see updates of my metadata in my files between LR and IMatch so I know I am correctly writing metadata into the files, but what I can't understand is why IMatch takes soooo long on my setup to write back metadata (at list 10's of seconds, even minutes at times), whereas LR is able to do it in an instant.
I have tried so many times to find out how to optimize IMatch performances to improve that problem but never managed. This is a pity as if I had managed I would certainly not be looking into switching to LR as I know all too well all the other benefits of IMatch which I will certainly miss in LR.
I wish I had some times to try to fix this problem but I simply don't have that time. So unfortunately I am giving up.

[Mario]

LR is able to write back metadata to my files in an instant (and I really mean write back to the file, not just write metadata in the LR database).

Different concept. Lr does not perform everything that ExifTool is doing when it comes to mapping metadata.
Also, Lr does 'in-place' updates, changing the original image when metadata is written. ExifTool uses a streaming approach, which means that it always rewrites the entire file, which allows it to properly deal with all the file variants, maker notes and stuff. Note that Lr may strip maker notes and other data from your files under some conditions. Ps does to.

Phil Harvey has written several informative texts about how he decided to implement ExifTool that way, and not supporting "in-line" updates. I would prefer in-line updates as well for some specific cases, e.g. changing only the rating of a file. But that of course does not only change the rating number text in the XMP record, it may also change the XMP digest, require mapping back to EXIF rating etc. Many inter-dependent things, really.

I don't see any performance problem with writing. I just updates metadata for about 5,000 JPEG files this  morning. IMatch did that in a few minutes (external USB 3.0 2.5" disk), while I was fetching a coffee. And I know that not only the XMP data is properly written, but also legacy IPTC (needed in this case) and EXIF data is properly synchronized, the maker notes of my camera remained intact etc.

ExifTool reports a runtime of ~ 0.1 per file on my local SSD for 36 MP JPEG files, and ~0.25s per file on my external USB disk. I would not call that slow at all.
What' are your timings? Change 10 files and write back. Attach the log file so I can see the performance data.

You can also open the ExifTool output panel  when you do this and see the write times reported by ExifTool. Add the time needed by IMatch to re-import the file, update data-driven categories and collections etc. About 0.2s per file or so.

[samisbp]

I regularly deal with very large files, up to 4 Gig and more, as I produce many stitched images from high resolution cameras. So if ExifTool always re-writes de whole file everytime for adding metadata, then I now know where the problem lies !!!
Is there a very quick way to configure IMatch so ExifTool does not re-write the whole file ?
I have to say, I find all that very confusing, and I am after all a photographer and don't want to have to deal with all the background blur that my DAM does. I guess this is why I think IMatch is no longer for me as I feel that you need to get your hands to much into the depth of how it works to be efficient with it. And as I said, I simply don't have the time.

I would love to try to send you logs for you to see more of what is happening in my configuration, but again, just no time right now. Although I really appreciate your time in trying to help.

[Mario]

There is no way to change how ExifTool writes files.
And, frankly, I doubt that there are many users out there which regularly update metadata in 4 GB files.
Usually you do that once, when the file is finished, and then never again.

You can write Phil Harvey from ExifTool about this. Maybe he changes ExifTool to support the file sizes you have to work with better.

I don't need logs in this case. Processing 4 GB files, maybe even over a network, will be slow when metadata needs to be updated. The 500 MB composites I regularly work with update in a few seconds, on a local SSD.

[Jingo]

I regularly deal with very large files, up to 4 Gig and more, as I produce many stitched images from high resolution cameras. So if ExifTool always re-writes de whole file everytime for adding metadata, then I now know where the problem lies !!!
Is there a very quick way to configure IMatch so ExifTool does not re-write the whole file ?
I have to say, I find all that very confusing, and I am after all a photographer and don't want to have to deal with all the background blur that my DAM does. I guess this is why I think IMatch is no longer for me as I feel that you need to get your hands to much into the depth of how it works to be efficient with it. And as I said, I simply don't have the time.

I would love to try to send you logs for you to see more of what is happening in my configuration, but again, just no time right now. Although I really appreciate your time in trying to help.

If you are using 4GB files and need to update them on a regular basis - you might consider using sidecar files to store you metadata... quick files that are only a few kb in size... quick to update and you never touch your huge files.

[Mario]

I considered mentioning that, but did not.

Such a schema would require

a) To remove EXIF/IPTC data contained in the images or to disable MWG compliance (bad)
b) To force IMatch to manage XMP in sidecar files, which causes compatibility issues in other apps which expect XMP to be embedded.

Not really an alternative, unless you have 100% control over your workflow and application suite - now and in the future.
Usually not worth it.
4 GB composite panorams are a very small niche in the market.