RFC: Import Lightroom Catalogs into IMatch?

[Mario]

Hi.

based on some additions I've made to IMatch WebServices (and thus the embedded App system in IMatch) I might be able to write a new app which can import data from Lightroom catalogs. Might!

For example, a new user could use that to tell IMatch which folders to import into the new datbase ("Same folders as I manage in Lr").
Or maybe we could translate some smart collections into IMatch data-driven categories. Things like that.

If you use Lightroom, would you see actual use for such an app?
If so, what would you like to transfer from Lr to IMatch?

(Note: Metadata travels automatically so no need for that).

Let me know below.

[Gerhard]

For me are the following 2 interesting:
- Smart collections ( as already mentioned)
- stacking of pictures

[Arthur]

Hi,

I am using LR6 currently and have imported my files in IMatch yesterday and am currently trying to reconstruct my LR catalog by hand in IMatch. The following aspects could be simplified:

1) Automatically insert the catalog roots which were present in Lightroom with the same directory structure
2) Auto translate non smart collection trees to categories and assign the images, so that the user albums are in place.
3) I am not using smart collections, so cannot say something about it.
4) Auto create assignable keyword trees (called thesaurus here?). These are not only the keywords which are used in files.
5) Ensure that the keywords categories are not duplicated. After the import I had each keyword category twice, as flat and as hierchical version. In LR I see the keywords once.
6) [ADVANCED] The killer feature for me would be, if it could be possible to automate Lightroom to generate a "farewell" export, so that there is a preview JPEG/TIF next to every raw with the last LR development settings. This previews could be auto configured as version stack proxies. This would really help user without technical affinity. But I read that LR is not automatable in that way. So a good compact tutorial which covers exactly that topic would also be a good start.
7) [ADVANCED] I do not know whether the information is in the catalog but if it is, you could also try to translate the users import pattern to a Renamer configuration. Yesterday I needed some time to figure out how to move new import files to my {YYYY}\{YYYY}-{MM}\{YYYY}-{MM}-{DD} directory structure. But because there is a timeline feature, I think about changing my directory structure in the future. Not sure here.

[Mario]

Thanks for providing feedback.

4) Auto create assignable keyword trees (called thesaurus here?). These are not only the keywords which are used in files.

1. IMatch can fill the Universal Thesaurus from the keywords in your files. Open the thesaurus manager and choose the corresponding import option.
2. You can export your keyword list in Lr and then import it into the IMatch Universal Thesaurus.
See the thesaurus topic in the IMatch help for details and this knowledge base article: https://www.photools.com/4588/free-controlled-vocabularies-imatch/

5) Ensure that the keywords categories are not duplicated. After the import I had each keyword category twice, as flat and as hierchical version. In LR I see the keywords once.

IMatch never duplicates keywords. If your files contain keywords both in a flat version (without hierarchy) and a hierarchical version, IMatch cannot tell without a filled thesaurus.

location|beach|daytona is different from daytona or race track|daytona or daytona|beach.

Make sure to fill the IMatch thesaurus before you import images with non-matching flat/hierarchical keywords. Then IMatch can lookup the flat keywords in your thesaurus and map them to the corresponding hierarchical keyword(s).

FARWELL FEATURE...

Not sure about this. Very complicated to implement. And most users use Lr and IMatch together anyway.

[Arthur]

Thanks, will look into these.

I have mentioned a farewell feature, because there are quite a view consumer/prosumer people which are stuck on LR6 now, because they never want to rent software. You can read it everywhere. So Lightroom Classic CC is not an option. As first step I have replaced my Develop module in LR by DxO, so that this important part is not done by the old LR6 anymore. What remains in Lightroom is the Import and the Library module with its Dam features. You can use this, but as far as I know, if a new/adapted raw format is out in the future, you will need to go on the DNG format to be able to further use LR as DAM. On the other hand IMatch uses external codecs/sources to display previews, so it never gets outdated here as long as for example FPV updates its codecs. Please tell me if I understood this wrong.

So, If you are not happy with Adobes licensing model but want to stay on software which is developed further, you have two possibilities to replace LR currently:

1) IMatch + DxO
2) Capture One

I would not underestimate this. But yes this is why I wrote ADVANCED :-)

[Mario]

I have finished the implementation of SQLite support for IMWS. It is pretty basic but works great for the intended purpose.
As a real world test I've copied/pasted together a small app which opens a Lr catalog and extracts some information. When I find the time I will see what and how things can be imported into IMatch.

I would need to allow the user to choose what to import ("Add these folders to IMatch", "Re-create these collections"). Flags may be useful (IMatch also has flags but Lr does not export them into XMP and hence we need  direct import).

The smart categories are stored as JSON in the Lr catalog, looks not too complex. Basically what the @MetadataTag category formula can no, not coming even close to what data-driven categories can do in IMatch.

[Mario]

Apparently this is not such a hot topic for IMatch users in this community, but it is for other users / potential new IMatch users.
I've spent some more time on this (also a good test case for new endpoints for IMatch WebServices).

My app can now display the root folders in the Lr catalog and allows the user to add the same folders to IMatch.
I can also import the Reject/Pick flags Adobe stores only in the Lr catalog but not in XMP.

The next step will be to import regular (non-smart) collections. This should be a real help for users with large collection hierarchies.
This type of collection should be relatively easily map into an IMatch category hierarchy.

Not sure yet if it would be worth the effort to attempt to map smart Lr collections into IMatch data-driven categories or, rather, categories using the @MetadataTag formula.

This is how it looks right now after I've selected a Lr catalog and clicked on the Analyze button:

IMAGE OUTDATED.

[Jingo]

If I was a LR user - I'm sure this would be very helpful for me to migrate to IM... having a method to migrate some data will be a great feature for those looking for alternatives to LR. 

[Mario]

This will also be helpful for users who keep using Lr.
Because proprietary Lr 'thingies' like collections and flags can be transferred this way whenever you need them in your IMatch database.
Of course this depends on whether you do actually asset management in Lr or you only use it as a RAW processor and leave the DAM part to IMatch.

This app is also a test for me and the new endpoints added to IMatch for working with SQLite databases.
I will write many of the new features for the next generation of IMatch in form of apps or a new concept based on apps which will be introduced later.
The development cycles are so much faster, I can do more with JavaScript/HTML in less time. Make changes quicker. Ship a bug fix for an app as a plain ZIP file if needed...

[snapzilla]

Hi Mario,

My two cents.

I'm in process of leaving LR6.  Not because I dislike LR6 but because I no longer trust Adobe.  I'm moving to ON1 Photo RAW 2018 which has a DAM component but even though it has a LR6 migration tool I'm choosing IMatch for DAM for two reasons.  One because ON1 is well behind photools in features but the main reason is I want my DAM separated from my photo-editor - I don't want to be back here again - migrating on the hurry-up - should I need to leave ON1.

I'm migrating manually and it is very time consuming.  My LR catalog has >25K images and many collections - smart and unsmart.

I've manually generated a small jpg for all LR edited files that don't have their own jpg/tiff/psd etc.  In IM5 I've excluded jpgs, tiffs and psds etc as a buddy files across the board.

I'm in process of keywording all images in all unsmart collections that I wish to migrate.

I plan to re-create Smart Collections manually as a last step.

Lack of a migration tool was not an issue in deciding my DAM landing place but I can see how it could be for others - esp for professional and semi-professional photographers.

Also I can see a time when those that do linger with a static unsupported LR6 will choose or need to move - possibly quickly.  An LR6 DAM migration tool(set) would very likely be a factor for them.

John

[Mario]

My Lr Importer can do this, so far:

Show you all root folders in your catalog

You can then select the folders you want to manage in IMatch and click  on "Add". IMatch then adds the same folders to your database and starts ingesting files.

Show you all flagged files (the Pick/Reject flag used by Lr).

IMatch can import the flag into IMatch flags, selected collections and mark files as rejected which have the Reject flag in Lr.

Show you all non-smart collections

You can select the collections you want to import. IMatch creates categories from them and assigns the files t these categories.

All this should at least make the initial transition easier.


Points to Ponder:

I know how Lr stores the setup for smart collections and I could probably write something that maps them to formula-based IMatch categories.
Some things are only possible inside Lr (smart collections based on proprietary Lr data) but others would work. I just don't know if it is worth the effort.

Or if a knowledge-base article that explains how to re-create the most frequently used smart collections in IMatch via the @MetadataTag formula would be better.
There is already the Using Adobe Lightroom and IMatch together which just needs some new sections.

Making Lr export JPEG files for files in the catalog so these can be used as visual proxy versions for the RAW file. This is something a user needs to do by hand. As fa as I know, this cannot be done from an external application via automation. I could write a plug-in for Lr, but that's a route I'd rather not follow. I'm wearing too many hats already.

[RobiWan]

I might be able to write a new app which can import data from Lightroom catalogs. Might!

Great idea.
Even better would be writing an app to import C1 catalog 

[Mario]

I might be able to write a new app which can import data from Lightroom catalogs. Might!

Great idea.
Even better would be writing an app to import C1 catalog 

I have started an app which handles C1 COS files but then stopped because of the increasing complexity of all the stuff C1 writes into the XML, how they repeat data for versions and all that. I don't have C1 nor the time to install it and try out all the features and how the affect the contents of the COS files. Just not worth the time giving the few users who will ever need this.
Naturally, neither the COS files not the C1 database is documented in any form.
For Lr, more info is available which makes reverse-engineering the database easier. And there are probably 50,000 Lr users for each C1 user 

[RobiWan]

I have started an app which handles C1 COS files but then stopped because of the increasing complexity of all the stuff C1 writes into the XML,

I know I was the guy asked this. Thank you very much again for your time and support!

Naturally, neither the COS files not the C1 database is documented in any form.
For Lr, more info is available which makes reverse-engineering the database easier. And there are probably 50,000 Lr users for each C1 user 

I agree with you unfortunately too. Even though I mean the C1 database is easier than LR

[Mario]

Please don't write answers inside quotes. It makes your answers very hard to read.

[RobiWan]

Sorry, was not my intention

[tspear]

Mario,

Nice to see the progress you are making. I am also looking to migrate to a new DAM solution away from Lr. I have more time than a few others, my subscription expires next fall.
I would be interested in either the directions, or the automated conversion of Lr smart collections. I use them extensively in my workflow, and for my image management.

Tim

[Mario]

Lr smart collections are a very special thing. Many rely on internal Lr data and cannot be translated.

For all other stuff, IMatch formula based categories using the @MetadataTag formula will do the job just nicely. Or data-driven categories.

While writing the Lr Importer App (which is now 'finished' and will be included in the next IMatch release) I have looked at what Adobe is doing with smart collections. The data is accessible in the database (they store it as JSON objects) but I don't think it will be worth the effort to write a converter. It's probably doable but it would take a long time. Much longer than for a user just to re-create his/her smart collections by creating an IMatch category.

Have a look at the IMatch Workflow Categories IMatch creates by default (In the Category View, orange color).
These categories use the @MetadataTag category formula to implement some common workflow categories based on metadata.

For example, the following formula returns all files with an empty title:

"@MetadataTag[title,novalue]"

Just create a new category and copy/paste this formula into the "Formula" property of the category.


Since IMatch category formulas support Boolean operators AND, OR and NOT, you can easily create any kind of combination:

"@MetadataTag[rating,between,4,5]" OR "@MetadataTag[label,regexp,^Red]"

returns all files with a rating > 3 or files with a red label.

"@MetadataTag[rating,between,4,5]" AND "@MetadataTag[label,regexp,^Red]"

By swapping OR with AND you get a category which contains files with a rating > 3 and a Red label.

If you want a category which shows you all files which have been modified today, you can just access the corresponding collection:

"@Collection[Updated|Today]"

or maybe all files which have been updated this week?

"@Collection[Updated|ThisWeek]"

Again, you can combine these formulas. A typical combination would be:

Show me all files added today which have no copyright info or no title:

"@Collection[Added|Today]" AND ("@MetadataTag[title,novalue]" OR "@MetadataTag[rights,novalue]")"

Category formulas are a very powerful concept and not limited to a set of pre-defined operations or metadata tags. You can use all of the 13,000 metadata tags IMatch currently knows and combine that with the contents of collections, other categories, folders and more.

The smart collection dialog box in Lr is easier to use at first, but rather limiting.
Maybe I find the time some day to write an app which does a similar job and outputs category formulas ;-)
Note: There is a category formula wizard already in IMatch, and you can build category formulas dynamically via drag & drop and the category @Builder.

Tip: See the Category Formulas help topic in the IMatch help for more info.

[RobiWan]

Is there a chance to become Alpha/Beta Version?

[Mario]

A Beta Version of what?

I will include the Lr Importer in the next regular update, which means it will be also included in the Trial version.

The Lr Importer currently has these features:

+ Add the same root folders you have In Lr to your database.
+ Import Lr Picks (Flags) into IMatch Flags and/or Reject rating
+ Import regular collections into IMatch categories
+ Transfer the Lr keyword catalog into the IMatch Thesaurus

Except for the flags and collection you can do all this manually already.

[RobiWan]

Ok I understand. I thought that will be an extra app in IMatch (like your C1 Importer)

[Mario]

Yes, the Lr Importer will be an app. but included in IMatch. It will show up in the Import & Export panel and I will also present it to the user when he creates a new database (in case he wants to base the database on his Lr catalog).

[Arved]

Yes, the Lr Importer will be an app. but included in IMatch. It will show up in the Import & Export panel and I will also present it to the user when he creates a new database (in case he wants to base the database on his Lr catalog).

Exactly what I'm hoping for. For planning purposes, when can we hope to get this next release? i.e. how much longer am I going to be stuck with Lightroom? 

[Mario]

I still have 23 stickers on my to-do list for the 2017.12.2 release.
Including updates of some major 3rd party components. This makes it very hard to estimate.
In the past, updating these components and solving the inevitable problems caused by such updates took between 4 hours and ~ 60 work hours...

I plan to ship the final update for this year in December, so there's your estimate: This year. Probably 

[Arved]

Fair enough! Vielen dank!

- Arved

[blackhead2]

Hello Mario,

just wondering whether it is also possible to write an app based on your LR import app that "exports" files selected in IMatch to the Lr catalog? The reason for my question is that till now I'm not really satisfied with my workflow using IMatch as DAM and Lightroom as Developer (I'm using Lightroom 5). Lightroom often needs a lot of time for regocnizing changes I did in IMatch. E.g. when I set the label "red" in IMatch for some pictures it sometimes takes several minutes till Lightroom sees the changes after I wrote back the metadata. So currently I'm faster with selecting the images manually in Lightroom  .

So if there would be the possibility to "export" a specific category to the Lightroom DB as a Lightroom collection this could be an excellent solution to exchange files between IMatch and Lightroom from my point of view.

Regards,

Jens

[Jingo]

Hello Mario,

just wondering whether it is also possible to write an app based on your LR import app that "exports" files selected in IMatch to the Lr catalog? The reason for my question is that till now I'm not really satisfied with my workflow using IMatch as DAM and Lightroom as Developer (I'm using Lightroom 5). Lightroom often needs a lot of time for regocnizing changes I did in IMatch. E.g. when I set the label "red" in IMatch for some pictures it sometimes takes several minutes till Lightroom sees the changes after I wrote back the metadata. So currently I'm faster with selecting the images manually in Lightroom  .

So if there would be the possibility to "export" a specific category to the Lightroom DB as a Lightroom collection this could be an excellent solution to exchange files between IMatch and Lightroom from my point of view.

Regards,

Jens

IMO - the slowness could party (or even mostly) be because you are using LR5... which is really old and very slow.  LR6 made things a bit better... though it too is no speed demon and the reason why I left LR for good a few years back.   Also, you can manually update the data in LR to force an update without waiting for LR to "pick them up". 

[Arthur]

Yes just go on your catalog folder in LR, right mouse button, sync folder, ensure metadata is checked at the bottom, click ok, done.

Ensure before syncronizing in LR that no metadata writeback is pending in imatch by selecting write back all pending files.

I have also both systems up and running, in case one of them fails in the future.