What's brewing...

[Mario]

I'm currently working on the last missing feature for IMatch 5's initial release: The Database Converter.

This converter takes your IMatch 3.6 database and converts it into an IMatch 5 database.
Databases created by pre 3.6 versions are not supported directly. If a user has a 3.4 database (six years out of date) he better starts over anyway.

The converter works as follows:

1. If there are pending XMP updates, let IMatch 3 write them to the file / sidecar file.
2. Ensure that your IMatch 3.6 database is OK by running a diagnosis.
3. Create a new IMatch 5 database and close it again.
4. Run the converter. Select he IMatch 3 database to convert and the fresh IMatch 5 database as the target.
5. Let the converter run.

The converter imports, maps and converts:

• Files and Folders
• Thumbnails from the old format into the new format
• Categories, with all attributes, colors, short codes, order etc.
• Bookmarks (Bookmarked files are put into the Bookmarks collection)
• Custom sort orders for folders and categories
• Virtual Transformations
• Properties into Attributes

It cannot convert:

• Data-driven categories (the concepts are too different)
• Category formulas referencing persistent selections (no concept in IMatch 5)
• Off-line Cache items

After the database converter has completed its work, you can open your database in IMatch 5. IMatch 5 now has to load and import the metadata from the files via ExifTool as usual. IMatch 5 uses a different metadata model and storage schema so there is no direct way to import the XMP data from the old database.

After this step has completed, your database will be fully usable.
You can now create cache files as needed, or even force a rescan of files for which IMatch 3 had no support and thus no thumbnail etc.
If your database contains off-line files, you have to bring the files on-line once to load the metadata and create cache files if required.

The implementation of the Database Converter is almost finished. Currently I'm working on the Properties -> Attribute importer and converter. The thinking for that has been done already when I implemented the Import module so I don't expect this to take too long.

The I need to run tests with several sample databases, small and large. Currently I test against a special database which contains all typical and many fringe test cases). I will include the converter with the next regular Beta build (5.0.128). It is a standalone program and you can run it separately from IMatch 5 (on a system where IMatch 5 is installed because the converter and IMatch 5 share components).

Run imports with your existing 3.6 databases and report your findings. The converter opens IMatch 3 databases in read-only mode so no harm can be done.

As said in the intro, this is the last missing feature for the initial release of IMatch 5. All that's left is finding and fixing the big bugs and crashes. Then I will release IMatch 5 officially. I may then take a day or two off, and then continue with the open bugs and feature requests.

Screen shot of the current development version:



[attachment deleted by admin]

[sinus]

I'm currently working on the last missing feature for IMatch 5's initial release: The Database Converter.

Ahhhhh!!! 

Great. Hope you will end with a fine and great tool to converts DBs from IM3 to IM5.

[cytochrome]

Will the converter handle also the special case where IM3 is on an old PC and IM5 on a new one (same images, image tree, but different path ?

Francis

[Lord_Helmchen]

Cool! 

Do you have an idea how it takes to convert a DB? I know that you can't give exact numbers, but maybe like "same speed as importing same amount of photos".

I ask because I've just realized that my IM3 DB is close to 10 GB big.

[kiwilink]

Can't wait to see it Mario!!!

Don't forget the "Light Table" sort!!!  I have several hundred hours into these kind of sorts so I hope it all works so I can keep doing my light table sorts in IMATCH 5.

Mike

[minstrel]

Also interested in 'seeing' this in action!
...my concern regarding categories: how are user added files in IMatch3.6 going to be translated into IMatch 5.0?
Minstrel

With user-added files you mean what, exactly?
All files are user added. Or do you mean you have modified the file format configuration file to add formats not natively supported by IMatch?
If the file format is not supported by IMatch 5 natively, you will have to add them to your target database as well before running the import. The import does not create 'new' file formats. The files will be added, but with an unknown format type.

edit - ...yes, file formats.  Thanks for the 'pre-load them' tip prior to import...I was thinking of adding them afterwards.

[Mario]

Will the converter handle also the special case where IM3 is on an old PC and IM5 on a new one (same images, image tree, but different path ?
Francis

This is no special case. You will have to relocate the drive/folders as usual, after the conversion.

[Mario]

Cool! 

Do you have an idea how it takes to convert a DB? I know that you can't give exact numbers, but maybe like "same speed as importing same amount of photos".

I ask because I've just realized that my IM3 DB is close to 10 GB big.

No performance measures yet. It should be quite fast because it's mostly a straight DB->DB transfer with some conversions in-between.
The initial read of metadata afterwards takes the same time as usual, but needs to be done only once.

[Mario]

Can't wait to see it Mario!!!

Don't forget the "Light Table" sort!!!  I have several hundred hours into these kind of sorts so I hope it all works so I can keep doing my light table sorts in IMATCH 5.

Mike

Custom sort orders for folders and categories are migrated automatically.

[sinus]

Can't wait to see it Mario!!!

Don't forget the "Light Table" sort!!!  I have several hundred hours into these kind of sorts so I hope it all works so I can keep doing my light table sorts in IMATCH 5.

Mike

Mike, so after the answer of Mario
"Custom sort orders for folders and categories are migrated automatically." I guess, I can hear a big "phewww" of you 

To be honest, I have never really trusted the "Light Table" or maybe better sayed: I have had always a bad feeling.
Hence I created in 3.6 an extra field in the properties "order", and a small script.
If I made big reordering of my "Light Table" I let run the script and IM took every image in the actual order and filled numbers of each image in acending order, means

1
2
3
...

Hence I had in my images a "written value" in the properties, where I could always go back to reorder after this field in the "sort - order".

With such a method we could even create several "light order - sortings", by simply create more than one field and let write the order in different files, and you could in IM3 have several orders very easy.

Since I did not look deeper in IM5, I do not know, what is possible there, but my "personal ordering" with attributes is also possible - but maybe it is not necessary.

[Mario]

The Attributes in IMatch 5 allow you to continue with your custom sort orders. The Database Converter converts your IMatch 3 properties into IMatch 5 Attributes. Then you setup your own sort profile under Edit > Preferences > Sort Profiles to use your "sort order" Attribute. That's it, you can work with IMatch 5 as before with IMatch 3.

You could also use Collections like flags, pins, dots or bookmarks for sorting. Or you agree on one of the XMP tags to be used to carry a sort order value. If you work with other software which allows you to sort by XMP data, your sort orders will be available there too.

IMatch supports all of these workflows, of course. 

[sinus]

IMatch supports all of these workflows, of course. 

Yep, thanks, Mario, I know, IMatch gives us a LOT of possibilites! And I enjoy and use them ... well, some of them 

(BTW: sometimes I write instead IMatch by mistake with "IMacht" and then I think always at "die Macht sei mit Dir" ... (I do not know the original phrase for this StarWars - phrase, like "the power/might is with you") 

[medgeek]

@sinus:  http://en.wikipedia.org/wiki/May_the_Force_be_with_you

[kiwilink]

Sinus:

I am interested in the Light Table conversion because I probably use this in a way that most users do not.  As an example, I had pictures of family members that were given to me from aunts/uncles and various other relatives.  Some pictures I scanned (original old prints) and some were given to me in Jpeg.  Many of the pictures came to me without a date taken.  The scans created a big problem for me because I had to guess at the date and use a utility to assign a date to the scanned image.  This became a problem when I started to run into problems using an outside program to create a date.  The utility would cause my image to not be displayed in IMATCH.  Eventually I was having so many problems with hundreds of photos that I contacted Mario and he suggested I look into EXIFtool.  In the process, I decided the best way for me to sort my photos would be to use the Light table sort just as I would if I had hundreds of pictures on a real light table and had to try and put the pictures in a chronological order as best I could.  I was going to try and assign a "Create" and "Last Modified" date to each scanned image using EXIFtool but this turned out to be a big task.  So, the Light Table became my solution.  I have the name of each relative in a folder and I have done the best that I could to put these pictures in a date order by manually grabbing the photo in IMATCH and dragging it to a locations within the light table.

It has worked for me very well.  Maybe there is a better way but that is what I have done.  I'm very happy that Mario will be able to save all of my work in the migration!!!

Mike   

P.S.  I have attached a picture of what I use to separate the "Years" as I organize pictures on the light table.  I created these using Photoshop and I drag and drop pictures between these "Year Separators"

[attachment deleted by admin]

[sinus]

@sinus:  http://en.wikipedia.org/wiki/May_the_Force_be_with_you

Ahhhh, thanks a lot! 

[sinus]

Sinus:

It has worked for me very well.  Maybe there is a better way but that is what I have done.  I'm very happy that Mario will be able to save all of my work in the migration!!!

Mike   

P.S.  I have attached a picture of what I use to separate the "Years" as I organize pictures on the light table.  I created these using Photoshop and I drag and drop pictures between these "Year Separators"

Hi Mike
thanks for your detailled information, it is very interesting. This is surely a good way to do, what you want. And finally, if it works fine for your, so it makes a lot of sense.

Yes, it is very good, that you can migrate these sortings into the new IM5.

[Mario]

Database Converter is finished, sans testing.

Performance is good. For a 35,000 files database the conversion takes about 8 minutes.
The performance is about linear so a database twice as large takes twice the time.

The slowest part is converting the thumbnails from the old CMP format into JPEG used by IMatch 5. Unfortunately due to restrictions in the imaging component used to convert the CMP format this step cannot be parallelized.

I've also add the option to convert/copy an existing off-line cache. This will be welcome by users with large off-line archives.

The only things I cannot convert are:

1. Data-driven categories (concepts differ to much)
2. Persistent selections (no longer supported / needed)
3. Categories with formulas referencing said data-driven categories or persistent selections.

I hope to ship build 5.0.128 very soon. Hopefully before Christmas 

[sinus]

Database Converter is finished, sans testing.

Performance is good. For a 35,000 files database the conversion takes about 8 minutes.
The performance is about linear so a database twice as large takes twice the time.

Well, very good news.
It is welcome to have ROUGHLY an idea, how much time such a conversion must have.

In my case:
if you write 8 minutes for 35'000, then I count for my 160'000 files about
70'000 = 16 Min.
140'000 = 32 Min.
160'000 = 40 Min. ROUGHLY

So I add more time, in my case, because I have a lot of categories and properties, so I count
160'000 = 80 Min.

I add again some time, because I think, I have an old computer:
160'000 = 90 Min.

And at least, I give some more "space", I think, for the conversion of my DB with 160'000 files the time will be about 2 hours.
This is very good for a conversion, specialy if I KNOW the time roughly.
So, even it takes 4 hours, no problem for me.

I will know, once I started the conversion, I will have to wait some hours ... an I can do something else in this time.

Thanks, Mario.

[Mario]

My runtime data is now invalid 

I decided to add the metadata import to the converter as well. Until last night I had planned to just put the files into the queue and when the user starts IMatch 5, the metadata is read in the background as usual.

Then I thought that I could probably do it faster in the converter, where IMatch does not have to keep the user interface running and no user interferes ;-)
If this works out, the database produced the the converter will be 'finished' and ready to use - which would be nice.

To, for a database of 160,000 files, plan for an entire night (fast disks or SSD). I don't have IMatch 3 databases that large so I cannot tell how long it will take. It also depends on how much metadata is to process. Since you (I think) work with NEF files, PSD etc. there will be a lot of metadata to read. We'll see.

[sinus]

My runtime data is now invalid 

I decided to add the metadata import to the converter as well. Until last night I had planned to just put the files into the queue and when the user starts IMatch 5, the metadata is read in the background as usual.

Then I thought that I could probably do it faster in the converter, where IMatch does not have to keep the user interface running and no user interferes ;-)
If this works out, the database produced the the converter will be 'finished' and ready to use - which would be nice.

To, for a database of 160,000 files, plan for an entire night (fast disks or SSD). I don't have IMatch 3 databases that large so I cannot tell how long it will take. It also depends on how much metadata is to process. Since you (I think) work with NEF files, PSD etc. there will be a lot of metadata to read. We'll see.

Your brain is working perfectly (well, I guess, mostly  ), I use indeed NEFs, jpg and psd, some tiffs.

I personally do not worry, if the converter works the whole night, even longer, because this is a task, what we have to do once. First I will test with only some files, then about 5'000, and if all works, I will let the whole DB convert.

Time is not so important, in my case, but good results 

[Ferdinand]

Then I thought that I could probably do it faster in the converter, where IMatch does not have to keep the user interface running and no user interferes ;-)
If this works out, the database produced the the converter will be 'finished' and ready to use - which would be nice.

I hesitate to use the "O-word", but could this be an option?

[Mario]

?

[sinus]

Then I thought that I could probably do it faster in the converter, where IMatch does not have to keep the user interface running and no user interferes ;-)
If this works out, the database produced the the converter will be 'finished' and ready to use - which would be nice.

I hesitate to use the "O-word", but could this be an option?

Ferdinand,
I hesitated to ask about your sentence, because I do not understand, but since Mario seems also have troubles to understand you, I dare to ask: what do you mean with that? 

[David_H]

Then I thought that I could probably do it faster in the converter, where IMatch does not have to keep the user interface running and no user interferes ;-)
If this works out, the database produced the the converter will be 'finished' and ready to use - which would be nice.

I hesitate to use the "O-word", but could this be an option?

Ferdinand,
I hesitated to ask about your sentence, because I do not understand, but since Mario seems also have troubles to understand you, I dare to ask: what do you mean with that?

Guessing, it would be could this be left as an option so that the convert stops prior to importing all the metadata (then IMatch picks them up on first run - slower, but if the user doesn't have the luxury of leaving the computer unattended for a weekend, it might be more preferable).

If its going to be importing everything (so the database is ready to use), could it have the option of importing into existing database, and therefore picking up versions and buddy relations that were predefined, or is that going to be quicker to do with the finished database ?

[Mario]

The Database Converter works on empty databases only.
The settings you make to the database before you run the converter will be applied, this includes relation rules and stuff.

I have not tested every combination yet, and it may be that you have to run a relation refresh on the Database node after opening the database to make sure all your relation rules are applied. This is not always possible when applying them on a folder-by-folder basis, depending on how you have structured your relation rules - and some of you built rules which spread multiple media, go up folder hierarchies by several levels etc. Not every fringe case can be handled.

The Database Converter has been written to be run unattended. I have no plans to add options to skip parts of the conversion process or something like that. If Ferdinand meant "add more options to it" with his "O" comment. I won't. Keep it simple. The only choice you have is to whether you want to import off-line cache files or not.

Users with specific requirements or twiddly things they want to do can just create a fresh database and import their folders in the usual way. This gives them the full control over the process  - and my guess is that quite a number of users will do that anyway. The most important assets here are properties and categories - and these can be exported from IMatch 3 and imported into IMatch 5 using the Import&Export features. No database converter needed for that.

We start with the converter as it is now. If we find that many users want additional features, we can change it later.

[Ferdinand]

Sorry if I was too cryptic.  Yes, I meant having an option to stop the conversion before metadata import.  I thought that this would give people the option to open the DB sooner, if needed.  I was thinking of people with very large DB.  But it was just a thought, and keeping it simple will suit most users.

[DigPeter]

@Mario - I have 2 questions:

My IM3 categories are all hierarchical, so the converter will import these into IM5 in this form. Once imported, will these automatically be written to metadata as LR hierarchical subject keywords?

Those who have already run FP's 3.6 to 5 keywords migraton script on images in IM3 will already have LR hierarchical subject keywords. Will this affect the operaton of the converter ?

[Mario]

1. No
IMatch 3 has no @Keyword category. All categories will be imported as regular categories.

2. I don't know this script (I don't look at scripts written by users). But the Database Converter does not modify metadata at all. It adds the files to the target database, and then runs a normal metadata import. The same operation that is performed when you add folders to an IMatch database yourself.

[DigPeter]

1. No
IMatch 3 has no @Keyword category. All categories will be imported as regular categories.

Does this mean that the IM3 categories will not be written into @keywords?  This is where I would want to find them.

[Ferdinand]

@Mario - I have 2 questions:

My IM3 categories are all hierarchical, so the converter will import these into IM5 in this form. Once imported, will these automatically be written to metadata as LR hierarchical subject keywords?Does this mean that the IM3 categories will not be written into @keywords?  This is where I would want to find them.

As Mario has said, the converter won't put your categories into @Keywords.  That is why you ran the script - the script puts them in the the correct fields, so that when you open the DB after the conversion they're already there

Those who have already run FP's 3.6 to 5 keywords migraton script on images in IM3 will already have LR hierarchical subject keywords. Will this affect the operaton of the converter ?

I haven't seen the converter, but I am confident that there's no link between what it does and what my script did.  Which is one reason why I wrote the script - to do something that wouldn't be done as part of conversion.

Does this mean that the IM3 categories will not be written into @keywords?  This is where I would want to find them

If you've run the script with the correct options, they should be there now, waiting for IMatch 5 to find them.

[BenAW]

Does this mean that the IM3 categories will not be written into @keywords?  This is where I would want to find them.

It seems you have the concepts of Categories and Keywords mixed up somehow.
Have a look in the IM5 Help under @Keywords. Imo it is explained very well there.

[DigPeter]

Does this mean that the IM3 categories will not be written into @keywords?  This is where I would want to find them

If you've run the script with the correct options, they should be there now, waiting for IMatch 5 to find them.

Thanks Ferdinand.  I am just dotting the 'i's' etc.  Mario's statement seems to imply that IM3 categories will not be migrated to @keywords, so they will presumably go into 'user' categories.

I have found your script very useful and intend to use it for the final migration.  As far I am concerned, all I want from the converter is the migration of properties.

[DigPeter]

Does this mean that the IM3 categories will not be written into @keywords?  This is where I would want to find them.

It seems you have the concepts of Categories and Keywords mixed up somehow.
Have a look in the IM5 Help under @Keywords. Imo it is explained very well there.

Thanks Ben - I am quite clear about this.  I am just trying to ascertain what will happen.  Please see my reply to Ferdinand's last post.

[BenAW]

Mario's statement seems to imply that IM3 categories will not be migrated to @keywords, so they will presumably go into 'user' categories.

Of course Categories won't go into @Keywords. Categories in IM3 will be converted to Categories in IM5.
Statements like I quoted above give the strong impression that you mix up the two concepts.

[DigPeter]

Mario's statement seems to imply that IM3 categories will not be migrated to @keywords, so they will presumably go into 'user' categories.

Of course Categories won't go into @Keywords. Categories in IM3 will be converted to Categories in IM5.
Statements like I quoted above give the strong impression that you mix up the two concepts.

@keywords are categories - there is no "of course" about it. I do not know what Mario intends. I realise that @keywords reflect XMP keywords and I have been under the impression that this is the central philosophy of IM5 and that user categories are extras.  I want confirmation that IM3 categories will not be migrated to @keywords by whatever mechanism Mario devises.  If, as seems to be the case, this is so, I will not use the converter, but rely on Ferdinand's script.  I think that IM5 is an immensley flexible system and we should avoid dogmatism.

[sinus]

1. No
IMatch 3 has no @Keyword category. All categories will be imported as regular categories.

Does this mean that the IM3 categories will not be written into @keywords?  This is where I would want to find them.

From the help-file:
@Keywords vs. 'Regular' Categories
When you look at the special @Keywords category you should think about it as a mirror of the keywords stored in your files. When you add or remove keywords in the Keyword Panel, the @Keywords category will automatically update to incorporate these changes. All child-categories you see under @Keywords are contained in the metadata of your files. They are public.

Regular IMatch categories are stored in the database exclusively. This keeps them private because only users with access to your IMatch database can see these categories.


So, like Mario wrote: the IM3 categories will NOT be written into @keywords.

[Mario]

There is only one kind of category. No such thing as a "user" category.

1. The Database Converter imports all categories found in the IMatch 3 database into the IMatch 5 database.

2. The special @Keywords category is basically a data-driven category on the hierarchicalKeywords metadata tag - with some extra features thrown in.
IMatch 5 automatically builds this category from the hierarchical keywords imported from your files. And the hierarchical keywords are imported/created as explained in the corresponding help topic in detail, from existing hierarchical keywords and/or flat keywords.

3. If you 'simulated' keywords in IMatch 3 categories, these categories will also show up in IMatch 5.
4. If you use some sort of script which writes keywords into IPTC or XMP metadata, these keywords will show up under @Keywords.

[DigPeter]

So, like Mario wrote: the IM3 categories will NOT be written into @keywords.

Thank you Markus - I missed the word "regular".

[DigPeter]

There is only one kind of category. No such thing as a "user" category.

Thank you Mario.  By "user" I  was meaning "regular" - just my sloppiness!

Any chance of a converter that just migrates IM3 properties?

[BenAW]

I have been under the impression that this is the central philosophy of IM5 and that user categories are extras.
I think that IM5 is an immensley flexible system and we should avoid dogmatism.

Perhaps you should consider that your impression and reality are not necessarily the same thing.
At the moment I have 0 (ZERO) images in @keywords  and I'm still using all major features of IM5 with great results.

I have no clue where your reference to dogmatism comes from.

[Mario]

Any chance of a converter that just migrates IM3 properties?

You can export properties from IMatch 3 to a schema file and then import them into IMatch 5 via Edit > Preferences > Attributes. Works great, all properties are imported and the data as well.

[Mario]

PS.: No need for a heated debate. I'll release IMatch 5.0.128 soon enough and then you all can try out the converter and see what it can do.

And look at the nice logo I just created for the community...at the top 

[DigPeter]

Any chance of a converter that just migrates IM3 properties?

You can export properties from IMatch 3 to a schema file and then import them into IMatch 5 via Edit > Preferences > Attributes. Works great, all properties are imported and the data as well.

Oh - what a system.  Thanks for the tip.

[Mario]

Oh - what a system.  Thanks for the tip.

You may also want to check out the other FAQ entries here:

https://www.photools.com/community/index.php?board=17.0

[Ferdinand]

Peter

Technically, @keywords are not categories - they exist in the files / sidecars, although they are displayed as a data-driven category.  So they aren't converted by the converter, since they're already in the files and will be displayed as such in any IMatch 5 DB,

If you intend to use my script to convert all regular categories to @keywords and don't want the regular categories in your V5 DB, then as Mario said, there's no need to run the converter.  All you need to do is run the script, and export / import properties.

You may also need to set your metadata preferences correctly.  I'm travelling with my Ubuntu netbook, so I can't check what I mean exactly.  I will reply again tomorrow.