Thread about 'Can I import my IMatch 3 database into IMatch 5?'

[sinus]

I think, this is a good decision.
IM5 is much more powerful than IM3, and hence you could change some things in your current workflow with IM5.

I would also say, we should "play" first with the IM5, and when we know IM5 better, we can modify our workflow better and finally import our "old" database. In my case I am even not sure, if I will import the properties into the new attributes.

[oldhank]

Excellent idea!

dnh

[NeilR]

I think it is a bad idea.

Part of beta testing anything is volume testing.  Without a migration aid now, fewer people will expend the time and energy to try to somehow else migrate their existing data and reproduce an environment at least approximating what they have now. 

That is the case with me.  I have about 350K image files cataloged in IM3.  I'm sure this beta version works well with small amounts of data.  The question is always what happens when it scales up, particularly with image catalog apps. 

I have many, many long standing performance issues with IM3. My main interest here is what, if anything, will change for the better with IM5.  Without a migration aid it may not be a very good use of my time to try to do it, and it will force me to do things I don't want to do with either my primary image collection, or a copy (of 4TB of images).  Such as trying to drive categories down into XMP files or the actual image files.

If Mario wants this thing volume tested in the beta phase he will expend the effort to do the migration code now, even if a few changes need to  be made down the road.  That will encourage volume users like to me to truly put this thing to a proper test in the real world in which it will be used.  Otherwise all that stuff will hit the fan after the initial public release, and that is the wrong time, for everyone's sake.

Just my 2 cents.

[Mario]

I don't do volume testing right now. The testers with the largest databases currently keep it about 100,000 files. It is always possible that a new beta version will require the database to be dropped and re-created.

350.000 files is an exceptionally large database and I expect that less than 1% of all users will ever work with databases that large.

I will implement and ship a database migration tool in due time, after my own schedule. It makes much more sense to implement this near the end of the Beta. And since you can already migrate your database (re-import all files into a new database, import categories and properties from your old database), you can do live end-to-end tests. Some minor things like virtual rotations are currently not supported by this process, but that's not too bad.

The Beta has not been tuned for performance in all aspects, and pushing your amount of images into it will do no good and reveal only that it has not been performance tuned yet.

In addition, the main goal for IMatch 5 was functionality and easy of use for standard-sized collections with 50,000 to 150,000 files per database. Over 90% of all users fall well within this range. A database with 350,000 files is way outside this limit. IMatch 5 should be able to handle it, more or less, as IMatch 3. But there is a price to pay for that amount of data. The metadata alone for this amount of images will use up gigabytes in the database, especially when these files are all RAW files. And this will naturally slow down everything.

I have many, many long standing performance issues with IM3

That's sad to hear. IMatch 5 will most likely not be faster in all aspects. And maybe even slower in some aspects, because it trades flexibility and ease of use over maximum performance. All the neat features like @Keywords, data-driven categories, versions, stacks and the like take their toll. IMatch 5 makes better use of memory, and fast SSD disks can speed up database operations by several 100% so there are ways to improve IMatch performance for such massive databases.

[HGF]

Hello,
I tested IM5 beta with a small db and for me it looks real good. Got no major problems. I've ~150.000 records (RAW, JPG) in the IM3 db with categories, rankings and labels. DB size ~ 5G.
For IM5 a created a new database including same records. Took about 10 h but it works . IM5 DB size 5G, too. Transfer of categories from IM3 to IM5 was easy  .
Question: How to import record (pic) related data?  (just IM3 categories, rankings and labels, no changes in pics META data, no xml)
Export into txt file in IM3 is easy. How to import in IM5? Maybe folder by folder.

"And since you can already migrate your database (re-import all files into a new database, import categories and properties from your old database), you can do live end-to-end tests"

Please let me know how to do as you mentioned above. I would really appreciate to be able to compare IM5 with IM3 working with my large DB.
Thank you, HGF


Habe nochmal einen Import der Kategorien durchgeführt, diesmal mit File-link.  Habe nun alle Kategorieinformationen sowie Labels transferiert. Einzig die Bewertungen sind nicht mit rübergekommen. Darauf kann ich im Notfall aber auch verzichten.
Gruß HGF

[Mario]

Question: How to import record (pic) related data? 

Which data do you mean?
Properties? See here.
IPTC, EXIF and XMP data is contained in the image and will be imported by IMatch 5 automatically.

[HGF]

Hi Mario,
I initiated in IM3 a category export again, this time I selected Schema+File links. Import into IM5 linked my records (pics) to categories. Also Labels were transferred. 
Just missing ratings.
HGF

[Mario]

Labels and Ratings are stored in the XMP record in your files (or sidecar files) and will be picked up by IMatch 5 automatically.
Unless you have set ratings in IMatch 3 but not yet written that data to disk. In this case the XMP data is only in the IMatch 3 database and not accessible for IMatch 5. You need to process pending updates first in IMatch 3.

[Buster]

I just downloaded 114 - it started with the offer to import a 3.xx Database.
First I created a new DB, killed it after testing and wanted to import my 3.6 Database then - but I couldn't find the offer again.
How can I import my old DB now?

[JohnZeman]

Welcome to the beta forum Buster.

If I remember right I think Mario has said that option won't actually be enabled until IMatch 5 is ready to launch.

For now you can do some exports in version 3 and import those into version 5.  Categories, version 3 properties into version 5 attributes.

Version 3 scripts have to be updated before they'll work in version 5.  There are a number of threads in this forum regarding importing things from version 3 to the beta.

[Richard]

Hi John,

You remember correctly.

I decided to implement the database converter near the end of the Beta. This allows me to be more flexible when it comes to make changes to the database due to bug reports or feature requests.

Also, since this is a Beta version, you should work with a separate set (copies!) of images anyway. You will be learning about IMatch 5, and when you make a mistake, you only risk a few copies of your image files.

[Richard]

I just downloaded 114 - it started with the offer to import a 3.xx Database.
First I created a new DB, killed it after testing and wanted to import my 3.6 Database then - but I couldn't find the offer again.
How can I import my old DB now?

Hi Buster,

I don't know what you saw but it could not have been an offer to import a 3.xx database. What you should do is copy some images to a new folder and use those in a IMatch 5 database. IMatch 5 is just a beta version and something could go wrong and testers would have to start over with a fresh database and thus all their work would be lost. That doesn't mean that you can't do some serious work in IMatch 5. It just means you need to be aware that it is a beta.

This thread has a lot of information about importing data from IMatch 3 and, as John stated, there is a lot more in other forums.

[JohnZeman]

I don't know what you saw but it could not have been an offer to import a 3.xx database.

Actually Richard, I think someone mentioned this same thing a few weeks ago.  That the option to import a version 3 database only appears one time when you're just starting without a database but even if you try to do it, it won't work yet.  I may be wrong about that, but I seem to recall something along those lines being mentioned awhile back.

[Richard]

Hi John,

I have started fresh several times and I didn't see it but I wasn't looking for it either.

[JohnZeman]

Nor have I Richard, but like you I wasn't looking for it.
I'm pretty sure Buster isn't the first one to mention this though.

[Richard]

When I thought more about it I sort of recall someone asking about it. Now I am wondering why you and I have not seen it.

[Connacht]

Hello,

I've just downloaded the beta version and I'm keen to see the new features and functionality - it looks good.

I agree with your restriction of the import function until Bets testing nears its end.

I note with interest your comment of the 8th July:

Quote:

350.000 files is an exceptionally large database and I expect that less than 1% of all users will ever work with databases that large.

My database holds over a million images and I guess I really ought to have created separate databases before letting it grow so much

I'm assuming you wouldn't recommend importing this to version 5 whenever  this function become available?

Thanks,
Connacht

[sinus]

I note with interest your comment of the 8th July:

Quote:
350.000 files is an exceptionally large database and I expect that less than 1% of all users will ever work with databases that large.

My database holds over a million images and I guess I really ought to have created separate databases before letting it grow so much

I'm assuming you wouldn't recommend importing this to version 5 whenever  this function become available?

Thanks,
Connacht

I think, some years ago, 350.000 files was really a large database.
But I tend more and more to think, nowadays, this is not that much. Today to hold about 1 million files (including txt, docs, movies and so on) is not that seldom, I guess.

A friend of me, not a professional, was on vacation in Venice for 3 days and came back with ... over 4'000 images! A bit crazy, but because we can with cams simply push a button to get (mostly) a technical good image, the pile of files grows faster then ever.

I am interested, what Mario says to your question.

[Connacht]

Hello Sinus,

Thanks for reply and I'm glad to see I'm not the only one

4K images on holiday, that's mad, he must have been snapping constantly!

I do the same myself, but not in such volume but try to be quick about the editing process when I get home and delete the photos of lesser quality etc.

I agree though, with digital imaging the volume has grown out of hand these last few years.

Connacht.

[Buster]

@connacht: Long long ago. In former times it took four or five films for a handball match - and the sharp AND good one was printed.
Nowadays you have lots of tasks in your bag: Trainer, screaming fans, cheerleaders, up to ten different players in action, and so on.
Result often are more than 1000 pics from one match. And I only use RAW .
Not long ago I thought of  the money I would have spent on films during the last 14 years, if I had taken all these pics as well - 100 000 pics per year, a films counts 36. That is in 14 years 1 400 000 pics. Divided by 36: 38888 films - multiplied with 5 Euro: 195 000 Euro!

[DigPeter]

My database holds over a million images and I guess I really ought to have created separate databases before letting it grow so much

I cannot get my mind round that!  I have a modest 40,000.  If you do not have your images already categorised in a database, I reckon you have so much work on your hands, if you want to do this in IM5, that you will not have time to take more potos.

[Connacht]

Hello DigPeter,

It seems a lot does it not, but I've been collecting digital images since 2004.

I could of course trim this but one always gets the feeling one might regret deleting something later.

I believe you're right, I have a lot of work on my hands.

I've created two databases v5 and managed to corrupt both using one folder for testing purposes!

Not an auspicious start but time will tell.

Connacht

[Mario]

I've created two databases v5 and managed to corrupt both using one folder for testing purposes!

IMatch 5 is in Beta test. I give users access to the software in order find out about such bugs and problems. I consider this a privilege and a call to help making IMatch 5 better. When I read comments like this, I wonder if the concept of a Beta test has been truly understood by everyone participating in this test  If users just ignore problems or 'start over', they will not help me or other users. Only when we get notice about such problems, we can make IMatch a better software.

In a case like this, where you managed to apparently cause damage to your database just like so, I would really much have liked to see a proper bug report.

As explained in How to report bugs and in the IMatch help (Beta Tester Guide), IMatch includes a number of features to help me diagnose such problems: The Application log file, the Database Diagnosis, the ExifTool Output Panel, the DUMP logging facility in case IMatch crashes.

Corrupted database are really rare. A corrupted database has usually been damaged by a disk or network problem, a power failure or similar. If the built-in recovery in the database system fails to recover a damaged database, we speak of a corrupted database. And I need to know about the exact conditions under which the corruption happened.

Without your input about what you did, what exactly you mean with a corrupted database, the log file from that session and a database diagnosis run I will never know about the problem you've ran into. And without knowing about it, I cannot look into it and fix the problem is this was caused by a bug in IMatch. So help me by providing the info, best attached to a bug report in the bug report board.

[Connacht]

Hello Mario,

Apologies, I wasn't addressing you directly, this was a comment to DigPeter, but I am aware now that it might have some across as such.

I do understand the concept of beta testing.

The first log was 46MB and I couldn't mail it and on the second incident I just scrapped the database, and subsequently the first one also.

Yes, this doesn't offer any help to you or others users' - I agree - but I was only playing around with the application getting the hang of its potential and admit I haven't taken the testing "seriously" as yet.

I will start afresh and log any issues once I start adding more folders etc.

Thanks for your comments, I'm sure you found mine exasperating.

Connacht

[Mario]

I maintain a dedicated FTP server for users who need to upload databases, DUMP files or large log files and who have no cloud storage available.
Just contact me via my support email (see link below) and I''s send you a user name and a password.

[Buster]

Nothing happened for hours - about 3800 files in the queue.
Log is attached.
Right now I'll start IM5 again to see what will happen.

[attachment deleted by admin]

[Mario]

Please don't post information about the same bug in multiple threads, unless you want to complicate things for me.
You did already report this here:

https://www.photools.com/community/index.php?topic=1165.msg7093;topicseen#msg7093

Or am I wrong?