New Database with > 400000 pics?

[Buster]

I'm planning To create a new base with more than 400000 pics.
Shall I risc this or shall I wait for the first final IM5?
Reiner

[Frank]

I have done it with more than 40000 pics it lasts approx 12 hours thats all.
It worked fine

regards
Frank

[Gerd]

Hi Reiner,

I have started in August this year and IM5 v.104 with 140.000 pics and now we are 8 versions further and through all crashes and stops and canceling I never lost a pic ... BUT have a copy of all of your pics ... for safety!

[Mario]

This is still a Beta test. The purpose is to find bugs and test IMatch 5 on as many computers as possible.
The current version is not performance-tuned in some areas, and adding 400,000 files will stretch things quite a bit. It's a test, though. Prepare for quite some ingest time. Run a Database > Tools > Compact from time to time to de-fragment the database file on disk (you can do that anytime, even when IMatch is ingesting files).

Managing 400,000 files with a < 100 US$ software is quite a challenge. And many of the cool things IMatch 5 can do, from collections to data-driven categories will naturally become slower the more files are added to the database. The largest test I've made so far is 200,000 files. It still works, but depending on which features you use, things may become slow. 400,000 files x an average of 150 metadata tags per file makes 60,000,000 metadata data values alone for the metadata storage. Indexing 60 million records, performing searches on that amount of data etc. will be necessarily slower than for a 100,000 files database.

The typical IMatch user has between 50,000 and maybe 100,000 files. Maybe splitting the 400,000 files into two databases a 200,000 files will be the better decision in the long end. I have no hard numbers for databases with 400,000 files yet. But for IMatch 3, which did a lot less with metadata and thus required less computing power, this would have been a stretch.

[Buster]

@mario: My 3.6 runs quite well with some 500000 pics. The only thing I use it for is to find files with special keywords and filenames quickly.
Do you suggest to use my 3.6er instead an IM5 beta?
Reiner

[Mario]

IMatch 5 is still a Beta. Nothing has been finally tuned yet - and I will start with that only when all breaking bugs are fixed.
IMatch 3 is a very mature product which has been through iterations over iterations of stability and performance improvement cycles.

If all you need to do is to find files by filename or keywords, IMatch 5 will be much faster than IMatch 3 - once the files are ingested. The full-text index system in IMatch can search 400,000 records very fast.

[BenAW]

I'm planning To create a new base with more than 400000 pics.
Shall I risc this or shall I wait for the first final IM5?

As long as you realise that your dbase may have to be re-created some time in the future when Mario changes the dbase structure, this would be an interesting test.
If I had that many images I would split the ingestion in 2 or even 4 batches, and not touch the computer while IM5 is ingesting files.
I also would select the Cache Working mode to OFF (or on demand). This speeds-up the ingestion considerably.

[Buster]

I will test the other week and tell you. Thanks a lot!
Reiner

[Buster]

400000 pics are a lot,  creating different bases was not what I intended.  So I decided to delete 200000 files.  From 2001 I used to take photos in raw data and saved every developed jpg in print and web size.  If you compare the results of raw converters from 2001 and today, you may understand that  I killed the jpg till 2010 and only kept the raw.  If I need some results as jpg, tiff or psd, it will take only a few minutes - IM5 will help me to find the originals in a mo.
I created the new base this afternoon from an USB 3 hdd. IM5 now says eight hours left.  I am looking forward to working with my ne and quick database after years of IM3, which was really great.  :-)
Best wishes from Kettwig,
Reiner

[Buster]

Oh, oh - patience ist not one of my attributes. Seems that I have to wait another 16 hours for the end of caching and indexing all 201000 files.
Best wishes,
Reiner

[Mario]

Since you  might need to delete your database before end of the Beta, I suggested in the Beta Tester Guide that creating cache images should be set to "on-demand" to save time. Creating cache images will slow down the ingesting phase considerably, especially if many "slow" file formats are used, or the WIC codecs cannot deliver a preview image with the minimum size and IMatch has to render the full RAW via the codec. I have seen the Canon and Nikon codecs take 10 seconds for one image in that case...

[Buster]

You're perfectly right - but I couldn't stop the runing process without riscing to start completely new.
My fault.
Are you sure, that the database has to be deleted at the end of the Beta?

[JohnZeman]

Are you sure, that the database has to be deleted at the end of the Beta?

You should count on it.  However if you export your categories, layouts, metadata, and other configuration when a new database needs to be created all you should need to do is to import the images again and then import your categories and other settings.

[Mario]

You're perfectly right - but I couldn't stop the runing process without riscing to start completely new.

You can shut-down IMatch any time, even when it's ingesting or writing back files. It will pick up there when you restart it.
You can also change the Options while IMatch is ingesting files, especially the Cache settings.

Are you sure, that the database has to be deleted at the end of the Beta?

Maybe. I explained this in the initial postings here, and also in the Beta Tester Guide in the help. A breaking change may cause all databases to become invalid and new databases need to be created. We are still running true Beta test here. Keep that in mind.