Data in IMD5 file: Complete file written at closing?

Started by Rene Toepfer, March 10, 2021, 07:12:02 AM

Previous topic - Next topic

Rene Toepfer

My IMD5 file has a current size of appx 20.3 GiB, containing 355k files. If I revise data, like adding keywords, face recognition etc, will the whole IMD5 file be rewriten while closing or just added new/revised data at the end of the file?

sinus

Wow, 355'000 files. Impressive.
I have only about 300'000.

The answer on your question I do not know.
Best wishes from Switzerland! :-)
Markus

Mario

The database system used by the IMatch engine for core disk storage is a relational database management system.
It has complex block allocation strategies, free block management, block reuse etc. It does never "rewrite" the entire database file.

New blocks are allocated at the "end" of the database file when needed (in bigger chunks), then used up as new data goes in. Blocks freed due to index updates or data being removed are reused.
This database system is also used by companies like Google, Apple, Microsoft, Adobe and others. For good reasons.

The only operation which rewrites the complete database file is the "Compact & Optimize" process. Which is supposed to.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Mario

Quote from: sinus on March 10, 2021, 07:32:48 AM
Wow, 355'000 files. Impressive.
I have only about 300'000.

My largest test database now has almost 900,000 files.
The largest database I see in telemetry has 2 million files.
The largest database I know of has about 4 - 5 million files (running on a dedicated high-end workstation in a research lab).
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Rene Toepfer

Quote from: Mario on March 10, 2021, 10:08:10 AM
The database system used by the IMatch engine for core disk storage is a relational database management system.
It has complex block allocation strategies, free block management, block reuse etc. It does never "rewrite" the entire database file.

New blocks are allocated at the "end" of the database file when needed (in bigger chunks), then used up as new data goes in. Blocks freed due to index updates or data being removed are reused.
This database system is also used by companies like Google, Apple, Microsoft, Adobe and others. For good reasons.

The only operation which rewrites the complete database file is the "Compact & Optimize" process. Which is supposed to.


Thanks for this explanation. To be honest, it is as I have expected it.  :D


Quote from: sinus on March 10, 2021, 07:32:48 AM
Wow, 355'000 files. Impressive.

Thanks, images since 1986.
I am really impressed how fast IMatch remains even with that much files and I can say it's the fastest DAM I have seen in my life.

Mario

Quote from: Rene Toepfer on March 10, 2021, 11:37:32 AM
I am really impressed how fast IMatch remains even with that much files and I can say it's the fastest DAM I have seen in my life.

Very good. Spread the word! Let others know about IMatch.

Word-of-mouth is the best marketing I can afford.
Adobe spends about 150 million US$ per year on marketing and 'endorsement'. This buys them a lot of exposure everywhere and makes it very hard for small businesses to attract attention.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook