Backup Strategy - Image Snapshot vs. consistent iMatch DB?

[ChrisMatch]

Hi Mario

Could you please help me to decide on choosing the right backup strategy?

_______________
The short questions:

Would an "image" backup of the iMatch-DB always get a consistent state without VSS (while iMatch is running)?
Would an "image" backup of the iMatch-DB get a consistent state when using VSS (is there a VSS writer for the DB used by iMatch)?

As far as I understood, it should at least work without problems when I start the 'snapshot' while iMatch is NOT running. So my question is if it would work even when iMatch is active (like suggested by the maker of DriveSnapshot).

_______________
The long story:

I am already using a file based backup (syncback) and I am investigating the addition of an image based backup.
DriveSnapshot seems to be widely accepted and getting good reviews and I love its simplicity.
One cool thing (and for me the main selling point) is, that you can...
"...   continue your work, while the Backup is in progress The new Snapshot technology ensures that all data are consistent, and reflect the PC's data at the start of backup. There are no difficulties with opened files."
(If someone is interessted, here is a description of how DriveSnapshot is working: http://drivesnapshot.de/en/backup.htm )

But as I found out, there are situations where that consistency of the snapshot can't be assured. Backing up Domaincontroller seems to be the worst case szenario but even on workstations (which is what I am talking about) there could be problems.

As far as I understand, it could happen that e.g. a database is writing parts of a transaction to the disk - then the snapshot is taken - and after that the database wants to write back the second part of that transactions (lacy write strategy of database).
To overcome this problem VSS can be used ( http://drivesnapshot.de/en/isnapshot_vss.htm#Details ) which raises the above questions!

Much seems to depend on how the DB is handling writing transactions to the disk.

So I see 3 possible scenarios:
1) I don't have to care about all those theoretical problems. The backup will work even while iMatch is running.
2) It is necessary to use VSS (which is supported by the iMatch DB) to get a consistent backup.
3) I have to start the backup while iMatch is NOT running. But could at least start working with iMatch after the backup has started.

Thanks
Chris

[Mario]

General rule: When dealing with live databases, use the built-in backup routines or shut-down the database before doing backups. So: Always close your database / IMatch before doing backups of the database file.

IMatch writes data to the database from multiple threads, almost all the time. The database system creates temporary journaling files in order to be able to rollback transactions, do post-mortem (post crash / system failure) recovery and all that. The state of these files at any given time is undefined, there is no 'right' time to backup an open database. Always close it to get a consistent state for backup.

[ChrisMatch]

Thanks for your reply.

Your recommendation doesn't surprise me
but is important to keep in mind.

Just to be sure: the DB used by iMatch doesn't use VSS writers - correct?

thanks
Chris

[Mario]

No. Just close IMatch, backup, done.
I think the only companies who ever supported VSS are Microsoft (SQL-Server, Exchange Server) and maybe Oracle. IMatch keeps all data in one single file, and you can easily close the application when you want to make a backup. Backup the single database file. Restart. Easy.

This is not 24/7 server land and you don't pay server-grade prices for IMatch...

[ChrisMatch]

Thanks again - no problem - just wanted to be sure 

[ChrisMatch]

(I just got an answer from the makers of DriveSnapshot and they confirm that you can
"safely backup what would survive a power failure at that moment").

Just to summarize the findings for others using DriveSnapshot (or a similar tool):

• At the moment you start Snapshot - be sure iMatch is NOT running.
• After a few seconds (when Snapshot has collected the necessary data and starts the actuall backup)
  it is save to start iMatch and work with it.

So I simply plan to start the Snapshot-Backup immediately after booting up the machine (or before shutdown).

[JohnZeman]

Although it doesn't surprise me I appreciate this information as well.  A couple months ago I switched from a file based backup system to Macrium Reflect (which has a new version out by the way) and it's required some rethinking on my part regarding the best ways to image/backup my hard drives.

[cthomas]

What is a file based backup? How can you tell what kind of backup system your using?

[Ferdinand]

What is a file based backup? How can you tell what kind of backup system you're using?

Tell us what you're using and well tell you, if we can.

Let's say you have a 5Gb file which is edited, like an IMatch database.  Let's also say that most of that file is unchanged, only small portions are changed.  A file-based backup will see that the file has changed and take a fresh copy of the whole file - consuming more backup space than is needed, given that not much has changed.

The alternative is to take a fresh backup of only the parts of the file that have changed.  This is often called a sector-based approach.  Only the disk sectors that have changed are backed up.  You typically take an initial full backup that acts as a base, and then a series of much smaller incremental backups, which record only the changes.  You need the base and all the increments to be able to restore a file or files or even a disk partition.

This might suggest that sector-based backups are always smaller than file-based backups, but this isn't necessarily so.  The operating system is often shuffling files around which the backup software sees as changed sectors, making increments larger than you'd expect.  Doing a defrag will often change a lot of disk sectors, and make the next incremental backup very much larger - often it's best to take a fresh base backup after a defrag.

Acronis True Image and Macrium Reflect are sector-based.  Something that takes incremental backups is likely to be sector-based.

[Carlo Didier]

"what would survive a power failure at that moment" says it all. You get the same status as if you PC crashed ... not clean and with a risk of data loss or corruption.

At work, we have snapshot technology for all our backups, BUT: for all databases, there are drivers with which the snapshot software works together with the database software which allows the db to flush all writes to get a consistent state when the snapshot is taken, essentially holding back all new updates during the snapshot (for ~1 second, the snapshot on the storage system is very fast).

[ChrisMatch]

It is a general knowledge that you are on the save side if you close 'all' programs before starting a backup (regardless of file based or sector based backups).
The interesting and confusing part was the promise of the makers of DriveSnapshot that you can "continue your work, while the Backup is in progress". That is true BUT there is an important trap: at the moment you START the backup all files must be in a consistent state on the drive - and for applications that use databases (like iMatch) this can only be assured if the applications are closed (or if the support VSS) - after that you can start the applications and continue your work.

[Mario]

The operating system is often shuffling files around which the backup software sees as changed sectors, making increments larger than you'd expect.  Doing a defrag will often change a lot of disk sectors, and make the next incremental backup very much larger - often it's best to take a fresh base backup after a defrag.

That's correct as long as no SSD is involved. For SSD's Windows disables defrag and the 'optimization' and re-arranging of files. At least in current Windows versions. If SSDs become larger and more common, it may start all over 

Something that takes incremental backups is likely to be sector-based.

Not necessarily. Incremental just means that not everything is backed up. File based systems can also make incremental backups, just backing up all files changed since the last backup. They can also do differential backups, like "imaging" backup software (TrueImage et.al.) do.

I personally use TrueImage for many years and with success. I could always restore, which is the key point for all backup solutions. Never trust a backup that was not tested for a proper restore. I also use simple built-in tools like RoboCopy (in scheduled tasks to copy files to other disks at regular intervals) and I use a version control system which stores by source code files into a protected cloud. And I use a combination of the German BoxCryptor software and SkyDrive to store other data in encrypted form. I never put anything in un-encrypted form into a cloud storage - not since Snowden.

[cthomas]

The operating system is often shuffling files around which the backup software sees as changed sectors, making increments larger than you'd expect.  Doing a defrag will often change a lot of disk sectors, and make the next incremental backup very much larger - often it's best to take a fresh base backup after a defrag.

That's correct as long as no SSD is involved. For SSD's Windows disables defrag and the 'optimization' and re-arranging of files. At least in current Windows versions. If SSDs become larger and more common, it may start all over 

Something that takes incremental backups is likely to be sector-based.

Not necessarily. Incremental just means that not everything is backed up. File based systems can also make incremental backups, just backing up all files changed since the last backup. They can also do differential backups, like "imaging" backup software (TrueImage et.al.) do.

I personally use TrueImage for many years and with success. I could always restore, which is the key point for all backup solutions. Never trust a backup that was not tested for a proper restore. I also use simple built-in tools like RoboCopy (in scheduled tasks to copy files to other disks at regular intervals) and I use a version control system which stores by source code files into a protected cloud. And I use a combination of the German BoxCryptor software and SkyDrive to store other data in encrypted form. I never put anything in un-encrypted form into a cloud storage - not since Snowden.

I also use True Image 2015. How does True Image compare to the ones talked about here? Also how does Cloud Storge compare with other types of storge?

[Mario]

Also how does Cloud Storge compare with other types of storge?

It's no replacement (yet) for local (disk) storage. I can upload with about only 1 MBit (~ 4-5 MB per minute) and for my daily backup volume of 10 to 30 GB it would take days to upload everything into the cloud...

Cloud storage has the benefits of being off-site (no danger in case of fire or theft), being (hopefully) professional backed up and convenient. Typical cloud storage works by mirroring a folder (hierarchy) from your local disk into the cloud. Automatically. Whatever you copy into that folder is uploaded into the cloud. If you change something in that folder, it is uploaded into the cloud. Easy. No-brain backup.

Problem with simple cloud storage (DropBox, SkyDrive etc.) are security & privacy (see Snowden) and more technical issues like archival and versioning. It's not wise to replace yesterdays backup with today's backup, in case you recognize a  problem in a week from now. You need daily, weekly and monthly backups and you need to be able to keep these for a longer time - so you can rollback to a file's version from two weeks ago if needed.

Most cloud services don't provide such features out of the box so you need some 3rd party solution for that. Duplicati comes to mind (free) or tools like rsynch which are command-line based and can be easily used in a scheduled task (better for tech folk).

[jch2103]

... it would take days to upload everything into the cloud...

The related issue is that downloading a backup, although usually much faster than an upload, will also take significant time (specifics depend on service provider speeds, volume of data, etc.). Some cloud services offer optional hard disk delivery for data restores, at a cost of course.

[Mario]

With SSD's and USB sticks getting bigger and faster by the week, many backup problems can be solved by that. I've just seen a 1 TB SSD with the size of a credit card and USB 3.0 interface. Goes up to ~ 400 MB/second (!) and at that speed and 1 TB capacity, you can do a lot of backing up. Two SSD's and you can manage several weeks of backup and keep one disk in another location to avoid risks theft or fire.

Unless you produce massive amounts of data, cheap 64 GB USB stick can hold lots of data, in several versions. I Pack & Go databases and IMatch settings on sticks, keeping at least five daily backups per stick. The typical backup volume of a normal user is only a few GB per day anyway.

Luring people into the cloud with cheap offerings has of course one iron-hard business reason: If you tie yourself to the cloud of a vendor, you barely switch if getting all your data from one cloud into the other is hard and risky. And so the big vendors, Apple being a leader, offer more and more cloud-only solutions so you cannot longer choose freely. Store locations of applications set by default to the cloud, marketing the "your data everywhere" idea (should be the more true "All your data in our cloud and under our control").

I don't go that route unless I have to. I also despise how smart phones with IOs or Android basically incapitate you. No more control over what apps do, no way to remove rights from apps after installing them, "security" systems which give you less and less control over where your data flows and which app is allowed to "phone home". I had a hard time recently to find a QR code reader which does not want access to my address book (!), browsing history (!) and full WLAN control (!!!). A QR reader - Jeezus. Rooting my old smart phone was the only solution to get control back. It's much faster now too, because all the crap software Google forces you to install is gone.

Control over my own systems and data is also the reason why I've chosen a Windows 8.1 tablet (full W8, not RT). All my software runs. And fast too. If I don't want an application to access whatever remote site, I just disable it in the firewall. And I stay in control of which software I install (no rubbish apps from whatever vendor), I can un-install what I don't want. What a difference to my current Android smart phone which gives me almost no choice and makes me trust Google - Pah!