WholesaleBackup Blog: The Latest Online Backup News and Opinion

Divinsa LLC’s WholesaleBackup software client, since it’s first version, has utilized the Microsoft Volume Shadow Copy Service (VSS) framework for backing up open files in what Microsoft terms as a “crash consistent state” (i.e. backups of all open files will be at a specific moment in time consistent with what has been written to disk at that time, as if the computer crashed at that point).

The WholesaleBackup client software now supports, in experimental mode, the VSS Component API for backing up, in a safe integral state, collections of files and folders associated with some application Component, such as Microsoft Exchange and Oracle.

The implementation of every VSS Component Writer was done by the application vendor itself (for example Microsoft, Oracle, VMware, etc.), thus the actual file backup process provided by a VSS Component is implemented by the actual author of the associated application.  Backup utilities that support Microsoft’s VSS interface are called Readers and are abstracted from details which the application vendor’s specific backup procedure requires, thus allowing a well written Reader to backup dozens of different applications using a single method.

The WholesaleBackup software client is now a VSS Reader that makes standardized calls to the VSS API, and thus WholesaleBackup can now backup any application, including some operating system functions supported by Microsoft, which have implemented a VSS Writer for their files.  WholesaleBackup’s VSS Reader backs up files in “full copy” mode by default, or optionally "full mode", and then WholesaleBackup will either use full or differential methods to transmit those files to the backup servers using the method you have specified within WholesaleBackup (the default is differential, a.k.a. delta block, to minimize the client’s Internet bandwidth).

In implementing account based storage quotas we faced two alternatives, the first was to use native quota managed packages in Linux itself (see http://www.yolinux.com/TUTORIALS/LinuxTutorialQuotas.html for example) or to implement alerting and management at a higher level.  To ease the burden on support and server administrators we elected to implement quotas at our application level rather then utilize the file system level.  This has resulted in a simple and robust implementation which you can customize!

WholesaleBackup's implementation is straight-forward and open source.  If you enable a quota for a backup account, it is checked each time a backup transaction occurs, and if the user is over the quota you set, your backup server administrative alert email account (which you specified at installation time) is sent an email indicating that quota has been exceeded for that user.  From the WholesaleBackup web portal you can easily edit the quota value, frequency of alerts, and see when an alert was last sent.  In addition, all alerts are logged for future reference.  

You can easily extend our implementation, since we've provided the schema for the underlying database as well as the simple script which checks for quota conditions and sends email alerts, to perform more sophisticated processing such as suspending accounts and emailing clients directly.

We’ve received a number of inquiries lately asking about scaling and redundancy (load-balancing and high-availability).  The WholesaleBackup Server Manual covers these topics, but we felt a high-level overview would be a good topic for a blog entry.

A decent quad-core server should be able to handle several hundred backup clients.  Disk I/O performance is the single most important factor in choosing hardware, in particular the server’s storage devices (both RAID controller cards and the physical disk drives) ability to handle multiple simultaneous readers and writers is critical.  We suggest you think ahead when making hardware purchases and configuring it.  For example, RAID10 is typically much quicker to rebuild multi-terabyte arrays then RAID5 or RAID6, so in the event of a disk failure (which is inevitable) you’ll be much happier with RAID10.  In addition, SAS drives typically handle multiple simultaneous readers better than SATA drives and SCSI and eSATA interfaces are much faster then USB.

If your server’s disk controller card and chassis will support it, you can add additional disk drives and make them available in Linux at any time.  If you run out of local storage capacity, it is straightforward to setup a Linux NAS or SAN and remote mount that network storage via NFS to your backup server (the file systems need to be Linux formatted).

Our backup server will automatically backup it’s key meta-data to the /backup/ directory, which you can then backup or sync to another device.  We also provide a script that automatically replicates this data to an Amazon EC2 Cloud server (the script will fire up the server, replicate the data, and then automatically stop the server to save you $s).  It is straightforward to modify this script to replicate to a server of your choosing.

The WholesaleBackup Server is architected for scaling and redundancy and can be thought of as having four major components:

• A MySQL database containing meta-data about every account’s backups and restores.  This is not mission critical data for a customer to backup and restore, rather it is created for you in supporting your clients and their ability to access their backup logs from a web portal.  This database is typically configured to run on one of your backup servers though it can be run on a separate machine.  If you desire you can custom configure replication of your MySQL database to an additional server using standard MySQL and Linux replication methods (there is nothing custom about our implementation which would preclude this).   If general your clients will be able to backup and restore, even if this database is offline, although we do support some load-balancing schemes that require to database to be online to perform optimally.
• A web-portal consisting of PHP and HTML pages which connect to the MySQL database.  This portal is specifically created for you to support your clients and for them to be able to access their backup logs when away from the software client.  Typically this web-infrastructure runs on the same server as the MySQL database, though that need not be the case.   Multiple servers can host these pages if you wish to implement high availability for your support web portal (though this is not requqired for your client’s backups and restores).
• The storage device(s) that hold the actual customer’s data, logs, and meta-data.  The devices can be local to the backup server(s) or remotely attached via NFS.  If you are using multiple storage devices, our support web-portal allows you to move customers between these devices.
• The actual backup server(s) running WholesaleBackup Server software.

A typical scenario then is for new WholesaleBackup Server customers to setup a single backup server with all four components on it, and then be sure that the /backup/ directory is safely backed up (or synced) to a separate device.   As one adds more users one may add more storage (either local or remote) and when the server’s processing ability (RAM and CPU) become maxed out after several hundred clients, one then adds a second server, and a third and so one. 

So the question then is, how are multiple backup servers supported?  Conceptually you have a few choices:

1. Setup different backup server implementations (such as backup1.mydomain.com, backup2.mydomain.com, etc.) and create unique software client installers for each where you manage where each client backups up to.  This scenario requires the minimum amount of configuration effort on your part but there will be a separate support web-portal and database for each backup server.  This is a non-load-balanced implementation.
2. Each backup server supports backups for certain client accounts, specifically those whose underlying storage device is mounted on that server.  So, when a client makes a request to WholesaleBackup to connect to your infrastructure our load-balancer makes a secure http call to one of your servers to request the network destination for that specific user, which is then returned back to the client so they can execute their backup/restore operation with the server that has access to their data.  This approach is most commonly used in Cloud implementations (which often have limitations on the size and number of volumes you can cross mount and which suffer from the Linux kernel bug described in #3 below) or when you want to minimize cross-mounting ‘slow’ network storage volumes or devices that aren’t fast enough to handle many simultaneous readers and writers.
3. Each backup server can provide backup and restore capability to every one of your clients, so each server must have all your storage devices mounted (either locally or remotely via NFS) so every backup account’s data is visible to each backup server.  This approach is most commonly used in implementations on your own hardware where you have extremely fast SAN or NAS devices with cross-mounted network storage volumes and is often the best if you are striving for ISO of Six Sigma compliance. In this scenario one needs to direct each client to a backup server either via:
   1. A simple DNS round-robin scheme or a dedicated load-balancing device (such as a box running pfsense).
   2. Sophisticated load-balancing mechanisms that look at the load of each backup server to determine where to direct a backup client (this is sometimes scripted with BSD machines running pf or by using a commercial load-balancing device such as made by f5 or other vendor).
   3. Please NOTE that there is a bug in the Linux kernel, which has not been fixed as of January 2012, with NFS buffers over-running and causing server crashes; we have seen this bug manifest itself on Amazon EC2 instances in high backup traffic situations where NFS mounts are being accessed from multiple backup servers simultaneously; if you think you may scale to this level and you are using EC2 servers we suggest you utilize scenario #2 above instead of this method of load-balancing.

In each of the above scenarios, the backup servers will need to be configured appropriately such as for Linux account replication if required, access to your MySQL backup server, potentially accepting load-balancing calls from your load-balancer or the one which WholesaleBackup provides, etc.  Given the intricacies of configuring and testing enterprise level load-balancing and high-availability schemes with multiple backup servers, you’ll need to enter into a professional services engagement with a trained WholesaleBackup engineer to configure and support load-balanced configurations #2 and #3 above.  

To get started, we suggest you setup a single backup server, knowing that our solution will scale as your business scales.  When you've hit several hundred users and exhausted the resources of your first server then it's time to add a second server and make some decisions as to whether to go with option #2 or #3 above (you can change between them so no decision is permanent)...

The following graph show’s four under-utilized backup servers (e03 – e06) plus a MySQL/web-server (e01) in production in the Amazon EC2 Cloud using load-balancing scheme #2 listed above.

If you're using our software, you DO NOT NEED this script ;-) 

That's because our software uses VSS properly to backup Outlook files, even though many other top-level backup applications fail to backup Outlook pst files. But if you're trying to cobble something together, as was a recent poster on Spiceworks, then you may need this advice. So here it is:

How to Safely Close Outlook Before Running Backup on a PST file

1. Create a visual basic script (let's call it outlookcloser.vbs) in a directory, with the following code:
   

   On Error Resume Next
   
   Set Outlook = GetObject(, "Outlook.Application")
   If Err = 0 Then
   Outlook.Quit()
   Set Outlook = Nothing
   End If
   Wscript.Quit

2. Create a batch file (let's call it StopOutlookBatch.bat) in the same directory, with the following contents, except that you need to ensure that the outlook file location matches yours, and that the backup destination (c:\outback\outlook.pst) is the right one, and exists:
   

   @echo off
   REM run the command to gracefully close outlook - THIS SHOULD BE IN THE SAME DIR!
   cscript.exe outlookcloser.vbs //B //nologo
   choice /T 10 /D y > nul
   REM copy the outlook file - VERIFY THAT THIS IS THE RIGHT FILE FOR THE PROFILE!
   robocopy.exe C:\Users\USER\AppData\Local\Microsoft\Outlook\ C:\outback\ outlook.pst

3. Run the batch file, which will stop outlook, then copy the file to the specified location, from where you can process it further as required.

You would think that pricing your online backup service would be simple: determine your costs, add a reasonable margin, and viola!  Increasing price competition from cheap commodity online backup vendors complicates the exercise, though. Naturally these services want to advertise the strength of their offerings and provide a service to many consumers, but the scale of their advertising has created great confusion in the marketplace against which a dose of reason must be injected.

I was reading a great post by the Harry McCracken at Technologizer: Mozy’s New Pricing is a Small Price Hike, or a Big Price Hike, or a Price Cut, Depending on How You Look at It, which linked to some scary reference pages that talk about some of the un-marketed facts about the cheap online backup solutions. Customers will hear that they can get "Unlimited Backup" for a few dollars per month, and compare your price points to that offer, but this is really comparing apples to apple seeds!  Many people ask us about the differences between what a good online backup service will offer vs these cheap commodity online backup services, so here is the summary list, followed by some discussion on each point below:

• Business-class upload speeds
• Unlimited upload speeds vs throttling
• Fast restores
• Live and responsive customer service
• Setup assistance
• Proactive backup monitoring and alerting by a tech support person

Business-class upload speeds:

According to the Carbonite website reviewed 2011 August 29th, Carbonite limits upload speeds. If I understand their webpage correctly, they say that all uploads are limited to 2mbps. My home connection has upload speeds of 5mbps, and our test backup server regularly sees individual customers at 50mbps. The difference in backup times can be staggering: the same upload that takes 1 hour at 50mbps will take over a day if you backed up to Carbonite! Think about this: limited to 2mbps, your backup wouldn't get done fast enough to do daily backups. But that's not the worst news: In bright red text the page says that those speeds are the maximum speeds that can be achieved, which I am guessing means you won't see those speeds unless conditions are perfect for fast uploads. 

Unlimited upload speeds vs throttling:

The same webpage says that Carbonite limits upload speeds for customers that have larger datasets. While you start at up to 2mbps only, that drops to 512kbps if you hit 35GB, and again drops to 100kbps if you hit 200GB. Just for the sake of clarifying this, the same dataset that takes 1 hour at 50mbps will take roughly 21 days at 100kbps! To make this more real, consider that the dataset we're now talking about would include changes of roughly 21GB per day. Many of our partners have customers who have individual databases that are larger than that, and must be backed up daily. Plainly, commodity online backup can't backup that kind of data. 

Fast Restores:

Not only does WholesaleBackup provide technology to its partners that can start downloads of restored files immediately without having to rebuild files during the restore operation, but our partners usually have faster potential download speeds than the low Carbonite restore bandwidth of 10mbps for restores. I will let you do the math on this one, but its pretty simple: you should charge your customers a fair amount for your service because they will get their data back faster from you than a commodity online backup service. Average households in the US have doubled their bandwidth in the last year to 7.5 mbps according to a recent CISCO report, and are expected to grow to 28mbps by 2015. Now imagine that we start talking about business backups rather than just household backups. It's clear that businesses have better internet connections than the average household, and also clear that for a large restore, you don't want your downloads limited by the backup provider!

Customer Support (Lumping the last 3 bullet points):

An informal poll of several of our partners' call center data shows that an average backup customer gets about 3-5 hours of customer support per year as part of their subscription. This may be for help during installation, dealing with issues with their internet connection or computer not backing up because the hard drive is full or has viruses, and also assistance with restores. At a typical IT consulting rate of $150/hour, that is a value of $450-750 a year which your customer would have to pay if they went with a commodity online backup vendor. If you add the $750 on top of say $50 a year, the backup service now costs $800 a year or $70 a month, which isn't all that competitive anymore. 

Add to this the datapoint that by far the biggest issue with online backup are the kind of errors that actually compromise the machine so it stops backing up, and the ugly issue of who watches your backup raises its head. If nobody is making sure you're backing up, your backup is worth nothing. And backups, like anything else, can fail. So paying for a service that monitors your backups and calls you when they're not working, or just fixes them, is worth a great deal of your customers' time, because if they spend 5 minutes a day checking it, that adds up to 31 hours a year, which even at the average wage of $21/hour adds up to another $651 a year. So now our cheap backup service costs: $50 + $750 + $651 = $1451, plus offers slow restores and limited uploads.

If that still sounds like a good deal to your customers, send them my way as I have a few bridges to sell!

So How should you price your service?

Oddly enough, after all the above math, I would still advise you doing the following: figure out how much your costs are, and then determine a fair markup, and charge that for your service. If you hear complaints on price, use the above to help your customers realize that when they see a cheap price, it often means they get a cheap result. When you compute your price, think about the following:

• your time to administer the solution
• tech support time to help customers
• bandwidth fees for the service
• computers required to run the service, using whatever amortization schedule you use internally
• Annual replacement goods like hard drives that you might replace annually
• Hosting fees (or electricity, rack space opportunity costs, and network)
• Your time required to do billing and service billing questions

One of our partners asked me today about how to backup the EBS volumes running on their Amazon AWS instance by automating snapshots, like in this discussion: http://aws.amazon.com/articles/1663?_encoding=UTF8&jiveRedirect=1.

In our case we've got to backup a MySQL database, plus some other directories, and we want to snapshot those. Since MySQL's files are not necessarily going to be in a consistent state through the snapshot window, we should also backup the database. Let's tackle that first.

Backing up the MySQL database with a cron job, and setup crontab lines for daily, monthly, and weekly snapshots:

1. use crontab -e to edit your crontab
2. Add a line to backup the database every day at 10:12am (this is usually a low load time for online backup), and a line for daily, weekly, and monthly snapshots:
   

   12 10 * * * /usr/bin/mysqldump --add-drop-table -uroot -pSHHHHHHHH divinsamgmtdb > /var/backup/divinsamgmdb.sql
   
   17 13 * * * /root/snapshotscripts/managesnapshots daily >/var/log/dailysnapshots 2>&1
   
   20 13 * * 0 /root/snapshotscripts/managesnapshots weekly >/var/log/weeklysnapshots 2>&1
   
   23 13 1 * * /root/snapshotscripts/managesnapshots monthly >/var/log/monthlysnapshots 2>&1

3. Make sure you have a directory called /var/backup and that the drive has enough space.
4. Verify that /var/backup is actually on your / (root) partition as that's the one we're going to backup using Amazon snapshots.

Install Amazon EC2 API Command line tools on your Ubuntu server

This is covered on the web in many places, so here is a great link to walk you through it (scroll down a bit): https://help.ubuntu.com/community/EC2StartersGuide

Deploy and customize the following script to create snapshots:

1. Create a file called /root/snapshotscripts/managesnapshots and paste the following script into it:
   

   #!/bin/bash
   
   		# backups and and deletes given a dailysnapshotlist (or monthly or weekly - make sure you run the cron accordingly)
   
   		
   
   		# Set env variables for Amazon as cron may run without them
   
   		export EC2_PRIVATE_KEY=/root/.ec2/YOURFILE.pem
   
   		export EC2_CERT=/root/.ec2/YOURCERT.pem
   
   		export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
   
   		
   
   		#if no $1, exit
   
   		if [[ $1 == "daily" ]] || [[ $1 == "monthly" ]] || [[ $1 == "weekly" ]]
   
   		then
   
   		  export runtype="$1"
   
   		else
   
   		  printf "Usage: managesnapshots daily|weekly|monthly\n"
   
   		  exit 1
   
   		fi
   
   		
   
   		#set date time stamp for temp files
   
   		export dts=$(date +"%m-%d-%Y")
   
   		
   
   		while read volume_id description daily weekly monthly
   
   		do
   
   		  #exit loop if no values - otherwise get infinite loop
   
   		  [ -z $volume_id ] && break;
   
   		  # skip lines with comments in them
   
   		  if [[ $volume_id =~ ^#.* ]]
   
   		  then
   
   		    true
   
   		  else
   
   		    #add runtype into the ec2 description so we can search for snapshot types for deletions later
   
   		    export description=$(echo "$description $runtype")
   
   		    #add 1 to the keepdays so the tail command get s right number of rows ($$ uses variable substitution)
   
   		    case $runtype in
   
   		     daily)
   
   		       export keepdays=$daily
   
   		       ;;
   
   		     monthly)
   
   		       export keepdays=$monthly
   
   		       ;;
   
   		     weekly)
   
   		       export keepdays=$weekly
   
   		       ;;
   
   		     *)
   
   		       exit 2
   
   		       ;;
   
   		    esac
   
   		    export tailnumber=$(($keepdays + 1))
   
   		    if [ $keepdays != 0 ]
   
   		      then
   
   		      # create the snapshot for today
   
   		      echo "snapshotting $volume_id -- $description"
   
   		      /usr/bin/ec2-create-snapshot "$volume_id" -d "$description"
   
   		      #delete the older shapshots now
   
   		      #get the snapshots that are older than the ones we want to keep
   
   		      printf "checking for snapshots to delete from $volume_id\n"
   
   		      /usr/bin/ec2-describe-snapshots --filter volume-id="$volume_id" | grep $runtype | sort -r -k 5 | tail -n +${tailnumber} > /var/tmp/delsnapshots$dts
   
   		      while read deleteline
   
   		      do
   
   		        #get snapshot_id
   
   		        export snapshot_id=$(echo $deleteline | awk '{print $2}')
   
   		        #delete the older snapshot
   
   		        printf "deleting snapshot $snapshot_id for $volume_id -- $description\n"
   
   		        /usr/bin/ec2-delete-snapshot "$snapshot_id"
   
   		      done < /var/tmp/delsnapshots$dts
   
   		      rm /var/tmp/delsnapshots$dts
   
   		    fi
   
   		  fi
   
   		done < /root/snapshotscripts/snapshotlist
   
   		echo "end"
   
   		exit 0

2. Now Create a file to list the volumes you want to snapshot, called /root/snapshotscripts/snapshotlist, and past the following contents, then modify them so that each line lists a volumeid of the volumes you want to backup, which instance they're on and which device(for tagging purposes so that you have a note atached to each snapshot), and how many daily, weekly, and monthly images you wish to keep:
   

   #volume_id description keepdays_daily weekly monthly [don't_use_spaces_in_values_-->_they_are_the_delimiter!]
   
   		vol-dcsdssa6 WSBU_Serv01:/dev/sda1 3 3 3
   
   		vol-32fja92r WSBU_Serv02:/dev/sda1 6 6 6
   
   		vol-89asd0fe WSBU_Serv03:/dev/sda1 3 3 3

3. make sure these are readable only by root, and that they're exectable only by root:
   

   chown -R root:root /root/snapshotscripts/
   
   		chmod -R 700 /root/snapshotscripts/

OK - you're now done!

Some notes and caveats: you should check to see that the script works in your environment, as even a small typo or configuration change can wreak havoc. Verify that you have snapshots by restoring one and testing it!

There will be logfiles in /var/log/ so you can see errors if they were logged and progress notes as they were logged.

You can also manually run the commands to test them, like:

/root/snapshotscripts/managesnapshots daily