Top 10 Drupal web site backups best practices

A rule I follow is that when you need to use a backup, you probably will need it very badly.  Here are my 10 best practices for setting up Drupal site backups so that when (not if) the day comes you need to do a restore, you will be able to recover with minimal downtime and impact.
#1 : Name  a person (not a team) who is personally accountable for monitoring the backup process.  This doesnt mean that person is doing the backups or fixing the process if it breaks down, but it is that persons job to ensure that alerts from backups are resolved and has tribal knowledge of how the backup process works.
#2 : A drupal site needs two different backups - file system (eg, document root) AND a mysql database dump.  Be certain that you create both backups when there are no changes being made to the site.  More on this later!
#3 : Prevent updates to the drupal site when creating backups.  For drupal sites that we host, this means shutting down apache web server and then we _know_ there wont be any changes while we create a backup of the file system and of the mysql database.  Our typical down time is about 3 minutes per server which can host _many_ drupal web sites.
#4 : Use a backup mechanism that is within the capability of your team.  If you are a small business and your unix admin is sitting right next to your drupal webmaster (or they are the same person!) , then use tar for your file system backup and mysqldump for your database.  If you are a global enterprise who has a common backup tool, then the best course of action is to go with the crowd.  Just be certain that whatever you use, be sure you can comply with testing best practice (next).
#5 : Test your drupal backups regularly and in different ways.  I recomend testing at an interval somewhere between once a month and once a quarter to ensure awareness and familiarity of the process. By testing differently, I mean creating test scenarios like removing a file from the web site, or changing its owner or its permission or its content - something different every time you test so that your team has confidence in recovering from a variety of failures.  The best test of all is to use the backups to recover to a disaster recovery site if you have such a thing.  Practice of this will pay dividends.
#6 : Monitor that your backups are both starting _and_ completing using a watch dog mechanism.  I like email based monitoring where the day starts where my backup person gets two mails : one saying the backup started and the second that the backup completed and files are _different_ from yesterday.  How you do this depends on the mechanism used, but it is critical to keep an eye on this.  I like two mails because it gives the troubleshooting team a headstart if you can say something like "backups are starting but not completing" or "backups didnt start" or "our backups arent changing".
#7 : Monitor the elapsed time of backup creation and investigate any step change.  Creation of backups is a surprisingly good benchmark of your hosting provider infrastructure and if there are sudden changes or a steady degradation over time, it is almost certain to be impacting users and visitors to your site.
#8 : Carefully consider the parameters for backup retention.  While tapes are no longer used at any of my clients, my preference are schemes based on Tower of Hanoi to retain the ability to go back _months_ if needed but with less resolution.  You should be sure to tailor this for your installation giving consideration to how frequently your site changes.  ContactMe if you need assistance.
#9 : Keep backups in two locations - one for convenience and one for security.  The first location - termed for convenience - is on the same server as the drupal site.  If you have to move fast to recover a site, there is no need to hunt around or transfer files from elsewhere.  The second location - termed the secure location - is physically far away.  At a minimum, it is on another server.  Preferably, that server is in another city.  And best practice, files at the secure location are only available "read-only"  just to avoid accidental destruction.
#10 : Use the backup process on demand to create a backup before making major changes.  Every auditor will tell you that change drives risk, and I always advise clients to get in the habit of creating backups before making major changes using the _same_ process and procedures used to create routine automatic backups.  This can require some cross training but when everyone is using the same process it will help all involved when the day comes when (not if) you need to use these backups.
#11 (Bonus Tip) : Should the drupal web site backup be encrypted?  If your site is hosting PCI or HIPPA data, you wont have any choice.  Or, if you get visited by an auditor who asks you if your backups are encrypted, it will likely make for a much shorter interview to say yes rather then explain why not.  Creating encrypting backups is a breeze today using openssl enc command against the tar file or mysqldump.  Having said this, the bigger challenge is managing the keys or certificates that you are using.  The best direction I can give is get some expert advice before implementing encrypted backups on your own.  While there is no better feeling to report that "our backups are encrypted" if you are advised that your secure site was compromised, not being able to read your own encyrpted files will make for a long day indeed.

ContactMe if your team needs assistance in designing drupal site backup procedures.  I would be delighted to assist!