On Backups

Summary: Your hard disk will fail. What's your backup strategy?

Recently (http://www.embedded.com/electronics-blogs/break-points/4435589/Software-updates) I wrote about losing some files from the laptop while traveling. Some readers responded with suggestions like using git or some cloud solution. Those are not really realistic options when working on an airplane, though more and more flights are now wi-fi enabled.

And I am quite suspicious of the cloud. While it surely has its merits and uses, it is yet another example of a new technology that we really don't understand. The technology is clear, but legal and other implications are not. A few of my concerns are:

 Lawyers tell me that data stored in the cloud is not protected by attorney-client privilege since it's not under one's direct control. Of course we don't expect to get sued so why worry? Well, in the last few months two friends, consultants both, one-person shops, have been sued for alleged product problems. The world has gotten extremely litigious, and I think only a poor businessperson would assume to be immune from legal hassles.

 Where is the data stored? If any copy is stored in an out-of-country data center are you violating Federal laws regulating the export of technology? Data files are subject these regulations, depending on their content.

 Since multiple copies are made, what really happens when you delete a file?

 If the hosting company disappears, what happens to your data? Amazon Web Services is losing $2B/year. Either they will learn to make money, or will shutter that business.

 We know at least some cloud providers scan files looking for illegal material. A scan of a binary could incorrectly appear to be a forbidden .JPG. Are you willing to engage the FBI in an expensive legal battle even when they are wrong?

I work from home and am the IT department, so am responsible for backups. Though my laptop is a Mac the other machines here run Windows.

Our backup strategy is multi-pronged and is designed to preserve files stupidly deleted or changed, to handle hard disk failures, and to be strong enough to survive some catastrophic event like a fire or zombie attack.

SecondCopy (www.secondcopy.com) is a $30 Windows application that I highly recommend for backups. It will do exact copies of directory structures to other directories or media (hard disks, USB drives, etc.). But, better, it can synchronize directories. That is, it copies changed files from one to another, deleting files that have been sent to the trash, so the desired directories on two media are identical. It can also save old versions of changed/deleted files to a third directory.

My computer is the backup nexus. Every morning at 3:00 SecondCopy starts itself up. The first thing it does is run a batch file I wrote that kills the email client (because that application's data file is locked and cannot be copied if the client is running). Then it uses xcopy to copy all changed files from various computers on the network to my machine.

SecondCopy then synchronizes my master directory (everything important is stored under it) to a second hard drive inside the computer. It saves the last ten versions of all changed/deleted files as well.

The program then does the same to a USB-connected external hard drive.

Every Friday my electronic TODO tool hounds me to check to make sure the backups look OK - too be sure something didn't go wrong during the week. I then swap the USB drive with another that lives locked up in the barn. That's fire insurance. Unfortunately the building is too far from the house for a wifi connection, but one of these days I plan to put up good antennas to permit daily automatic backups to a computer there.

Also each Friday I run another script that copies the changed files to one directory, which in turn gets zipped and encrypted. That gets sent to a family member many hundreds of miles from here. I'm told this preserves attorney-client privilege.

A reader who prefers to remain anonymous responded to my column about losing files with his solution:

Although I'm sure that our needs and equipment are different, I thought I'd share my list of IT-related tools, since my office is in my home, as I believe yours is. Perhaps something will be interesting or useful.

I get internet service from one of the local cable companies. We are fortunate to have three to choose from, so prices and speeds are reasonable, $90 for 110Mbps in, 25 Mb out.

As a firewall, I have a dedicated PC running Astaro UTM (Sophos now owns Astaro, but the UTM software remains mostly unmolested by Sophos). The only drawback to this is the license is sold per-IP address on the local network, so that (with the kids' friends' devices logging onto our WiFi) we fall just north of the 25-IP license, and have a 50-IP license instead. The Astaro UTM/Firewall provides a first level of anti-virus defense and provides a VPN port into the network when I'm away.

Within our network, I have a windows (server 2012) machine which functions as a file repository, and runs typical server applications:

 DHCP server,

 DNS server, with external DNS queried to OpenDNS,

 Subversion file repository server,

 Mercurial file repository host,

 Atlassian JIRA issue tracker (for bugs, anything I need to report to the customer) application

 Atlassian Confluence wiki

 typical file server, including reference docs, pictures, music, common software archives

 Storagecraft Shadowprotect server.

 Kaspersky anti-virus (Small-business Endpoint Security)

All machines on the network run Kaspersky and Storagecraft Shadowprotect.

Backup strategy:

1) Each Shadowprotect client and the server periodically (once/3hrs from 8 AM to 9 PM) save to the server a snapshot of incremental changes to the disk(s) since the previous snapshot. Once a month, they create a new whole-disk baseline. The server consolidates the snapshots into daily and weekly images, and keeps one week's worth of 3-hrs, a month of dailies, and all the weeklies. This way, I can mount previous images in the event of hardware problems, accidental deletions, or infections.

2) My phone runs an app that executes when [on home-wifi, & on charger, & >= 24 hrs since last] and copies pictures and other data to the file server (which is backed up as noted).

3) The Shadowprotect backups are stored to a dedicated 3 TB WD Red drive that sits in a USB-SATA adapter. Once every two months, I pull the drive, and start a fresh one. Every-other old drive gets put in a "Hard drive transporter classic" from Silicon Forensics and shipped off-site to an out-of-state relative, so that I have some protection against fire/calamity.

I've considered Cloud storage, but the economics aren't quite as good, and security??

I used to use a desktop machine at the office and carry a laptop when I traveled. However, I found that preparing for trips was a headache, as I had to make sure to update all relevant files at the last moment. Now, I use a laptop as my primary machine, and plug it into a docking station at the office. The docking station has two large monitors, a keyboard, mouse, and hard-wired Ethernet. When I need to travel, I just undock and go, all my files and shortcuts remain with the laptop. I typically carry two (or three) USB sticks on the road: 1 with customer-viewable materials (PowerPoints, spreadsheets, etc.), potentially another one that's blank for exchanging files, and one that always stays in my pocket, with a copy of my latest files, in case my laptop gets stolen while traveling. My laptop has a Samsung SSD drive, which is speedy, quiet, low power, and doesn't care about G-forces.

I don't claim this is the perfect setup, but it has served me pretty well. I do get frustrated by spending most of a Wednesday/month (following patch Tuesday) walking up to each machine, and verifying that the MS updates have been applied, that Java is up to date, etc.

I'm sure that I could automate this with some IT magic or windows domain rules, but I am leery of investing too much effort into automation that's likely to need tweaking in a few months anyway. Kaspersky also offers a module that can monitor and centrally administer machines across the LAN, but it seemed to be a resource hog. I probably update software more aggressively than is needed, but I do try to avoid performing any updates during the week prior to a customer release date.

If I had to change one thing, I'd probably get a laptop with a smaller screen. I have a 15" screen, and only use it while traveling. As a result, it weighs more, and is barely usable on most flights.

What's your backup strategy?

Published October 6, 2014