Case Study: Syndeo Communications
How this MSP got a failed server in a doctor’s office up and running
Like most managed service providers (MSPs), Andrew Hartman understands it’s not if a disaster is going to happen, but when.
As the founder of Syndeo Communications, an IT consulting firm in the San Diego area, he specializes in business continuity:
We take proactive measures to keep businesses running, no matter what happens.
That includes backup and disaster recovery planning with his small-to-medium-size-business clients.
We assess where the client might be exposed to technological threats, then we recommend prevention and recovery strategies. The goal is for them to continue their business operations, even under adverse conditions.
Adverse conditions bring to mind hurricanes, floods, and other natural disasters. However, even the simplest hardware failure can be a disaster for a small business. Especially if that business is the healthcare industry, in which downtime can be a matter of life and death.
In November 2012, Hartman installed a business continuity solution for one of his healthcare clients, a small doctor’s office with 12 employees.
We set up a BCDR (business continuity and disaster recovery) box at the customer’s site. The customer was getting backups every 15 minutes, and the backups automatically replicated offsite to our data center.
The Syndeo Communications BCDR appliance is usually a beefed-up PC configured with Windows 7, StorageCraft ShadowProtect (which includes StorageCraft VirtualBoot technology), and StorageCraft ImageManager, along with 4GB to 16GB of RAM (depending on the size of the installation) and 2TB or more of storage.
The doctor’s computer network ran smoothly until July 2013, when the office reported some blue-screen incidents. As Hartman recalls,
Our diagnostics showed something wrong with the motherboard on their Windows Small Business Server, which was running on Dell hardware from 2008. At that time, we tried to get them to upgrade their hardware, but the customer’s response was, “We have a backup, don’t we?”
The doctor wasn’t worried about the hardware because the office could still reboot its server, and they had backups every few minutes. The “if it ain’t broke, don’t fix it” philosophy is common in the SMB world, but Hartman knew it could be disastrous for his client.
They Didn’t Know How Lucky They Were
Of course, Hartman quoted the client a new server.
The doctor wanted to take some time to decide between putting everything in the cloud or buying a new onsite server. Then one day, the five-year-old server wouldn’t reboot.
We booted with VirtualBoot, and they were up and running within 40 minutes. Then we discovered that two of the 12 Exchange mailboxes had to be rebuilt. As the server was dying, some backup snapshots were getting corrupt. It turned out that the server was beginning to corrupt the Exchange database about a month before the server finally failed. Our clients didn’t have any idea how lucky they were. Just because you’re backing up doesn’t mean you won’t lose data. If there’s a hardware issue on the server, it can corrupt your backups.
The doctor ordered a new server, and in the meantime, the office used VirtualBoot, which boots ShadowProtect backup images as virtual machines, to run the business on the BCDR appliance for more than two weeks. During that time, Hartman used StorageCraft HeadStart Restore® technology to pre-stage the new server as a virtual machine and get it ready for production.
We wanted to test the restore first in our lab and make sure the OS would boot and the hardware and drivers and everything would work properly.
The doctor’s office had a chain of backup files covering more than eight months, so using HeadStart Restore to start the recovery beforehand allowed Hartman to minimize the downtime for his customer when the new server was installed at the doctor’s office.
HeadStart Restore allowed us to restore all but the most recent files to the new server, which we built as a virtual machine so we could perform a bare metal restore to the new hardware onsite. We arrived at 5 p.m., got the new server in the rack, gave it the last few backup files, and by 5:20 the server was back, running the domain controller, Exchange and everything. Our client was only down for 20 minutes during the entire restore process.
The incident at the doctor’s office had a happy ending—thanks to ShadowProtect, VirtualBoot, and HeadStart Restore—and Hartman learned a valuable lesson: test backups and restores more frequently.
If we had been testing the customer’s backups more often, we would’ve caught the issue with the two Exchange mailboxes.
Based on this incident, Hartman’s advice to other MSPs is to:
- Make sure your backup chain is long enough to find a restore point with non-corrupt data
- Have more restore point options rather than fewer, so you can go back far enough to cover any corruption that may have occurred
- Remember that hardware can fail at any time, even if it’s new and under warranty
It’s Not Snake Oil; It’s Magical
As an IT consultant for almost 10 years, plus several years working in corporate IT, Hartman has seen a lot of new technology solutions come and go.
There’s lots of stuff out there being sold as the latest and greatest snake oil. IT people are constantly being pitched by companies trying to cure all. “This will fix everything,” they tell us, when more often than not, their product doesn’t do what they say.
So when Syndeo Communications evolved from a break-fix support shop to a managed service provider in 2009, Hartman realized he needed to recommend only the best products to his customers.
If you’re going to be responsible for fixing something, you don’t want to recommend a product that causes you extra work.
After trying three different backup and disaster recovery solutions with poor results, he starting looking for a more robust, affordable, and flexible solution—one that would also offer bigger margins for MSPs. Then he found StorageCraft.
We started using ShadowProtect and StorageCraft ImageManager in 2012. Shortly after using it here we became a StorageCraft partner. There’s no other backup and disaster recovery solution I would sell now. I can take an image of a server that’s stored in the cloud, put it on a five-year-old computer until my new one comes in, then do a Hardware Independent Restore. That’s magical!
About Syndeo Communications
Founded by Andrew Hartman in January 2005, Syndeo Communications is a San Diego IT consulting firm that provides onsite and offsite IT support, monthly maintenance services, business phone systems, network security, cloud computing, and business continuity solutions.
Syndeo is Greek and means to connect or bring together. “I chose the name because it was unique and symbolized what we do,” Hartman says.
Syndeo Communications also offers 24/7 phone support, with service desk staff certified in Microsoft network and desktop operating systems and all versions of Outlook, Exchange Server, and SQL Server. They also have a staff of Apple IT consultants for Mac products such as iPhones, iPads, and MacBooks.
Syndeo specializes in efficient IT systems that save small businesses from losing time and money because of a slow network or sluggish computers. Here’s how Hartman describes it:
Let’s say you own a small business with 25 employees. Each day your employees have to wait two minutes to launch a program and on average, they launch five programs a day. This means each employee loses at least 10 minutes per day due to technical problems. If you take those 10 minutes, multiply them by 250 working days per year, and then by your 25 employees, the result is nearly 1,050 hours of lost time in one year. Are professional IT services worth it? You do the math.