Opinion: Recovery Happens In Steps

Opinion: Recovery Happens in Steps

A long career in IT has taught me some hard-learned lessons. Perhaps the most important of these is that good judgment comes from experience. You can read a book, attend a seminar, or hear a great speaker, but nothing will hardwire a lesson in your brain like the experience of touching the hot stove your parents told you not to touch, or of facing down a great calamity and coming out on top. Permit me to tell you about one of these experiences.

I was in my office on G Street in Washington DC. I was the CIO for the White House. The year was 2004 and it had been a quiet day – a rarity which, of course, did not last. Quiet days were few and far between. One of my employees, Bob, came into my office to tell me that the database that supported the Social Office for the White House had crashed. Crashed as in, corrupted, unusable, not recoverable. Bob had a pale look on his face that alerted me to the fact that there was more bad news. The backup, he continued, was also corrupted. It was unusable. Recovery was impossible. 

Now, the Social Office is much more important than you might think. The Social Office supports the First Lady of the United States. It is responsible for, among other things, planning all the state dinners and social functions of the White House. The Social Office’s database is a massive archive that contains the information for decades of White House functions and the tools for planning upcoming events. 

“They,” Bob said, meaning the women who worked in the office and reported to the First Lady “were furious and they wanted to speak with me.” I was to report to their office ASAP to tell them how I was going to fix what I knew was unfixable. The first thought that ran through my head was, “Does She already know? Have they told Her?” And, “Is she furious? Is she going to tell The Boss?” My last thought, before I called my own boss to tell him I was headed to the East Wing was to think, “Why the hell did I take this job?” 

I asked Bob another question to which I suspected I knew the answer: “Had we, by any chance, done an offline backup?” Of all the work to create offline backups for recovery, I knew that the Social Office database had not been in that priority list. But we hadn’t and now I was going in to face the gauntlet. Hindsight is 2020. My boss wished me good luck, glad that he was not taking the walk to the East Wing. 

The elevator chimed and I was about to step through its doors when Bob found me, breathless. He had a smile on his face. He told me that one of the techs, the week before had printed the entire database out on paper. We had the data. I knew, just as Bob knew, that everything else could be reconstructed. We had the data! On the way to the Social Office, I sketched out a plan. Bob was tailing along with me. We would put a task force together to reconstruct the database and to enter all the data manually, night and day. Could we do it in a month? Bob thought so. 

That is what we did. It took a month of herculean effort, but we got it done.

The lesson is simple. Okay, there is more than one lesson, but the one I want to convey is simple.  Don’t overthink recovery. Think of it in tiers. Printing the database had turned out to be the first tier – not the most efficient of ways, but still effective. Of course, the tech had printed the database for other reasons, so we got lucky. My career was pulled back from the trash bin by a stroke of luck. We could have just as easily run a data conversion to a spreadsheet, saved it to a thumb drive, and saved the thumb drive in a safe. And it would have taken a very minor effort to restore the database. But, we didn’t. 

What is cyber resilience? It is the ability to operate your business, to perform your mission even when under attack. It is learning how to plan, organize, and recognize the tiers of recovery. It means having air-gapped, alternative sources of electricity to run your vital systems and backup communications ready for when the utility service, your internet connection, or your mobile phone service is down. It’s the ability to begin recovering as soon as possible. 

It means planning for a recovery carried out in steps. Get the computers back up so you can use communication functions. Next, get those computers connected to the data that was backed up and protected. Then restore full functionality. Figure out what tiered recovery means for your business, put a plan together, test the plan, and establish the expectation that even in a crisis, there is control. And remember to include the whole organization in this conversation and to think outside the box. Couriers can replace emails if needed, and services like HR and CRMs can operate from alternative platforms if the necessary preparations are made. 

In all of this planning, there should be one overriding consideration, and that is to take care of your customers first. Figure out a way to make the disruption to your customers minimal, always back up the data, and remember: resilience is the ability to recover in tiers – not tears. 

Carlos Solari

Carlos C Solari was born in Colombia, South America and grew up in Huntington, NY. His career started in government service: U.S. Army for 13+ years, FBI senior executive in the 1990's and Chief Information Officer for the White House (2002 - 2005).

In the private sector, Carlos was VP of Cybersecurity at Bell Labs, VP / GM of Global Security Solutions at CSC, SR VP at Mission Secure Inc and VP of Cybersecurity Services at Comodo Group. He is currently in several roles including Advisory Board Chair for CyberFortis focused on cybersecurity education / training for the corporate leadership including the Board of Directors, the C-Suite and Compliance Officers. He is the author of several books including Security in a Web 2.0+ World published by Wiley in 2009. He has taught cybersecurity in various settings and is an international speaker on this topic.

He is a graduate of Washington and Lee University with a BS in Biology and the Naval Postgraduate School with a MS in Systems Technologies.

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top