METAIRIE, Wis.-It was only seven years ago that Hurricane Katrina forever altered the Gulf Coast and the Crescent City. Now, with Hurricane Isaac having just passed over the region, credit unions are benefiting from lessons learned in testing disaster recovery systems.
"At the time (of Katrina), our back-up process involved manual transfer of data at day-end to physical tapes, which an employee of the credit union would store offsite, or in the event of a disaster ship to a recovery site that was located in California at that time," recalled Judy De Luca, CEO of the $143-million New Orleans Firemen FCU.
While many credit unions have a disaster recovery plan in place, testing is often a gray area.
"It's really more of business continuity issue than disaster recovery issue, and while disaster recovery is a big part of it, the challenge for smaller credit unions is that they don't demand sophisticated disaster recovery solutions, often because they don't have budgets to support it," said Kirk Drake, founder and CEO of Maryland-based CUSO Ongoing Operations (OGO). "When it comes to testing, smaller credit unions just don't have the same rigor."
De Luca had a chance meeting with Drake in October 2009 as they were both presenting at the National Association of Firefighter Credit Unions Conference.
"I was talking about our Katrina experience and we opened a dialogue on how a partnership would be beneficial," said De Luca.
Beyond the 28 employees who lost their homes, the credit union and its branches took weeks to recover. De Luca noted that there were no or unreliable communications in the affected area, including no cellular service or landline service. The postal service was disabled and due to systems being offline, shared branches could not identify members.
Compounding The Challenge
The challenges the credit unions faced were compounded by the human error factor, corrupted data on the back-up tapes, security of shipping the physical tapes and a limited plan for testing its back-up systems.
"We also faced the fact that the entire process took place from our main office, and beyond the problem of someone having to be physically present to conduct the back-up, we were relying on the assumption that the main office would always be functional," said De Luca. "If we wanted to close our offices in anticipation for a hurricane evacuation, our IT staff would always have to be the last to leave."
After her meeting with Drake, De Luca and her team researched Ongoing Operations, while meeting with XP Systems, its core solution provider. "We chose Ongoing first because the solutions offered were unique and not offered by other providers, and Kirk had worked at XP Systems, so we felt his background possibly gave his company an edge."
Ongoing Operations recently acquired Cloudworks, a provider located in Southern California. "We see this as a continued expansion of how business continuity plays into disaster recovery," said Drake. "We are seeing more and more the value in both offering a back-up of data as well as supporting a day-to-day fully integrated business platform including online banking, payment systems, Check 21, website hosting and secure e-mail and many other solutions."
An Old Theory That's Outdated
Matt Gerber is CEO of IT-Lifeline, a company similar to Ongoing Operations that offers cloud-based fully managed IT disaster recovery services. It counts among its clients approximately 35 credit unions. "The old theory of sending five IT people 300 miles away to a facility to address disaster recovery is outdated; those days are gone," said Gerber. "When it comes to recovery, and our Black Cloud Service, we can be accessing all the information as long as there is a wi-fi connection."
Like Ongoing Operations, IT-Lifeline works closely with it clients on training and testing, an often intuitive process for IT and C-level executives. "We go through the first test cycle and there is no additional hardware required. They will buy a very small space on one of our remote racks for their server so we essentially become part of their network," said Gerber.
A question that often comes up with disaster recovery is who is backing up the companies that are backing up credit unions?
"We will either back up all of our information offsite or we will move our back up to Amazon Web Services and we do that because the data is encrypted," said Gerber.
While historically clients have been wary about a public cloud platform, Gerber said that he is seeing an increase with more than 50% of clients agreeing to platform because it is a fully encrypted storage solution. "However, we don't have single financial institution client yet comfortable with storing production systems in the public cloud."
Peace of Mind
With the support of OGA, De Luca said the credit union's disaster recovery solution is now remotely backed-up with XP Systems in Troy, Michigan. "We signed up with Ongoing Operations for remote back-up of NCAT, file and intranet servers, along with business continuity planning."
It took nearly six months for the remote backup rollout, and De Luca conceded that "The BCP process is more work-intensive on our part and requires a lot of proprietary information for complete documentation. We've realized that we need to designate the coordination of this as a major task within a staff member's job description in order to get to the next level."
Since signing with Ongoing Operations, the credit union has participated in one on-site table top exercise.
"We were impressed with how much valuable information we gained during a simulation of what might happen during a main office fire, for example," De Luca said.
When questioned about the return on investment, De Luca responded, "Our ROI might best be quantified by placing a value on the reputation risk we would incur if we are not able to provide service to our membership before, during and after an emergency situation. We consider ourselves as financial first responders, and we guard our post-Katrina reputation very carefully."