Newsfeed for Disaster Recovery Planning

You can get the full news feed by going to http://disaster-recovery-planning.org/news/disaster-recovery-planning-org.xml . Below are the latest the items from that feed.

---

Major Disaster Recovery Failure with an Outsource Provider

08/30/2010

Virginia’s Department of Motor Vehicles along with 25 other state agencies  hasn’t been able to process requests for licenses and ID cards. These systems are supposed to be up and running six days after the outages started to appear.Northrop Grumman  manages Virginia’s IT infrastructure under a $2.3 billion IT services contract.

The Virginia Information Technologies Agency (VITA) said in a statement that teams have been working throughout the weekend to restore data. In a nutshell, the IT infrastructure of the state of Virginia was reportedly crushed by an EMC storage area network failure. The Richmond Times-Dispatch reports that several systems are still down. The same paper said that Northrop Grumman will have to pay a fine for the failure. And the real kicker is that recently revised its contract with Northrop Grumman and extended the deal for three years. The state paid an additional $236 million for better service from Northrop Grumman.

Highlights of the Revised Contract - Operational Efficiencies

• Consolidates and strengthens Performance Level Standards with a 15% increase in penalties across the board if Northrop Grumman fails to perform on clearly identified and measured performance standards. - PAY-UP 
• Improves Incident Response teams to determine technology failures and expedite repair - FAILED
• Institutes clear performance measurements for Northrop Grumman that agencies can easily track - FAILED
• Adds new services to contract such as improved disaster recovery and enhanced security features - FAILED

Among the key parts of the VITA statement:

Successful repair to the storage system hardware is complete, and all but three or possibly four agencies out of the 26 agency systems have been restored. Agencies continue to perform verification testing.

Progress continues, but work is not yet complete for the three or four agencies that have some of the largest and most complex databases. These databases make the restoration process extremely time consuming. The unfortunate result is the agencies will not be able to process some customer transactions until additional testing and validation are complete.

According to the manufacturer of the storage system (EMC), the events that led to the outage appear to be unprecedented. The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time.

The outage was blamed on the failure of two circuit boards installed and maintained by EMC. It is a big disconcerting that two circuit boards can bring down a state’s IT infrastructure for nearly a week.

Among the things that don’t add up in the Virginia IT outage:

• Why wouldn’t these boards be replaced quickly?
• Why was there a single point of failure?
• Service was restored for 16 agencies, but 10 require “a lengthy restoration of data.” Where was the disaster planning? After all, Northrop Grumman touted its disaster recovery for the state just two years ago.
• Where did the IT management fail?

- more info

---

How to request funding for DRP BCP

08/25/2010

In these tough economic times how can CIOs get the budget necessary to support Disaster Recovery and Business Continuity Planning.

The following steps should be taken when planning a presentation seeking to gain management support of a Disaster Recovery and Business Continuity program.

• Define the scope, objectives, and requirement - It is not enough to have an objective of getting more funding or gaining executive support.  Define exactly how much funding is needed, or exactly what form the executive support should take.
• Verify expectations - Define what management's expectations for the meeting are.
• Focus on business continuity - It makes more sense to get the commitment for resources to achieve a 24-hour recovery time objective (RTO) than to demand the resources for a two-hour RTO and get nothing.
• Anticipate objections - realize that the number one objection is the cost, and prepare accordingly. Let the results of the business impact analysis (BIA) justify the "investment" (not "cost").
• Prepare a competitive analysis - Executives care what their competition is doing. Annual benchmark studies and surveys are good sources of information on the investments in DPR/BCP being made by industry, by size of organization, etc.
• Prepare examples of what has happened to others - Remind the executives of the regulations that affect their business, and the impact of not complying with them. Examples of such regulations are Sarbanes-Oxley, HIPAA, Foreign Corrupt Practices Act, and Gramm-Leach-Bliley. In addition,  research companies that have been damaged significantly in highly publicized news stories because of their failure to act responsibly.
• Define the Risk/Reward of DRP/BCP - Research and develop the business continuity program's return on investment.
• Package Resources - Work with vendors like Janco Associates who can package infrastructure solutions like the Disaster Recovery Business Continuity Template to accelerate the process and minimize the cost.
• Get buy-in for key decision makers before you meet to ask for a decision - The effort will have greater success if key decision makers and other departments within the organization support the DRP/BCP program. The power of a presentation supported by key executives, marketing, IT security, physical security, human resources, facilities, and risk management is highly significant.

- more info

---

Backup requirments defined

08/18/2010

CIOs, CSO's, Disaster Recovery Managers, and Business Continuity Mangers constantly are working to improve their recovery point objective (RPO) and recovery time objectives (RTO) by performing fast, non-disruptive backups, and by performing data restoration.  All comprehensive data protection solutions involve many considerations and contingencies.

Here are some of the things that can go wrong with your data and the backup requirements that need to be addressed:

• Accidental or malicious deletion of critical data - Requirement that provides the ability to quickly and easily restore individual files and folders.
• Data that is lost or corrupted over a period of time - Requirement to roll back individual records to fix  database corruptions. The ability to recover data from any previous point in time, and have it as granular as possible.
• A crashed disk - Requirement to recover a disk volume is different than recovering a single file, but it should be done just as quickly, and with automation to help keep operational disruptions to a minimum.
• A server failure - Requirement to restore operations when replacing a broken server may be complicated by the need to install different drivers on the new system if the hardware is not an exact match. It helps to have the capability to move the application workload to a standby server (with different hardware) or virtual server while the system is being replaced or repaired.
• A local or regional disaster - Requirement when you lose an entire office to fire, flood, or other disaster, have a current copy of your important information in another location that is outside the disaster zone.
• Remote offices and branch offices - Requirement  to have a process in place to restore with minimal technical support as remote and branch offices often do not have the luxury of having an on-site technical resource to assist in backups and restores.
• Resource-intensive backup processes - Requirement frequent or even continuous backup that is not resource-intensive .
•  Security breaches - Requirement to secure data. When moving data between sites, it needs to be protected from potential security breaches. A breach of data security, whether actual damage is done or not, can be devastating to your company's reputation, as dozens of large enterprises and government agencies have found in recent years.

- more info

---

DRP versus BCP

08/11/2010

Disaster recovery planning is one of the most important jobs of the IT professional. It includes working with upper management and winning the cooperation of all departments to make a working recovery plan. The two main parts are the Business Continuity Plan (BCP) and the Disaster Recovery Plan (DRP). These have to go hand-in-hand procedurally. The BCP focuses more on the schedule and timing of the DRP, so that in the event of a disaster the business can function normally. The three stages of a DRP are Prevent, Detect and Correct.

 

- more info

---

Disaster Recovery and Business Continuity a critical part of enterprise operations

08/07/2010

Disaster recovery is becoming an increasingly important aspect of enterprise computing. As devices, systems, and networks become ever more complex, there are simply more things that can go wrong. As a consequence, recovery plans have also become more complex. According to Janco Associates (the author of the Disaster Recovery Business Continuity Template). For example, fifteen or twenty years ago if there was a threat to systems from a fire, a disaster recovery plan might consist of powering down the mainframe  and other computers before the sprinkler system came on, disassembling components, and subsequently drying circuit boards in the parking lot with a hair dryer. Current enterprise systems tend to be too large and complicated for such simple and hands-on approaches, however, and interruption of service or loss of data can have serious financial impact, whether directly or through loss of customer confidence.

Appropriate plans vary from one enterprise to another, depending on variables such as the type of business, the processes involved, and the level of security needed. Disaster recovery planning may be developed within an organization or purchased as a software application or a service. It is not unusual for an enterprise to spend 25% of its information technology budget on disaster recovery.

Nevertheless, the consensus within the DR industry is that most enterprises are still ill-prepared for a disaster. According to the Janco Associates Disaster Recover Business Continuity web site, Despite the number of very public disasters since 9/11, still only about 50 percent of companies report having a disaster recovery plan. Of those that do, nearly half have never tested their plan, which is tantamount to not having one at all.

- more info

---

Security and DRP play a role in CIO Infrastructure Design

07/24/2010

Designing IT Infrastructure requires CIOs to consider the globalized world they are now in. It is necessary and valuable for CIOs to understand the fundamental trends that are pushing businesses to redesign their operations around this new reality.  Factors they need to consider are:

• Security - With the growing importance of digital applications and data, the sources of threats to enterprise data have multiplied dramatically. Everything from natural disasters to criminals to corrupt sources within the company might try to steal or corrupt data. While businesses do everything that they can to stop these threats in the first place, they still must be prepared to recover from these threats as quickly as possible.
• Business Continuity and Disaster Planning - As businesses have expanded the need for anytime, anywhere application access has become a requirement. At the same time, “follow the sun” (global 24/7) operations have shrinking maintenance windows and a need for applications to be running at all times. Delay or loss of data for any reason – system failure, natural disasters – has a domino-like effect across the entire organization, at any time of the day or night.
• Flexibility - Most businesses now operate across international borders and CIOs must be able to respond to opportunities and challenges faster than ever before. CIOs are usually battling well-resourced organizations that may be based where the opportunity originated, or another globalizing company that is reaching out for new opportunities. In order to compete, a business has to be faster to deliver a product or service as good, or better, than that of potentially any other company in the world.
• Simplicity - Increases in technology have typically led to increased complexity. While per unit costs of technology are always decreasing, in aggregate companies see an increase in cost. With the pressure on IT to act less as a cost center and more as a way to increase the profitability of business units, just adding more storage, more bandwidth, or additional technologies throughout the organization is no longer an acceptable approach to managing information technology. Successful CIOs are investing in numerous technologies including; continuous data protection, virtualization, and wireless connectivity.  They are trying slim down IT’s footprint while increasing their business’s competitive advantages. The CIO is typically in a difficult position, assessing where to try and cut costs while still moving forward with a plan to continually enhance IT services to the business.

- more info

---

Nature can distroy anything that man can make

07/13/2010

Nothing man-made can withstand the forces of nature. In certain regions of the country, natural disasters are not a question of if, but of when. The main headquarters of many companies are located in North Carolina, right in the heart of Hurricane Alley. In addition, Southern California is earthquake and brush fire central.

 

 

They know a hurricane, earthquake, or brush fire is going to be coming along at some point; it is inevitable.  At the worst, you are looking at physical damage to facilities and systems, or flooding. At minimum, it will knock out power and your network circuit. Even if power and network stay up, just the fact that you do not have physical access to your system may prevent you from doing a crucial operational task.

- more info

---

How a CIO should chose a backup site

06/18/2010

 Disasters cost money, interrupt business operations and may cause the enterprise or government agency to fail, which makes planning a business continuity issue. Disasters can interfere with or even terminate IT and communications services. It does not matter whether the disaster affects the enterprise, government or service provider. Floods, fire, volcanoes, earthquakes and other events can destroy a primary and backup site if they are too close together.

Telecom service providers can offer expert advice on where to locate a backup facility and should position themselves with CIOs to offer both consulting and services. After all, they have experience planning for their own primary and backup facilities, as well.

A CIO's selection of the backup site location will always have risks and liabilities attached to the decision. Adequate and reliable communications to the backup site and communications between the primary and backup sites are what most service providers can successfully offer to the CIO.

      

In choosing a backup site, CIO's must first determine how big a disaster plan for and budget for it. The level of disaster planning increases as you goes down the following list:

• Building closed/evacuated
• Loss of power
• Loss of communications
• Facility damaged/destroyed
• Community disaster (10-to-30 mile range)
• Regional disaster (30-to100 mile range)

- more info

---

Successful Disaster Planning and Business Continuity Planning Processes

06/10/2010

The success of most business depends on Information Technology. However, business and technology environments are becoming more complex. Being prepared to respond to non-typical events - both planned and unexpected - that threaten to disrupt essential business systems and processes, is a major corporate concern.

A recent survey found that disaster recovery planning is a priority for many organizations. Eighty-six percent of IT executives said they have a disaster recovery plan in place at their organization. While the economy has affected IT budgets overall, 43 percent of IT respondents indicated the economy has not affected their disaster recovery investment (including planning) - with another 33 percent, saying investment in disaster recovery has become more important.

Organizations cannot control whether or not they will be affected by a natural disaster, power outage or other unplanned incident, but they can work to help ensure their business is prepared to respond to and recover from these events with minimal impact. Disaster recovery planning is an organizational requirement that can help reduce risk and help companies effectively respond to situations that threaten to disrupt essential business processes.

Janco Associates has found that enterprises that are successful:

• Focus on employee safety. Every disaster recovery plan needs to begin by addressing the physical safety and psychological well-being of employees. That means the plan must include alternative locations where employees can go if a primary work site is unavailable, as well as incident notification and escalation strategies. In addition, the plan needs to be well communicated throughout the organization so everyone knows how to respond in a disaster situation.
•   Conduct a business and IT impact analysis. Carry out a thorough analysis of people, information, application, and other resources to build an understanding of the consequences - financial and operational - of losing vital components. Take particular care to uncover interdependencies across the organization that is critical to staying in business. This analysis will provide a solid foundation for establishing recovery priorities and timeframes in your plan, allowing you to make informed decisions on where and how much to invest in disaster recovery.
•  Plan with business operations in mind. Involve all key stakeholders in the planning process, including IT, business leaders, human resources, corporate communications, and physical and information security managers. Be sure that in planning you coordinate with other business units in your organization to avoid potential conflicts, such as multiple business units depending on the same facility as a secondary site in response to an interruption.
•  Make the disaster recovery plan a living document. Business processes and IT systems undergo constant change in every organization. Your disaster recovery plan needs to keep pace with new workflows, business applications, and computer systems. Disaster recovery planning software can provide best practice methodologies to help you navigate through planning decisions and plan updates. In addition, regular testing will help you demonstrate your ability to recover and pinpoint areas for plan improvements.

- more info

---

Disaster recovery and business continuity planning issues

05/27/2010

Disaster recovery and business continuity management and contingency planning are essential especially in these economic times. However, the creation, testing, and updating  of a sound disaster recovery and continuity and contingency plan is costly and complex.

For example, initially it is necessary to understand the underlying risks and the potential impacts of disaster. This is the primary building block upon which sensible and cost effective business continuity plan or disaster recovery plan is built. When the plan itself is created, there are the maintenance and testing phases, to ensure that the plan remains current. Even having arranged all these matters there are the external auditors to consider - and of course, there is the not so small matter of ISO 27000, SOX, HIPAA, and PCI-DSS compliance.

The industry standard solution is the Disaster Recovery and Business Continuity Template by Janco Associates. The template includes all of the right tools to assist with business impact analysis and risk analysis. You can quickly create a core plan (some of Janco's clients have created an operational plan in less than thirty days), maintain the plan, audit the DRP BCP, and create a cost effective budget to support the disaster recovery business continuity process.

- more info

---

DRP Critical Component of Risk Management

05/22/2010

Disaster Recovery (DR) is a critical component of IT disaster planning and risk mitigation strategies, and compounded in difficulty by ever growing data volumes, distributed computing, and new technologies. How can you get creative in protecting more data, recovering more swiftly, but also saving some money?

Download this outline learn how the Janco Disaster Recovery Business Continuity Template can reduce RPOs and RTOs even more. 

Disaster Recovery Guide
Business Continuity Planning

ISO 27001, ISO 27002, ISO 17799, Sarbanes-Oxley, and HIPAA Compliant

         

What is Disaster Recovery and how does the Disaster Recovery Planning Template help?

This DRP Template can be used for any sized enterprise.  

The template and supporting material have been updated to be Sarbanes-Oxley compliant.  The complete package includes:

• Disaster Recovery Planning and Business Continuity Template
• Business and IT Impact Analysis Questionnaire
• Work Plan
• Disaster Recovery / Business Continuity Audit Program

With lost data being a competitive liability, there is no room for downtime in today's business world.

- more info

---

Backup strategies for Disaster Recovery and Business Continuity

05/11/2010

In order to meet current demands for application and data availability, many enterprises are increasingly relying on the wide-area network (WAN) as a storage transport resource for Disaster Recovery and Business Continuity Planning. This enables Disaster Recovery and Business Continuity operations to be centralized - reducing redundancy and lowering overhead - and to leverage innovative disk-based backup and replication technologies offered by the leading storage vendors. Decentralized, tape and other removable media  Disaster Recovery and Business Continuity strategies are simply too costly and labor-intensive. In practice, they fail to meet the recovery time and recovery point objectives (RTOs/RPOs) demanded by companies facing increasingly stringent customer service and regulatory requirements. - more info

---

Disaster Recovery versus Business Continuity

04/26/2010

Disaster recovery is about re-establishing IT services in the face of large-scale hardware failure or sabotage, facilities failure and/or regional natural disaster. Disaster-recovery capabilities are measured by the amount of time it takes to re-establish services and the amount of data loss. Business continuity is the ability to continue operations with little or no downtime in some of these scenarios. - more info

---

Lesssons learned from Iceland Volcano

04/23/2010

The Icelandic volcanic dust cloud provides some concerns for disaster planning and business continuity planning:

• Business redundancy - businesses that have a natural redundancy and resilience capability built in as part of everyday operations were well placed before, during and after these events.
• Crises will happen - crises are a routine part of business of as usual and not a special event.

While recent events may well be an opportunity for business continuity, disaster recovery and crisis management consultants, businesses that focus on implementing redundancy and resiliency into business because it is good for business anyway and achieve business continuity, disaster recovery and crisis management capabilities as a bonus, are likely to get more from these activities than those that just use consultant to develop business continuity, disaster recovery and crisis management capabilities in isolation.

- more info

---

Volcanic event business continuity and disaster planning requirements

04/21/2010

Until recently most enterprise did not consider volcanic events as something they should plan for.  However the volcanic eruption in Iceland impacted global transportation and commerce.   With that in mind, disaster recovery - business continuity plan for volcanic events should contain the following elements:

• Identification and mapping of the hazard zones;
• Impact of travel limitations
• Register of valuable movable property (excluding easily portable personal effects);
• Identification of safe refuge zones to which the enterprise operations will be moved in case of a dangerous eruption;
• Identification of evacuation routes; their maintenance and clearance;
• Identification of assembly points for transport for evacuation;
• Means of transport, traffic control;
• Shelter and accommodation in the refuge zones;
• Inventory of personnel and equipment for search and rescue;
• Security in evacuated areas; and
• Alert procedures;
  

- more info