MSI Establishes High Availability Between Physically Separate Data Centers

The largest health care provider in its region, SMDC Health System of Duluth, Minn., operates four hospitals and 17 clinics. The organization serves a rural area larger than the state of Delaware. Its 7,000 physicians and employees handle more than 400,000 patient visits each year. In an organization this large, databases hosting vital medical records, patient billing, and other business-critical information - sometimes life-saving information -- had better be readily available on time and at all times.

"Healthcare delivery today relies on a complex set of tightly interfaced digital systems [requiring zero data loss and recovery times measured in minutes]. MSI has been a great partner in building and executing our plan to provide our caregivers and the patients they serve a highly resilient and expandable infrastructure." - Dennis Smith
Director, Technology
SMDC Health System

These databases are built on EPIC, a widely used records management software program that uses applications that are integrated from the beginning. This allows people to access the right information at the right time and offers web access to extend the user's available resources.

While many health systems have set up high availability failover within a single data center, SMDC mapped out a plan with MSI Systems Integrators (MSI) to take resilience one step further. SMDC now has the first EPIC installation, based on the IBM hardware platform, implementing an environment that facilitates automated failover between physically separate data centers utilizing an extended distance (XD) software component.

The Challenges

Today's most sophisticated data centers boast redundancy in virtually all aspects of their infrastructure. SMDC realized there was still a huge risk of losing critical data from a broader site failure that could occur with a major power outage or through catastrophic physical damage to the data center. They just couldn't take the gamble.

SMDC wanted a disaster resilience solution that could be implemented across geographically separate data centers while maintaining subsecond response times of high-volume transactions and ensuring data integrity in the case of a major disaster.

In addition to continuous operation, the solution would need to increase processing power for SMDC's EPIC applications and bring built-in scalability to accommodate new applications down the road.

The MSI Solution

MSI conducted a comprehensive Proof of Concept (POC) workshop to prove the effectiveness of two IBM technologies -- HACMP-XD and Metro Mirroring Peer-to-Peer Remote Copy (PPRC) -- which together instigate a highly reliable, automatic failover process for a variety of problem situations.

Metro Mirror is an IBM storage service designed to create and maintain a real replica of data between locations. The service in two separate storage subsystems communicates to create and maintain copies of data between locations. The XD component of HACMP-XD facilitates communications and actions between servers in separate locations. PPRC configuration menus are built into the XD software, allowing the two technologies to work together to enable automated failover and recovery for data centers typically within 18 miles of one another.

During a Proof of Concept Plan and Design, MSI set up two IBM pSeries production servers and a third IBM pSeries server, which held a shadow database to verify the integrity of the EPIC database after failover. The shadow server also acted as the EPIC transaction driver to generate a representative mix of transactions accessing and updating the 600-gigabyte database. The transaction driver was set to push the EPIC production system to high workloads, over 70 percent utilization, to test the effects of a realistic workload.

The POC demonstrated successful failover between the two production servers in three key types of outages: planned outages, unplanned outages and forced or multiple error situations.

The Proof of Concept allowed MSI to fine-tune and document processes that ensure the software works automatically and in both directions - that is, from the first production server to the second, as well as from the second back to the first.

MSI's workshop methodology accelerated SMDC's understanding of how the components work together and identified operational or setup issues unique to our environment.

The Benefits

MSI's lab environment provided a true-to-life demonstration of how well pSeries, HACMP-XD and PPRC solutions integrate to achieve a highly available, robust and scalable environment for EPIC across geographically separate data centers. SMDC no longer has the risk that many of its competitors that use EPIC have. SMDC can continue doing business confidently knowing that not even a broad site failure can harm its most precious assets.

Because proper procedural documentation and appropriate security is essential to maintain system availability and data integrity, MSI also left SMDC with a great deal of knowledge and detailed documentation in terms of how the software and hardware would respond in various types of disasters, and what actions would need to be followed to rectify each of the conditions created.

Summary

SMDC Health System, a world-class organization, is committed to innovation and excellence in medical care, education and research. A key element to the delivery of high quality health care is the use of electronic health records (EHR). EHR offers health care providers with an integrated view into the medical history of healthcare clients. This integrated view enhances provider efficiency, quality of service and lower costs of care delivery.

The SMDC electronic health record system is designed using integrated technologies that include IBM servers, IBM storage and EPIC Systems software solutions. Using multiple, interconnected data centers that are distributed across the SMDC campus, the electronic health records system has been designed to protect this service from the most common local disasters.

By automating the EHR recovery processes, SMDC has the capability to return electronic health record services to production in a matter of minutes following a disaster. The value that EHR brings to patient care means that legacy disaster recovery techniques, which provided recovery of the service in a matter of hours or even days, no longer meet the demanding SMDC standards for patient care.