> UK’s leading paediatric specialist hospital
> Operates in London, established in 1852
> Over 100,000 patients treated per year
> Principally supported from Hitchin Recovery Centre
> Customer for IT Recovery services, supported by Guardian iT since 1998 and during an invocation in 1999, now supported by SunGard Availability Services following their acquisition of Guardian iT in 2001.
"Before signing with Guardian iT we did look at a few other data recovery services, but frankly we felt that they weren’t up to the job," comments Chris Gregory, systems administration manager, Great Ormond Street Hospital.
About Great Ormond Street Hospital
Fire, floods and explosions are just a few examples of the more dramatic type of incident that can create a potential disaster for a company. Just as serious, however, are the 40% of incidents managed by SunGard Availability Services worldwide that are the result of software or hardware malfunctions, like the one that happened at Great Ormond Street Hospital for Children - one of most respected paediatric hospitals in the world.
Opened in 1852, Britain’s first children’s hospital offers the widest range of paediatric specialities in the country, including 21 medical, 11 surgical, 8 diagnostic specialities, as well as eight paramedical and other clinical support services including pharmacy, physiotherapy, psychology, dietetics and speech and language therapy.
The hospital is the largest paediatric research and training centre in the UK with all areas of the hospital taking part in research, including a focus on research done by nurses and professions allied to medicine. Over 100,000 patients are admitted each year, many from overseas, and are attended by the hospitals 2,100 staff and 500 student paediatric nurses.
Why business continuity?
In 1998 the hospital took the practical and responsible decision of implementing a comprehensive data recovery plan to ensure continued provision of critical IT systems.
Broadly the IT in the hospital is used to run three essential functions, using large Sequent Hewlett Packard and ICL systems. The first is administration, which includes all the financial and logistical requirements of the hospital.
The second deals with the clinical systems. For example, the pathology dept has to be able to match blood samples for transfusions at any time of the night or day, and also to supply the results of blood tests. Similarly, the radiology dept makes around 50,000 examinations per year and the system handles appointments, film tracking as well as the doses of radioactivity that are required to get each image.
The third function is the Patient Information Management System (PIMS) which carries patient demographics; details like check-in and records of a patient’s previous appointments.
Keeping all this together is Chris Gregory, the systems administration manager and his team of seven. "It may sound dramatic, but it’s not an exaggeration to say that lives may depend on the integrity of the IT we look after, so it is essential that we have a contingency plan in the event of any problems. Before signing with Guardian iT we did look at a few other data recovery services, but frankly we felt that they weren’t up to the job," he comments.
Invocation!
Within a month of signing with Guardian iT Chris was confronted with a serious software problem, which could not be solved. "When this happened, we were in the process of putting a an emergency plan in place. Events simply overtook us before we were able to rehearse or perform a test recovery of data with our consultants – we were totally in their hands".
When Chris invoked the business continuity firm, he was asked to fax details of the hospital’s file system to Guardian’s technical department, then located at Hitchin. When he arrived at the Recovery Centre a few hours later he discovered that the recovery technicians on the account had already started to build the hospitals file system on the recovery computer.
Within a matter of hours the hospital’s vital services were recovered on a parallel system created by the business continuity firm, and circumstances dictated that this system was kept live for a further six weeks.
According to Chris: "It was a very complicated recovery, but I’m happy to say that Guardian performed a very professional job. In addition to this, they were able to establish our link from the Hitchin Recovery Centre to Great Ormond Street using NHS Net and this was a major achievement as far as we were concerned. NHS Net can be viewed as an extremely secure intranet for the NHS, acting broadly as a central information resource and it is an important tool for many of the staff at the hospital."
At the time, Guardian iT was the first disaster recovery provider to have been granted such access by the NHS Executive, following an extensive security evaluation.
This provides the NHS Trusts with the opportunity for testing and rehearsing disaster recovery scenarios and processes, which will be of particular benefit to those Trusts without dedicated personnel for system testing. "Speaking as a taxpayer, I believe that the more NHS Trusts that develop their own business continuity strategies the better it is for us all,” says Chris.
The picture today
Since the 1999 invocation Great Ormond Street's technical infrastructure has changed considerably. Whilst remaining a Novell site for networking purposes, increasingly it is implementing systems based on Windows 2000 using SQL Server databases. This means that many of the old UNIX systems, such as the one recovered in 1999, have been migrated to Windows 2000 or are earmarked for migration.
Similarly, whilst Oracle and Sybase were the main relational database management systems (RDBMS) GOSH is now predominantly a SQL Server site. The healthcare systems market seems to prefer the combination of Windows 2000 and SQL Server to the UNIX and Oracle/Sybase combination favoured in 1999, with economic reasons representing the chief driver for this change.
In July 2001, Guardian iT was acquired by SunGard Availability Services which continues to support Great Ormond Street Hospital as well as Guardian’s other clients. In terms of the business continuity planning required for the new system, this has made things somewhat more straightforward. The majority of the systems now run on Intel-based servers so that the same server specified in the contract with SunGard Availability Services could be used as a recovery platform for a network file and print server or, for example, a pathology database server. Although some isolated UNIX servers are still retained, particularly for the financial systems, and therefore included in our SunGard contract, UNIX is definitely a dying breed amongst purely medical systems.
Testing the system
Fortunately, Great Ormond Street has not had to invoke its business continuity supplier since 1999, although Chris and his technical team recently undertook a test of the hospital's disaster recovery procedures at SunGard's Hitchin Recovery Centre. “This was the centre at which the 1999 recovery took place, and it was good to see some of the same faces from Guardian days who had been so helpful during our previous visit,” says Chris.
The objective of this recovery exercise was to undertake a test recovery of the newly migrated Detente Omni lab Pathology system. This had previously been housed on a UNIX server but currently resides on a pair of clustered Windows 2000 servers. Whilst DR tests of the UNIX version had previously been undertaken this was the first opportunity to test the Windows version and Detente, the system supplier based in Australia were keen to assist in the process.
“As expected, SunGard staff were extremely helpful throughout the two day exercise from planning and booking the recovery days to readying the equipment for our visit.” Chris said.
“Due to an oversight on our part, our technical staff arrived with backup tapes in the Super DLT format to find, as per our contract, the recovery machine had a DLT drive. This is exactly the sort of thing that disaster recovery exercises are for, as we had updated our tape drives without updating the business continuity contract! We will now be changing the SunGard contract accordingly – you live and learn!”
SunGard was able to replace the DLT drive with a Super DLT device overnight between the first and second days meaning that Great Ormond Street Hospital was able to continue its recovery. Once the correct media was placed in the correct drive they were able to recover the system with the help of Detente whose representative logged in the middle of the night to restore the database. The comms experts were able to establish a link back to Great Ormond Street via NHS Net and staff from the Pathology department were able to verify the integrity of the recovered system.
SunGard allowed GOSH to retain the recovered system for an extra day to compensate for the time lost earlier in the exercise, allowing further assurance testing to be undertaken.
Summary
> The situation
Opened in 1852, Great Ormond Street is Britain’s first children’s hospital which offers the widest range of paediatric specialities in the country. When it encountered a serious software problem, which could not be solved in-house, it called upon the help of SunGard Availability Services (then Guardian iT).
> The solution
Within hours of the invocation call SunGard rebuilt the hospital’s file system on a recovery computer.
> The result
The hospital’s vital services were fully recovered in next to no time thanks to SunGard Availability Services.