Northgate Information Solutions plc ‘At A Glance’
Northgate Information Solutions is a market leader of software applications and outsourcing solutions to the public safety, local government, education and human resources sectors.
Headquartered in Hemel Hempstead, with 25 major UK offices, Northgate employs approximately 3,200 staff
The Group has three divisions – Human Resources, Public Services and Managed Services – each with market leading positions
Northgate has more than 2,700 large/medium customers and in excess of 7,000 small/medium customers in the UK and Ireland
www.northgate-is.com
The Largest Explosion In Peacetime Europe?
Occasionally an incident occurs which is on such a huge scale that its name becomes synonymous with disaster. In December 2005, a new name entered this category: Buncefield.
Just after 6.00am on Sunday 11th December, an explosion of monumental proportions occurred at the Buncefield Oil Depot, close to Junction 8 of the M1 motorway in Hemel Hempstead. The depot was one of the largest fuel supplying facilities in the country and stored aviation fuel as well as petrol and other oil-based substances. The explosion was later considered to be the largest ever in peacetime Europe and, for one company, it resulted in what may be the largest and most complex business recovery project ever undertaken.
The headquarters building of software and IT services company, Northgate Information Solutions, was one of the closest to the source of the explosion. Extensive blast damage occurred and several parts of the building were set on fire. When damage assessors were able to reach the building, the devastation was all too evident.
Advance Planning Pays Off
Northgate had invested significantly in business continuity and high availability provision. Two backup generators were installed on site; full UPS protection was in place; and multiple communications networks were in place. The entire on-site business continuity infrastructure was destroyed as a result of the explosion. This created a major business continuity problem, not just for Northgate but also for the company’s clients. Northgate is a leading UK supplier to the human resource and public service markets and the company works with approximately 90 percent of UK local authorities, all UK police forces and is active in the education, utilities and corporate sectors. Northgate’s clients outsource various critical processes to the company; including payroll, web services and data processing. All of these were crucial services, none more so than a £1.4 billion pay roll run which was due by the end of December; if this was not carried out the citizens within four London Boroughs and the staff of 186 payroll clients would be personally disadvantaged at a critical time of the year.
The responsibility to ensure that the company responded quickly and effectively to the myriad business continuity issues arising from Buncefield fell on the shoulders of Northgate’s Emergency Recovery Team, coordinated by Mark Farrington.
Invocation!
At 6.10am on that Sunday morning, Mark received a telephone call notifying him of the disaster. He immediately placed SunGard Availability Services on standby and convened a tele-conference with the rest of the Emergency Recovery Team and the company’s executive. This took place at 7.15am and crisis management roles were confirmed. The decision was taken to meet at Northgate’s Holborn office at midday; from here the team would manage and coordinate resources in line with the business continuity plan.
Concurrently SunGard, having being invoked, started to make systems and assets available for both IT and Workplace Recovery.
By the Monday morning, a team of over 100 project and technical staff commenced the recovery of customer systems within the SunGard London Technology Centre facility and other Northgate designated recovery locations.
The first step was to perform a triage and determine recovery priorities, with customer systems being prioritised over Northgate’s own internal systems. The scale of the recovery soon became apparent: a total of 212 systems relating to 209 customers needed to be re-established.
Concurrent with the systems recovery work, another urgent issue had to be quickly resolved by the Emergency Recovery Team; how to accommodate Northgate staff displaced by the loss of the headquarters building. The team decided to temporarily relocate these staff in other company sites whilst suitable alternative accommodation in the Hemel area was located.
A shift working system was established with recovery staff working back to back 12-hour shifts across 24 hours; and the recovery team now settled into a daily routine. A conference call was made at 5.00pm each day, which was chaired by Mark Farrington. Here the Emergency Recovery Team gave an update on activities and set priorities for the coming 24 hours. This was conducted daily for the first month of the recovery.
After the first week, the majority of production services had been restored enabling core activities to recommence. The crucial £1.4 billion pay roll run was conducted on time - a great achievement given the scale of the difficulties surmounted to get to that point
Lessons Learned
According to Mark Farrington the following lessons have been learned by the Emergency Recovery Team to date:
The importance of adequate insurance
Northgate’s financial exposure was limited by its insurance policies. These provided sufficient cover for the building and its contents as well as for business interruption. This enabled the board and the company’s investors to be confident that the long-term impact of this incident would not threaten the company’s profitability.
The importance of having a good crisis response team in place
However good your risk assessment process – and Northgate’s was very thorough – the possibility that an unpredictable disaster will come into play still remains. Buncefield was such for Northgate. Nothing could have been done to mitigate such a disaster, but having a designated and well trained Emergency Response Team in place meant that the company could immediately begin to plan its recovery.
The importance of effective and timely crisis communications
Within 24 hours of the incident, Northgate had issued a statement to the City. This helped reassure customers, stakeholders and the financial markets that effective recovery plans were in place and that the company’s future was secure.
Specific crisis communications measures also ensured that Northgate employees were kept up-to-date. Within 36 hours a staff intranet site had been set up. This provided employees with the latest information and was regularly updated. In addition a video message from Northgate’s CEO was included; explaining the situation and detailing the business continuity processes that were underway. A text-based cascade communications system was also used and a 0800 number was established, giving a recorded ‘message of the day’. This proved so popular with staff that it had to be extended with additional lines.
The importance of managing expectations
When commencing a recovery of this nature there is a relatively low level of expectation; but as the recovery gathers steam, expectations can run ahead of what is actually possible. It is therefore important that expectations are managed. For Northgate this was done through frequent and clear communications; both within the Emergency Response Team and then out from the team to the executive, staff, customers, and other stakeholders.
At the time of writing, three months on from Buncefield, although it has returned to ‘business as usual’ operationally, Northgate is still using SunGard facilities and will be doing so until the company establishes a new HQ and data centre. The recovery is complete, and such was its success that Northgate collected the Most Effective Recovery of the Year award at the 2006 Business Continuity Awards ceremony held in London in May this year.