Beating IT Risks

•Fortnightly reporting to the programme team with regular, planned-for audits on project and technical delivery.

•Recognizing the value of offshore development but the real challenges faced in communication between on-shore and offshore.

•For offshore development projects, the project manager remains onshore while the project focus, and the balance of the project team, typically moves on-shore (definition, outline design), offshore (detailed design, development, system test) and finally on-shore (acceptance and implementation).

•A project is not finished until the benefits are delivered.

The result has been the creation of effective project teams delivering projects, and their benefits, on target.

Printed with permission of PA Consulting Group

9 Infrastructure

Servers on wheels

In 2003, two thieves, who posed as authorized computer technicians from the outsourced provider EDS, were allowed access to a highly secure mainframe data centre and after two hours emerged wheeling two servers away. Worryingly, in the days of international terrorism, this theft was from Australia’s Customs department in Sydney’s international airport (ABC, 2003; SMH, 2003). It later emerged that two further computers were missing and this was not picked up for some months due to poor asset tracking.

Upgrades and merger integration don’t mix

In April 2004 ANZ bank stopped a major project that had been running since 2001. The Next Generation Switching project had aimed to deploy new infrastructure technologies into the core transaction network of ATMs and EFTPOS machines. Continuing with the project at the same time as completing integration with the recently acquired National Bank of New Zealand was considered too risky (AFR, 2004e).

Difficulties with the US Navy fleet (of PCs)

Early in 2004 EDS realized they were in trouble on a US$ 8.8 billion contract to support the desktop fleet of the US Navy. Some of the contributing reasons include: a failure to understand how military requirements differ from the requirements of corporate customers, vast underestimations of scope and complexity and poor estimating of the cost and difficulties of servicing the fleet of 345 000 PCs. EDS project losing US$ 400 million on this deal in 2004 (WSJ, 2004).

While the boxes and wires are, for most of us, the least sexy part of IT, they are nevertheless prone to their share of risks. This chapter helps you divide and conquer the IT infrastructure landscape, understand the evolving nature of infrastructure risks and guide you on both keeping the lights on and successfully executing major transformation.

How IT infrastructure failure impacts your business

The delivery of IT services requires a solid foundation. Do you have adequate capacity to process and carry the volumes of data that will flow through your organization tomorrow?75 Will systems stay up and running when they are needed? Can they change to meet evolving requirements?

If your infrastructure fails, the applications and IT services can come tumbling down like a deck of cards. In thinking through the potential impact, it is most useful to decompose the IT infrastructure landscape into five generic areas: facilities, centralized computing, distributed computing, data networks and voice networks.

Additional types of IT infrastructure are required in different industries – these are outlined here at only a high level.

Facilities

All of your physical sites and buildings are points of vulnerability. The essential ingredients for an IT installation are power, air-conditioning and physical safety and security of the installed devices and their operators.

Recent widespread power failures in Europe and the USA have illustrated the reliance many organizations have on ‘the basics’ continuing uninterrupted – or being rapidly restored.

While some environmental threats – earthquake, fire and flood – can dramatically strike the facilities, less dramatic but occasionally impressive impacts can result simply from a rodent’s gnawing.

When co-locating you are sharing facilities with others. While risk reduction advantages due to the professional management of the facilities and the building compliance with engineers’ data centre specifications are obvious, there may be a downside. More people will be coming and going and deliberate attack – from outside or turned insiders – may be more likely on the site due to the accumulation of targets.

The risk mitigation measures for facilities include: back-up sites, uninterruptible and redundant power supplies including back-up power generators, redundant air-conditioning units, bunker-like construction, access control security measures, monitoring and escorting of visitors. Managing IT risks in this space overlaps with ordinary building and facilities management, but has plenty of special ‘twists’.

75 The World Health Organization experienced an unprecedented spike in demand for its online services when the SARS outbreak suddenly positioned them as the authoritative source for the public. Capacity planning based on historical trends would not have been sufficient to prepare the organization and Maximum Tolerable Outage perceptions changed overnight.

Centralized computing

The centralized computers, the enterprise application and database servers, are most often identified as the heart of the IT landscape. Certainly the data centre managers would have us all think so!

Failure

If these centralized systems keel over then all hosted services go down, information processing ceases and all users are impacted. Depending on the nature of the outage, information assets are also at risk of loss or corruption.

In cases of deliberate and malicious attack on your systems these risks are heightened – if indeed the objective of attack was to damage rather than merely snoop.

Unfortunately often the knowledge of the mapping of IT services to infrastructure is incomplete – the consequences of Box A ‘going down’ are poorly understood and in times of crisis confusion reigns.

Here is the focus of the classic incident and disaster recovery response: bring the standby system into life, recover and restore data, bring up the systems and confirm that all is fine. In Chapter 6 on IT service continuity risks the full recovery of service to users is the objective and this step is simply one of many along the way.

We will proclaim again here – given the continuing lack of focus of organizations on these practices – it is essential these basic procedures exist in a documented form and that rehearsals and tests of the DRP are undertaken regularly.

The ‘gold-plated’ technology solution that many aspire to and few attain, often because of the price tag – is to have no single points of failure, full and automatic fail-over to a ‘hot’ standby site that is a mirror of the production platform. If chasing this technology dream, do not overlook the need to bring the right people on the journey.

Performance degradation

Perhaps the less obvious risks relate to degradation of service from the centralized computing platforms.

When the volume and timing of IT transactions is unpredictable – an increasingly common situation for all Web-based systems which cater for customers as end-users – an unacceptable performance degradation can occur with relatively little notice.

This occurred in early 2004, when Qantas announced its new bargain domestic Australian airline – Jetstar – with a fanfare offer of discounted tickets, and Virgin Blue – its competitor in the space – immediately responded with a ‘see you and double’ counter-offer. As bargain-hunting customers went into frenzy –

both on the phone and via the Web – those customers of Virgin Blue who had already bought full-fare tickets (including one of the authors!) were caught in ‘holding patterns’ on the phone and Web as the underlying systems ground to a halt (AFR, 2004g).

The need for tight capacity management, including proper planning and allocation of resources in advance of the demand and facility to scale (expand) the infrastructure is key to being able to avoid severe consequences from systems running ‘like a dog’.

Third party reliance

As almost universally the infrastructure components are procured rather than built in house, other wider risks relating to centralized computing platforms tie back to technology vendor risks (explored in more detail in Chapter 7 on IT service provider risks):

•The willingness for vendors to sign up for a ‘fix on fail’ deal within guaranteed response times is contingent on you staying on the current versions of hardware and software;

•The provision of proactive patching and maintenance releases of technology products is similarly reserved for current and supported versions (Lawson, 1998); and

•Operations and integration problems in a multi-vendor environment can increasingly result in finger-pointing when failure occurs that can’t be slated back to an individual product’s performance.

To keep in step with vendor-supported infrastructure components you’ll need to make timely decisions to migrate and bring the business community along with an appropriate justification. Many companies have found themselves in a tight spot due to hardware obsolescence and expiring vendor support.

Distributed computing

Assets in the distributed computing infrastructure – most obviously the individual user’s client device such as a PC – are less important in their own right than the critical servers. Loss of a single device will typically impact only one user’s IT service experience.

While frustrating for the user – who amongst us hasn’t ranted at the poor helpdesk staff about PC failings! – this typically isn’t a headline risk item.

The issue of information asset loss or exploitation – certainly made much easier when IT assets are portable – is dealt with in Chapter 6. The key mitigating action is to host enterprise data on servers rather than PCs, or at the very least

to offer some form of back-up solution for those with enterprise data on ‘their’ PCs.

Sitting between the centralized and the end-user IT asset are a host of other network, access, local services and security computing assets, including for example local area network file and print servers, remote access and firewall devices. These are typically exposed to a mix of the risks associated with centralized computers (many users can be impacted when they are down) and end-user assets (they are typically easier to steal or damage than the data centre kit).

When we draw together the entire distributed computing base, a number of key issues arise:

•Viruses and other unwanted software can spread rapidly and infest a distributed computing environment;

•Attacks on the centre can be staged from an outpost;

•Liabilities for operating with unlicensed software can be significant;

•Costs can be notoriously difficult to understand and control; and

•Service delivery consistency continues to be elusive to even the specialist managed IT services providers.

Best practices in control of distributed computing assets require the establishment and maintenance of a baseline – a ‘source of truth’ asset register – introduction of standard operating environment and software loads, a ‘lockdown’ approach where only administrators can alter the base configuration and remote administration techniques. Specific products have their part to play – such as subscription-based anti-virus software.

Maintaining currency – again important for vendor support – is also often required to satisfy the appetite of increasingly demanding and IT-sophisticated end-users. However, the costs of ‘refreshing’ the desktop fleet can be far from refreshing for the managers who foot the bill!

Data networks

Connectivity is an increasingly important feature of the corporate IT environment. When links are down, or bandwidth constrained, IT service quality can degrade rapidly.

Network redundancy and resiliency – alternative routes for the traffic to follow

– are key traditional design principles to adopt to reduce risks of individual link outages.

However, when you or your customers are using shared and public networks, such as the Internet, your business remains vulnerable to system-wide service disruptions. A key question for every business manager at risk of IT failure is:

How vulnerable is your core business to failures or extended outages of the Internet?76

In answering this question, consider four common IT services that are Internetenabled:

•Customer access to on-line services offered by you;

•Supply chain partners linking with e-business applications;

•Your staff accessing on-line services for business purposes; and

•Use of email for business transactions.

In addition to Internet reliance issues, specific carrier and managed network service provider dependencies are common, along with the associated third party reliance.77 Many customers will be subject to standard terms and conditions which are set out by the network carriers and may not truly reflect their service standards and risk management requirements; larger customers may be the only ones with negotiating strength to dictate terms.

As the world increasingly converges to Internet Protocol standards for data networks, issues of security and bandwidth management reign supreme:

•A plethora of technology products and services purport to offer at least part of the ‘answers’ to the continually evolving security ‘problem’. These offer to authenticate and identify users, establish secure sessions and encrypt traffic, protect from intrusion, detect intrusion, gather forensic evidence, etc. However, for every new security defence there is a new form of attack, requiring the ongoing vigilance of specialists.

Further network security management considerations are covered in Chapter 6 on Information asset risks.

•Issues of effective bandwidth management – achieving delivery and performance certainty – over mostly shared, public and packet-switched networks remain problematic.

In the meantime the use of connection-oriented services goes on – the old ‘leased line’ remains the only real alternative for securing point-to-point data network performance – and this suffers from being an obvious single- point-of-failure.

76In a recent online survey (Whitman, 2003) of IT executives, 95% of respondents use the Internet to provide information, 55% to provide customer services, 46% to support internal operations, 27% to integrate value chain partners and 18% to collect orders.

77A telecommunications failure in Australia resulted in the loss of half of a major bank’s ATMs and the Australian Stock Exchange was forced to close, suspending trade for a morning (ABC, 2000a).

Emerging technologies continue to offer up challenges. For example, many early adopters of wireless data networks have exposed themselves unwittingly to security vulnerabilities.

Voice networks

The integration of voice and data networks is a contemporary trend that will perhaps make this part of the IT infrastructure landscape defunct in the next few years.

For now, most companies have the ‘fall back to phone’ option when the computing and data network infrastructure is down. Rarely is the PABX and public switched telephony service (PSTN) down at the same time.

Those migrating to ‘Voice over IP’ may well benefit from lower costs, but most will have a heightened vulnerability to outage due to the consolidation of both voice and data communication over the same transport.

As the traditional voice network component is a relatively stable and mature part of the enterprise’s IT infrastructure, with an acceptable performance track record, it is likely not to be the focus of risk management attention. Those with tightly integrated computer and telephony, inbound or outbound, call centre infrastructure are perhaps the exception to this general rule.

Industry-specific infrastructure and risks

The ubiquity of computing technology was highlighted in the Y2K scare which gave coverage to the risks of flaws in distributed and embedded computers: lifts not working, office building doors not opening, electricity generation and dispatch functions stalling, robots on manufacturing plants going wild, aircraft control systems failing, etc.

While the specific issue – inability to handle the date change event – is now behind us, what became obvious during the Y2K period is the high reliance on computing technologies across all spheres of human endeavour.

As an example, recent work on process control systems highlighted particular vulnerabilities in the changing risks of the industry-specific infrastructure challenges:

•Historically, process control systems were designed and built using proprietary technologies and installed in isolation from ordinary IT systems. However, recent trends have been to base newer systems on more cost-effective platforms (e.g. standard PCs).

•Furthermore, the desire for remote control and management information has led to the adoption of common network protocols and the connection of many of these systems to the corporate IT network.

•While these changes have certainly yielded many business benefits, it has also meant that control systems now possess more security vulnerabilities and are

increasingly exposing these weaknesses to the same threats faced by corporate networks – notably viruses and malicious hacking (US NIST, 2001).

IT infrastructure’s evolving risks

As your IT infrastructure evolves so do the risks. Understanding this evolution will help you to frame your risk management approach accordingly.

Migration of IT application features into the infrastructure layer

Traditionally when programmers were set a task of writing a new application they started from scratch. Computers needed to be directed at a low level and in precise detail, tailored to the hardware, to perform even basic tasks – like storing a piece of information in memory.

Over time the features of operating system software grew to fill out the common requirements of managing input, output, processing and control of peripherals.

Other software was developed to handle files, databases, certain types of system operations – such as back-up and recovery – and various utilities were built to run as companion and support tools. Some were bolted onto the operating systems and others became adjuncts.

Application features that might have required laborious coding and customization many years ago can be programmed quickly now with a ‘hook’ to an infrastructure layer feature.

While application development has become more efficient, the reliance on the infrastructure layer has become more significant. Vulnerabilities in acquired infrastructure components are inherited by any system that is built on them.

For example, a database management system may allow ‘back door’ access so that data is able to be changed directly. This infrastructure element and its associated tools, such as low-level file maintenance programs, can detract from the integrity of the application. Thus critical information can be easily corrupted without the normal controls imposed by the application.

Market dynamics of infrastructure

The cost of IT progress – new IT infrastructure that is faster and better and has more features – is obsolescence. New products attract new customers – and this is where vendors fight to stay in business: growing and retaining market share. If you have chosen an IT vendor without a new product pipeline they won’t be in business for long.