Cybersecurity: Water Utility Security Part 9

In this final post on cybersecurity in the water utilities industry, we’ll look at remaining areas of interest that center around information and communication. It’s evident by this point that lack of knowledge is one of the greatest threats to infrastructure security. The more awareness an organization has, the simpler it is to take the necessary steps to create an environment that is resistant to cyber threats.


Who needs to be educated about security practices, policies, and procedures? Internal employees are at the core of any training program. But clients and service providers shouldn’t be overlooked. Any third party that is involved with PCS systems or other vital technology should be aware of potential risks. SLAs should include relevant certifications. As a whole, the water utility should participate in water sector programs that are designed to identify and implement best practices.

Here are a few of the other actions that the AWWA recommends in terms of education:

  • Regular testing of security procedures to evaluate security awareness and incident response
  • Cross-training for IT and PCS staff so knowledge is shared and potential gaps in security can be more readily found
  • Training for all staff to identify common risks and threats such as social engineering
  • Communications training (network & radio) for PCS technicians to ensure proper, secure communication and crisis response capability

Personnel Security

From hiring through termination—and even after—employees and contractors can hold the keys to cybersecurity for an organization. Ensuring that each worker has only the access level required to accomplish their assigned duties is an important aspect of personnel security. Network and facility access should be set up for immediate, automatic revocation upon termination so that no unauthorized individuals are able to infiltrate the system after leaving the organization.

Vetting prospective workers prior to hiring may help spot red flags and at-risk candidates. A formal (standardized) background check process should be implemented for each level of responsibility within the agency. All new hires should also be required to sign appropriately worded confidentiality and cybersecurity policies. Such policies should be reviewed and signed again annually by all workers.

That’s it for our Water Utility Cybersecurity series. We hope you have found this overview helpful. For more information on keeping your facilities and workers safe, contact DKF Solutions for a consultation.

Cybersecurity: Water Utility Security Part 8

Our series on cybersecurity for critical water infrastructure now focuses on how to ensure ongoing operation based on vendor agreements, procedures, and workflows. While details like contract negotiation and employee policies might not seem like a strong line of defense, they can make a big difference in whether a utility will remain up and running in a crisis.

Service Level Agreements

Although process control systems should be designed to run independently when necessary, the day-to-day operation of a water utility typically involves quite a bit of third party infrastructure. As with the majority of organizations today, public utilities rely on a range of providers for telecommunications, internet connectivity, power, network capacity, data storage, and other resources. The contracts that govern factors such as availability (uptime), bandwidth, and technical support are referred to as Service Level Agreements (SLAs).

Selecting an appropriate level of support is important for a utility. For critical infrastructure, guaranteed uptime and a fast response time in the event of disruption may be two areas of particular concern. As the AWWA points out, the bandwidth required to run PCS equipment is often not high. But it must meet minimum requirements. SLAs should be negotiated with each contracted vendor based on how emergencies might impact process control systems and related infrastructure. Agreements with third party integrators and companies tasked with servicing the PCS equipment itself should also be reviewed to ensure that the utility is appropriately prioritized as a preferred customer when it comes to response times. Limiting the total number of external vendors involved may help simplify this process.

Operations Security

OPSEC, the formidable acronym for Operations Security, can cover almost any area of procedures and workflows in an organization. Limiting access to information is one important aspect of cybersecurity. For example, a utility might have a social media policy that prohibits workers from posting information about internal procedures online. Such policies should be in writing and the accompanying training might give examples of the types of postings that might seem innocent to employees but that could reveal potential vulnerabilities to hacker or parties interested in doing harm. Cybersecurity training should alert employees to suspicious behaviors—such as people fishing for information about security protocols or other protected information (social engineering).

Within the organization, OPSEC should also cover isolation of PCS functions from other business functions. This includes ensuring that the equipment’s interfaces are blocked from accessing the internet, email, and other remote systems—up to and including removable media. IT staff and other involved personnel should receive ongoing training in cybersecurity for PCS and water utilities in general to ensure best practices are kept up to date. As with all areas of security, change is inevitable.

In next month’s post, we’ll wrap up this series with a look at education and personnel security, since a water utility’s security is only as good as its employees. 

Cybersecurity: Water Utility Security Part 7

This month, it’s time to take a look at both virtual and physical access for process control systems security. When both of these areas are addressed, infrastructure is better protected from on-site and remote attacks. The American Water Works Association (AWWA) offers guidance on these topics. Here’s a quick overview.

Telecommunications, Network Security, and Architecture

The wired and wireless aspects of network infrastructure come under scrutiny in this phase of cybersecurity. On a physical level, computer rooms, network server closets, and individual cables should be secure from tampering. Port level security can serve as a second layer of protection if physical security is compromised. It shouldn’t be possible for anyone to simply plug a device in to an open port and gain access to (and control over) the system.

In the virtual sphere, data must be secure as it travels from point A to point B. On the network level, using dedicated hardware, separate IP subnets, and virtual local area networks (VLANS) can make systems and processes easier to protect both within the server architecture of the organization and where the network must interface with field equipment and 3rd party systems. It may be wise to create an architecture that allows for critical equipment to continue operating in isolation (in the event that other parts of the network are compromised).

More about Physical and Network Security

In the words of the AWAA “Once physical access to a network device or server is achieved, compromising equipment or systems is usually a trivial matter.” That’s a chilling thought given the percentage of malicious security breaches that are carried out by internal parties (about 25% according to a 2014 Forrester survey). For critical infrastructure, it is vital that only authorized personnel have access to hardware—and only for needed activities. Control rooms, removable media, cabinets, ports, and communication pathways should all be hardened against intrusion.

Physical locks and electronic access control help keep unauthorized personnel away from critical equipment while monitoring systems provide an alert of potential trouble. Security information and event management (SEIM) detection within the network can also report on anomalous activity in real time. In some situations, video surveillance may be beneficial for identifying unauthorized entrants. But bear in mind that prevention is always the primary goal. As with all monitoring programs, having personnel in place to evaluate and swiftly respond to incidents is essential.

Operational Security and Service Level Agreements are up for exploration next month!

Cybersecurity: Water Utility Security Part 6

The issues surrounding cybersecurity in the water utility industry become more complex and technically demanding as the focus becomes more granular. This month, we continue the shift from overarching “big picture” best practices that remain fairly stable to a moving target of security and encryption protocols that may change on a monthly basis.

In fact, application security and encryption can’t be pinned down perfectly because there are too many variables to consider—and hackers are always looking for new ways to penetrate online systems. For this reason, cybersecurity measures in these areas must be subject to ongoing testing and review with resources committed to frequent fixes, updates, and upgrades. Any other approach makes it simply a matter of time until a breach occurs.

Application Security

Securing an application begins in the design phase and continues through testing, deployment, monitoring/maintenance, and through to obsolescence. One common area of vulnerability is Insufficient Transport Layer Protection that fails to protect network traffic, leaving data and session IDs exposed. Knowing enough to ask the right questions is essential during vendor selection to avoid these risks. Software vendors and system integrators must demonstrate that their applications and processes have an appropriate level of integrity. There are also actions that can be taken at an administrative level within the organization to promote better application security.

Examples of current smart practices: Each PCS user should have their own login (username and strong password). This login should be different from the user’s login for other business apps and provide access only to those program capabilities required for the user to perform their job. Administrator privileges should be given only to administrators, and all application usage should be logged, monitored, and reviewed regularly.

No application can be guaranteed to be entirely secure, but the level of security is enhanced with constant vulnerability monitoring to identify weak points and address them quickly.


In simple terms, encryption is about keeping information away from prying eyes through the use of cryptography (codes). Data must be protected both in storage and during transmission from one point to another. Encryption schemas may include compression algorithms, Virtual Private Networks (VPNs), and other components to provide well-rounded security. The appropriate type and level of encryption should be applied to databases, laptops and computers, mobile devices, wireless and wired communication, removable storage devices, and so forth. The best practice is typically to use the highest level of encryption available for a given piece of equipment or system.

Encryption keys themselves should be treated with special care throughout the lifetime of the keys and the data they are intended to protect. They must be backed up and managed to prevent loss, theft, or unintentional destruction. Key vaults and similar environments with restricted access and redundant storage capacity may provide a solution.

Above all, water utilities should ensure that encryption is more than window dressing. According to the AWAA, Weak encryption schemes are particularly dangerous because they provide little protection and create a false sense of security and complacency. Proprietary encryption schemes should be avoided since they typically have not gone through comprehensive testing and often contain flaws. Also, only encryption schemes that are referenced by appropriate standards and use keys of proper length should be considered secure.” Encryption only works if it addresses real world risks.

Next up, Telecommunications, Network Security, Architecture and more!

Cybersecurity: Water Utility Security Part 5

This month, we’ll take a look at some technical and administrative aspects of cybersecurity for the network systems that keep a water utility going. The goal of these AWWA guidelines is to maintain the confidentiality, integrity, and availability (CIA) properties of the computing resources used by a water agency.

Server and Workstation Hardening

Application, web, and database servers as well as the individual workstations they support offer many potential openings for attack. There are a wide variety of best practices for hardening these resources against intrusion.

Workstation hardening begins with physically securing the client-side devices (for example, ensuring locations where workstations and control consoles reside are protected using an electronic access control system.) On the IT side, here are some more security measures:

  • Patching/upgrading vulnerable apps and services
  • Prohibiting addition of new services without IT review and approval
  • Eliminating unused, unnecessary, and non-secure programs and services

Similarly, it’s important to keep up with hotfixes and patches on the server side. It may be wise to disable unnecessary network services, registries, executables, and test scripts that are known to be insecure. Properly restricting permissions for files, services, end points, and registry entries is also smart. Of course, the specific security measures vary based on operating system and require careful review to ensure functionality isn’t compromised.

Access Control

Here are some basic security steps for access control to water utility networks:

  • Restrict permissions for files and data according to end-user roles
  • Automate user provisioning (and de-provisioning) to standardize and streamline the process, reducing the risk of errors and delays
  • Use dual-factor authentication such as a passcode and a chip-enabled card or fingerprint scanner for critical systems
  • Audit access on a regular basis so you know who is accessing which resources

The most obvious step in access control is ensuring that user passwords throughout the system are strong and difficult to guess. As a next step, implementing Single Sign On (SSN) gives users access to multiple applications without re-entering their information. Making it simple to sign in supports the best practice of signing out whenever a person is away from their workstation. However, the AWWA points out that SSN shouldn’t be used to allow users to access both process control systems and enterprise systems. As always, shielding PCS is the key to safeguarding critical infrastructure.

More to come: Next month, we will explore application security and encryption. 

Cybersecurity: Water Utility Security Part 3

The threat to infrastructure from online attackers has become a matter of national security over the past few years. According to the Department of Homeland Security, “The Water and Wastewater Systems Sector is vulnerable to a variety of attacks, including contamination with deadly agents, physical attacks such as the release of toxic gaseous chemicals and cyberattacks. If these attacks were realized, the result could be large numbers of illnesses or casualties and/or a denial of service that would also impact public health and economic vitality.” The DHS points out that critical services and systems such as firefighting and healthcare, energy, food, agriculture, and transportation systems would all face challenges or break down altogether if water service was interrupted.

Process Control Systems Are Already Under Attack

The American Water Works Association offers additional insight into the scope of the problem: “Cybersecurity is an increasingly important issue for water systems for a few reasons. First, water systems have become more automated to improve operational efficiency, including the expanded use of supervisory control and data acquisition systems for treatment plant and distribution system operations. Second, hackers and state-sponsored organizations are increasingly targeting process control systems (PCSs) for malevolent attacks. Discussions with water systems and reported incidents reveal that many systems receive hundreds of attempted attacks and probes on a daily basis.”

It’s evident that, as one of the 16 critical infrastructure sectors in the U.S., water utilities need to do a better job of protecting their process control systems from cyber-threats. But how exactly can this be accomplished?

A Framework for Cybersecurity Has Been Created

The National Institute of Standards and Technology has collaborated with the AWWA to develop guidelines to help water and wastewater utilities better protect their critical systems. As of 2014, this cybersecurity framework covers 12 key areas:

  1. Governance and Risk Management
  2. Business Continuity and Disaster Recovery
  3. Server and Workstation Hardening
  4. Access Control
  5. Application Security
  6. Encryption
  7. Telecom, Network Security, and Architecture
  8. Physical Security of PCS Equipment
  9. Service Level Agreements
  10. Operations Security
  11. Education
  12. Personnel Security

In the ensuing months, we will explore each of these areas of cybersecurity for the water utility sector. Stay tuned next month for a look at Governance & Risk Management—the big picture stuff that needs to be understood at the outset of developing a comprehensive plan for reducing security risks.


Cybersecurity: Water Utility Security Part 2

Last month, we began an exploration of the risks posed to water utilities by lack of adequate cybersecurity. Now, it’s time to understand why organizations aren’t doing more to address these modern threats.

How Widespread Is Lack of Preparedness?

Very. According to a 2014 report by Unisys and Ponemon Institute (“Critical Infrastructure: Security Preparedness and Maturity”), more than two-thirds of utility and infrastructure agencies admit to having at least one incident of compromised security that led to data exposure or operational disruption. Yet fewer than one out of three of these organizations considered security one of their top 5 priorities. Less than 20% of companies interviewed were operating at a mature level of cyber security.

What’s Holding Utilities Back?

Most public agencies have, historically, been very slow to adopt new technology. There are several reasons:

  • High costs
  • Perceived risks
  • Practical and technical challenges

The last problem is one of the trickiest to resolve. Even systems that were once considered state-of-the art weren’t built with today’s digital world in mind. For example, it’s common for utility companies to run their technology infrastructure on very old or unpatched versions of Windows—the same O/S that was initially put in place when systems were first computerized.

Upgrading can’t be achieved simply by throwing money at the problem. Legacy modernization requires a great deal of preplanning and risk mitigation to avoid disruption of services. After all, the operation of utilities has physical, real-world consequences. A failed upgrade doesn’t just mean people complaining on message boards about a bug in a smartphone app. It can entail people living without water and sanitation.

The Wait and See Approach Remains the Norm

It’s no wonder that the time and cost involved in updating utility systems to make them more secure is often viewed as prohibitive. Cybersecurity as an unavoidable cost of maintaining a system can be a hard sell. The only upside to making a system secure is that it continues to function as customers expect. There’s no visible benefit to having better security as long as things are going well. Unfortunately, many agencies simply cross their fingers, hoping to avoid a catastrophic event. What should they be doing differently?

In Part 3 of this series, we’ll explore some of the ways water utilities can become more secure.

Cybersecurity: Water Utility Security Part 1

Public utilities are a vital part of national and local infrastructure. No corporation, organization, government, or residence can operate for very long without essentials like running water, sanitation, electricity, and communications. Yet in an increasingly computerized and connected world, the convenience associated with these necessities also poses a risk.

The delivery processes for utilities are increasingly automated. Relying on modern software and remote communication is a benefit overall because it makes systems much more efficient. Web-based software and cloud computing deliver highly scalable and reliable management of critical applications at a low cost. With the advent of the “internet of things”, systems can be monitored, diagnosed, and even repaired remotely. Yet these advances also open up new areas of risk. Any system that is not completely isolated on its own network can be remotely hacked.

Who Would Attack a Water Provider?

Public sector organizations like water and wastewater utilities might not appear to be likely targets. Compared to a major retailer, a local water company simply doesn’t process enough transactions to be an attractive victim. The Dallas Water Utility, one of the largest public water utilities in the U.S., has only 300,000 meters in its system. Even the largest private water utility in the U.S. (American Water) only serves 15 million customers. As a comparison, the Target data breach exposed the financial data of 40 million customers and the name and contact information of 70 million more.

But if the goal is disruption of infrastructure rather than profit, hacking a water/wastewater agency’s computer network makes a lot of sense. A city without water could turn into a disaster zone in short order in the event of a security breach. If the cyberattack on a utility managed to damage the infrastructure itself, the results could be devastating for a community or region.

What Challenges Do Water Utilities Face in Preventing Attacks?

For utility companies seeking to increase the security and reliability of services, the very criticality of undisrupted service is a barrier to implementing appropriate safeguards. Banks have been called “too big to fail.” In the same way, utilities might be considered “too essential to take offline for upgrades.” Unfortunately, that way of thinking could lead to disaster.

In Part 2 of this series, we’ll take a closer look at why so few water utilities make security a top priority.