When assessing critical infrastructure environments, security teams should always ask: What is considered an acceptable risk? Is there a limit on the danger to life? Is it one life, 100 lives, 1,000 lives? According to recent news reports, a hacker gained unauthorized entry to the system controlling the water treatment plant of Oldsmar, Florida, a city of 15,000. The hacker tried to taint the water supply by increasing the level of a caustic chemical, lye, in the water supply. This act exposes a danger that has grown as systems become more computerized and accessible via the internet and remote connectivity. Was this an act of domestic terror or was it a prank gone terribly wrong? Is this a target considered worthy of a nation-state bad actor? Maybe not, but it surely is a target worth protecting from a hacker who can stumble upon an open remote session where they can do damage. This is a small municipal water treatment plant, one that probably didn’t consider itself a target. This plant has now disabled remote access. The management team realized the risk and anticipated this attack could happen. Over the years, many other municipal providers have observed that they are not targets of potential attacks. But this case shows the risk is now very real.
What was amazing about the water hack is that a supervisor saw a mouse moving on its own, and he stopped the attack. This is great threat detection, but with all the remote work being performed, what are the chances that the hacker could have accessed a system that wasn’t well monitored? Would the lye have made it into the water supply? How many of the 15,000 residents could have been injured? The fact that the remote session was hacked is significant and points to the basic lack of cybersecurity in some critical infrastructure networks. Taking down the power grid may be problematic but contaminating the water supply can be deadly. The fact that the hacker “briefly increased the amount of sodium hydroxide by a factor of one hundred (from 100 parts per million to 11,100 parts per million)” indicates that the numbers were changed … that fact equals danger! Are there backup controls? I am sure there are, but if a hacker has access to your internal systems, there is a potential for anything to be overridden.
The Oldsmar water plant was also in two prior data breaches dating back to 2017. It appears their credentials may have been exploitable for a while. Although smaller organizations do not have large security budgets, there is a need for more effective account monitoring and control. This does not necessarily require a big investment. If they had been more in tune with this important security control, then uncontrolled access would have been eliminated back in 2017 and the plant would have been rendered more secure.
What should be done to protect municipalities? Power plants that provide 1,500 MW of power are subject to NERC CIP regulations. What about the smaller power plants or water treatment plants such as the one in Oldsmar, FL? This is not to suggest that these smaller providers should be more formally governed, as regulations drive compliance, but not security. Public utilities should be held responsible for the communities they serve. These smaller plants are often underfunded and understaffed, which increases risk. Ensuring security requires a commitment not just in words but also in the funding of effective cyber controls.
Aside from regular risk compliance assessments, there is a need to ensure municipalities are providing necessary controls that address cybersecurity in their operations too.
Key steps to ensure basic cybersecurity in ICS environments:
- First and foremost, develop a cybersecurity policy that aligns with best practices, reference: CIS Controls to Industrial Control System environment, and reevaluate it regularly.
- Enforce strong passwords with frequent changes (at least 90-day intervals). This could possibly have alleviated the unauthorized access in the Oldsmar case, but in any case, change your passwords, folks!
- Consult with any of your OEM automation providers and maintain their recommended updates. Almost every OEM provides its customers with a list of approved updates. They will try to sell you automated solutions to help you push this information but consider investing in a resource to maintain these updates. Don’t shortchange basic cybersecurity best practices.
- Use change control for everything. This is a pretty common practice, keep it going.
- Patch your machines – these security patches are free but do require resources. Again, consider using the patch list, but invest in the resource. This could be a contracted resource whose sole purpose is to come in once a month and manage your security updates.
- Do not overlook your network equipment, lockdown basic ports, and keep your firmware updated.
- Use secure remote access methods – this may require some research or some professional consultancy. With the increasing remote workforce, remote access is a growing reality. Take the time and money to secure this important resource.
- Use and update anti-malware on any machine that will accept it. Again, consult the OEM and get a list of approved updates for your systems.
- Backup your critical machines and test those backups, regularly. These machines don’t change their configuration often, but they should be backed up and tested at least quarterly.
- Again, do not ignore your network equipment, back it up, compare configurations with the last backup, and investigate any changes. If your network switch, router, or firewall has changed in the last month, be suspicious.
- Set basic limits on controls that would reduce the risk of hazardous outcomes, check with your OEM provider to set these limits so you don’t affect production.
This list can be a foundational guide, as every plant no matter its size should prioritize its obligation to its communities to keep it safe as a critical responsibility. Additionally, cybersecurity needs to be considered in every budgeting cycle and cannot be shortchanged. If you feel the organization needs help addressing these issues, get some professional guidance, have a risk assessment done, and follow basic cyber best practices to keep your community safe.
Follow me on LinkedIn.
To find out more about how we can help you, visit our Secure IoT/OT Services page.
OT Solution Architect | NA Cyber Center of Excellence
Experienced Senior Solutions Architect with a demonstrated history of working in the oil & energy industry. Skilled in Firewalls, Network Engineering, Network Security, Wireless Networking, and Cross-functional Team Leadership. Strong engineering professional with an AAS focused in Computer Science from Tampa Technical Institute.