During the last three decades a lot has happened in the tech space, although comparatively much less has changed in SCADA due to the industrial automation industry as a whole being very conservative.
- Is SCADA old tech?
- What does next generation SCADA look like?
- What are the benefits to uplifting old SCADA?
- Why are we so cautious about OT systems?
- The great challenge to renew industrial software
- What is the greatest threat to operational sustainability?
- What should we learn from the Industrial Automation Software Vendors?
- Where do I start with renewing SCADA?
Is SCADA old tech?
The industrial automation industry on the whole is very conservative. As consumers of automation in industry, we fully appreciate this fact when it comes to automation that affects our personal safety. In more recent times, this cautious approach to automation is creating new types of risks for those operating assets using SCADA. As each year passes without uplift, disaster looms closer and closer. What are some of the risks?
New tech has emerged, and it’s not IoT or IIoT that we speak of. IoT is not a replacement for SCADA, except in the simplest of applications and in most scenarios IoT and SCADA are complementary though IoT is mostly used on non-critical assets. Surprisingly, basic SCADA from 3 decades ago still operates today. Since then, a new generation of SCADA emerged and is used in manufacturing plants, heavy industry, building management, and virtually everywhere throughout the world. If we are not on at least this second generation of technology, we are fully exposed. Now some people enjoy the simpleness or the older technology and the familiarity that it brings. Similarly, some people enjoy driving antique cars, and there is nothing wrong with antique cars, however, antique cars have some challenges with today’s safety standards. Just don’t be in an accident. But consider this. The fastest touring car in the late 80s is slower than the average modern sports hatchback today. That’s out of the box, no modifications and at a fraction of the cost using present value.
As systems evolved through the 90’s, new functionality was added to old platforms as vendors struggle to keep their cash cows alive. This has produced very mixed results. Concatenation of new functionality to existing products without the opportunity to revisit fundamentals and change them, inhibits the software developers. In most cases you just can’t start again. Just to keep up, sometimes legacy products can be rendered as “next generation” software with new brochures and a face lift. Concatenation can easily become the norm for “major” functional uplifts but really, underneath, the core remains with its dependencies on the old technologies. That isn’t all bad, but perhaps if it remains constant for more than a decade it could be questioned.
What does next generation SCADA look like?
Singling out one obvious difference with new generation SCADA, these technologies are built around object hierarchies for organising data points and metadata. They are not limited by a flat file singular point definition like older systems have. They contain unlimited extensions to the basic structure without the need for antiquated customisations that makes the best programmers dizzy. If the next gen SCADA platform doesn’t have this functionality, move to the next in your quest.
When SCADA was the only technology option available (except for maybe using Basic or C), it was customized to do more than what it was built for. It was customised to load recipes for production, it was customised for reporting on asset performance, it was customised to automatically control entire systems. SCADA has become a fat cat stretching well beyond its intended boundaries. What choice did we have? None. What choice do we have today; many. We no longer need to use SCADA like we have done and perhaps SCADA should revert to only being the real-time Operational Management software component of critical systems.
These days (modern times post 2000!), we should be using software designed for purpose, which could mean removing unsuitable functionality from SCADA, so it can do its job properly. So, what software can we get now to do what we tried do with SCADA?
And there are more. What this really means is that SCADA should have less, not more business application. But who wants to take the risk to simplify their operational SCADA to remove the complexities? There is so much to weigh up. Starting with all the challenges is overwhelming for the faint hearted. Start with the upside.
What are the benefits to uplifting old SCADA?
The upsides to modern SCADA include the same benefits of using other modern software systems that are not inhibited by old architecture. This is especially true for operating systems and all their vulnerabilities. The ability to have SCADA applications segregated goes a long way to keeping them cyber safe, but isolation can no longer be the singular default security strategy like it has been for decades. Whatever is isolated can always be breached, so the systems must be current to increase their chances of threat detection/avoidance while also being supported by a new age work force. Support for new SCADA is an upside not to be undervalued.
In some cases, a quick evaluation means – throw out the old SCADA. Yes, there is bad news and there is no hope for almost all the old stuff. Using the same technology for over 2 decades is such a long time. It simply must go.
Why are we so cautious about OT systems?
There is a cautious disposition toward replacing industrial software system because they are delicate, fail without warning and make everyone feel nervous. Ironically, this is one of the most important reasons it’s time for it to go.
Despite what you see on the “Terminator” movies, software systems don’t have a mind of their own. In the industrial automation space, when the environment is well managed including the operating system and networks, these software systems can be extremely deterministic and very reliable. That is partly why they have aged so well. When they were tested thoroughly at inception, the issues around confidence and repeatability were taken off the table. Add spaghetti code and that confidence rightly disappears. We don’t engineer bridges and airplanes this way, but somehow, we tolerate it with Industrial Automation systems engineering. It all comes down to budgets and the appetite for risk.
What is the greatest challenge to renew industrial software?
The history around how we arrived with old SCADA helps us understand how we need to change our thinking about SCADA as a renewal technology asset. Decades ago (and more recently we have replicated it) we usually start with capital funding to put in new systems with new assets. When the tangible assets continue to operate there is no cash to replace the control systems as it is operationalized. On the flip side, as assets are upgraded and replaced incrementally item by item, the software systems remain untouched. So tangible assets are refreshed (by good practice), however the less tangible software systems are too often left status quo. In isolation, the costs to upgrade the software systems is high, however, amortized across the total asset count it is a low cost. A single asset can often be replaced with reduced perceived risk in contrast to replacing a software asset that could affect everything!!!
What is the greatest threat to operational sustainability?
Systems that were implement 30 years ago are way past their use by date for several reasons which are not related to their reliability. Many are operating with old unsupported operating systems which means the core applications cannot be updated and made cyber safe. Rest assured, when an event occurs, it will be updated then, but at great loss and without very good control of quality and the ongoing certainty. The more alarming issue is that systems that were implemented 30 years ago have little to no one supporting them. Do you know anyone over the age of 20 that who wants to be servicing old software systems for a living? I don’t. Ask the rising generation (not the ones applying for a job) how keen they are to learn about something which is on its last legs. That doesn’t sound very exciting to me either. So, the best time to deprecate old software systems is before the people who know how they work, retire from the workforce. We have just arrived at that point in time. It’s time for next generation SCADA which means a few things. Here are just a few of the upsides:
Another threat to operational sustainability is the need to modify stable systems to cater for new operational and business requirements. New business drivers demand reconfiguration or replacement of assets to drive peak asset performance. With a mindset to inhibit change for the sake of stability, there is a very real risk that the assets usefulness will diminish in time and never be fully realised. Today’s best practice supersedes best practice from decades ago implying we need to revisit all aspects of systems to remove constraints that prevent peak asset performance. New requirements always demand more from existing assets and we have to find ways to measure if we have reached the performance limit or if we have more improvements to make before assets are repurposed.
What should we learn from the Industrial Automation Software Vendors?
It’s an expensive exercise to churn a SCADA system. It’s even more expensive when the SCADA system has other cool code that isn’t out of the box. If modern SCADA systems (current generation) don’t do things out of the box, you must ask yourself why don’t they? If someone is building new products, why not put it all into one application duplicating all the cool stuff we did with our “old” top line programmers? Keeping things simple increase’s reliability and maintainability without question. The software vendors didn’t get where they are today by being naïve about quality control and risking their reputation. They have learned their lessons about what should be in SCADA and what should be in other software systems, but has industry watched? Has industry listened and then adapted? We may complain that the automation vendors are behind the times, but perhaps it’s the user base that is holding the vendors back from leaping forward.
It’s taken 20-30 years to refine some of these systems or work around their limitations just to keep them alive. It’s difficult to let go of years of good solid hard work. It’s also true that we have all done amazing things. Perhaps letting go is the first step to changing how we view our whole career’s technology experience. It’s hard to tear up a day’s work let alone years of effort for the sake of a better solution. The good news is the principles remain the same. Not all is lost for those who choose newer technology and if there is a persuasion to learn something new, it may be easier than you think. With the flexibility of software development platforms in modern times, software vendors are not inhibited like they once were, so making things easy to do is now the default stance, as is quality. The focus is more about functionality, interoperability, and performance. With the cloud becoming the default position, interoperability is reigning king, but be careful, performance will be the first line of defence that will fall as we open the flood gates to one size/cloud/service fits all.
Where do I start with renewing SCADA?
When it comes to “okay, I know I have to do something”, too often the first question is “how much is the new software?” Starting with software price, will smoke screen the greater issues. Engineering system development and lifecycle is the great hidden cost. Software pricing is just the wrong place to start, although the choice of software is still an important factor. The wrong software is likely to be a minimum 5-year business mistake as it takes time to implement new systems and likewise time to replace them. The greater costs are hidden. Start with the system lifecycle. Fund the project based on the projected life of the software system (not the assets it controls); include external and internal costs, support fees, internal quality costs, engineering and reconfiguration, documentation, and standards management. If you are planning for another 3 decades of set and forget, stop reading this article right now and quit your job. This approach is no longer a reality. The lifecycle of software is significantly reducing as product development cycles shorten and the breadth of options expands.
Realising that next generation SCADA (call it Gen 2) is already mature and stable, should make the oldies feel more comfortable. The truth is, no matter what our age, we are okay to cycle our corporate software systems regularly but don’t apply the same philosophy to maintaining the industrial solutions that operate critical assets for many embarrassing reasons:
The sacredness of a tightly managed Industrial Control System (ICS) is not to be trifled with. We are not suggesting that we should blatantly update ICS’s with every minor change rolled out by vendors. Systems Lifecycle Management (SLM) may have special events; however, we advocate that SLM is ongoing, consideration should be frequent, and the events are planned by stratagem rather than forced upon us by a failure or because we failed to move early enough. What we are suggesting is being too cautious about change, and in particular with mission critical systems, is now the riskiest path of all to take.
When we could no longer buy kerosene at the garage to run our heaters, we coped. When we cannot easily get a horse and buggy to get to the grocery store to buy fresh milk and warm eggs, somehow we survive. To move somewhere before we don’t have any options left may take some courage indeed, but it certainly puts us ahead of the looming change. Wouldn’t you rather be in control of change than have it forced upon you? If you have Gen 1 SCADA software, move on, before there isn’t anyone left who can help lead it to its graveyard. Call the wreckers, get something for it, and maybe get a loan for a new set of wheels. If you have Gen 2 SCADA, do you have a Systems Lifecycle approach and are you already planning for what next? You should be, because the speed of change for OT systems is accelerating.
We may seem to be critical of industrial automation software being the “same same” for so long. To make a point, yes, we confess in jest we inferred that, however a quick reality check highlights that no other software systems have lasted this long! SCADA has surpassed everyone’s expectation. In many cases control devices have lasted far beyond their usable life of the product. This is a testament that the Industrial Control Systems industry has done a stellar job. It’s a critical time to phase out the old so that this industry’s reputation remains in tack.
With operational risk set to increase with constant change to systems that have traditionally been stable by the “no touch” approach, the impetus is we need better systems engineering and better lifecycle support, not less. As systems become more dynamic and adaptive to constant business demand and produce better returns, the investment in people and technology who support these systems is sure to increase. New technology is less likely to be set and forget like our systems of old. Welcome to the era of industrial automation where change is constant and not an annual or biannual event.