Today’s Challenges Of Maintaining Legacy Control Systems
The struggle in maintaining aging automation platforms is very real. According to ARC Advisory Group, there are $65 billion worth of installed distributed control systems (DCSs) nearing their end of life, with many of those systems over 25 years old. Unfortunately, manufacturers experience a much greater rate of failure with aging components, along with a host of other associated issues and risks, not least of which is the scarcity of suitable replacement components. It should be noted that most electronic components have a usable life of ten to twelve years before they start to dry-up or become at risk, so overcoming these obstacles and finding the best path forward toward a more effective automation solution is key to future success. Areas that we believe are today’s biggest challenges are as follows:
No Spare Parts
Sourcing spare parts becomes increasingly difficult as control system suppliers can no longer source component parts to build their control systems, or as replacement parts to existing installed systems. Suppliers may choose not to redesign the old circuit boards with new components either due to significantly increased costs, or impractical and cost prohibitive recertification. This forces users to rely on the aftermarket for used parts or remanufactured components, which simply don’t have the reliability of new parts. Failures in systems without redundancy often cause immediate production downtime, even systems with redundancy will eventually experience failure rates high enough to impact production due to multiple failures occurring before parts can be replaced.
Fortunately for Athena’s customers, we maintain a large stock of component parts for our most popular controllers and are able to manufacture replacement boards with relative ease. In the case where components are made obsolete by our suppliers, we typically make a substantial last buy and our engineering teams start to develop direct replacement boards using newer parts and components.
Tribal Knowledge
Not only do parts become difficult to find, having personnel knowledgeable with the legacy platform is also a challenge. Again, according to ARC, over 20% of personnel familiar with legacy DCS platforms have retired with many more approaching retirement, leaving many facilities without people able to modify or even maintain the control system. The options for replacing this tribal knowledge are limited because DCS suppliers often no longer provide training on older platforms, most commonly due to a lack of demand. Even if training is offered, millennials are none too excited about learning a “new” technology which will not give them skills to enhance their career or find a job in the future.
Documentation, a major part of this tribal knowledge, allows the new developers to understand the system. Any lack of documentation turns systems into unchartered territory for a new developer; in such cases mission critical applications suffer huge losses as even small batch work will take a really long time, as people who actually developed the application and process are either no longer in the company or have retired.
At Athena we have always maintained a superior documentation system and if any of our personnel retire, their in-depth product knowledge has been captured and transcribed into engineering notes for each product, ready for tomorrow’s replacement to pick up where retirees left.
Scalability issues
Though not faced by all legacy systems, this certainly can be a big hiccup. When additional workload is presented to the system, additional hardware resources should be efficiently utilized to service the load increase. In this case with older control systems, hardware availability is not the only issue; new hardware may be incompatible with older hardware and the knowledge on how to integrate old and new hardware may be hard to come by.
For Athena’s customers, we make a concerted effort with all our new products to make them backwards compatible so that they can talk with older controllers. If this is not feasible because the new and older technologies are totally incompatible, then conventional wisdom indicates that the embedded control system is long overdue for an upgrade.
Code fragmentation
Up to this point, we have only considered hardware concerns and its support; there is another critical part of control system operation that needs to be considered, namely software and firmware. Over the years, there are multiple implementations of codes around the core software done by different developers. This results in fragmented code. Incorporating new functionality within the core system is difficult, hence people start building new code around it or adding middleware and front end systems which increases the complexity of the system. The result is a cluster of code which requires a lot more manpower for maintenance. There is also a lot of redundant code (as a byproduct) in the system making it even more difficult to fix the errors whenever they show up. In addition to some code fragmentation over time, the addition of new functionality and added new code can make the resultant program more bug-prone, making it more difficult to test and prove, especially for newer code developers unfamiliar with earlier original code.
At Athena, good code design and structure are paying dividends. Firmware in Athena’s controllers is structured in a way that key routines are compartmentalized; if, for example, the code that addresses how an analog to digital converter (ADC) operates needs to updated, it can be worked on, altered, tested and implemented without affecting any other part of firmware ‘system’. Additionally, while Athena’s code is sophisticated, it has been designed, and well documented, such that it will be easily understood by future generations.
Limited connectivity
Most of the legacy systems were developed before the concept of internet was introduced to the masses. These systems were designed and developed to work without internet. Introducing web solutions to such legacy systems is a daunting task which is filled with issues like security, new business requirements and compatibility. The limited ability of legacy to interact with other systems also poses a challenge when expanding the scope of business. Interoperability issues are mostly tackled by workarounds which are not foolproof and prone to errors. Integrating the existing system with today’s network of mobile, cloud, web services etc is again very challenging and results in more fragmented code.
Athena’s more recent products have communications abilities and can talk, using familiar protocols, with MES and ERP software. It should be noted that while the Internet Of Things (IOT) and Industry 4.0 are the subject of much current hype, reality shows that there is a slow implementation of these tools driven primarily by concerns over data security and data ownership. This indicates that we should be asking whether the upside to upgrading to Industry 4.0 technologies is worth the investment; the hype promises all kinds of plant capabilities, data reporting and data analyses, but how much of all this potential will be used has yet to be seen – after all, who uses all the awesome power of Excel or Word?
What’s the Risk?
The risks for keeping a legacy automation system are numerous. As the failure rate of components increases, so does the impact to production. Facility outages lasting several weeks in duration can occur due to a significant control system failure. The risk escalates when you combine failures with a lack of resources able to troubleshoot and make repairs. The cost of this lost production quickly exceeds the cost to upgrade the control system to a modern platform. And nearly every production upset comes with associated safety and environmental risks.
Fortunately for Athena’s customers, our ability to keep our legacy automation systems running with the ability to replace older boards and systems, means that this risk is not eliminated but significantly reduced.
Hidden Costs
Even if the legacy system is still working, there are hidden costs to keeping it around. OEM parts and support costs are higher for older platforms and there is often a lack of functionality when compared to a modern distributed control systems. Limitations in the older technology prevent open communication to smart field devices, subsystems and higher-level enterprise resource planning (ERP) systems. New operators are less effective using the older style human-machine interfaces (HMIs) in legacy DCS platforms, and their response to abnormal situations is inhibited by unfamiliar legacy alarm systems. Additionally, older platforms often utilize unsupported operating systems and slower technologies that have early iterations of communication capabilities, may be more vulnerable to cyber-attacks, with limited options to adequately secure them.
Modernization Is Key
What can be done to mitigate all these risks and find the best path forward? Some users will adopt a strategy of accumulating a quantity of spare parts hoping to extend the life of their system but this approach still leaves them vulnerable to all the risks previously mentioned.
The other option for long-term operational efficiency is to modernize the automation system. Modernization is best done in a planned, disciplined fashion. As with any project, you will want to utilize proven best practices and implementation resources that will deliver value throughout the new system’s entire lifecycle. Of utmost importance is to begin with a front-end loading engineering effort for successful planning and budgeting. This will allow you to:
- define a scope aligned with business needs and facility requirements
- evaluate and select the best platform and project options
- develop an execution plan and schedule
- develop an accurate cost estimate and associated justification
If you look closely all the issues are overlapping in nature but solutions are mutually exclusive of each other. It implies that all the issues have to be fixed individually, indirectly stating a lot of investment is required. Sticking to legacy systems/software and trying to squeeze out every drop of service can save a cash-strained organization some significant investment but this strategy may be shortsighted in the long run. The Industrial Internet of Things (IIoT) and Industry 4.0 have been launched with a very rocky start with major fears about security, privacy and data ownership making headlines in the news almost every day, but the advantages and efficiencies that IIoT and Industry 4.0 offer are too enticing and are here to stay (and be developed); the learning curve will be a shallow one.
All said, however, the scenario is not grim. There are effective solutions available in the market to ensure that your legacy applications can be converted into modern updated applications without incurring loss in terms of investment or downtime.