How long have we had Big Data Solutions?

Big data concepts have been embodied inside critical infrastructure companies for nearly two decades. Historian technologies have been used to save our environment and manage the most stringent critical infrastructure in the world. Tier 1 process historians have key features such as high compression, extremely fast data access, API access to proprietary stored data, standard out of the box interfaces for enterprise integration and the list goes on. The information systems “industry” attempted to utilise relational database like MS SQL, Oracle, MySQL and so forth to avoid the Automation vendor proprietary industrial offerings. Many tried these alternatives but all failed, unless they compromised on the performance of the system or more importantly the scope of data to be stored and retrieved. This was true from process plant to nuclear power plant.

For the organisations that compromised on requirements, they quickly realised a limited data set means limited benefits to the enterprise in the longer term. A decade ago, interoperability between OT and IT was starting to blur the lines in terms of providing data storage and analysis functions, but if you didn’t want to wait a day for your report to be generated then you were still using a process historian. The process historian never really received the accolades it deserved, mainly because of its unique complexity and sometimes poor implementation. Let’s face it, it’s a back office black hole, not the cool desktop application we would show our friends. It’s an industrial application that the nerds use to solve some serious problems.

What is the life cycle of a Process Historian?

Process historians are akin with the SCADA and DCS systems used in industrial environments. They are mostly considered to be OT infrastructure; however, Enterprise Historians that are typically implemented in the corporate environment blur this simplification.

Assuming that process historians are located in the control system environment remain in this location quarantined from the organisation’s corporate systems, the process historian is likely to remain as long as the SCADA or DCS application remains. When SCADA or DCS systems are transitioned out, we are likely to see new technologies considered.

Can Relational Databases be used as a process historian?

With the improvement of relational databases in the cloud, new SCADA providers provide out of the box connectors to allow “easy” interfacing for on premise and cloud-based warehousing. Dozens of TSDBs (Time Series Data Bases) are readily available and dozens of standard interfaces and tools are available to the more general data consumer. As with relational databases, TSDBs are not all suitable for process data. There are a number of key factors that impact the suitability of technology. Connectivity is over promoted as the key fundamental and should not be the only determinant due to its relative ease.


Process data is unique and highly technical. The planning for and management of process data needs to be done with consideration and transported intelligently. This is particularly important on low bandwidth infrastructure, however even with high bandwidth infrastructure the solution design still needs to purify what is stored and manage the significant volumes of data that can be made available. Higher speed systems also create “more bad data” if it is not conditioned appropriately. This is a big warning for novice data providers. No matter what the technology or location of the data storage, the quality of the data source is key to any solution. The speed of data acquisition multiplies the effect of good and bad data, not only amplifying the choice of technology but in principle exposing the architecture and its final configuration. The openness of relational databases on live production systems has been causing operational headaches since their introduction. Unfairly, relational databases continue to be blamed for poor performing systems when they have been selected as the wrong technology. Most often, relational databases are part of the solution rather than the primary component. Relational databases are not a replacement for the process historian, and the process historian is not a replacement for relational databases. They are apples and oranges.

What are the important factors when implementing a process historian?

The principle of managing process information with integrity will never change. In particular, for control systems with complex and customised process, the challenge will always require it to be defined, configured and delivered by a specialist. The collection and aggregation of good quality process data is critical, and a device is not able to produce that outcome no matter who supplies the device in isolation to the system.  In the past and until process historians disappear from view, process historian data collectors will continue to take away some of the headaches of this data aggregation activity described above. Process historians have good interfaces and adapters which collect information from standard SCADA and automation devices. This is a very strong reason to stay with the process historian for most if not all industrial applications.


If you were about to implement a new system, what technology would you recommend today? The following list may provide a few hints about what to consider.

  • Is the solution to be on premise now and forever.
  • Does the architecture require online/offline buffering of process data
  • Do you need out of the box data interfaces or will you develop these yourself.
  • Does the solution require out of the box client tools for trending and dashboards.
  • Is redundancy a factor for data collection, networks, and data archiving
  • Is the system small, medium, or large in terms of data points to be configured, storage size, and data concentration.
  • Do you need a tiered secure solution with data is replicated or mirror in different network zones.

Can I used cloud-based technology for process control applications?

Hybrid solutions are being used in the world at large by military and critical infrastructure companies because the benefits are hard to refute. Hybrid solutions leverage the process historian for storage of data and leverage new technologies for the unification and presentation of analytics to the wider enterprise. This would normally include the organisation’s preferred Business Intelligence (BI) platform.


Some of the technical benefits of new tech in the cloud include:

  • On demand storage.
  • On demand numerical processing power.
  • “Unlimited” client access and programmatic performance adjustments in real time that customise the user experience. This leads into AI and extended business intelligence applications.

It’s time to think next generation because for some non-industrial applications it is the end of the road for the process historian. When all that we had was a process historian for time series data, the choice was easy. Today we have options, historian on premise, historian in the cloud, alternative TSDB big data solutions and the hybrid/blended solution.


Previously published 2016 and updated 2019.

Model Predictive Control emulates plant operation  


Our top 5 problems with Process Historians
Operational versus Enterprise Historians, what’s the difference?