Creating Industrial Data Lakes for the management of remote assets, plant and equipment is not usually a bed of roses.
Many data science and AI industry thought leaders and analysts are claiming that up to 60% of AI projects are failing.
This is an alarming statistic given the claims that artificial intelligence is going to change our world for the better. For systems with more complex data, namely time series data associated with managing critical infrastructure including machines, refineries and heavy equipment, we would expect this statistic to be far worse given the complexity of the data and the complexity of rigorous processes to make “things safe”. With the fast adoption of simplistic IoT applications which do not cater for complex systems but do provide very cool analytics and sexy dashboards, pilot projects are under-delivering on promises and proof of concept studies are littering the streets with disappointment.
Generational Change
As our technologies evolve and legacy systems need to be phased out and new systems staged in, a rudimentary review of the value of legacy system data should be made to weigh up the cost of translating old operational data and porting it into new environments with new promises. Profiling data including any compensation that data consumers may make either unconsciously or programmatically are important to consider. Also, as we transfer the helm to a new generation of tech savvy born workers we also do not want to lose the lessons learned.
These considerations are essential lessons bought with a price and need to be incorporated for next time. These essential lessons may lead to the very reasons that legacy data should be discarded when the refresh button is pressed or to the contrary lead to early adoption of new tech. When operational data has been used with precision to make reliable decisions in the past, the value proposition for keeping the old data and shaping it for reuse is an easier decision. In some cases because what has been learned about the assets is also about how the data can be organised more efficiently (grouping), new dimensions and strong causational relationships can be inherently saved within new data formats paving the way for simplified data retrieval and a wider acceptance of what often has been labelled as data to be “handled with care”.
What increases the changes of successful data project?
There are a number of parameters and factors that help prepare our plan for a successful data project that produces a trusted data solution. Some of these factors include:
- Where does the data come from
- Periodicity of samples
- The accuracy and resolution of the source instruments
- Reliability and repeatability of the data sampling technology
- The method of capturing data and how is time synchronised with UTC
- The volume of data to be retained and does the system capture everything that is available or is data lost when certain events occur
- Who consumes raw data, who transforms raw data for further consummation and who owns the data solution
- What other systems will need to be connected (finance, asset management, GIS etc)
This teaser list is far from a blueprint and only highlights the importance of understanding data fundaments from the outset. The approach to data migration and adding new context to data is not a spreadsheet based activity either. The preparation process begins with establishing a common language about the fundamentals of data and ends with creating hypothesis about how other data may or may not have casual links or interdependencies.
Where has this been done before?
Barwon Water (Barwon Region Water Corporation) is Victoria’s largest regional urban water corporation. Barwon Water has a proud history of supporting regional prosperity by providing excellence in water, sewerage and recycled water services to their customers and the community for more than 110 years.In April 2019 Barwon Water and Parasyn commenced implementing an Operational Intelligence Platform.
The data solution delivered by December 2019 included the reuse of the existing process historian, decommissioning a legacy data repository, providing tools and instructions for the translation of legacy data, tools to add quality assurance measures to ongoing data capture, and implementing the core platform to allow Operational Intelligence to reach out beyond SCADA and the traditional SCADA user group.
The Barwon Water “trusted data” solution occurred at a time when highly skilled staff who supported legacy systems retired from service. An essential input to the successful project was for the project team (including new data owners) to understand how data was being managed historically, how it was currently being captured and integrated into data management practices and where the limitations in existing practice and technologies were. One of the primary goals was to simplify the road ahead in terms of access to new data types without suffering a loss in the data quality. One may reasonably conclude that because a system for managing water assets is old, the legacy historical data integrity is poor due to age, however, the ranking and profiling of data is not about shades of grey (because it is digital) but it is more about does the data system contain the essentials for reuse. Ironically, some of the older database systems though slow in performance and antiquated in terms of connectivity and flexibility, supersede data quality in terms of some of todays new IoT platforms which are built with domestic grade quality in terms of data capture and network synchronisation of data.
What are the bare essentials to great outcome?
No one knows your car like the driver. That is why service personnel ask you questions before they take your keys and begin a service. When they ask you questions that demonstrate authentic interest, somehow we pass the keys over with less reservations. Perhaps in contrast to having your car serviced where the vehicle owner is never permitted in the workshop, a data project cannot succeed in isolation. Putting technology capability, integration and a fundamental understanding of data first principles to the side, who needs to be in the “working team” is the “main thing”. We have learned over two decades that no matter how good the new infrastructure works, if you don’t have stakeholders sharing their stories, it’s a project for keepsake not a project to improve the business.
For data projects to succeed like with Barwon Water, core stakeholders decide the success or failure of the project. This too is digital. Essential roles are either on or off. The roles include:
- Project Coordinator (usually leverages existing change management framework, tracks who is doing what, and when it is required to keep activities on schedule and rallies all other stakeholders to be prepared to offer their support when it is required)
- IT & Infrastructure Managers (security, networks, servers and client/desktop support)
- Business Analyst or Process Control engineers (understand data sources and can investigate and place value on data)
- Solution architect (understands enterprise systems, APIs, databases, licensing)
- Chief Data Consumer (the owner of information and chief advocate)
Be Prepared for Change
With almost every data project there are unknowns to be discovered at both the early analysis stages and then during testing. Having a tight knit working team that understand and share the fundamentals of why decisions were historically made and later during the project, empowers everyone to contribute and support agile organisational structures to work. This is a great human capital platform to help realise future benefits when early data project outcomes are realised and more mature and ambitious undertakings can be planned for with increased confidence.
What were the main challenges faced in this story?
| “There have been some technical issues and I always feel comfortable having an honest and open discussion with Parasyn expecting to receive a considered response in a timely manner.” |
It is not incriminating to share the lessons learned about data projects because they are far from cookie cutting parties and they are never the same thing.
The data science industry and Industrial Automation adaptation of AI means infrastructure and applications are constantly being improved. With new interfaces, performance improvements, support for cloud technologies and new analytics including operational tools, the baseline technologies are always on the move. It is not so much 10 years of experience on the latest version of software that counts, because that doesn’t exist any more, it’s the approach to staging, testing, managing change, handling expectations and challenging unexpected results that gives each data project uniqueness, its own set of value and most importantly the trust of its users. The successful management of this journey is what qualifies the business for the longer term benefits some of which are yet to be discovered.
Common challenges also seen in the Barwon Operational Intelligence Platform Implementation
- Software bugs and performance limitations
- Learning about technology new features
- Creating new and unique data relationships valued by stakeholders
- QA failures when processing historical data expected to be perfect
- Inadequate vendor documentation for some elements of software interfaces
- Licensing – you always need more software licenses for something else
If it wasn’t for the challenges, data projects would be like baking cakes, follow the recipe and 9 times out of 10 its edible and easily consumed. As long as legacy remains, new evolving tools are being developed and new advanced applications must be applied to improve existing business, the value of successful data projects is going to increase in it’s importance.
| “We like working with Parasyn because we know Parasyn takes into consideration Barwon Water’s needs in their approach to delivery. It’s like a partnership which is very valued.” |
To see more about the application of technology see: https://www.parasyn.com.au/articles/
Creating Industrial Data Lakes for the management of remote assets, plant and equipment is not usually a bed of roses.
Many data science and AI industry thought leaders and analysts are claiming that up to 60% of AI projects are failing.
This is an alarming statistic given the claims that artificial intelligence is going to change our world for the better. For systems with more complex data, namely time series data associated with managing critical infrastructure including machines, refineries and heavy equipment, we would expect this statistic to be far worse given the complexity of the data and the complexity of rigorous processes to make “things safe”. With the fast adoption of simplistic IoT applications which do not cater for complex systems but do provide very cool analytics and sexy dashboards, pilot projects are under-delivering on promises and proof of concept studies are littering the streets with disappointment.
Generational Change
As our technologies evolve and legacy systems need to be phased out and new systems staged in, a rudimentary review of the value of legacy system data should be made to weigh up the cost of translating old operational data and porting it into new environments with new promises. Profiling data including any compensation that data consumers may make either unconsciously or programmatically are important to consider. Also, as we transfer the helm to a new generation of tech savvy born workers we also do not want to lose the lessons learned.
These considerations are essential lessons bought with a price and need to be incorporated for next time. These essential lessons may lead to the very reasons that legacy data should be discarded when the refresh button is pressed or to the contrary lead to early adoption of new tech. When operational data has been used with precision to make reliable decisions in the past, the value proposition for keeping the old data and shaping it for reuse is an easier decision. In some cases because what has been learned about the assets is also about how the data can be organised more efficiently (grouping), new dimensions and strong causational relationships can be inherently saved within new data formats paving the way for simplified data retrieval and a wider acceptance of what often has been labelled as data to be “handled with care”.
What increases the changes of successful data project?
There are a number of parameters and factors that help prepare our plan for a successful data project that produces a trusted data solution. Some of these factors include:
This teaser list is far from a blueprint and only highlights the importance of understanding data fundaments from the outset. The approach to data migration and adding new context to data is not a spreadsheet based activity either. The preparation process begins with establishing a common language about the fundamentals of data and ends with creating hypothesis about how other data may or may not have casual links or interdependencies.
Where has this been done before?
Barwon Water (Barwon Region Water Corporation) is Victoria’s largest regional urban water corporation. Barwon Water has a proud history of supporting regional prosperity by providing excellence in water, sewerage and recycled water services to their customers and the community for more than 110 years.In April 2019 Barwon Water and Parasyn commenced implementing an Operational Intelligence Platform.
The data solution delivered by December 2019 included the reuse of the existing process historian, decommissioning a legacy data repository, providing tools and instructions for the translation of legacy data, tools to add quality assurance measures to ongoing data capture, and implementing the core platform to allow Operational Intelligence to reach out beyond SCADA and the traditional SCADA user group.
The Barwon Water “trusted data” solution occurred at a time when highly skilled staff who supported legacy systems retired from service. An essential input to the successful project was for the project team (including new data owners) to understand how data was being managed historically, how it was currently being captured and integrated into data management practices and where the limitations in existing practice and technologies were. One of the primary goals was to simplify the road ahead in terms of access to new data types without suffering a loss in the data quality. One may reasonably conclude that because a system for managing water assets is old, the legacy historical data integrity is poor due to age, however, the ranking and profiling of data is not about shades of grey (because it is digital) but it is more about does the data system contain the essentials for reuse. Ironically, some of the older database systems though slow in performance and antiquated in terms of connectivity and flexibility, supersede data quality in terms of some of todays new IoT platforms which are built with domestic grade quality in terms of data capture and network synchronisation of data.
What are the bare essentials to great outcome?
No one knows your car like the driver. That is why service personnel ask you questions before they take your keys and begin a service. When they ask you questions that demonstrate authentic interest, somehow we pass the keys over with less reservations. Perhaps in contrast to having your car serviced where the vehicle owner is never permitted in the workshop, a data project cannot succeed in isolation. Putting technology capability, integration and a fundamental understanding of data first principles to the side, who needs to be in the “working team” is the “main thing”. We have learned over two decades that no matter how good the new infrastructure works, if you don’t have stakeholders sharing their stories, it’s a project for keepsake not a project to improve the business.
For data projects to succeed like with Barwon Water, core stakeholders decide the success or failure of the project. This too is digital. Essential roles are either on or off. The roles include:
Be Prepared for Change
With almost every data project there are unknowns to be discovered at both the early analysis stages and then during testing. Having a tight knit working team that understand and share the fundamentals of why decisions were historically made and later during the project, empowers everyone to contribute and support agile organisational structures to work. This is a great human capital platform to help realise future benefits when early data project outcomes are realised and more mature and ambitious undertakings can be planned for with increased confidence.
What were the main challenges faced in this story?
It is not incriminating to share the lessons learned about data projects because they are far from cookie cutting parties and they are never the same thing.
The data science industry and Industrial Automation adaptation of AI means infrastructure and applications are constantly being improved.
“There have been some technical issues and I always feel comfortable having an honest and open discussion with Parasyn expecting to receive a considered response in a timely manner.”
With new interfaces, performance improvements, support for cloud technologies and new analytics including operational tools, the baseline technologies are always on the move. It is not so much 10 years of experience on the latest version of software that counts, because that doesn’t exist any more, it’s the approach to staging, testing, managing change, handling expectations and challenging unexpected results that gives each data project uniqueness, its own set of value and most importantly the trust of its users. The successful management of this journey is what qualifies the business for the longer term benefits some of which are yet to be discovered.
Common challenges also seen in the Barwon Operational Intelligence Platform Implementation
If it wasn’t for the challenges, data projects would be like baking cakes, follow the recipe and 9 times out of 10 its edible and easily consumed. As long as legacy remains, new evolving tools are being developed and new advanced applications must be applied to improve existing business, the value of successful data projects is going to increase in it’s importance.
“We like working with Parasyn because we know Parasyn takes into consideration Barwon Water’s needs in their approach to delivery. It’s like a partnership which is very valued.”
To see more about the application of technology see: https://www.parasyn.com.au/articles/