Build an Industrial Data Foundation
I4.0 Insight
4 minutes
Jul 18, 2024
This article seeks to demystify a concept that both supports and challenges digital suppliers, providing clear explanations and best practices for implementation.
Current situation
Industrial data foundation typically refers to consolidating various types of data sources into a single place for easy consumption by other applications.
In today's IIoT(Industrial Internet of Thing) market, the industrial data foundation is implemented as follows:
Integrating with existing customer software through a RESTful API to interpret and process upper-layer industrial protocols transmitted over TCP/UDP.
Extracting data from existing customer relational databases using scripts or ETL tools and storing it in the database.
Modeling or formatting the collected data and publishing it through RESTful API.
However, this approach has drawbacks: it turns data silos into data spaghetti. Consumers of the platform data would find it no more convenient, and existing systems would not communicate more efficiently through the platform.
Why is this happening?
In the Information Age, RESTful APIs and web services have become the primary methods for system data integration. With the rise of the IIoT concept, RESTful APIs have emerged as the dominant method for data consumption layer integration.
However, it's important to note that they were not designed for bidirectional data transmission and interoperability. For instance, when a customer wants to integrate an app that consumes various types of data:
Before adopting an IIoT platform ⬇️
After adopting an IIoT platform ⬇️
Even after adopting the platform as a "hub," each individual line still exists. Clearly, the data integration task cannot be simplified or standardized just by changing the address of RESTful requests.
The stateless design of RESTful APIs brings great convenience, but it also means that two systems cannot perceive each other and must continually poll to obtain the latest information. After adopting an IIoT platform, the step of writing polling scripts still cannot be omitted; this task is merely transferred to the engineering team of the IIoT Platform.
The above examples assume that systems are equipped with comprehensive RESTful APIs. However, in reality, IIoT platform vendors often end up custom-coding various microservices and scripts, directly interacting with databases to achieve integration and communication. This approach is far from the standardized and productized vision of IIoT.
What should we do?
As mentioned above, popular protocols such as RESTful, web server + ODBC/JDBC, OPC, and Modbus are all based on polling mechanisms.
Even if the data source hasn't changed, there will still be periodic requests flooding the network, leading to bandwidth wastage, computational overhead, and additional issues such as delays, lack of scalability, and difficulty in layering (designing and using RESTful requests often becomes an obstacle for industrial OLTP+OLAP). Most importantly, using integration architecture centered around RESTful cannot form a standardized base product that simplifies and standardizes data integration.
Therefore, Event-driven frameworks have emerged
When the temperature changes (an event occurs), the temperature sensor will push (publish) a message to the network announcing its new temperature. This announcement will be broadcast simultaneously to all consumers (MES, ERP) interested in (subscribe to) this change.
The subscription and publication mechanism of the MQTT protocol enables event-driven systems and networks. Given the characteristics of industrial data, we can boldly build a star topology based on Message Broker instead of continuing to string together data spaghetti for point-to-point communication!
Unified Namespace based on an EDA (Event-Driven Architecture)
Utilize an MQTT Broker to integrate all data sources, enabling seamless subscription and bidirectional communication.
Perform semantic modeling on all incoming data based on ISA-95, Sparkplug B, or OPC UA. This will achieve semantic uniformity in communication formats across different systems.
Unified Namespace (UNS) can be divided into two layers:
The MQTT Broker and its subscriber layer serve as the hub for real-time data exchange. Information flowing here is instantaneous (Snapshot).
The Data Lake layer, where information from the first layer is stored for time-series use cases (e.g., predicting motor failures based on a period of three-axis sensor data, querying historical tag trends, and aggregating employee information).
Due to the protocol-level limitations of MQTT 3.1 (actually a tradeoff for simplicity, which will be detailed in a future article), it can sometimes be unreliable for stream data processing. To ensure reliability, pioneers in UNS tend to use Kafka and MQTT in parallel. However, based on our experience, even using only MQTT is far more reliable than RESTful "data spaghetti." Event-driven architecture, utilizing a Message Broker and the publish-subscribe mechanism, can establish a standardized industrial data foundation.
Acknowledgments to Walker Reynolds, President of Industrial 4.0 Solutions, for proposing and actively promoting the UNS concept.
Next steps
Some readers might wonder, what about systems that do not support messaging? How should data formatting be handled? How can we ensure transactions are processed correctly? Manufacturers like United Manufacturing Hub and HiveMQ have conducted extensive research and work on these issues, and I will also detail these topics in future articles.
If you have different opinions or are interested in discussing this, please contact me at mercy@freezonex.io. We're actively seeking passionate individuals, offering opportunities that far exceed industry standards, and we are proud to serve some of the world's largest and most visionary manufacturing and process companies.
For those in the manufacturing or process industries looking for a practical data platform, I'd like you to stay tuned for my next article. I will explore why and how open standards are the best way to achieve an effective "data foundation."
If you're an industrial digitalization supplier or part of an IIoT company, I encourage you to accelerate your efforts with EDA. It has the potential to bring significant transformation to our industry.
Disclaimer: Some content in this article references the communities of United Manufacturing Hub (UMH) and HiveMQ.
