All posts by vogler

Bring MQTT Payload to OPC UA?

 I wanted to get my Home-Automation values to SCADA, it’s a “self-made” JSON message format. I tried it with Ignition and the MQTT Module. Btw.: it’s great that they have the Makers Edition for non-commercial use at home 👍. But I don’t know why, it only got one topic and one value from my MQTT Broker, and it did not receive any updates. Don’t know what went wrong…

Anyhow, I decided to add a custom JSON format to the Automation-Gateway.com. It’s simple, just define the JSON-Path to the value and optionally to a timestamp in milliseconds since epoch or to an ISO 8601 format.

Now I can use the Automation-Gateway’s OPC UA server in any SCADA system to visualize my MQTT values…

Here is the config.yaml configuration file for the Automation-Gateway.

Servers:
  OpcUa:
    - Port: 4841
      Enabled: true
      LogLevel: INFO
      Topics:
        - Topic: mqtt/home/path/Original/#
Drivers:
  Mqtt:
    - Id: "home"
      LogLevel: INFO
      Host: 192.168.1.3
      Port: 1883
      Format: Json
      CustomJson:
          Value: "Value"
          TimestampMs: "TimeMS"

Use the Unified Namespace (UNS) as a graph?

🤔 I thought about if it could make sense to connect to a MQTT broker and write the topic-path in a structured and connected way to a graph database. It could reflect a companies UNS with ISA structure in a graph with query possibilities.

🤝 The UNS could be enriched by adding additional meta information to the database and be linked to the MQTT nodes. Graph queries could be used to combine the enriched data with the current values of the machines…

👉 I reactivated the Neo4J Logger in the Automation-Gateway.com, and enabled MQTT for it. It creates the nodes in a neo4j graph database based on the incoming MQTT messages.

Also interesting to explore the graph of the test.mosquitto.org hashtag#MQTT broker…

saw some go-eChargers, some solax inverters, some json scada demos (never heard about it before), and also a “Atomkraftwerk1234567890” 😀

MQTT for Unity

“MQTT for Unity” is a Unity Package designed to seamlessly integrate MQTT (Message Queuing Telemetry Transport) functionality into Unity projects, offering a user-friendly solution for enabling real-time communication and data exchange within Unity applications.

Tested on Windows, OSX, WebGL, UWP + HoloLens2, and Android. iOS not tested, but should work as well.

You can find it at the Unity Asset Store here

Key Features:

  1. Streamlined Integration: “MQTT for Unity” provides a straightforward and hassle-free integration process, enabling developers to quickly set up MQTT communication in their Unity projects.
  2. Real-Time Communication: Harness the power of MQTT to establish real-time communication channels within your Unity application, perfect for multiplayer games, IoT applications, and more.
  3. Customizable Configuration: Easily configure MQTT parameters, such as broker settings, topic subscriptions, and message handling, to tailor the communication to your specific project needs.
  4. Cross-Platform Compatibility: “MQTT for Unity” is designed to work seamlessly across various Unity-supported platforms, including Windows, OSX, WebGL, UWP + HoloLens2, and Android. iOS not tested, but should work as well.

With “MQTT for Unity,” developers can unlock the potential of MQTT communication in their Unity applications without the complexities of manual integration, making it an essential tool for creating interactive and connected experiences in Unity.

Online documentation can be found here.

SCADA Real Time Data and Apache SPARK

The integration of SCADA with Spark and WinCC Open Architecture offers a powerful and versatile solution that combines real-time data processing, advanced analytics, scalability, and flexibility. This combination empowers you to optimize industrial processes, make data-driven decisions, and stay ahead in a rapidly evolving technological landscape.

By utilizing my 5-year-old project that implemented a native Java manager for WinCC Open Architecture, I have enabled the integration of SCADA with Spark for the current WinCC OA Version 3.19.

Very simple example is to analyze tags and the corresponding amount of values in your SCADA system can provide valuable insights into the distribution and characteristics of the data.

res = spark.sql('SELECT tag, count(*) count FROM events GROUP BY tag ORDER by count(*) DESC')
data = res.toPandas()
plt.figure( figsize = ( 10, 6 ) )
sns.barplot( x="count", y="tag", data=data)
plt.show()

Another simple example is to calculate the moving average of 10 preceding and following values for a given data point in a time series, you can use a sliding window approach:

data = spark.sql("""
SELECT ROUND(value,2) as value, 
  AVG(value) OVER (PARTITION BY tag ORDER BY ts 
  ROWS BETWEEN 10 PRECEDING AND 10 FOLLOWING) avg 
  FROM events 
  WHERE tag = 'System1:ExampleDP_Trend2.'
  ORDER BY ts DESC
  LIMIT 100
  """).toPandas()
data = data.reset_index().rename(columns={"index": "nr"})
sns.lineplot(data=data, x='nr', y='value', label='Value')
sns.lineplot(data=data, x='nr', y='avg', label='Average')
plt.show()

By leveraging the distributed file system, you can take advantage of Spark’s parallel processing capabilities. The distributed file system ensures that the data frame is partitioned and distributed across the nodes of the Spark cluster, enabling simultaneous processing of data in parallel. This distributed approach enhances performance and scalability, allowing for efficient handling of large volumes of real-time SCADA data.

I have achieved real-time data streaming from WinCC OA to a Spark cluster with a Websocket-Server based on the Java manager. This streaming process involves continuously transferring SCADA real-time data from the WinCC OA system to the Spark cluster for further processing and analysis.

url='wss://192.168.1.190:8443/winccoa?username=root&password='
ws = create_connection(url, sslopt={"cert_reqs": ssl.CERT_NONE})

def read():
    while True:
        on_message(ws.recv())
Thread(target=read).start()

cmd={'DpQueryConnect': {'Id': 1, 'Query':"SELECT '_online.._value' FROM 'ExampleDP_*.'", 'Answer': False}}
ws.send(json.dumps(cmd))

Once the data is received by the Spark cluster, I store it as a data frame on the distributed file system (DFS). A data frame is a distributed collection of data organized into named columns, similar to a table in a relational database. Storing the data frame on the distributed file system ensures data persistence and allows for efficient processing and retrieval.

schema = StructType([
    StructField("ts", TimestampType(), nullable=False),
    StructField("tag", StringType(), nullable=False),
    StructField("value", FloatType(), nullable=False)
])
df = spark.createDataFrame(spark.sparkContext.emptyRDD(), schema)
bulk = []
last = datetime.datetime.now()
def on_message(message):
    global bulk, last, start
    data = json.loads(message) 
            
    if "DpQueryConnectResult" in data:
        values = data["DpQueryConnectResult"]["Values"]
        for tag, value in values:
            #print(tag, value)
            data = {"ts": datetime.datetime.now(), "tag": tag, "value": value}
            bulk.append(data)
            
    now =datetime.datetime.now()
    time = datetime.datetime.now() - last
    if time.total_seconds() > 10 or len(bulk) >= 1000:
        last = now

        # Create a new DataFrame with the received data
        new_df = spark.createDataFrame(bulk, schema)
        
        new_df.write \
            .format("csv") \
            .option("header", "true") \
            .mode("append") \
            .save("events.csv")
        
        bulk = []

Once the SCADA data is stored as a distributed data frame on the Spark cluster’s distributed file system, you can leverage Spark’s parallel processing capabilities to efficiently process the data in parallel.

df = spark.read \
    .format("csv") \
    .option("header", "true") \
    .option("timezone", "UTC") \
    .schema(schema) \
    .load("events.csv")
df.createOrReplaceTempView("events")

By combining SCADA (Supervisory Control and Data Acquisition) with Spark’s powerful data processing capabilities, I have created a solution that can handle large volumes of real-time data efficiently. This enables faster and more accurate decision-making based on the insights derived from the processed data.

Collect Factory data and write it to Kafka, with the Automation Gateway…

Did a test with one gateway and collected 250000 😲 value changes per second(!) from 10 OPC UA servers and wrote it to 10 topics on a single Kafka Broker… Kafka was bored 😴… one and a half core of an i7 PC. The gateway had a lot of work to do, collect and bulk all the single values and then send it to Kafka … roughly 10 cores of an i7 PC.

The gateway has a GraphQL and MQTT interface where you can read and write OPC UA values. On an internal MQTT topic the logger publishes how many values are getting in and out…

WinCC Unified V18 exposed to the Internet…

This article will show how WinCC Unified can be accessed through a public available server in the internet.

Disclaimer: I only did this for testing and demo purposes!!!

First you need to have a public domain name and a public accessible host. Or a host running somewhere in the cloud and you will get a IP and/or an URL, which will point to your host. In my case I have a public IP address from my internet provider and my public sub domain name points to my server at home.

My registered public domain name is rocworks.at. Additionally I have used a sub-domain name unified.rocworks.at. Because I have multiple services running on my machine at home. With the subdomain the service can be easily be distinguished. At my internet provider I have configured a DDNS services, so that my subdomain unified.rocworks.at points to my IP at home. You can also use other DDNS services (noip.com) , also if you have a dynamic IP address.

If you have it running at home, then you have to setup a port forwarding from your modem to your web server IP at home.

At the WinCC Unified Runtime Host we have to change some settings in files, to set the right public URL for the identity provider (UMC). After doing this, you should reboot the computer.

Config.level (C:\Program Files\Siemens\Automation\WinCCUnified\config)

	[IdentityProvider]
	Url = "https://unified.rocworks.at/umc-sso/"
	
Web.config (C:\Program Files\Siemens\Automation\WinCCUnified\WebRH)

	<appSettings>
	    <add key="appvirtdir" value="/WebRH" />
	    <add key="origins" value="https://unified.rocworks.at" />
	  </appSettings>

Config.json (C:\Program Files\Siemens\Automation\WinCCUnified\SimaticUA)

        "dnsname": "unified.rocworks.at"

Umcd.cfg (C:\Program Files\Siemens\LocalUserManagement\etc)

	Search and replace hostnames

HKEY_LOCAL_MACHINE\SOFTWARE\Siemens\User Management\WebUI\Settings

        ipaddress = "https://unified.rocworks.at/umc-sso/"

Note: instead of "unified.rocworks.at" use your public domain name. 
    

At the web server at home I have NGINX running in a Docker Container together with Let’s Encrypt. With Let’s Encrypt and Certbot we can get valid Certificates for our Webserver. But that’s another story. Here is a docker-compose.yml file for NGINX and Let’s Encrypt:

version: '3'
services:
  nginx:
    image: nginx
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./data/www:/var/www
      - ./data/letsencrypt:/etc/letsencrypt
      - ./config:/etc/nginx/conf.d
       
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"

  certbot:
    image: certbot/certbot
    restart: unless-stopped
    volumes:
      - ./data/www:/var/www
      - ./data/letsencrypt:/etc/letsencrypt
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; date; do certbot renew --webroot -w /var/www/certbot; sleep 12h & wait $${1}; done;'"

Before you start with a new sub domain you have to initially get a certificate:

docker run --rm -ti -v $PWD/data/www:/var/www -v $PWD/data/letsencrypt:/etc/letsencrypt certbot/certbot certonly --webroot -w /var/www/certbot -d <your-public-domain-name> --email <your-email-address>

NGINX Configuration: default.conf :

server {
        listen 80;
        server_name unified.rocworks.at;
        location /.well-known/acme-challenge/ {
            root /var/www/certbot;
        }
        location / {
            root /var/www/html;
        }
}

NGINX Configuration: unified.conf:

server {
        server_name unified.rocworks.at;

        root /var/www/html;
        index index.html index.htm;

        location / {
            proxy_pass https://<ip-of-wincc-unified-host>/;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;
        }

        location /umc-sso {
            proxy_pass https://<ip-of-wincc-unified-host>/umc-sso;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;
            proxy_buffer_size 128k;
            proxy_buffers 4 256k;
            proxy_busy_buffers_size 256k;
        }

        #location /graphql { # Optionally you can also publish GraphQL
        #    proxy_pass http://<ip-of-wincc-unified-host>:4000/graphql;
        #    proxy_http_version 1.1;
        #    proxy_set_header Upgrade $http_upgrade;
        #    proxy_set_header Connection 'upgrade';
        #    proxy_set_header Host $host;
        #    proxy_cache_bypass $http_upgrade;
        #}
        

        listen 443 ssl; # managed by Certbot
        ssl_certificate /etc/letsencrypt/live/unified.rocworks.at/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/unified.rocworks.at/privkey.pem; # managed by Certbot
        include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

GraphQL for Unity with WinCC Unified

In this post I will show how data from WinCC Unified V18 can be brought into Unity with the GraphQL for Unity Pro Asset.

First you need WinCC Unified V18 with the GraphQL Server. The GraphQL server comes out of the box with the new version and should be up and running automatically when you download and start your runtime. You should see a process “WCCILgraphQLServer.exe” in your task manager.

In Unity you have to create a project and download the GraphQL for Unity Pro Asset in the Package Manager.

  1. Prefab: Drag and drop the WinCC Unified Prefab from the Prefabs folder into your scene.
  2. Connection: In the properties you have to set your GraphQL Host, optionally the port, the path (typically /graphql), and if you want to have a secured TLS connection (HTTPS, WSS).
  3. Websocket: If you want to subscribe to tag value changes, then you have to open additionally a Websocket connection. Without the Websocket connection you can still read and write tag values, but a subscription to tag value changes is not possible.
  4. Authorization: Set the username and password to connect to WinCC Unified (at the time of writing, this user must have the role HMI Administrator).
  5. Logon: Check the “Logon” checkbox, if you want to start the connection at startup (you can also set this “on-demand” during runtime in your code.

Then you can already start the project in the editor to see if the connection can be established. If everything works fine, then the “Logged On” property turns to checked.

Now you can create your own C# script and read/write/subscribe tag values in an easy way in C# scripting. In that case I have simple added the script as additional component to the WinCC Unified GameObject (Prefab).

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using GraphQLWinccUnified;
using Newtonsoft.Json.Linq;

public class WinccUnified3 : MonoBehaviour
{
    private WinccUnified _connection;
    private bool _ready;

    // Start is called before the first frame update
    void Start()
    {
        _connection = GetComponent<WinccUnified>();
    }

    // Update is called once per frame
    async void Update()
    {
        if (!_ready && _connection.IsWebsocketReady()) 
        {
            _ready = true;

            // SubscribeTagValues
            _connection.SubscribeTagValues(
                new string[] { "HMI_String_1" }, 
                data => { Debug.Log(data.GetValue<string>()); });

            // WriteTagValues
            await _connection.WriteTagValues(
                new string[] { "HMI_String_1" },
                new JValue[] { new JValue("Hello World!") },
                new DateTime[] { DateTime.Now });

            // ReadTagValues
            var result = await _connection.ReadTagValues(
                new string[] { "HMI_String_1" });
            Debug.Log(result[0].GetValue<string>());
        }
    }
}

If you deal with with a self-signed certificate, then you must uncheck “Validate Certificate” at the “Graph QL Http” Component. But the Websocket connection in Unity does not yet support self-signed certificates. So, it is better to use an insecure connection. Typically the GraphQL port is 4000, so be sure to set the Port to 4000. And don’t forget to open the firewall for that port on the host where the Unified runtime is running.

WinCC Unified GraphQL Server for Data Integration

With the GraphQL Server in WinCC Unified V18 we can now integrate various (IT) data platforms with simple programs. Those programs can be written in Python, Java, Kotlin, Go, JavaScript, or whatever kind of programming language you prefer.

In my case I have used Kotlin to implement a simple Apache Kafka Consumer, which maps and writes values from my Home-Automation to the WinCC Unified SCADA system.

I can now use WinCC Unified for visualisation, even if I don’t have any PLC at home. WinCC Unified can be used as a Low- or No-Code platform to create fancy Web-Based visualisations with real-time values from any kind of data source.

WinCC Unified also has great alarming features. Alarm handling can be done in WinCC Unified and alerts could also be published back to the streaming platform with a producer.

I collect my Home-Automation values from a Raspberry Pi, which reads values from power meters, temperature sensors, or data via Bluetooth from my PV Converter. The values are published to a MQTT Broker. And from that MQTT Broker I bring my values to Apache Kafka and then to the WinCC Unified system.

With the GraphQL Server of WinCC Unified it would also be easily possible to implement an Apache Kafka Producer, so that values from PLC’s can easily be published to Apache Kafka, or any other data streaming platform.

From Apache Kafka I write my values to WinCC Unified and additionally to a CrateDB. CrateDB is a great NoSQL database with the power of SQL and it is highly scalable. It can be used for data analytics, machine learning, Grafana Dashboard, and more…

For sure you can also grab the data directly from the MQTT broker and bring it directly into WinCC Unified via the GraphQL Server without a streaming platform, but a streaming platform has additional benefits, which are not covered in this post…

AWS AppSync query with Cognito User Pool

If you want to use the Cognito User Pool, then you first have to get somehow the JSON Web Token (jwt). Once you have this token, then you should pass the token in the header within the “Authorization” key.

graphQLHttp.Headers = new List<Header>()
{
      new Header() {Key = “Authorization”, Value = userSessionCache.getIdToken()}
}

Here you will find more information about jwt tokens.