How to Transfer Data from OPC UA to Kafka

How to Transfer Data from OPC UA to Kafka

Open Automation Software can be used to transfer data from an OPC UA server or device to a Kafka cluster, locally or over a network. This guide walks you through downloading and installing OAS, configuring an OPC UA connector, a tag and a Kafka publisher, and finally publishing the tag using the Kafka publisher.

For this guide on how to transfer data from an OPC UA source to a Kafka cluster you will need:

  • Docker installed in Linux or Windows including the docker-compose utility

ℹ️ For simulation purposes, you will use the OPC UA server available in the OAS platform to access a Tag value. If you already have an OPC UA server or device available, you can also adjust your configuration to suit your own OPC UA server or device.

1 - Download and Install OAS

If you have not already done so, you will need to download and install the OAS platform.

Fully functional trial versions of the software are available for Windows, Windows IoT Core, Linux, Raspberry Pi and Docker on our downloads page.

On Windows, run the downloaded setup.exe file to install the Open Automation Software platform. For a default installation, Agree to the End User License Agreement and then click the Next button on each of the installation steps until it has completed.

If you'd like to customize your installation or learn more, use the following instructions:

The OAS Service Control application will appear when the installation finishes on Windows.

OAS Service Control

Click on each START SERVICE button to start each of the three OAS services.

2 - Configure OAS

Configure OAS is the main application used to configure local and remote OAS instances.

OAS Logo

  1. From your operating system start menu, open the Configure OAS application.

  2. Select the Configure > Tags screen.

    Important

    If this is the first time you have installed OAS, the AdminCreate utility will run when you select a screen in the Configure menu. This will ask you to create a username and password for the admin user. This user will have full permissions in the OAS platform.

    For further information see Getting Started - Security.

  3. If this is the first time you are logging in, you will see the AdminCreate utility. Follow the prompts to set up your admin account. Otherwise, select the Log In menu button and provide the Network Node, username and password.

    Log In Menu

    Log In Dialog

Info

In this guide you will use the Configure OAS application to configure the local Network Node which by default is localhost.

If you have installed OAS on a remote instance you can also connect to the remote instance by setting the relevant IP address or host name in the Network Node field.

3 - Create a Kafka test cluster using Docker

In this step you will create a local Kafka cluster. To simplify the deployment you will need to have Docker installed including the docker-compose utility. This method allows you to create a cluster very quickly for testing purposes and remove it again when you are done.

  1. Create a new folder such as Kafka and inside the folder create a new docker-compose.yml file with the following definition.

    services:
        zookeeper:
            image: confluentinc/cp-zookeeper:latest
            container_name: zookeeper
            ports:
                - "2181:2181"
            environment:
            ZOOKEEPER_CLIENT_PORT: 2181
            ZOOKEEPER_TICK_TIME: 2000
    
        kafka:
            image: confluentinc/cp-kafka:latest
            container_name: kafka
            ports:
                - "9092:9092"
            environment:
            KAFKA_BROKER_ID: 1
            KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
            KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
            KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
            KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    
    
  2. Open a command line terminal or bash terminal and use the following command to start a new container. This will download the image and all its dependencies and may take a few minutes.

    docker-compose up -d

  3. Create a new topic called temperature using the following command.

    docker exec kafka kafka-topics --create --topic temperature --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1


Info

Once you are done testing, you can stop and remove the containers that you created by using the following command:

docker-compose down

4 - Create Security Group and User

When connecting to the local OAS OPC UA server, you need to configure a security group and a user to provide Tag read/write access. You'll need these credentials when creating the OPC UA connector instance later.

ℹ️ You can skip this step if you are connecting to your own third party OPC UA server.

  1. Select Configure > Security from the top menu.

    Configure drivers menu

  2. Provide a Group Name such as OPC UA Access.

    OPC UA security group name

  3. Click on the DISABLE ALL button to disable all access.

    Disable all access button

  4. In the Tags tab, ensure Get Tag Group Names and Get Tag Names is checked. This setting is required when browsing the OPC UA server tag structure once you get to the step of assigning the OPC UA data source to a Tag.

    Allow get tag groups and tag names

  5. In the Read Tags tab, ensure Disable All Tags From Reading is NOT checked.

    Read tags not disabled

  6. In the Write Tags tab, ensure Disable Add Tags From Writing is NOT checked.

    Write tags not disabled

  7. Click on the ADD GROUP button on the left hand side to add this security group configuration. Once added, the security group name should appear in the list of security groups.

    Add security group button

  8. Select Configure > Users from the top menu.

    Configure drivers menu

  9. Provide a User Name such as opcuauser, a password and set the Security Group as OPC UA Access.

    Set username and security group

  10. Click on the ADD USER button on the left hand side to add this user configuration. Once added, the user name should appear in the list of users.

    Add user button

5 - Add Tag for Temperature Simulation

In this section you will create a Tag to represent a Temperature sensor that will then be read from the OAS OPC UA server.

ℹ️ If you have your own third-party OPC UA server where you can simulate a Tag value then you can skip this step.

  1. Select Configure > Tags from the top menu.

    Configure tags menu

  2. Select the root Tags group and click on the ADD TAG button.

    Add tag button

  3. Provide a Tag Name such as TemperatureSensorSimulation and click the OK button.

    Add sim tag to root node dialog

  4. In the Enter Value field enter a value to represent the simulated temperature.

    Temperature value

6 - Configure OPC UA Data Source

In the following steps you will create and configure a OPC UA Connector to connect to the OAS internal OPC UA server on port 58728.

ℹ️ If you want to use your own third-party OPC UA server you can provide the relevant Server URL and security credentials instead of the details provided below.

  1. Select Configure > Drivers from the top menu.

    Configure drivers menu

  2. Enter a meaningful Driver Interface Name to give this driver interface instance a unique name.

  3. Ensure the following parameters are configured:

    • Driver: OPC UA
    • Server Url: opc.tcp://localhost:58728
  4. Click on the BROWSE button to select a Security Profile. Choose the appropriate security profile. For this guide we will choose None:None:Binary.

    OPC UA security profile

  5. Enabled User Security and enter the Username and Password credentials created in the earlier section.

    OPC UA connector configuration

  6. Click on the ADD DRIVER button on the left hand side to add this driver configuration. Once added, the driver interface name should appear in the list of drivers.

    Add OPC UA driver button

7 - Add Data Source Tag

In this section you will create a Tag to represent your data point in the field (for example a temperature sensor). This can then be transferred your desired destination.

  1. Select Configure > Tags from the top menu.

    Configure tags menu

  2. If you want to add a Tag to the root Tags group make sure the Tags node is selected in the tag list and click on the ADD TAG button.

    Add tag button

    If you want to add a Tag to a Tag Group, select the Tag Group first and then click on the ADD TAG button.

    You can also add Tag Groups by using the ADD GROUP button.

  3. Provide a Tag Name such as TemperatureSensor and click the OK button.

    Add tag to root node dialog

8 - Assign OPC UA as Tag Data Source

You will now set the Tag's data source to the OPC UA driver interface that you created previously.

  1. Select the Tag that will source data from the local OPC UA data source.

    Tag

  2. Set the Data Source to OPC UA.

  3. Set the Select Driver Interface drop-down to the OPC UA Connector interface created previously.

  4. Click on the BROWSE button next to the NodeId field. In the BROWSE OPC UA SERVER window you can select the Local node, then select TemperatureSensorSimulation. In the right hand VARIABLE section select - Value. You should see the Node ID is now TemperatureSensorSimulation.Value.

    Click on the OK button to select this Tag.

    Browse OPC UA Server window

  5. Click on the Apply Changes button to apply the changes.

    OPC UA tag configuration

  6. Check that the quality status is Good Quality and the data in the Value field is as expected.

    OPC UA tag quality

  7. If you change the value of the TemperatureSensorSimulation Tag then you should see the same value update in the TemperatureSensor Tag. This means your OPC UA interface is working.

9 - Configure Kafka Producer

In the following steps you will create and configure a Kafka Connector connecting to a local Kafka cluster on port 9092. This connector will act as a Producer where Tags are published to a topic in the cluster.

  1. Select Configure > Drivers from the top menu.

    Configure drivers menu

  2. Set the Driver Interface Name to Kafka Producer to give this driver interface instance a unique name.

  3. Ensure the following parameters are configured:

    • Driver: Kafka
    • Bootstrap Servers: localhost:9092
    • Security Protocol: Plaintext
    • Client Id: oas-kafka-producer

    Kafka producer configuration

  4. Click on the ADD DRIVER button on the left hand side to add this driver configuration. Once added, the driver interface name should appear in the list of drivers.

    Add Kafka producer driver button

10 - Publish Selected Tags in Kafka Producer

In this step you will select the Tags that you want to publish to the Kafka cluster.

  1. In the Configure > Drivers screen, select the Kafka driver instance that you created in the previous section (for example Kafka Producer).

  2. Make sure the Publish Selected Tags checkbox is ticked.

    Publish selected tags

  3. In the table at the bottom click on the ADD button.

    Publish selected tags ADD button

  4. Select the Tag you want to add in the left hand panel and then ensure the Value property is selected. By default the name of the property will be the full Tag path (e.g. TemperatureSensor.Value). If you want to set your own property name, you can change the Id field to your own custom value.

    Tag browser

  5. The Tag has now been added to the list. You can add other Tags by repeating steps 3 and 4.

    Tag browser

  6. Click on the Apply Changes button.

11 - Verify Messages are Published to Kafka Cluster

In this step you will confirm that OAS is successfully publishing your selected Tags to the Kafka cluster.

  1. Run the following command in a terminal or command line window to start listening for new messages on the oas_tags topic which is the default topic for the Publish Selected Tags configuration.

    docker exec kafka kafka-console-consumer --topic oas_tags --bootstrap-server localhost:9092
    

    Info

    You may receive a warning message, because the oas_tags topic doesn't exist yet. You can safely ignore this as it will be created automatically.

  2. Any changes your TemperatureSensor tag value should be published within 10 seconds to the terminal window.

    {
        "values": [
            {
                "id": "TemperatureSensor",
                "value": 24.902344,
                "quality": true,
                "timestamp": "2025-05-20T09:23:41.332Z"
            }
        ]
    }
    
  3. To stop listening for new messages press the Ctrl-C shortcut combination on your keyboard.

12 - Save Changes

Once you have successfully configured your OAS instances, make sure you save your configuration.

On each configuration page, click on the Save button.

If this is the first time you are saving the configuration, or if you are changing the name of the configuration file, OAS will ask you if you want to change the default configuration file.

If you select Yes then OAS will make this configuration file the default and if the OAS service is restarted then this file will be loaded on start-up.

If you select No then OAS will still save your configuration file, but it will not be the default file that is loaded on start-up.

Change Default Configuration Files dialog

Important

Each configuration screen has an independent configuration file except for the Tags and Drivers configurations, which share the same configuration file. It is still important to click on the Save button whenever you make any changes.

For more information see: Save and Load Configuration

Info

  • On Windows the configuration files are stored in C:\ProgramData\OpenAutomationSoftware\ConfigFiles.
  • On Linux the configuration files are stored in the ConfigFiles subfolder of the OAS installation path.