How to Transfer Data from Kafka to Sparkplug B

Open Automation Software can be used to transfer data from a Kafka cluster to a Sparkplug B Edge of Network (EoN) node locally or over a network. This guide walks you through downloading and installing OAS, configuring a Kafka data source with a Tag and a Sparkplug B host app and EoN node connector.

Typically you will have your own Edge of Network (EoN) node configured. For the purposes of this guide, you will create a Sparkplug B hosting app and EoN node. You can also use your own EoN node if you already have one.

For this guide on how to transfer data from a Kafka cluster to a Sparkplug B EoN you will need:

• Docker installed in Linux or Windows including the docker-compose utility

1 - Download and Install OAS

If you have not already done so, you will need to download and install the OAS platform.

Fully functional trial versions of the software are available for Windows, Windows IoT Core, Linux, Raspberry Pi and Docker on our downloads page.

On Windows, run the downloaded setup.exe file to install the Open Automation Software platform. For a default installation, Agree to the End User License Agreement and then click the Next button on each of the installation steps until it has completed.

If you'd like to customize your installation or learn more, use the following instructions:

• Windows Installation
• Linux Installation
• Raspberry Pi Installation
• Docker Installation

The OAS Service Control application will appear when the installation finishes on Windows.

Click on each START SERVICE button to start each of the three OAS services.

2 - Configure OAS

Configure OAS is the main application used to configure local and remote OAS instances.

1. From your operating system start menu, open the Configure OAS application.

2. Select the Configure > Tags screen.

   Important

   If this is the first time you have installed OAS, the AdminCreate utility will run when you select a screen in the Configure menu. This will ask you to create a username and password for the admin user. This user will have full permissions in the OAS platform.

   For further information see Getting Started - Security.

3. If this is the first time you are logging in, you will see the AdminCreate utility. Follow the prompts to set up your admin account. Otherwise, select the Log In menu button and provide the Network Node, username and password.

   

   

3 - Create a Kafka test cluster using Docker

In this step you will create a local Kafka cluster. To simplify the deployment you will need to have Docker installed including the docker-compose utility. This method allows you to create a cluster very quickly for testing purposes and remove it again when you are done.

1. Create a new folder such as Kafka and inside the folder create a new docker-compose.yml file with the following definition.

   services:
       zookeeper:
           image: confluentinc/cp-zookeeper:latest
           container_name: zookeeper
           ports:
               - "2181:2181"
           environment:
           ZOOKEEPER_CLIENT_PORT: 2181
           ZOOKEEPER_TICK_TIME: 2000
   
       kafka:
           image: confluentinc/cp-kafka:latest
           container_name: kafka
           ports:
               - "9092:9092"
           environment:
           KAFKA_BROKER_ID: 1
           KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
           KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
           KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
           KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
   
   

2. Open a command line terminal or bash terminal and use the following command to start a new container. This will download the image and all its dependencies and may take a few minutes.

   docker-compose up -d

3. Create a new topic called temperature using the following command.

   docker exec kafka kafka-topics --create --topic temperature --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

4 - Configure Kafka Consumer Connector

In the following steps you will create and configure a Kafka Connector connecting to a local Kafka cluster on port 9092. This connector will act as a Consumer where Tags can subscribe to a topic in the cluster.

1. Select Configure > Drivers from the top menu.

   

2. Set the Driver Interface Name to Kafka Consumer to give this driver interface instance a unique name.

3. Ensure the following parameters are configured:

   • Driver: Kafka
   • Bootstrap Servers: localhost:9092
   • Security Protocol: Plaintext
   • Client Id: oas-kafka

   

4. Click on the ADD DRIVER button on the left hand side to add this driver configuration. Once added, the driver interface name should appear in the list of drivers.

   

5 - Add Data Source Tag

In this section you will create a Tag to represent your data point in the field (for example a temperature sensor). This can then be transferred your desired destination.

1. Select Configure > Tags from the top menu.

   

2. If you want to add a Tag to the root Tags group make sure the Tags node is selected in the tag list and click on the ADD TAG button.

   

   If you want to add a Tag to a Tag Group, select the Tag Group first and then click on the ADD TAG button.

   You can also add Tag Groups by using the ADD GROUP button.

3. Provide a Tag Name such as TemperatureSensor and click the OK button.

   

6 - Assign Kafka as Tag Data Source

You will now set the Tag's data source to the Kafka driver interface that you created previously.

1. Select the Tag that will source data from a Kafka data source.

   

2. Set the Data Source to Kafka.

3. Set the Select Driver Interface drop-down to the Kafka Consumer interface created previously.

4. Set the Topic to temperature. This is the topic that you created previously when setting up the local Kafka cluster.

   

5. Click on the Apply Changes button to apply the changes.

6. To publish a data value from your Kafka cluster, you can use the following command.

   docker exec -i kafka kafka-console-producer --topic temperature --bootstrap-server localhost:9092

   You can now type in any data such as a numeric value and it will be published to the temperature topic.

7. Check that the quality status is Good Quality and the data in the Value field is as expected.

   

7 - Check that the MQTT Broker is enabled

If you are using the OAS MQTT Broker, you can follow the steps below to ensure it is enabled, otherwise you can skip this step.

1. Select Configure > Options from the top menu.

   

2. Select the MQTT Broker tab.

3. Check that the OAS MQTT Broker Enable checkbox is checked and the OAS MQTT Broker Port is configured as 1883.

   

8 - Create Security Group and User

When using Sparkplug B over the OAS MQTT Broker, you need to configure a security group and a user to provide Tag read/write access. You'll need these credentials when creating the Sparkplug B connector instances later.

ℹ️ You can skip this step if you already have your own MQTT Broker.

1. Select Configure > Security from the top menu.

   

2. Provide a Group Name such as SparkplugAccess.

   

3. Click on the DISABLE ALL button to disable all access.

   

4. In the Read Tags tab, ensure Disable All Tags From Reading is NOT checked.

   

5. In the Write Tags tab, ensure Disable Add Tags From Writing is NOT checked.

   

6. Click on the ADD GROUP button on the left hand side to add this security group configuration. Once added, the security group name should appear in the list of security groups.

   

7. Select Configure > Users from the top menu.

   

8. Provide a User Name such as sparkpluguser, a password and set the Security Group as SparkplugAccess.

   

9. Click on the ADD USER button on the left hand side to add this user configuration. Once added, the user name should appear in the list of users.

   

9 - Set up Sparkplug B Host App

In this section you will create a Sparkplug B driver configured using Host App mode. When you link EoN nodes to this host using the Host ID OAS will automatically create the Tags provided by the EoN node.

1. Select Configure > Drivers from the top menu.

   

2. Enter a name such as SpB Host App in the Driver Interface Name to give this driver interface a unique name.

3. Ensure the following parameters are configured:

   • Driver: Sparkplug B
   • Host: localhost
   • Port: 1883
   • User Name: sparkpluguser
   • Password: the password you configured in the previous step
   • Protocol Driver: V500
   • Client ID: OAS_Client_Host
   • Mode: Host App
   • Host ID: OAS_Host (you will need to use this in your EoN node configuration)

   

   ℹ️ If you are using your own MQTT Broker, ensure that you configure the Host, Port, User Name, Password and Protocol Driver accordingly.

4. Click on the ADD DRIVER button on the left hand side to add this driver configuration. Once added, the driver interface name should appear in the list of drivers.

   

10 - Create Destination Sparkplug B EoN node

In order to simulate the Edge of Network (EoN) node acting as a data destination, we can use the features available in OAS to create an EoN using a Sparkplug B connector instance and leverage the built-in MQTT broker.

ℹ️ You can skip this section if you want to use your own existing EoN node. You'll have to ensure that your EoN node is configured with the details in step 3 below so that it can talk to the OAS Host App.

1. Select Configure > Drivers from the top menu.

   

2. Enter a name such as EoN Destination Node in the Driver Interface Name to give this driver interface a unique name.

3. Ensure the following parameters are configured:

   • Driver: Sparkplug B
   • Host: localhost
   • Port: 1883
   • User Name: sparkpluguser
   • Password: the password you configured in the previous step
   • Protocol Driver: V500
   • Client ID: OAS_Destination_Node
   • Mode: Edge Node
   • Group ID: DestinationGroup
   • Edge Node ID: DestinationNode

   

4. Click on the ADD DRIVER button on the left hand side to add this driver configuration. Once added, the driver interface name should appear in the list of drivers.

   

11 - Set Host Configuration on Tag

In this section you will configure the Sparkplug B Host parameters for the EoN node where the Tag value should be sent to.

1. On the Tags configuration screen, select the data source Tag.

   

2. Configure the Host parameters according to your Sparkplug B destination EoN node.

   • Host Group ID: DestinationGroup
   • Host Edge Node ID: DestinationNode
   • Host Device ID: EoN Data Destination
   • Host Metric Name: Temperature

   

3. Click on the Apply Changes button to apply the changes.

12 - Verify Host App Tag Generation for Data Destination

You will now check to make sure the Tag Group folder structure and Tag for the Temperature metric was automatically generated and that any updates to the TemperatureSensor tag will flow through to the Sparkplug B tag structure.

1. On the Tags screen, click on the SELECT button to refresh the tag list.

2. You will see a Tag Group structure starting with a parent folder called SpB Host App and then a sub-folder for DestinationGroup, another sub-folder for DestinationNode and finally a sub-folder for the EoN node called Eon Data Destination. The Temperature tag representing the Temperature metric is inside this final sub-folder.

   As you can see the Sparkplug B host app driver has automatically generated the Tag Groups and Tags.

   

3. Select the Temperature tag in the EoN Data Destination sub-folder. You should see the value updating according to your data source and the Client parameters configured according to the EoN node Host properties. These properties were automatically configured by OAS.

   

13 - Save Changes

Once you have successfully configured your OAS instances, make sure you save your configuration.

On each configuration page, click on the Save button.

If this is the first time you are saving the configuration, or if you are changing the name of the configuration file, OAS will ask you if you want to change the default configuration file.

If you select Yes then OAS will make this configuration file the default and if the OAS service is restarted then this file will be loaded on start-up.

If you select No then OAS will still save your configuration file, but it will not be the default file that is loaded on start-up.