How to Transfer Data from Sparkplug B to Kafka
Open Automation Software can publish data from Sparkplug B Edge of Network Nodes to local Kafka brokers or cloud-hosted Kafka brokers. Use the following guide to set up data transfer from Sparkplug B to Kafka. This tutorial walks you through downloading and installing OAS, defining Sparkplug B Metric tags using the OAS Sparkplug B Host Application interface, and publishing data to Kafka in a JSON format. The message topic to publish is automatically formatted to include tag id, value, quality, and timestamp. All Sparkplug B and Kafka parameters are adjustable to deliver the message in the desired format.
Step 1. Download and Install the Open Automation Software and Start the OAS Service
If you have not already done so, you will need to download and install the OAS platform. Fully functional trial versions of the software are available for Windows, Windows IoT Core, Linux, Raspberry Pi and Docker on our downloads page.
On Windows run the downloaded Setup.exe file to install one or more of the Open Automation Software features. Select the default Typical installation if you are not sure what features to use or the Custom installation if you want to save disk space on the target system. When prompted agree to the End User License Agreement to continue the installation.
For more detailed instructions and video tutorials, visit the installation guide for your system:
Windows Installation | Linux Installation | Raspberry Pi Installation | Dockers Installation
The OAS Service Control application will appear when the installation finishes on Windows. Use this application to start the 3 Services. Run the Configure OAS application on Windows and select Configure-Tags; if the first time running, the AdminCreate utility will run to create an Administrator login as shown in Step 1 of Getting Started – Security.
Step 2. Set Up Sparkplug B Host App to Access EoN Node Data.
Select Configure-Drivers.
Enter a meaningful Driver Interface Name that you will refer to this connection and define the properties.
Set the driver to Sparkplug B.
Enter the Host property to the IP Address of the of the host broker or just use localhost for a local broker.
Set the Port property of the host broker to connect to. The default port is 1883 or 8883 for SSL.
Note: The OAS MQTT Broker can be used to host data locally. To use another broker on the default ports on the same computer change the OAS MQTT Broker Port under Configure-Options-MQTT Broker.
Enter the User Name and Password if required.
Set the MQTT Protocol Version to V311 or V500.
Set the Client ID to a unique value, each driver interface must have a specific value different than all other clients connecting to the host.
Set the Mode to Host App for OAS to control the online state of each Edge of Node defined to the shared Host Application ID. Or set to Client App to access EoN Node data without controlling their state.
Set the Host ID to the specific value that the Edge of Nodes will be using as the Host Application ID.
Set the Sparkplug B Version to 2.2 or 3.0.
To automatically add tags for each Metric from the matching Group, Edge of Nodes, and Devices enable the property Add Client Tags Automatically.
When enabled optionally set the Filter Groups, Edge Nodes, and Devices that tags should be added from for each of their Metrics. Leave the Filter fields blank to include all Edge of Nodes publishing data to the broker.
Select the Add Driver button in the left part of the form to add the Driver Interface as an available selection when defining Tags in the next step.
You are now ready to add tags for specific Sparkplug B EoN Node Metric values.
Note: If the Driver Interface property Add Client Tags Automatically was enabled tags will be created for you, you can skip this step to add tags manually.
Select Configure-Tags.
Note: You can add organizational Groups as many levels deep as you prefer and add tags to groups. To do this first add a Group to Tags Group at the root level, then right click on the Group in the right window to add additional Groups or Tags.
Select Add Tag.
Change the Data Source Tag property to Sparkplug B.
Select the correct Driver Interface from the Driver Interface pull down list.
Specify the Client Group ID, Client Edge Node ID, Client Device ID, and Client Metric Name for the Edge of Node Metric value to read and write.
Select the Data Type of the Tag to match the data type of the Metric.
Select Apply Changes and the current value for the Metric should appear with Good Quality.
Select the Save button from the toolbar to save your configuration.
You can view the Getting Started with Sparkplug B Video to familiarize yourself with the above steps to setup OAS as a Sparkplug B Edge of Network Node and a Host Application
- 00:00 – Introduction
- 01:10 – Download OAS
- 01:46 – Quick Start
- 02:08 – Edge of Network Node
- 05:52 – EoN Tags
- 07:04 – Host Application
- 09:20 – SpB Client Tags
- 10:22 – Programatic Interface
- 10:59 – CSV Export / Import
- 12:27 – Multiple Metric Tags
- 13:46 – Modbus
- 16:10 – DCMD Metric Write
- 17:00 – Data Route
- 17:58 – Networking
- 18:17 – Save Configuration
- 18:56 – OAS Platform
Step 3. Define Kafka Topic and Tags to Publish
Select Configure-Drivers.
Enter a meaningful Driver Interface Name that you will refer to this connection and define the properties.
Set the driver to Kafka. Set the Bootstrap Servers of the brokers you wish to publish to.
If sending data to the cloud set the appropriate Security Protocol.
When connecting with Confluent Cloud set the SASL Username with the provided API Key and the SASL Password with the Secret Key.
Enable Publish Selected Tags at the bottom of the Driver configuration.
Select to publish data continuously at a specified interval, based on event, or at a specific time of day.
If Event Driven browse for a local or remote OAS tag that will trigger the publish. Select a Boolean tag that will change state from false to true, true to false, or both. Or choose an Integer tag that trigger a publish anytime the value changes other than 0.
Specify the Publish Topic Id if choosing to Publish All Tags As One Topic.
When publishing as one topic to optimize communications with Kafka multiple messages are sent for each publish when the number of tag value changes exceeds the Max Tags Per Publish limit.
Specify the Tag Id, Value Id, an optional Quality Id, and Timestamp Id for each tag value that is sent.
When including the Timestamp Id also specify the timestamp format, use Custom to specify your own date and time format.
Specify local and remote OAS tag variables to include in each publish and specify the Id. Value is the current value from the data source or you can select any of the over 600 tag variables of each tag to publish.
Optionally use CSV Export and CSV Import buttons to set up additional tags to publish using Microsoft Excel.
When selecting remote tags use Basic Networking syntax or Live Data Cloud syntax in the tag path.
Select the Add Driver button in the lower part of the form to add the Driver Interface.
Data is now publishing to the Kafka broker.
Select the Save button from the toolbar to save your configuration.
You can view the Getting Started with Kafka Video to familiarize yourself with the following steps to setup OAS as a producer and consumer of Kafka.
- 00:00 – Introduction
- 01:20 – Kafka Producer
- 09:07 – Kafka Consumer
- 10:20 – Save Configuration
- 11:00 – Transaction Logging
- 11:45 – Publish 100,000 Values per Second
- 15:50 – Contact Us