How To Articles > How to Transfer Data from OPC to Kafka

How to Transfer Data from OPC to Kafka

OPC Kafka


Open Automation Software can be setup to transfer data from classic OPC DA 2.xx and 3.0 OPC servers to Kafka brokers.  Use the following guide to setup data transfer from OPC servers to Kafka. This tutorial walks you through downloading and installing OAS, defining the OPC server interface and automated tag creation with the One Click OPC feature, and publishing data to Kafka in a JSON format. The message topic to publish is automatically formatted to include tag id, value, quality, and timestamp. All JSON packet parameters are configurable to deliver the Kafka topic message in the desired format.

Step 1. Download and Install the Open Automation Software and Start the OAS Service

If you have not already done so, you will need to download and install the OAS platform.  Fully functional trial versions of the software are available for Windows, Windows IoT Core, Linux, Raspberry Pi and Docker on our downloads page.

On Windows run the downloaded Setup.exe file to install one or more of the Open Automation Software features. Select the default Typical installation if you are not sure what features to use or the Custom installation if you want to save disk space on the target system.  When prompted agree to the End User License Agreement to continue the installation.

For more detailed instructions and video tutorials, visit the installation guide for your system:
Windows Installation | Linux Installation | Raspberry Pi Installation | Dockers Installation

When the installation is finished the OAS Service Control application will appear.  Use this application to start the 4 Services. If this is the first time installing the software it will automatically enter Runtime with an example Tag Configuration.

Step 2. Configure OPC Server Tags

OAS provides multiple ways to add and define tags:

To add a Tag manually:

  1. In the OAS Configure Application, select Configure >> Tags from the top menu.


  2. Select localhost or the remote service you wish to modify with the Select button to the right of the Network Node list.


  3. Click on the Add Tag button located at the top of the Tag browser on the left portion of the screen.


  4. A dialog box will appear. Enter a name for your new tag and click ok.

  5. A configuration screen will appear for your new tag. Select OPC in the Data Source dropdown box.


  6. Use the Browse button to the right of the OPC Item to browse OPC Servers for the desired OPC Item.


  7. Select Local, the desired OPC Server, branch within the OPC Server, and OPC Item and click OK.


  8. Specify the desired OPC Update Rate for the Tag.

  9. Click Apply Changes at the bottom right of the window.

To add Tags with One Click OPC:

  1. In the OAS Configure Application, select Configure >> Tags from the top menu.


  2. Select localhost or the remote service you wish to modify with the Select button to the right of the Network Node list.


  3. To begin the One Click OPC process select the Group you would like to import to in the Tag configuration. If you want to import to the Root Level, select the Tags Group at the top of the tree. Then select the One Click Import button on the top menu bar.


  4. Click on the Import OPC DA 2.XX or 3.0 Items Button in the pop up window.


  5. Use the One Click OPC Wizard to browse for a branch as a starting position within an OPC Server or just select the OPC Server name itself to add all items from the selected OPC Server. For the best networking design select OPC Servers from Local even if you are configuring a remote OAS Service.


  6. Select to enable the options to Get Data Type from OPC Server and optionally the Descriptions. Additionally if you want to specify to Trend all of the points select Trend Points.

  7. Click Add Tags and it will automatically add all of the OPC Items from the OPC Server Branch you have selected and all of the sub Branches beneath it.

  8. Select the Save button on the toolbar at the top.

For more detailed instructions on Configuring OPC Server Tags, visit our Getting Started OPC tutorial or the One Click OPC tutorial or watch the video tutorial below:

Step 3. Define Kafka Topic and Tags to Publish

Select Configure-Drivers.

Configure Drivers

Enter a meaningful Driver Interface Name that you will refer to this connection and define the properties.

Set the driver to Kafka. Set the Bootstrap Servers of the brokers you wish to publish to.

Kafka localhost

If sending data to the cloud set the appropriate Security Protocol.

When connecting with Confluent Cloud set the SASL Username with the provided API Key and the SASL Password with the Secret Key.

Kakfa Sasl Ssl

Enable Publish Selected Tags at the bottom of the Driver configuration.

Select to publish data continuously at a specified interval, based on event, or at a specific time of day.

IoT Publish Type

If Event Driven browse for a local or remote OAS tag that will trigger the publish.  Select a Boolean tag that will change state from false to true, true to false, or both.  Or choose an Integer tag that trigger a publish anytime the value changes other than 0.

IoT Publish by Event

Specify the Publish Topic Id if choosing to Publish All Tags As One Topic.

When publishing as one topic to optimize communications with Kafka multiple messages are sent for each publish when the number of tag value changes exceeds the Max Tags Per Publish limit.

Max Tags per Publish

Specify the Tag Id, Value Id, an optional Quality Id, and Timestamp Id for each tag value that is sent.

IoT Publish Ids

When including the Timestamp Id also specify the timestamp format, use Custom to specify your own date and time format.

Specify local and remote OAS tag variables to include in each publish and specify the Id.  Value is the current value from the data source or you can select any of the over 600 tag variables of each tag to publish.

IoT Publish Tags

Optionally use CSV Export and CSV Import buttons to set up additional tags to publish using Microsoft Excel.

When selecting remote tags use Basic Networking syntax or Live Data Cloud syntax in the tag path.

Select the Add Driver button in the lower part of the form to add the Driver Interface.

Add Driver

Data is now publishing to the Kafka broker.

Select the Save button from the toolbar to save your configuration.

Save Button

You can view the Getting Started with Kafka Video to familiarize yourself with the following steps to setup OAS as a producer and consumer of Kafka.

  • 00:00 – Introduction
  • 01:20 – Kafka Producer
  • 09:07 – Kafka Consumer
  • 10:20 – Save Configuration
  • 11:00 – Transaction Logging
  • 11:45 – Publish 100,000 Values per Second
  • 15:50 – Contact Us