How To Articles > How to Transfer Data from MTConnect to Kafka

How to Transfer Data from MTConnect to Kafka

MTConnect Kafka


Open Automation Software can be setup to transfer data from MTConnnect data streams to Kafka brokers.  Use the following guide to setup data transfer from MTConnect to Kafka. This tutorial walks you through downloading and installing OAS, defining MTConnect interface with automated tag creation, and publishing data to Kafka in a JSON format. The message topic to publish is automatically formatted to include tag id, value, quality, and timestamp. All JSON packet parameters are configurable to deliver the Kafka topic message in the desired format.

Step 1. Download and Install the Open Automation Software and Start the OAS Service

If you have not already done so, you will need to download and install the OAS platform.  Fully functional trial versions of the software are available for Windows, Windows IoT Core, Linux, Raspberry Pi and Docker on our downloads page.

On Windows run the downloaded Setup.exe file to install one or more of the Open Automation Software features. Select the default Typical installation if you are not sure what features to use or the Custom installation if you want to save disk space on the target system.  When prompted agree to the End User License Agreement to continue the installation.

For more detailed instructions and video tutorials, visit the installation guide for your system:
Windows Installation | Linux Installation | Raspberry Pi Installation | Dockers Installation

When the installation is finished the OAS Service Control application will appear.  Use this application to start the 4 Services. If this is the first time installing the software it will automatically enter Runtime with an example Tag Configuration.

Step 2. Configure Your MTConnect Data Source

  1. First, you will need to open the Configure OAS application from the program group Open Automation Software.

  2. Select Configure >> License from the top menu and verify that MTConnect is one of the available Drivers in the lower left of the form. The demo license will have this by default. If you do not see MTConnect available, contact support@openautomationsoftware.com to update your license.

  3. Select Configure >> Drivers from the top menu.


  4. Select localhost or the remote service you wish to modify with the Select button to the right of the Network Node list.


  5. The Configure Drivers Screen will appear. Select MTConnect from the Driver dropdown box.


  6. Enter a meaningful Driver Interface Name that you will refer to this physical connection when defining Tags with a MTConnect Data Source.

  7. Leave Enable and Add Tags Automatically enabled.

  8. Specify the Live Data Url for the MTConnect stream.

  9. Click the Add Driver button above the Driver list in the left pane to add the Driver Interface as an available selection when defining Tags in the next step.

For more detailed instructions on configuring your MTConnect data source, click here to see our Getting Started MTConnect tutorial or watch the video tutorial below:

Note: OAS can automate the tag creation under the driver interface for MTConnect.

Step 3. Configure Your MTConnect Tags

OAS provides multiple ways to add and define tags:

To add a Tag manually:

  1. In the OAS Configure Application, select Configure >> Tags from the top menu.


  2. Select localhost or the remote service you wish to modify with the Select button to the right of the Network Node list.


  3. Click on the Add Tag button located at the top of the Tag browser on the left portion of the screen.


  4. A dialog box will appear. Enter a name for your new tag and click ok.

  5. A configuration screen will appear for your new tag. Select your data source type in in the Data Source dropdown box.


  6. Specify the correct data type in the Data Type dropdown box.

  7. Click Apply Changes at the bottom right of the window.

For more detailed instructions on configuring your tags, click here to see our Getting Started Tags tutorial.

Step 4. Define Kafka Topic and Tags to Publish

Select Configure-Drivers.

Configure Drivers

Enter a meaningful Driver Interface Name that you will refer to this connection and define the properties.

Set the driver to Kafka. Set the Bootstrap Servers of the brokers you wish to publish to.

Kafka localhost

If sending data to the cloud set the appropriate Security Protocol.

When connecting with Confluent Cloud set the SASL Username with the provided API Key and the SASL Password with the Secret Key.

Kakfa Sasl Ssl

Enable Publish Selected Tags at the bottom of the Driver configuration.

Select to publish data continuously at a specified interval, based on event, or at a specific time of day.

IoT Publish Type

If Event Driven browse for a local or remote OAS tag that will trigger the publish.  Select a Boolean tag that will change state from false to true, true to false, or both.  Or choose an Integer tag that trigger a publish anytime the value changes other than 0.

IoT Publish by Event

Specify the Publish Topic Id if choosing to Publish All Tags As One Topic.

When publishing as one topic to optimize communications with Kafka multiple messages are sent for each publish when the number of tag value changes exceeds the Max Tags Per Publish limit.

Max Tags per Publish

Specify the Tag Id, Value Id, an optional Quality Id, and Timestamp Id for each tag value that is sent.

IoT Publish Ids

When including the Timestamp Id also specify the timestamp format, use Custom to specify your own date and time format.

Specify local and remote OAS tag variables to include in each publish and specify the Id.  Value is the current value from the data source or you can select any of the over 600 tag variables of each tag to publish.

IoT Publish Tags

Optionally use CSV Export and CSV Import buttons to set up additional tags to publish using Microsoft Excel.

When selecting remote tags use Basic Networking syntax or Live Data Cloud syntax in the tag path.

Select the Add Driver button in the lower part of the form to add the Driver Interface.

Add Driver

Data is now publishing to the Kafka broker.

Select the Save button from the toolbar to save your configuration.

Save Button

You can view the Getting Started with Kafka Video to familiarize yourself with the following steps to setup OAS as a producer and consumer of Kafka.

  • 00:00 – Introduction
  • 01:20 – Kafka Producer
  • 09:07 – Kafka Consumer
  • 10:20 – Save Configuration
  • 11:00 – Transaction Logging
  • 11:45 – Publish 100,000 Values per Second
  • 15:50 – Contact Us