How to Transfer Data from MTConnect to Kafka
1 - Download and Install OAS
If you have not already done so, you will need to download and install the OAS platform.
Fully functional trial versions of the software are available for Windows, Windows IoT Core, Linux, Raspberry Pi and Docker on our downloads page.
On Windows, run the downloaded setup.exe file to install the Open Automation Software platform. For a default installation, Agree to the End User License Agreement and then click the Next button on each of the installation steps until it has completed.
If you'd like to customize your installation or learn more, use the following instructions:
The OAS Service Control application will appear when the installation finishes on Windows.
Click on each START SERVICE button to start each of the three OAS services.
2 - Configure OAS
Configure OAS is the main application used to configure local and remote OAS instances.
From your operating system start menu, open the Configure OAS application.
Select the Configure > Tags screen.
Important
If this is the first time you have installed OAS, the AdminCreate utility will run when you select a screen in the Configure menu. This will ask you to create a username and password for the admin user. This user will have full permissions in the OAS platform.
For further information see Getting Started - Security.
If this is the first time you are logging in, you will see the AdminCreate utility. Follow the prompts to set up your admin account. Otherwise, select the Log In menu button and provide the Network Node, username and password.
Info
In this guide you will use the Configure OAS application to configure the local Network Node which by default is localhost.
If you have installed OAS on a remote instance you can also connect to the remote instance by setting the relevant IP address or host name in the Network Node field.
3 - Create a Kafka test cluster using Docker
In this step you will create a local Kafka cluster. To simplify the deployment you will need to have Docker installed including the docker-compose utility. This method allows you to create a cluster very quickly for testing purposes and remove it again when you are done.
Create a new folder such as Kafka and inside the folder create a new
docker-compose.yml
file with the following definition.services: zookeeper: image: confluentinc/cp-zookeeper:latest container_name: zookeeper ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 kafka: image: confluentinc/cp-kafka:latest container_name: kafka ports: - "9092:9092" environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
Open a command line terminal or bash terminal and use the following command to start a new container. This will download the image and all its dependencies and may take a few minutes.
docker-compose up -d
Create a new topic called temperature using the following command.
docker exec kafka kafka-topics --create --topic temperature --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
Info
Once you are done testing, you can stop and remove the containers that you created by using the following command:
docker-compose down
4 - Configure MT Connect Data Source
In the following steps you will create and configure an MT Connect Connector that connects to a publicly available endpoint. It will also make use of automatic tag generation.
Select Configure > Drivers from the top menu.
Enter a meaningful Driver Interface Name to give this driver interface instance a unique name.
Provide a Live Data Url to connect to an MTConnect endpoint.
Specify the Interval at which to poll the endpoint.
Click on the ADD DRIVER button on the left hand side to add this driver configuration. Once added, the driver interface name should appear in the list of drivers.
5 - Automatic Tag Generation
The MTConnect endpoint provides a schema that allows OAS to automatically create Tags for the data points defined in the schema.
Select Configure > Tags from the top menu.
You will see a Tag Group that matches the name of the MTConnect driver you created. The MTConnect driver creates all of the Tag Groups and Tags within this parent Tag Group.
You can remove any Tags or even entire Tag Groups that you do not need by selecting them and pressing Delete button on your keyboard or right-clicking on a Tag or Tag Group and selecting Remove.
Tips
Some of the Tags may appear red. This indicates that the Tag value is in a Bad Quality state. This usually means that the Tag did not receive a value from the MTConnect endpoint.
6 - Configure Kafka Producer
In the following steps you will create and configure a Kafka Connector connecting to a local Kafka cluster on port 9092. This connector will act as a Producer where Tags are published to a topic in the cluster.
Select Configure > Drivers from the top menu.
Set the Driver Interface Name to Kafka Producer to give this driver interface instance a unique name.
Ensure the following parameters are configured:
- Driver: Kafka
- Bootstrap Servers: localhost:9092
- Security Protocol: Plaintext
- Client Id: oas-kafka-producer
Click on the ADD DRIVER button on the left hand side to add this driver configuration. Once added, the driver interface name should appear in the list of drivers.
7 - Publish Selected Tags in Kafka Producer
In this step you will select the Tags that you want to publish to the Kafka cluster.
In the Configure > Drivers screen, select the Kafka driver instance that you created in the previous section (for example Kafka Producer).
Make sure the Publish Selected Tags checkbox is ticked.
In the table at the bottom click on the ADD button.
Select the Tag you want to add in the left hand panel and then ensure the Value property is selected. By default the name of the property will be the full Tag path (e.g. MTConnect1.HCN001.avail.Value). If you want to set your own property name, you can change the Id field to your own custom value.
The Tag has now been added to the list. You can add other Tags by repeating steps 3 and 4.
Click on the Apply Changes button.
8 - Verify Messages are Published to Kafka Cluster
In this step you will confirm that OAS is successfully publishing your selected Tags to the Kafka cluster. By default, the publishing type is set to Continuous and the interval is 10 seconds.
Run the following command in a terminal or command line window to start listening for new messages on the oas_tags topic which is the default topic for the Publish Selected Tags configuration.
docker exec kafka kafka-console-consumer --topic oas_tags --bootstrap-server localhost:9092
Info
You may receive a warning message, because the oas_tags topic doesn't exist yet. You can safely ignore this as it will be created automatically.
Any changes your HCN001_Available_State tag value should be published within 10 seconds to the terminal window.
{ "values": [ { "id": "HCN001_Available_State", "value": "AVAILABLE", "quality": true, "timestamp": "2025-05-20T09:23:41.332Z" } ] }
To stop listening for new messages press the Ctrl-C shortcut combination on your keyboard.
9 - Save Changes
Once you have successfully configured your OAS instances, make sure you save your configuration.
On each configuration page, click on the Save button.
If this is the first time you are saving the configuration, or if you are changing the name of the configuration file, OAS will ask you if you want to change the default configuration file.
If you select Yes then OAS will make this configuration file the default and if the OAS service is restarted then this file will be loaded on start-up.
If you select No then OAS will still save your configuration file, but it will not be the default file that is loaded on start-up.
Important
Each configuration screen has an independent configuration file except for the Tags and Drivers configurations, which share the same configuration file. It is still important to click on the Save button whenever you make any changes.
For more information see: Save and Load Configuration
Info
- On Windows the configuration files are stored in C:\ProgramData\OpenAutomationSoftware\ConfigFiles.
- On Linux the configuration files are stored in the ConfigFiles subfolder of the OAS installation path.