Kafka Videos
Two videos cover the Kafka driver. The first is a two minute introduction to what the connector is for; the second is a full walkthrough that builds a producer and a consumer and then pushes them to 100,000 values per second.
Apache Kafka Connector
Where the connector sits in a plant. The point it makes is translation rather than transport: the equipment that holds the data does not speak Kafka, and Kafka does not speak Modbus, Siemens S7, OPC UA and DA, Allen-Bradley ControlLogix, CompactLogix, GuardLogix, Micro800, MicroLogix, SLC 500 and PLC-5, MTConnect or MQTT. The driver sits between them in both directions.
There is no configuration in this one. Watch it to decide whether the driver fits what you are building, then go to Getting Started to build it.
Kafka IIoT
The full walkthrough, and the one to watch if you are configuring the driver for the first time. It builds a producer that publishes tags from any OAS data source to a topic, then a consumer that reads a topic back into a tag, saves the configuration, and finishes on a throughput demonstration.
The last third is worth watching even if you already have a working driver - the 100,000 values per second segment is where the batching and buffering parameters stop being abstract.
Chapters
| Time | Chapter | Covered in |
|---|---|---|
| 0:00 | Introduction | Kafka Driver |
| 1:20 | Kafka Producer | Configure Kafka Producer |
| 9:07 | Kafka Consumer | Configure Kafka Consumer |
| 10:20 | Save Configuration | Save the configuration |
| 11:00 | Transaction Logging | Data Logging guides |
| 11:45 | Publish 100,000 Values per Second | Producer tuning |
| 15:50 | Contact Us | - |
Info
The walkthrough was recorded in October 2022 and shows the driver screen as it was then. Everything in it still applies, but the screen has since gained the Message format, Schema mapping and Schema Registry options, none of which appear in the video. The messages it publishes are the Default JSON format, which is still the default today.
Where next
- Getting Started with Kafka - the walkthrough in written form, with a Docker test cluster.
- Config Reference - every parameter shown in the videos, with its default.
- Videos - the full OAS video library.