Kafka Config Reference

Below is a comprehensive list of each configuration option on the Kafka driver with its default, a description of its purpose and what effect it will have.

The driver options are set on the Drivers screen. The options a tag uses to consume from Kafka are set per tag on the Tags screen, and are listed under Consumer tag properties.

Minimum configuration requirements

These are the fields that need a value before the driver will do anything. Everything else has a working default.

Field	Needed for	Notes
Driver Interface Name	Both	The name tags refer to in Select Driver Interface.
Driver	Both	Set to Kafka.
Bootstrap Servers	Both	Defaults to `localhost:9092`, which is a real address - the driver will try it.
Enable	Both	The interface is not active until this is on and the changes are applied.
Publish Selected Tags	Producer	Off by default. A driver interface with this off is a consumer only.
Tags To Publish	Producer	Nothing is published until tags are added.
Topic	Consumer	Set on the tag, not on the driver.

Driver interface

Property	Default	Notes
Driver Interface Name	-	The name used to identify this driver interface. Tags reference it from their Select Driver Interface property.
Driver	Modbus	Set to Kafka to produce and consume with Apache Kafka clusters.
Enable	on	Enable communications. Hidden when Enable by Tag is on.
Enable by Tag	off	Enable and disable the driver from a Boolean tag instead, named in the field beside it.
Good Quality When Disabled	off	On: tags on this driver keep good quality while the driver is disabled, and can be written to. Off: they go to bad quality.
Simulate	off	Disable communications and hold every value static with good quality.
Return to Online	60 Seconds	How often to re-check communications after the interface has been taken offline.

Connection

Property	Default	Notes
Bootstrap Servers	`localhost:9092`	Comma-delimited list of broker addresses as `host:port`. This is a seed list only - the addresses are used to fetch cluster metadata and the full broker set is then learned from the cluster, so listing more than one buys startup and reconnect resilience rather than load balancing. Every entry must be a broker of the same cluster; a disaster-recovery cluster belongs in Failover Bootstrap Servers, not here.
Security Protocol	Plaintext	`Plaintext`, `Ssl`, `SaslPlaintext` or `SaslSsl`. Plaintext suits a cluster on the local network; a hosted cluster will want SSL or SASL.
SASL Mechanism	Plain	`Gssapi`, `Plain`, `ScramSha256`, `ScramSha512` or `OAuthBearer`. Shown for the SASL protocols.
SASL Username	blank	Shown for `Plain`, `ScramSha256` and `ScramSha512`. For Confluent Cloud this is the API key.
SASL Password	blank	As above. For Confluent Cloud this is the secret.

TLS certificates

Shown when the Security Protocol is Ssl or SaslSsl. Each certificate can be given as a file path or pasted as PEM text - use one or the other, not both.

Property	Default	Notes
SSL CA Location	blank	Path to the CA certificate file.
SSL CA Pem	blank	The CA certificate as PEM text.
SSL Certificate Location	blank	Path to the client certificate file.
SSL Certificate Pem	blank	The client certificate as PEM text.
SSL Key Location	blank	Path to the client private key file.
SSL Key Pem	blank	The client private key as PEM text.
SSL Key Password	blank	Passphrase for an encrypted private key.
SSL Keystore Location	blank	Path to a PKCS#12 keystore, as an alternative to the separate certificate and key.
SSL Keystore Password	blank	Password for the keystore.

Producer tuning

These tune throughput against latency on the producer side. Each has a working default, and the defaults suit most clusters.

Property	Default	Notes
Acks	Leader	`None`, `Leader` or `All`. How many brokers must acknowledge a write before it counts as delivered. `All` is the durable choice; `None` is the fastest and can lose messages.
Client Id	`oas-kafka`	The identifier the driver sends with its requests, and what the broker logs it as.
Compression Type	None	`None`, `Gzip`, `Snappy`, `Lz4` or `Zstd`.
Batch Num Messages	10000	Maximum number of messages in one producer batch.
Batch Size	1000000	Maximum size in bytes of one producer batch.
Linger	5 ms	How long to wait for more messages before sending a batch. Higher gives larger, better-compressed batches at the cost of latency.
Max In Flight	1000000	Requests allowed in flight per broker connection before the producer blocks.
Message Max Bytes	1000000	Maximum size of a single message. Must not exceed the broker's own limit.
Message Timeout	300000 ms	How long the producer keeps trying to deliver a message before it fails.
Queue Buffering Max Kbytes	1048576	Maximum total size of the producer's send queue.
Queue Buffering Max Messages	100000	Maximum number of messages in the producer's send queue.
Request Timeout	30000 ms	How long to wait for a broker to answer a produce request.
Socket Connection Timeout	30000 ms	How long to wait for a broker socket to be set up.
Socket Keepalive Enable	off	Enable TCP keep-alives on broker sockets.
Socket Timeout	60000 ms	How long a broker socket may stall before it is treated as failed.
Transaction Timeout	60000 ms	How long a transaction may remain open.

Topic creation

Property	Default	Notes
Automatically Create Topics	on	Before a topic is written to for the first time, create it if it does not exist already.
Replication Factor	-1	Used when a topic is created. `-1` leaves it to the broker's default. Shown when Automatically Create Topics is on.
Number Partitions	-1	Used when a topic is created. `-1` leaves it to the broker's default. Shown when Automatically Create Topics is on.

Property	Default	Notes
Output Format	Default	`Default` - JSON, with field names taken from the payload field names. `ISA-95 Lite` - the built-in unified namespace schema. `Custom AVRO Schema` - your own schema.
Batched Messages	on	On: the tags of each publish that share a topic go in one message, up to Max Tags Per Publish. Off: each tag value is published as its own message, still using the same structure. Shown with the `Default` format, and New with `Custom AVRO Schema` when Publish All Tags As One Topic is on: there, a schema needs a tag array or tag map to hold many tags, and without one each message holds one tag. The same switch is in the Edit AVRO schema dialog.
AVRO Schema	blank	The AVRO schema the message is published against, and which part of a tag fills each of its fields. Both are built in the Edit AVRO schema dialog, opened with Edit… - see Schema mapping. Shown when the Output Format is `Custom AVRO Schema`.
Force AVRO Binary New	off	Publish AVRO binary using the Avro single-object encoding, which carries a fingerprint of the schema in place of a Schema Registry ID. The consumer needs a copy of the same schema, so click Download .avsc beside Output Format and give the file to the consumer - that is what keeps the two ends in step. When this is off, messages are published as JSON. There is no JSON fallback while this is on: a value that cannot be encoded is discarded and logged. With the `Default` format, changed payload field names, or Publish All Tags As One Topic off, are encoded with a schema that matches them, and Download .avsc gives that schema. Shown when the Schema Registry is off - with the registry on the wire format is already AVRO binary and the schema ID comes from the registry.

Field	Filled from
`equipmentHierarchy.enterprise` `.site` `.siteId` `.area`	The ISA-95 level assigned to the tag groups that enclose the tag
`equipmentHierarchy.workCenter` `.workCenterType`	Work Center tier - Process Cell, Production Unit, Production Line or Storage Zone
`equipmentHierarchy.workUnit` `.workUnitType` `.workUnitId`	Work Unit tier - Unit, Work Cell or Storage Unit
`equipmentHierarchy.equipmentModule` `.equipmentModuleId` `.controlModule` `.controlModuleId`	The lower two tiers
`groupPath`	The tag's group path verbatim, as an array, at any depth
`tagName`	The full tag path
`timestamp`	The tag's own timestamp, as epoch milliseconds
`dataType`	`BOOLEAN`, `INT`, `LONG`, `FLOAT`, `DOUBLE` or `STRING`
`value`	The tag value
`quality.isGood` `quality.severity`	OAS quality, normalised to `GOOD` or `BAD`
`units` `sourceProtocol`	The tag's Units and Data Source
`metadata`	Additional key/value pairs

The hierarchy fields are resolved by walking the tag groups that enclose the tag and reading the ISA-95 level assigned to each group. A group with no level assigned contributes nothing, and a tag at the root of the configuration publishes null for the hierarchy fields. Where a group has an export name override, the override is published instead of the group's own name.

Schema mapping New

A custom AVRO schema has to be told which part of a tag fills each of its fields. Both the schema and that mapping are built in the Edit AVRO schema dialog, which the MQTT driver and the schema library use too. Its three tabs, Schema Editor, Field Mapping and Data Test, are covered on the shared page: The Edit AVRO schema dialog. Nothing is written back to the driver until you click OK.

On the Kafka driver:

A message holds many tags only with Custom AVRO Schema, a schema with a tag array or tag map, Batched Messages on and Publish All Tags As One Topic on. Otherwise each message holds one tag.
The new Custom Format and Sequence sources work as they do on the MQTT driver.
Data Test shows each message with the Kafka topic it would go to.

Schema Registry New

Property	Default	Notes
Enable Schema Registry	off	Connect to a Confluent Schema Registry for AVRO schema management.
Schema Registry URL	blank	For example `http://localhost:8081`.
Username	blank	If the registry requires authentication.
API Key	blank	Entered as a password.
Subject Name	blank	The subject to register or retrieve schemas under. Leave blank to register under `<topic name>-value`. The name is used exactly as written: it cannot hold variables, braces are kept as typed, and the web app warns about them. New Required when Publish Topic Id is a topic pattern and Publish All Tags As One Topic is on: left blank, the driver does not connect, raises a schema alarm, and the web app shows an error on the setting.

Warning

With the Schema Registry enabled and the Default output format, the message always follows the standard Default schema, with its fixed field names - values, id, value, quality, timestamp - and always includes quality and timestamp. That holds with Publish All Tags As One Topic off too. The payload field names below do not apply and are hidden; Timestamp Type and its Format still do.

With a topic pattern, the schema is registered once, when the driver connects, under the one Subject Name, and every topic's messages carry its ID. A subject for each topic is not supported.

Publishing

Publishing is off until Publish Selected Tags is turned on. A driver interface with it off is a consumer only.

Property	Default	Notes
Publish Selected Tags	off	Turns publishing on for this driver interface.
Tags To Publish	-	The tags to publish, each with the Id that names it in the message. The Id is also the topic when Publish All Tags As One Topic is off. A tag can be listed more than once, each time with a different Id, and is published under every Id. An Id can be made of parts separated by a vertical bar, such as `SITE-A\|PQM-1\|P01`, which `{tagIdPart1}`, `{tagIdPart2}` and so on read - see Message Variables. The spaces around each part are dropped.
Publish Type	Continuous	`Continuous` on an interval, `Event Driven` from a trigger tag, or `Specific Time of Day`.
Publish Interval	10 Seconds	Continuous only.
Publish On Startup	on	Continuous only. Publish immediately on startup rather than waiting out the first interval.
Trigger Tag	-	Event Driven only. A Boolean tag triggers on transition; an Integer tag triggers whenever its value changes to a non-zero value.
Digital Trigger Type	False to True	Event Driven only. `False to True`, `True to False` or `Both`.
Time of Day	-	Specific Time of Day only.
Publish Latest Value Only	on	On: publish only the current value of each tag. Off: publish every value change since the last publish. Turn this off when the target is a historian.
Publish Boolean As Number	off	Publish `1` and `0` rather than `true` and `false`.
Include All Tags Each Publish	off	Include every selected tag in each publish, even the ones that have not changed.
Include Tags For Period	off	Include all tag values in each publish for the period set beside it, then drop the unchanged ones from further publishing. The period defaults to 1 second.
Publish All Tags As One Topic	on	On: all values go to Publish Topic Id. Off: each tag goes to its own topic, named by its Id in Tags To Publish.
Max Tags Per Publish New	1000	The most tags in one message, with Batched Messages on - in the `Default` format, or in a schema with a tag array or tag map. A publish with more is sent as several messages on the same topic, in order. With a topic pattern it limits each topic's message. `0` means no limit: every tag in one message.
Publish Topic Id	`oas_tags`	The topic used when Publish All Tags As One Topic is on. New It can be a topic pattern, such as `oas.{groupPath}`, that gives each tag a topic of its own.
Override Value When Bad	None	What to publish when a tag's quality is bad: `None` (publish it bad), `Do Not Send`, `Hold Last Value`, `Set To Value` or `Hold Then Set`.
Override Time	60 Seconds	How long to hold the last good value. Shown for `Hold Last Value` and `Hold Then Set`.
Value When Bad	0	The value to publish instead. Shown for `Set To Value` and `Hold Then Set`.

Upgrading

A Default format driver with Batched Messages on that publishes more tags at once than Max Tags Per Publish now sends several messages, each with its own values array, where before it sent one. Set Max Tags Per Publish to 0 to keep one message.

Topic patterns New

With Publish All Tags As One Topic on, Publish Topic Id can hold message variables in braces, as on the MQTT driver. Each tag goes to the topic the pattern gives it, the topic is created under that name, and the tags of one publish that share a topic share a message.

Each topic must also be a legal Kafka topic name: 1 to 249 characters of letters, digits, ., _ and -, and not . or ... So:

use {groupPath}, {tagPath} and {hierarchyPath}, not their slash forms, which always fail - oas.{groupPath} gives oas.Plant.Line1,
a group, tag or override name with a space or any other character outside that set fails, and so does a {property:Name} whose value holds one.

A tag whose topic breaks the rule, or has no value for a variable in it, is not published, and System Errors gets one entry for it. The line under Publish Topic Id in the web app shows an example topic, offers Insert a variable… and Try a tag…, and warns about a topic Kafka would refuse before you apply. Topics are worked out again once a minute.

With the Schema Registry on, a topic pattern needs a Subject Name.

Payload field names

These name the fields inside a Default format message. They do not apply to ISA-95 Lite or Custom AVRO Schema, where the schema's field mapping sets the names, and they do not apply when the Schema Registry is enabled. New They are shown only where they apply: with the Default format and the Schema Registry off. Timestamp Type and its Format are also shown with the Schema Registry on, whose message always carries the timestamp. Values saved before are kept.

Property	Default	Notes
Values Id	`values`	The name of the array that holds the values.
Publish Tag Id	`id`	The field carrying each tag's identifier.
Publish Value Id	`value`	The field carrying the value.
Include Quality	on	Include the quality with each value.
Id (quality)	`quality`	The field carrying the quality.
Include Timestamp	on	Include the timestamp with each value.
Id (timestamp)	`timestamp`	The field carrying the timestamp.
Timestamp Type	ISO 8601	`ISO 8601`, `Unix`, `Unix ms`, `Ticks` or `Custom`.
Format	`yyyy-MM-dd HH:mm:ss.ffffff`	Shown when the Timestamp Type is `Custom`.

Store and Forward

Property	Default	Notes
Enable Store and Forward	off	Buffer values when the cluster cannot be reached, rather than discarding them, and forward them once communication is restored.
Last In First Out	off	Process buffered data newest first. Shown when Store and Forward is on, and requires Store Buffer to Disk.

Info

Where the buffer is kept is not a driver property. It is set once for the whole engine under Configure → Options → Store and Forward: Store Buffer to Disk, Directory for Buffer, Limit Disk Buffering and, for buffering data published from a remote engine, Buffer Data for Remote IoT Publish.

Failover

Shown when Enable Failover is on. These describe a second, independent cluster - not more brokers of the primary cluster, which belong in Bootstrap Servers.

Property	Default	Notes
Enable Failover	off	Enable fail over to the backup connection when the primary interface fails.
Failover Bootstrap Servers	`localhost:9092`	Comma-delimited broker addresses of the failover cluster.
Failover Security Protocol	Plaintext	As the primary, for the failover connection.
Failover SASL Mechanism	Plain	Shown for the SASL protocols.
Failover SASL Username	blank	Shown for `Plain`, `ScramSha256` and `ScramSha512`.
Failover SASL Password	blank	As above.
Failover SSL CA Location	blank	Shown for `Ssl` and `SaslSsl`.
Failover SSL CA Pem	blank	As above.
Failover SSL Certificate Location	blank	As above.
Failover SSL Certificate Pem	blank	As above.
Failover SSL Key Location	blank	As above.
Failover SSL Key Pem	blank	As above.
Failover SSL Key Password	blank	As above.
Failover SSL Keystore Location	blank	As above.
Failover SSL Keystore Password	blank	As above.
Failover Schema Registry URL	blank	Shown when the Schema Registry is enabled.
Username	blank	The failover registry's own, beneath Failover Schema Registry URL.
API Key	blank	As above.

Warning

The failover cluster needs its own Schema Registry. Schema IDs are issued by, and only mean anything within, the cluster that issued them, so the failover brokers cannot borrow the primary cluster's registry.

Consumer tag properties

A tag consumes from Kafka through its own properties on the Tags screen, not through the driver.

Property	Default	Notes
Data Type	-	String or JSON. A JSON tag parses the payload into its members; a String tag holds it as text.
Data Source	-	Set to Kafka.
Select Driver Interface	-	The Kafka driver interface the tag subscribes through.
Topic	blank	The Kafka topic that supplies this parameter's value.
Read Only	off	The value cannot be written to.
Write Only	off	The value is not read from the topic, but writes are allowed.

Where next

Getting Started with Kafka - these options in a working configuration.
Kafka How-To guides - logging, transferring and routing Kafka data.
Message Schemas - the schema library and the Edit AVRO schema dialog.
Message Variables - every variable a topic pattern or a schema can use, and the Kafka rules.