Overview – Kafka

The OAS Kafka communication interface is both a producer and consumer, to both local and cloud clusters.

Kafka Consumer

A Kafka Consumer configuration is one where you configure the parameters of a Kafka driver to connect to a Kafka cluster and you create Tags with the Data Source set to Kafka. You then specify the topic that the Kafka consumer should subscribe to in order to receive the data.

👉 For more details see Kafka Consumer

Kafka Producer

A Kafka Producer configuration is one where you configure a Kafka driver and you enable the Publish Selected Tags property to specify the list of Tags that should be published to a given topic. The chosen Tags will be published based on a configurable interval, a boolean Tag event trigger or a time based schedule. You can control a number of different parameters to customize the JSON payload structure. See IoT Publish for details.

👉 For more details see Kafka Producer

Connection Parameters

The Kafka driver supports the following connection parameters:

  • Bootstrap Servers: The Kafka cluster bootstrap server IP or hostname and port.
  • Security Protocol: The security type to use for the connection. [Plaintext | Ssl | SaslPlaintext | SaslSsl]
  • Acks: The acknowledgement strategy to use in order to control durability. [None | Leader | All]
  • Client Id: An id string to pass to the server when making requests.
  • Compression Type: The compression level to use. [None | Gzip | Snappy | Lz4 | Zstd]

Networking Parameters

The Kafka driver supports the following networking and message parameters for tweaking the performance of the connection:

  • Batch Num Messages: The number of messages to send in one batch when using async mode. [10,000]
  • Batch Size: The batch size in bytes. [1,000,000]
  • Linger: The number of milliseconds a producer is willing to wait before sending a batch out. [5]
  • Max In Flight: Controls the maximum number of outstanding requests a producer can send to a Kafka broker before waiting for a response. [1,000,000]
  • Message Max Bytes: Maximum Kafka protocol request message size. [1,000,000]
  • Message Timeout: Local message timeout in milliseconds. This value is only enforced locally and limits the time a produced message waits for successful delivery. A time of 0 is infinite. [300,000]
  • Queue Buffering Max Kbytes: Maximum total message size sum allowed on the producer queue. This queue is shared by all topics and partitions. This property has higher priority than Queue Buffering Max Messages. [1048576]
  • Queue Buffering Max Messages: Maximum number of messages allowed on the producer queue. This queue is shared by all topics and partitions. [100,000]
  • Request Timeout: Timeout for sending messages in milliseconds [30,000]
  • Socket Connection Timeout: Maximum time allowed for broker connection setup (TCP connection setup as well SSL and SASL handshake). If the connection to the broker is not fully functional after this the connection will be closed and retried. [30,000]
  • Socket Keepalive Enable: Enable TCP keep-alives on broker sockets. [False]
  • Socket Timeout: Default timeout for network requests in milliseconds. [60,000]
  • Transaction Timeout: The maximum amount of time in milliseconds that the transaction coordinator will wait for a transaction status update from the producer before proactively aborting the ongoing transaction. [60,000]

Store and Forward

Enable data buffering when communication failure occurs. Values will be stored to directory specified under Configure > Options > Store and Forward.

To buffer data from remote OAS Engine enable Buffer for Remote IoT Publish under Configure > Options > Store and Forward on each remote data source system.

Failover

You can enable the Enable Failover property and configure a secondary bootstrap server and security protocol. When a secondary interface is configured, OAS will attempt to use it if the primary interface fails to communicate.