Kafka

Apache Kafka is a distributed open-source message broker. The Tenzir integration can publish (send messages to a topic) or subscribe (receive) messages from a topic.

Internally, we use Confluent’s official librdkafka library, which gives us full control in passing options.

URL Support

The URL scheme kafka:// dispatches to load_kafka and save_kafka for seamless URL-style use via from and to.

Examples

Send events to a Kafka topic

from {
  x: 42,
  y: "foo",
}
to "kafka://topic" {
  write_ndjson
}

Subscribe to a topic

The offset option controls where to start reading:

from "kafka://topic", offset="beginning" {
  read_ndjson
}

Other values are "end" to read at the last offset, "stored" to read at the stored offset, a positive integer representing an absolute offset, or a negative integer representing a relative offset counting from the end.