Fluent Bit

Fluent Bit is a an open source observability pipeline. Tenzir embeds Fluent Bit, exposing all its inputs via from_fluent_bit and outputs via to_fluent_bit

This makes Tenzir effectively a superset of Fluent Bit.

Fluent Bit parsers map to Tenzir operators that accept bytes as input and produce events as output. Fluent Bit filters correspond to Tenzir operators that perform event-to-event transformations. Tenzir does not expose Fluent Bit parsers and filters, only inputs and output.

Internally, Fluent Bit uses MsgPack to encode events whereas Tenzir uses Arrow record batches. The fluentbit source operator transposes MsgPack to Arrow, and the fluentbit sink performs the reverse operation.

Usage

An invocation of the fluent-bit commandline utility

fluent-bit -o input_plugin -p key1=value1 -p key2=value2 -p…

translates to Tenzir’s from_fluent_bit operator as follows:

from_fluent_bit "input_plugin", options={key1: value1, key2: value2, …}

with the to_fluent_bit operator working exactly analogous.

Examples

Ingest OpenTelemetry logs, metrics, and traces

from_fluent_bit "opentelemetry"

You can then send JSON-encoded log data to a freshly created API endpoint:

curl \
  --header "Content-Type: application/json" \
  --request POST \
  --data '{"resourceLogs":[{"resource":{},"scopeLogs":[{"scope":{},"logRecords":[{"timeUnixNano":"1660296023390371588","body":{"stringValue":"{\"message\":\"dummy\"}"},"traceId":"","spanId":""}]}]}]}' \
  http://0.0.0.0:4318/v1/logs

Imitate a Splunk HEC endpoint

from_fluent_bit "splunk", options = {port: 8088}

Tip

Use the dedicated to_splunk operator to send events to a Splunk HEC.

Imitate an ElasticSearch & OpenSearch Bulk API endpoint

This allows you to ingest from beats (e.g., Filebeat, Metricbeat, Winlogbeat).

from_fluent_bit "elasticsearch", options = {port: 9200}

Send to Datadog

to_fluent_bit "datadog", options = {apikey: "XXX"}

Send to ElasticSearch

to_fluent_bit "es", options = {host: 192.168.2.3, port: 9200, index: "my_index", type: "my_type"}