Zeek

The Zeek network monitor translates raw packets into structured logs. Tenzir supports various Zeek use cases, such as continuous ingestion, ad-hoc log file processing, and even generating Zeek logs.

Zeek logs come in three forms in practice, all of which Tenzir can parse natively:

1. Tab-Separated Values (TSV) with a custom header.
2. One NDJSON file for all log types combined (aka. JSON Streaming)
3. One NDJSON file per log type.

Examples

Ingest logs into a node

To ingest Zeek logs into a Tenzir node, you have multiple options.

Easy-button import with the official Zeek package

Our official Zeek package makes it easy to ship your Zeek logs to a Tenzir node. Install the package first:

zkg install zeek-tenzir

Then add this to your $PREFIX/share/zeek/site/local.zeek to send all logs that Zeek produces to a Tenzir node:

@load tenzir/import

# Uncomment to keep the original Zeek logs.
# redef Tenzir::delete_after_postprocesing=F;

For ad-hoc command line processing you can also pass tenzir/import to a Zeek invocation:

# Ship logs to it and delete the original files.
zeek -r trace.pcap tenzir/import

# Ship logs to it and keep the original files.
zeek -r trace.pcap tenzir/import Tenzir::delete_after_postprocesing=F

Run an import pipeline when rotating logs

If you cannot use our Zeek package, it is still possible to let Zeek trigger an import pipeline upon rotation. Zeek’s logging framework can execute a shell script whenever it rotates a log file.

This requires setting Log::default_rotation_interval to a non-zero value. The default of 0 secs means that log rotation is disabled. Add this to $PREFIX/share/zeek/site/local.zeek, which is the place for local configuration changes:

redef Log::default_rotation_interval = 1 day;

Then redefine Log::default_rotation_postprocessor_cmd to point to your shell script, e.g., /usr/local/bin/ingest:

redef Log::default_rotation_postprocessor_cmd=/usr/local/bin/ingest;

Here is an example ingest script that imports all files into a Tenzir node:

ingest

#!/bin/sh

file_name="$1"
base_name="$2"
from="$3"
to="$4"
terminating="$5"
writer="$6"

if [ "$writer" = "ascii" ]; then
  read="read_zeek_tsv"
elif [ "$writer" = "json" ]; then
  read="read_zeek_json"
else
  echo "unsupported Zeek writer: $writer"
  exit 1
fi

pipeline="load_file \"$file_name\" | $read | import"

tenzir "$pipeline"

Our blog post Native Zeek Log Rotation & Shipping provides further details on this method.

Run Zeek on a packet pipeline

You can run Zeek on a pipeline of PCAP packets and continue processing the logs in the same pipeline. A stock Tenzir installation comes with a user-defined zeek operator that looks as follows:

tenzir.yaml

tenzir:
  operators:
    zeek: |
      shell "eval \"$(zkg env)\" &&
             zeek -r - LogAscii::output_to_stdout=T
             JSONStreaming::disable_default_logs=T
             JSONStreaming::enable_log_rotation=F
             json-streaming-logs"
      read_zeek_json

This allows you run Zeek on a packet trace as follows:

tenzir 'load_file "/path/to/trace.pcap" | zeek'

You can also perform more elaborate packet filtering after light-weight decapsulation:

tenzir 'load_file "/path/to/trace.pcap"
        read_pcap
        this = decapsulate(this)
        where ip.src in 10.0.0.0/8 || community == "1:YXWfTYEyYLKVv5Ge4WqijUnKTrM="
        write_pcap
        zeek'

Process logs with a pipeline on the command line

Zeek ships with a helper utility zeek-cut that operators on Zeek’s tab-separated logs. For example, to extract the host pairs from a conn log:

zeek-cut id.orig_h id.resp_h < conn.log

The list of arguments to zeek-cut are the column names of the log. The select operator performs the equivalent in Tenzir after we parse the logs as Zeek TSV:

tenzir 'read_zeek_tsv | select id.orig_h id.resp_h' < conn.log

Since pipelines are multi-schema and the Zeek TSV parser is aware of log boundaries, you can also concatenate logs of various types:

cat *.log | tenzir 'read_zeek_tsv | select id.orig_h id.resp_h'

Generate Zeek TSV from arbitrary data

You can render any data as Zeek TSV log using write_zeek_tsv:

For example, this is how you create a filtered version of a Zeek conn.log:

subscribe "zeek"
where @name == "zeek.conn"
where duration > 2s and id.orig_p != 80
write_zeek_tsv
save_file "filtered_conn.log"