Kafka

• Supported Kafka versions: 3.x

Kafka can be used as an intermediary buffer between collector and an actual storage. Jaeger can be configured to act both as the collector that exports trace data into a Kafka topic as well as the ingester to read data from Kafka and write it to a storage backend.

Writing to Kafka is particularly useful for building post-processing data pipelines.

Kafka also has the following officially supported resources available from the community:

• Docker container for getting a single node up quickly
• Helm chart by Bitnami
• Strimzi Kubernetes Operator

Configuration

Please refer to these sample configuration files:

• collector: config-kafka-collector.yaml
• ingester: config-kafka-ingester.yaml

Jaeger uses Kafka exporter and receiver from opentelemetry-collector-contrib repository. Please refer to their respective README’s for configuration details.

• Kafka exporter
• Kafka receiver

Topic & partitions

Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to the Kafka quickstart documentation to learn how.

You can find more information about topics and partitions in general in the official documentation. This article provide more details about how to choose the number of partitions.

At-least-once delivery

With the default ingester configuration the Kafka receiver commits an offset as soon as the pipeline accepts the record, and the pipeline accepts it before the storage has written it, so a backend outage loses spans that Kafka considers delivered. The ingester can instead be configured so that an offset is committed only after the spans it covers are durable. The pipeline half of that configuration, no batch processor and an exporter queue with wait_for_result: true, is the same for every receiver and is described on the Delivery Guarantees page. The Kafka-specific half is:

receivers:
  kafka:
    message_marking:
      after: true     # commit the offset only after the pipeline accepted the record
      on_error: false # a failed record is not skipped

exporters:
  jaeger_storage_exporter:
    trace_storage: some_storage
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0 # never give up on a batch; giving up pauses the partition
    queue:
      wait_for_result: true
      block_on_overflow: true # a full queue waits instead of failing the record
      sizer: bytes
      num_consumers: 1
      queue_size: 104857600
      batch:
        sizer: bytes
        flush_timeout: 200ms
        min_size: 1048576
        max_size: 4194304

The storage must return an error when a write fails. Cassandra and ClickHouse always do; Elasticsearch and OpenSearch need write_mode: sync, and should run with poison_pill_handling: drop or the dead-letter pipeline so that a document the backend rejects on every attempt cannot hold the offset forever.

The configurations the Kafka end-to-end tests run against are config-kafka-ingester-sync.yaml and, with the dead-letter pipeline, config-kafka-ingester-dead-letter.yaml. Both follow the recommended shape.

Sizing

Batch size is bounded by the number of partitions the ingester consumes. The receiver processes each partition serially and partitions concurrently, so at most one record per partition is waiting in the exporter’s batcher at a time, and a topic with few partitions produces small storage writes. Add partitions or ingester replicas to increase write throughput; raising queue.batch.max_size alone does not help.

For Elasticsearch and OpenSearch, keep queue.batch.max_size well below the storage’s bulk_processing.max_bytes. The collector measures a batch in OTLP protobuf bytes while the storage measures the encoded _bulk body, which is larger, so a batch at the limit is otherwise split across several _bulk requests. Both values must stay below the http.max_content_length limit of Elasticsearch, 100 MB by default.