All Jaeger client libraries support the OpenTracing APIs. The following resources provide more information about instrumenting your application with OpenTracing:
- OpenTracing tutorials for Java, Go, Python, Node.js and C#
- A deep dive blog post Tracing HTTP request latency in Go
- The official OpenTracing documentation and other materials at opentracing.io
opentracing-contriborg on GitHub contains many repositories with off-the-shelf instrumentation for many popular frameworks, including JAXRS & Dropwizard (Java), Flask & Django (Python), Go std library, etc.
The rest of this page contains information about configuring and instantiating a Jaeger tracer in an application that is already instrumented with OpenTracing API.
We use the terms client library, instrumentation library, and tracer interchangeably in this document.
The following client libraries are officially supported:
Libraries in other languages are currently under development, please see issue #366.
Initializing Jaeger Tracer
The initialization syntax is slightly different in each languages, please refer to the README’s in the respective repositories. The general pattern is to not create the Tracer explicitly, but use a Configuration class to do that. Configuration allows simpler parameterization of the Tracer, such as changing the default sampler or the location of Jaeger agent.
Jaeger tracers use reporters to process finished spans. Typically Jaeger libraries ship with the following reporters:
- NullReporter does nothing with the span. It can be useful in unit tests.
- LoggingReporter simply logs the fact that a span was finished, usually by printing the trace and span ID and the operation name.
- CompositeReporter takes a list of other reporters and invokes them one by one.
- RemoteReporter (default) buffers a certain number of finished spans in memory and uses a sender to submit a batch of spans out of process to Jaeger backend. The sender is responsible for serializing the span to the wire format (e.g. Thrift or JSON) and communicating with the backend components (e.g. over UDP or HTTP).
EMSGSIZE and UDP buffer limits
By default Jaeger libraries use a UDP sender to report finished spans to the
The default max packet size is 65,000 bytes, which can be transmitted without segmentation when
connecting to the agent via loopback interface. However, some OSs (in particular, MacOS), limit
the max buffer size for UDP packets, as raised in this GitHub issue.
If you run into issue with
EMSGSIZE errors, consider raising the limits in your kernel (see the issue for examples).
You can also configure the client libraries to use a smaller max packet size, but that may cause
issues if you have large spans, e.g. if you log big chunks of data. Spans that exceed max packet size
are dropped by the clients (with metrics emitted to indicate that). Another alternative is
to use non-UDP transports, such as HttpSender in Java (not currently available for all languages).
Jaeger tracers emit various metrics about how many spans or traces they have started and finished, how many of them were sampled or not sampled, if there were any errors in decoding trace context from inbound requests or reporting spans to the backend.
SpanContext is encoded on the wire as part of the request to another service, Jaeger client libraries default to the Jaeger native propagation format specified below. In addition, Jaeger clients support Zipkin B3 format and W3C Trace-Context.
- Case-insensitive in HTTP
- Lower-case in protocols that preserve header case
- 64-bit or 128-bit random number in base16 format
- Can be variable length, shorter values are 0-padded on the left
- Clients in some languages support 128-bit, migration pending
- Value of 0 is not valid
- 64-bit random number in base16 format
- Value of 0 is not valid
- 64-bit value in base16 format representing parent span id
- Deprecated, most Jaeger clients ignore on the receiving side, but still include it on the sending side
- 0 value is valid and means “root span” (when not ignored)
- One byte bitmap, as one or two hex digits (leading zero may be omitted)
- Bit 1 (right-most, least significant, bit mask
0x01) is “sampled” flag
- 1 means the trace is sampled and all downstream services are advised to respect that
- 0 means the trace is not sampled and all downstream services are advised to respect that
- We’re considering a new feature that allows downstream services to upsample if they find their tracing level is too low
- Bit 2 (bit mask
0x02) is “debug” flag
- Debug flag should only be set when the sampled flag is set
- Instructs the backend to try really hard not to drop this trace
- Bit 3 (bit mask
0x04) is not used
- Bit 4 (bit mask
0x08) is “firehose” flag
- Spans tagged as “firehose” are excluded from being indexed in the storage
- The traces can only be retrieved by trace ID (usually available from other sources, like logs)
- Other bits are unused
- Value: url-encoded string
- Limitation: since HTTP headers don’t preserve the case, Jaeger recommends baggage keys to be lowercase-kebab-case,
Example: the following code sequence:
span.SetBaggageItem("key1", "value1") span.SetBaggageItem("key2", "value2")
will result in the following HTTP headers:
uberctx-key1: value1 uberctx-key2: value2