Sampling
Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: A -> B -> C
. When service A receives a request that contains no tracing information, Jaeger tracer will start a new trace, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its spans will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.
Client Sampling Configuration
When using configuration object to instantiate the tracer, the type of sampling can be selected via sampler.type
and sampler.param
properties. Jaeger libraries support the following samplers:
- Constant (
sampler.type=const
) sampler always makes the same decision for all traces. It either samples all traces (sampler.param=1
) or none of them (sampler.param=0
). - Probabilistic (
sampler.type=probabilistic
) sampler makes a random sampling decision with the probability of sampling equal to the value ofsampler.param
property. For example, withsampler.param=0.1
approximately 1 in 10 traces will be sampled. - Rate Limiting (
sampler.type=ratelimiting
) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, whensampler.param=2.0
it will sample requests with the rate of 2 traces per second. - Remote (
sampler.type=remote
, which is also the default) sampler consults Jaeger agent for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a central configuration in Jaeger backend, or even dynamically (see Adaptive Sampling ).
Adaptive Sampler
Adaptive sampler is a composite sampler that combines two functions:
- It makes sampling decisions on a per-operation basis, i.e. based on span operation name. This is especially useful in the API services whose endpoints may have very different traffic volumes and using a single probabilistic sampler for the whole service might starve (never sample) some of the low QPS endpoints.
- It supports a minimum guaranteed rate of sampling, such as always allowing up to N traces per seconds and then sampling anything above that with a certain probability (everything is per-operation, not per-service).
Per-operation parameters can be configured statically or pulled periodically from Jaeger backend with the help of Remote sampler. Adaptive sampler is designed to work with the upcoming Adaptive Sampling feature of the Jaeger backend.
Collector Sampling Configuration
Collectors can be instantiated with static sampling strategies (which are propagated to the respective service if configured with Remote sampler) via the --sampling.strategies-file
option. This option requires a path to a json file which defines the sampling strategies.
If no configuration is provided, the collectors will return the default probabilistic sampling policy with probability 0.001 (0.1%) for all services.
Example strategies.json
:
{
"service_strategies": [
{
"service": "foo",
"type": "probabilistic",
"param": 0.8,
"operation_strategies": [
{
"operation": "op1",
"type": "probabilistic",
"param": 0.2
},
{
"operation": "op2",
"type": "probabilistic",
"param": 0.4
}
]
},
{
"service": "bar",
"type": "ratelimiting",
"param": 5
}
],
"default_strategy": {
"type": "probabilistic",
"param": 0.5,
"operation_strategies": [
{
"operation": "/health",
"type": "probabilistic",
"param": 0.0
},
{
"operation": "/metrics",
"type": "probabilistic",
"param": 0.0
}
]
}
}
service_strategies
element defines service specific sampling strategies and operation_strategies
defines operation specific sampling strategies. There are 2 types of strategies possible: probabilistic
and ratelimiting
which are described
above
(NOTE: ratelimiting
is not supported for operation_strategies
). default_strategy
defines the catch-all sampling strategy that is propagated if the service is not included as part of service_strategies
.
In the above example:
- All operations of service
foo
are sampled with probability 0.8 except for operationsop1
andop2
which are probabilistically sampled with probabilities 0.2 and 0.4 respectively. - All operations for service
bar
are rate-limited at 5 traces per second. - Any other service will be sampled with probability 0.5 defined by the
default_strategy
. - The
default_strategy
also includes shared per-operation strategies. In this example we disable tracing on/health
and/metrics
endpoints for all services by using probability 0. These per-operation strategies will apply to any new service not listed in the config, as well as to thefoo
andbar
services unless they define their own strategies for these two operations.