Nagarjunanm opened a new pull request, #20559:
URL: https://github.com/apache/druid/pull/20559
Adds optional sampling of ServiceMetricEvents to the kafka-emitter extension,
so operators can drop a percentage of high-volume metrics before they reach
the
Kafka producer.
New properties under druid.emitter.kafka:
- samplingRate (0-100, default 100 = no sampling); values outside 0-100 are
rejected
- sampledMetrics: exact metric names eligible for sampling (empty = all)
- sampledNodeTypes: services eligible for sampling (empty = all)
Only metric events are sampled; alerts, request logs and segment metadata are
always emitted. When an event has an "id" dimension, the keep/drop decision
is
based on a hash of that id, so all metrics for one query are kept or dropped
together; otherwise events are kept at random. The emitter counts sampled and
emitted metric events and logs a summary every five minutes.
Fixes #20246
---
# PR title
feat: add configurable metric sampling to the Kafka emitter
# PR description
Fixes #20246.
### Description
High-traffic clusters emit a very large number of metric events, especially
per-query metrics such as
`query/time`. When those metrics are used for trends and percentiles,
keeping all of them costs Kafka
throughput, network and storage without adding much value. Filtering on the
consumer side doesn't help,
because the events have already been serialized, produced and stored by then.
This PR adds opt-in sampling to the Kafka emitter. Events are dropped in
`KafkaEmitter.emit()`, before
they're queued for the producer.
#### New configuration
| Property | Default | Description |
|---|---|---|
| `druid.emitter.kafka.samplingRate` | `100` | Percentage (0-100) of
matching metric events to keep. Values outside 0-100 fail at startup with an
`INVALID_INPUT` error. |
| `druid.emitter.kafka.sampledMetrics` | `[]` | Metric names to sample
(exact match). Empty means all metrics. |
| `druid.emitter.kafka.sampledNodeTypes` | `[]` | Services to sample, such
as `druid/broker`. Empty means all services. |
The default `samplingRate` of `100` keeps today's behavior.
#### Behavior
- Only `ServiceMetricEvent`s are sampled. Alerts, request logs and segment
metadata are always emitted.
- An event is sampled only if it matches both `sampledMetrics` and
`sampledNodeTypes`. An empty list matches everything.
- If a matching event has an `id` dimension (for example the query ID), the
decision is based on a hash of
that ID, so every metric for the same query is kept or dropped together.
This keeps per-query metrics
consistent with each other. Events without an `id` are kept at random
using `ThreadLocalRandom`.
- Sampled-out events are counted separately from lost events. The emitter
logs a sampling summary
(total, emitted, sampled out) every five minutes, only when something was
sampled.
#### Example
```properties
druid.emitter.kafka.samplingRate=10
druid.emitter.kafka.sampledMetrics=["query/time", "query/bytes",
"sqlQuery/time"]
druid.emitter.kafka.sampledNodeTypes=["druid/broker"]
```
#### Release note
The Kafka emitter can now sample metric events with the new
`druid.emitter.kafka.samplingRate`,
`druid.emitter.kafka.sampledMetrics` and
`druid.emitter.kafka.sampledNodeTypes` properties. This reduces
Kafka produce volume for high-traffic metrics such as `query/time`. Sampling
is off by default.
<hr>
##### Key changed/added classes in this PR
* `KafkaEmitterConfig`
* `KafkaEmitter`
* `KafkaEmitterConfigTest`
* `KafkaEmitterTest`
* `docs/development/extensions-contrib/kafka-emitter.md`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
- [x] added Javadocs for most classes and all non-trivial methods.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for code coverage is met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
