amaechler opened a new pull request, #20648:
URL: https://github.com/apache/druid/pull/20648
Fixes #13924.
### Description
With the configuration the README describes, every Druid service fails to
start once the `opentelemetry-emitter` extension is loaded.
`OpenTelemetryEmitterModule.getEmitter` builds the SDK with
`AutoConfiguredOpenTelemetrySdk`, which defaults every signal to the OTLP
exporter, but the extension does not ship one, so the provider throws during
injector creation.
This PR:
- Bundles the OTLP exported `opentelemetry-exporter-otlp` with `runtime`
scope
- Exports only spans by default
- Fixed the README test setup
<details>
### Additional details from Claude analysis
The exporter was declared in the first commits of #12015, then replaced by
the API and SDK artifacts in the "Add used undeclared dependencies" commit,
presumably because `dependency:analyze` flagged it as unused (it is only loaded
through SPI). Since then the extension has shipped without any span exporter.
Adding the exporter jar to the extension directory does not help either,
because the extension relocates `io.opentelemetry` to
`org.apache.druid.opentelemetry.shaded.io.opentelemetry`, and an unshaded
exporter does not implement the relocated SPI.
#### Bundled the OTLP exporter
Added `opentelemetry-exporter-otlp` with `runtime` scope, so it is shaded
and relocated with the rest of the SDK. Instead of its default OkHttp sender,
which would add unshaded `okhttp`, `okio` and `kotlin-stdlib` jars to the
extension directory, the module uses:
- `opentelemetry-exporter-sender-grpc-managed-channel` for the `grpc`
protocol, which is the SDK's default. It runs on the `grpc-netty-shaded` the
extension already bundles. The only new third-party artifact is `grpc-stub`,
which is pinned to `shade.grpc.version` so it moves with the bundled gRPC.
- `opentelemetry-exporter-sender-jdk` for `http/protobuf`, built on
`java.net.http` with no further dependencies.
The dependency-reduced POM, which `pull-deps` resolves, lists the same
unshaded dependencies as before. The shaded jar grows from about 16.5 MB to
17.0 MB.
#### Exported only spans by default
The SDK also defaults `otel.metrics.exporter` and `otel.logs.exporter` to
`otlp`, and it exports its own self-metrics (`queueSize`, `processedSpans`,
`otlp.exporter.seen`, `otlp.exporter.exported`) every metric interval. Against
a collector with only a traces pipeline, such as the one in the README, every
Druid process would log `Failed to export metrics. Server responded with
UNIMPLEMENTED` once and `Exporter failed` every minute.
`OpenTelemetryEmitterModule` now passes `otel.metrics.exporter=none` and
`otel.logs.exporter=none` to `addPropertiesSupplier`. Supplier values have the
lowest precedence, so users can still enable either with an environment
variable (`OTEL_METRICS_EXPORTER`) or a system property
(`-Dotel.metrics.exporter`), and system properties win over environment
variables.
#### Fixed the README test setup
The collector config in the README did not parse (a duplicated `receivers:`
key and a top-level `version:`), used the removed `logging` exporter, and
relied on the Zipkin exporter, which is only in the contrib collector
distribution. The README now uses `otel/opentelemetry-collector-contrib`, the
`debug` exporter, explicit `0.0.0.0` receiver endpoints, and documents the HTTP
protocol option.
#### Testing
- Added `testDefaultExportersOnlyExportSpans`, which builds the SDK with the
module's defaults. Without the exporter on the classpath it fails with the same
missing-exporter error (reported for `otel.traces.exporter`), and removing
either `none` default from the module also makes it fail.
- Added `testSystemPropertyOverridesDefaultMetricsExporter` for the override
path.
- `testModuleCreatesEmitter` now only sets `otel.traces.exporter=none`,
since the module defaults the other two.
- Built the shaded jar and ran a small program through
`OpenTelemetryEmitterModule.getEmitter`, loading the extension jar and
`perfmark-api` in a child `URLClassLoader` over the Druid core classpath,
matching the dependency-reduced POM and Druid's parent-first extension loading.
With the README's environment variables and the README's docker-compose stack
(current `latest` images), spans arrived in Zipkin over both `grpc` (port 4317)
and `http/protobuf` (port 4318), with the trace ID and parent span ID from the
`traceparent` query context. Against a collector with traces, metrics and logs
pipelines, the default configuration sent only spans,
`OTEL_METRICS_EXPORTER=otlp` added metrics, and `-Dotel.metrics.exporter=otlp`
added metrics even with `OTEL_METRICS_EXPORTER=none`.
- Built the distribution with `-Pdist`, installed the extension with
`pull-deps`, applied the README configuration (`loadList` and
`druid.emitter=opentelemetry`) and ran `bin/start-micro-quickstart` with the
README's environment variables:
- With the extension from master, all five services exited with `Unable to
create injector` and the error above.
- With this change, all services started. A SQL query with a `traceparent`
in its context produced a `druid/broker` span in Zipkin with the expected trace
ID and parent span ID, and no exporter warnings were logged after the first
metric interval.
#### Not changed
- `licenses.yaml`: the CI license check only scans core and
`extensions-core` modules, and this extension has no entries today. The new
artifacts are Apache 2.0.
- `OpenTelemetryEmitter.flush()` and `close()` are no-ops and do not shut
down the SDK, so batched spans are only flushed by the SDK's JVM shutdown hook.
This predates the change; wiring `close()` to `OpenTelemetrySdk.close()` could
be a follow-up.
</details>
#### Release note
The `opentelemetry-emitter` extension now bundles the OTLP trace exporter,
so it starts with the default OpenTelemetry configuration and exports query
spans over OTLP gRPC (default) or HTTP. Previously the extension failed at
startup unless every exporter was set to `none`. The extension no longer
exports the OpenTelemetry SDK's own metrics and logs by default; set
`OTEL_METRICS_EXPORTER` or `OTEL_LOGS_EXPORTER` to enable them.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
