bhuvan-somisetty opened a new issue, #13945:
URL: https://github.com/apache/apisix/issues/13945
### Description
The `opentelemetry` plugin emits exactly one span per request, with `kind =
span_kind.server`, and never emits a `CLIENT` span for the call APISIX makes to
the upstream. Verified directly against current `master` (2026-09-15):
```
$ grep -n "span_kind\." apisix/plugins/opentelemetry.lua apisix/tracer.lua
apisix/plugins/opentelemetry.lua:429: kind = span_kind.server,
```
`span_kind.client` does not appear anywhere in `opentelemetry.lua`. All
internal phase spans created via `apisix/tracer.lua` (`apisix.phase.access`,
`apisix.phase.header_filter`, `apisix.phase.body_filter`, `http_router_match`,
`resolve_dns`, ...) are `server` or `internal` kind and describe APISIX's own
phases, not the outbound call to the upstream.
Originally raised and investigated in #13828 (credit: @alice101-dev, who
supplied the reproduction below). I independently re-verified the root cause
against current master and traced a concrete fix path through the existing
tracer internals (details below), and I'm filing it as a tracked issue since it
has a clear, actionable solution and real impact but no PR yet.
### Reproduction / evidence
Same application, same OpenTelemetry Collector, same APM backend
(HyperDX/ClickStack) — only the gateway in front differs.
**Fronted by APISIX**: the service map shows two disconnected clusters.
`api-gateway` is joined to the callers in front of it, but not to the services
behind it (e.g. `orders-service`).
**Fronted by Kong** (which emits a `kong.balancer` CLIENT span for the same
hop): one continuous chain from browser through to backends.
Every cross-service parent/child span pair in the trace data, with span
kinds:
| parent | parent kind | child | child kind | service-map edge drawn |
|---|---|---|---|---|
| orders | CLIENT | inventory | SERVER | yes |
| orders | CLIENT | payments | SERVER | yes |
| storefront-web | CLIENT | api-gateway | SERVER | yes |
| **api-gateway** | **SERVER** | orders | SERVER | **no** |
| **api-gateway** | **SERVER** | auth | SERVER | **no** |
Context propagation itself is not broken — child spans exist and are
correctly parented to the gateway's span. There is simply no `CLIENT` span for
the proxy hop for APM tools to anchor a `CLIENT -> SERVER` edge to, which is
the pairing that OpenTelemetry-based service-map generation (Jaeger, Grafana
Tempo, HyperDX/ClickStack, Datadog APM, etc.) relies on by convention.
Environment: APISIX 3.17.0 and master, official `apache/apisix` image,
`apisix-ingress-controller` 2.1.0 with Gateway API, OTel Collector ->
ClickHouse.
### Root cause
`apisix/plugins/opentelemetry.lua:428-431` starts a single request span with
a hardcoded `kind = span_kind.server` and never opens a corresponding `CLIENT`
span around the proxy-to-upstream leg:
```lua
local ctx = tracer:start(upstream_context, span_name, {
kind = span_kind.server,
attributes = attributes,
})
```
This was not addressed by the two most recent tracing PRs on this plugin
(#12686, merged 2026-02-07, added more *phase* spans; #13008, merged
2026-02-13, fixed SSL span nesting) — neither introduces a `CLIENT`-kind span
for the upstream call, which #13828's author confirmed by inspecting master
after both had landed, and which I re-confirmed today with the grep above.
Note this isn't a missing capability in the tracing layer:
`apisix/secret.lua:158` already creates a `tracer.kind.client` span for
outbound secret-manager fetches, so the `client` kind is already wired through
`apisix/tracer.lua` and simply never applied to the (far more common) upstream
proxy hop.
### Impact
Any APISIX deployment using the `opentelemetry` plugin with an APM backend
that builds a service topology from span-kind pairs (which is most of them)
shows APISIX as a disconnected node rather than the gateway hop it actually is.
This is a correctness gap in the plugin's tracing semantics, not a
configuration issue on the user's side — there is no config option in the
plugin's schema (`sampler`, `additional_attributes`,
`additional_header_prefix_attributes`) that can work around it. For anyone
using APISIX as their edge/gateway with standard OTel-based observability
tooling, this silently degrades one of the main reasons to adopt distributed
tracing.
### Affected components
- `apisix/plugins/opentelemetry.lua` (primary; this issue's scope)
- `apisix/plugins/zipkin.lua` has the analogous gap (`["span.kind"] =
"server"` only, confirmed via inspection) but is **not** in scope for this
issue — worth a follow-up once the pattern here is settled, to avoid conflating
two plugins' review cycles.
### Acceptance criteria
- When the `opentelemetry` plugin is enabled and a request is proxied to an
upstream, a `CLIENT`-kind span is emitted as a child of the existing `SERVER`
span, bounding the upstream leg (started before the upstream call, finished
once the upstream response is known).
- The `CLIENT` span carries standard OTel HTTP client attributes at minimum:
`server.address`, `server.port` (available via `ctx.balancer_ip` /
`ctx.balancer_port`, set in `apisix/balancer.lua`), and
`http.response.status_code`.
- Span timing reflects the real upstream latency rather than the full
request latency (`ngx.var.upstream_connect_time` /
`ngx.var.upstream_response_time` are already read elsewhere, e.g.
`apisix/init.lua:1213-1214`, confirming they're available at
`log`/`header_filter` time).
- No regression in existing `t/plugin/opentelemetry*.t` tests; new test
coverage added in the same suite for: span present, correct parent/child
relationship, correct `kind`, and correct attributes.
- Behavior is off by default in no way that requires a new schema field
unless the implementation direction settled on in review needs one (to be
confirmed during design discussion, not assumed here).
### Solution direction (starting point, not a mandate)
`apisix/plugins/opentelemetry.lua` already has the mechanism needed via
`create_child_span()` (line ~454), which accepts an arbitrary `span.kind` plus
explicit `span.start_time` / `span.end_time` to backdate a span rather than
requiring a live start/finish hook at the exact call site — this is the same
technique that would let a `CLIENT` span be constructed after the fact from
`ngx.var.upstream_connect_time` / `upstream_response_time`, without needing to
hook into `balancer_by_lua` directly (nginx's actual `proxy_pass` to the
upstream is not itself Lua-driven, so there's no single call to wrap
synchronously).
The concrete open question, flagged by the original reporter and still open:
whether the `CLIENT` span's create/finish should live in `header_filter`
(attributes and timing are known but the request log's `upstream_response_time`
isn't finalized as a var yet in some codepaths) or `log` (all upstream vars are
final, but this is after the response has already started streaming to the
client, which is fine for tracing since APM ingestion is always async). This is
the main design decision a contributor would need maintainer input on before
implementing.
Given the scope (touches the tracing plugin's span-emission logic, requires
understanding of nginx phase boundaries and upstream timing vars, and needs
solid regression tests against the existing Test::Nginx-based suite) without
being an open-ended research problem (the data needed is already available in
`ctx`, and the span-construction pattern already exists in the same file), this
seems like a good fit for a mentored contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
