yeganeahmadnejad opened a new issue, #13980:
URL: https://github.com/apache/apisix/issues/13980
## Description
The vendored `opentelemetry-lua` dependency (pinned `opentelemetry-lua =
0.2-6` in `apisix-master-0.rockspec`) has a bug in `batch_span_processor.lua`'s
`create_timer()`: when `ngx.timer.at()` fails because the worker is exiting --
expected near the end of every graceful shutdown/deploy -- it falls back to a
*synchronous* span export (`self:flush_all()`, opening a cosocket).
Since spans finish inline inside whatever phase the `opentelemetry` plugin
is running (`body_filter_by_lua*` or `log_by_lua*` for HTTP spans -- see
`apisix/plugins/opentelemetry.lua`'s `phase_func`), that synchronous cosocket
call hits APISIX's own phase guard in `apisix/patch.lua` and crashes:
```
2026/09/20 09:31:05 [error] 70#70: *1277095 failed to run log_by_lua*:
/usr/local/apisix/apisix/patch.lua:406: API disabled in the context of
log_by_lua*
```
Full traceback (captured on 3.18.0):
```
batch_span_processor.lua:97: create_timer(): failed to create timer: process
exiting while logging request
stack traceback:
[C]: in function 'error'
.../resty/core/socket/tcp.lua:298: in function 'original_tcp'
.../apisix/patch.lua:406: in function 'ngx_socket_tcp'
.../resty/http.lua:227: in function 'new'
.../opentelemetry/trace/exporter/http_client.lua:36: in function
'do_request'
.../opentelemetry/trace/exporter/otlp.lua:58: in function 'export_spans'
.../opentelemetry/trace/batch_span_processor.lua:45: in function
'process_batches'
.../opentelemetry/trace/batch_span_processor.lua:215: in function
'flush_all'
.../opentelemetry/trace/batch_span_processor.lua:99: in function
'create_timer'
.../opentelemetry/trace/batch_span_processor.lua:183: in function
'on_end'
.../opentelemetry/trace/recording_span.lua:98: in function 'finish'
/usr/local/apisix/apisix/plugins/opentelemetry.lua:559: in function
'phase_func'
/usr/local/apisix/apisix/plugin.lua:1442: in function 'common_phase'
/usr/local/apisix/apisix/init.lua:1213: in function 'http_log_phase'
```
## Steps to reproduce
1. Enable the `opentelemetry` plugin with `tracing: true` (or the default
per-route tracing) on any route -- including one with no other plugins, e.g.
just a health-check route.
2. Deploy/restart the worker (trigger a graceful shutdown) while traffic is
hitting that route.
3. Near the end of the outgoing worker's shutdown, the last span(s) to
finish will hit the race and crash with the above error.
Because this only requires the worker to be exiting -- not any specific
plugin config -- it fires ~consistently on every rolling deploy of any release
with tracing enabled.
## Root cause & fix
Filed and fixed upstream in opentelemetry-lua:
yangxikun/opentelemetry-lua#107, fix at yangxikun/opentelemetry-lua#106. Once
merged/released there, bumping the pinned `opentelemetry-lua` version here
would pick up the fix. In the meantime, APISIX could consider wrapping
`phase_func`'s span-finishing call in a `pcall` as defense in depth, since this
class of "third-party tracer code doing something disallowed in a restricted
phase" isn't fully preventable from the library side alone.
## Environment
- APISIX 3.18.0 (Ubuntu-based image), `opentelemetry-lua = 0.2-6`
- OpenResty/CentOS or Ubuntu base, not relevant to the phase-restriction
behavior (standard OpenResty `body_filter_by_lua*`/`log_by_lua*` cosocket
restriction)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
