aglinxinyuan commented on issue #5913:
URL: https://github.com/apache/texera/issues/5913#issuecomment-5276349925
The cause described here is fixed, but a second race survives it — and it is
the one still failing CI, so this issue's remedy no longer matches the symptom.
**What is already fixed.** #5915 replaced the 100-row `smallCsvScanOpDesc`;
the spec has used `slowRegionSourceOpDesc(numTuple = 30, delaySeconds = 0.25)`
ever since, so the run stays in flight for ~7.5s and the pause lands mid-run as
intended. Making the source bigger again would change nothing.
**The surviving race is in pyamber, and is independent of how long the
source runs.** `MainLoop._process_end_channel` ends with `complete()`, which
transits the worker to `COMPLETED`. Nothing between the last control check and
that call drains control, but the coordinator pauses on its own schedule — so a
`PauseWorker` landing in that window leaves the worker `PAUSED` when the
transition runs. `PAUSED -> COMPLETED` is forbidden by
`WORKER_STATE_TRANSITIONS`, so `transit_to` raises and kills the main loop
thread:
```
main_loop.py:674 _process_end_channel -> self.complete()
main_loop.py:305 complete() -> state_manager.transit_to(COMPLETED)
state_manager.py:80 -> InvalidTransitionException:
Cannot transit from PAUSED to
COMPLETED
```
The Scala worker cannot hit this: `DPThread`'s input selection only picks
*control* channels while `pauseManager.isPaused`, so a paused Scala worker
never advances to completion. Only the Python runtime reaches `complete()`
while paused.
**It is repo-wide, not specific to any one branch.** Two unrelated branches
produced the identical traceback within hours on 2026-08-13 —
`chore/remove-progressive-utils` (run 31664512736) and
`test/worker-state-updated-handler-spec` (run 31653669034), both on `build /
amber-integration (macos-latest, 17)`.
**Fix in #7624**: wait the pause out before completing, mirroring the Scala
behaviour. Two notes for whoever reviews it — adding `PAUSED -> COMPLETED` to
the graph would be the smaller diff, but `WORKER_STATE_TRANSITIONS` mirrors
Scala's `WorkerStateManager` by contract and the two graphs are currently
identical; and the wait is guarded on `is_paused()` rather than draining
control unconditionally, because `_check_and_process_control` blocks while the
data lane is disabled and backpressure disables that lane too, so an unguarded
drain would park a merely-backpressured worker.
The regression test pauses inside `all_ports_completed()` — the last call
before `complete()`, with no control check in between — which reproduces the
production traceback deterministically. `amber-integration` on macOS passes on
that PR.
Happy to retitle this issue to match the surviving cause if that is
preferred over closing it via #7624.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
