aglinxinyuan opened a new issue, #6891:
URL: https://github.com/apache/texera/issues/6891
### What happened?
`LoopIntegrationSpec` fails intermittently in `amber-integration` (roughly 1
in 10 runs, both OSes), with region terminations logging:
```
java.lang.IllegalStateException: worker still has unprocessed messages
at ...promisehandlers.EndHandler.endWorker(EndHandler.scala:51)
[WARN] [RegionExecutionManager] Failed to terminate region N on attempt 1 of
150. Retrying in 200 ms.
```
This is the same fail-fast race previously seen in `DataProcessingSpec`
(#5614). Loops amplify it: every loop iteration terminates and re-executes
regions, so each iteration is another roll of the dice — and the suite's single
retry doesn't save runs where it strikes twice.
### Root cause
The "unprocessed" message is always a **`ReturnInvocation`** — the
coordinator's ack for an RPC the *worker itself* sent fire-and-forget (e.g.
`workerExecutionCompleted`, `portCompleted`). The race is inherent: that very
RPC is what makes the coordinator decide to end the worker, so the ack and
`EndWorker` are concurrently in flight. The worker never awaits these acks; an
ack-only backlog carries no work.
| Side | Behavior at `EndWorker` with a queued ack |
|---|---|
| Scala `EndHandler` | throws `IllegalStateException` → termination retry
loop → flaky/slow CI |
| Python `EndWorkerHandler` | **consumes one message as a side effect of
logging it**, then `assert empty` — accidentally tolerates exactly one ack;
with two queued it crashes, and if the queued message were real work it would
be silently swallowed |
### Expected behavior
`EndWorker` should distinguish acks from work: an ack-only backlog is safe
to leave/drop at termination (warn + proceed); any other queued message
(control invocation, data, ECM, actor command) must still fail loudly so the
coordinator's retry lets the worker drain it — **without** consuming it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
