aglinxinyuan opened a new issue, #6920:
URL: https://github.com/apache/texera/issues/6920
### What happened?
The 150-attempt region-termination retry budget only bounds **failures** —
it is bypassed entirely by **non-delivery**, so a single failed `gracefulStop`
pass can wedge the whole workflow silently, with the give-up ERROR never
firing. This is exactly the silent-hang class that the bounded retry (#5737)
was meant to eliminate.
Sequence:
```
attempt K: terminateWorkers
├─ endWorker OK for all workers
├─ for each worker: removeActorRef (:198), removeControlChannel x2
(:201-202) <- state already torn down
└─ gracefulStop fails for ONE worker (e.g. a slow Python PVM > 5 s)
<- attempt K fails
attempt K+1: re-sends endWorker to ALL workers
(RegionExecutionManager.scala:177-184)
├─ removed worker, ref gone -> message parks FOREVER in messageStash
(PekkoActorRefMappingService.scala:69-73)
└─ ref still resolvable -> coordinator seq was reset to 0 by
removeControlChannel
(NetworkOutputGateway.scala:97-101) while
the worker's channel
`current` is still > 0 -> classified
duplicate and silently
dropped (AmberFIFOChannel.scala:60-61;
debug-only log at :46-49)
either way: Future.collect(endWorkerRequests) (:186-187) never resolves
-> rescue never runs -> retry budget never consumed -> give-up ERROR
(:237) never fires
-> terminationFutureRef (CAS-set once, :157-171) wedges
advanceRegionExecutions
(WorkflowExecutionManager.scala:108-113) -> no later region ever starts
```
Aggravating details:
- There is **no timeout anywhere** on the path: not on the `endWorker`
collect, not on AsyncRPCClient promises (a repo grep for
`.within(|raiseWithin|TimeoutException` in amber main sources hits only
unrelated files).
- `actorRefService.getActorRef(workerId)` at
`RegionExecutionManager.scala:196` is a bare map apply
(`PekkoActorRefMappingService.scala:50-52`) and throws `NoSuchElementException`
for a worker whose ref was removed by a previous attempt.
- The 30 s resend timer cannot rescue the seq-reset case: it re-sends the
same (now duplicate) sequence number forever. Its suppression guard is
separately broken (filed separately).
A per-attempt `.within(...)` timeout on the `endWorker`/`gracefulStop`
collects (using the existing `killRetryTimer`) would make the budget honest;
the channel/ref teardown at :198-202 probably needs to move after the *whole
region's* stop pass succeeds, not run per-attempt.
### How to reproduce?
Make one worker's `gracefulStop` exceed the 5 s bound during region
termination (e.g. a Python worker whose PVM is slow to exit) on a multi-worker
region. Attempt 2's `endWorker`s either stash forever or are dropped as
duplicates; the workflow hangs with no ERROR log.
### Version/Branch
main (observed at 429be110a7; discovered during the investigation for #6916).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
