eugenegujing opened a new pull request, #8855:
URL: https://github.com/apache/texera/pull/8855
### What changes were proposed in this PR?
`WorkflowWebsocketService` only noticed a lost connection through
`retryWhen`, which reacts to **errors**:
```ts
const wsWithReconnect = this.websocket.pipe(
retryWhen(errors =>
errors.pipe(
tap(_ => this.updateConnectionStatus(false)),
tap(_ => console.log(`websocket connection lost, reconnecting in 3
seconds`)),
delayWhen(_ => timer(WS_RECONNECT_INTERVAL_MS)),
tap(_ => this.send("HeartBeatRequest", {}))
)
)
);
```
When the server closes the socket with a close frame, the browser reports
the close as `wasClean: true`, and rxjs's `WebSocketSubject` then **completes**
the subject instead of erroring it (`WebSocketSubject.js`, `socket.onclose`:
`e.wasClean ? observer.complete() : observer.error(e)`). A completion passes
straight through `retryWhen`, so the subscription simply ended:
- `updateConnectionStatus(false)` never ran, so `isConnected` stayed `true`:
the connection badge stayed green, the run button stayed enabled, the kill
button stayed enabled mid-run, and the Form View's Stop from #7944 stayed
enabled;
- nothing redialed;
- every later `send()`, including the 10 s heartbeat, went into the
subject's buffer, which is only flushed on a reconnect that never came, so the
heartbeat could not expose the dead socket either;
- `ComputingUnitStatusService` refreshes the unit list only on a
connection-status change, so that did not happen either.
A server goes away cleanly whenever it is stopped gracefully: a restart or
redeploy of the service behind `/wsapi` (Jetty sends a 1001 close frame on
shutdown), or a Kubernetes computing-unit pod being deleted, evicted or
drained. An unclean close (code 1006, process killed) already took the retry
path, so only the graceful case was broken.
**The change** adds `concatWith(throwError(() => new Error("websocket closed
by the server")))` immediately before `retryWhen`. `concatWith` does nothing
while the socket is live; when the socket completes it appends an error, so a
server-side close now takes the existing retry path: report the drop, wait
`WS_RECONNECT_INTERVAL_MS`, redial, and deliver the buffered requests on the
new socket. `closeWebsocket()` unsubscribes the chain before completing the
subject, so a client-initiated close still does not redial, and
`openWebsocket()` starts with `closeWebsocket()`, so re-opening during the
reconnect gap cancels the pending redial.
`retryWhen` is deprecated in rxjs 7 and is deliberately left untouched here;
replacing it with `retry({ delay })` is a separate change.
Three lines in the service (import, operator, comment).
#### Before: the socket is closed, the UI says connected
The black panel is a debugging overlay injected into the page for the
reproduction (it records every workflow socket created and every close event,
and reads the badge colour from the DOM); it is not part of Texera.
Normal state: one `/wsapi/workflow-websocket` socket is OPEN and the badge
is green.
<img width="1568" height="562" alt="bug2-01-normal-socket-OPEN-badge-green"
src="https://github.com/user-attachments/assets/9d8477e2-1040-4a1a-9181-676708cbd96c";
/>
The socket is closed cleanly. Eleven seconds later it is CLOSED, no second
socket has been created (the reconnect delay is 3 s), the badge is still green,
and the "websocket connection lost, reconnecting" line never appeared.
<img width="1568" height="562"
alt="bug2-02-clean-close-11s-socket-CLOSED-badge-green-no-redial"
src="https://github.com/user-attachments/assets/67d9d633-1470-4ef0-9123-6ac31a6d4479";
/>
34 seconds later, unchanged.
<img width="1568" height="562"
alt="bug2-03-clean-close-34s-still-CLOSED-still-green-BUG"
src="https://github.com/user-attachments/assets/35d8d779-09d5-480e-9b54-6229c659e495";
/>
Toolbar close-up while the socket is closed: the green dot and the enabled
unit picker.
<img width="1018" height="84"
alt="bug2-04-toolbar-zoom-green-dot-while-socket-closed"
src="https://github.com/user-attachments/assets/3cdcd344-95e7-4c97-a225-71f77542e20d";
/>
#### After: the same clean close, on this branch
Same workspace and overlay, served from this branch. Two seconds after the
clean close the socket is CLOSED and the connection badge has turned red; the
console shows "websocket connection lost, reconnecting in 3 seconds".
<img width="1568" height="562"
alt="fixed-bug2-02-clean-close-2s-socket-CLOSED-badge-RED"
src="https://github.com/user-attachments/assets/6242be86-291a-4cd8-814d-a58b1715c4a4";
/>
Nine seconds after the close a new socket has been created (the third in
this session, after two clean closes) and is OPEN, and the badge is green again.
<img width="1568" height="562"
alt="fixed-bug2-03-clean-close-9s-new-socket-OPEN-badge-green-again"
src="https://github.com/user-attachments/assets/e988ed1f-902a-454d-a0f9-48c1770e1d78";
/>
The connection badge itself, before the close, 2 s after it, and 7 s after
it (the dot at the far right):
<img width="980" height="80" alt="fixed-bug2-04-toolbar-before-close-green"
src="https://github.com/user-attachments/assets/91f66c70-4a4c-49cf-8e83-d6a3c28efa1f";
/>
<img width="980" height="80"
alt="fixed-bug2-05-toolbar-2s-after-clean-close-RED"
src="https://github.com/user-attachments/assets/e6352d2c-0829-41f0-9ccd-431178aabd9d";
/>
<img width="980" height="80"
alt="fixed-bug2-06-toolbar-7s-after-close-reconnected-green"
src="https://github.com/user-attachments/assets/7d2bd38e-9204-4d06-8216-84aca4eb90d4";
/>
### Any related issues, documentation, discussions?
Closes #8853
The reproduction closes the socket from the browser; a server-side close
frame takes the same `wasClean: true` path in rxjs, and the fix handles clean
and unclean closes alike.
### How was this PR tested?
- **New unit tests** in `workflow-websocket.service.spec.ts` (red before the
fix where noted): a clean server-side close reports the drop and redials after
`WS_RECONNECT_INTERVAL_MS`, for close codes 1000, 1001, 1005 and 1012 (red); a
clean close before any message was received still redials (red); two
consecutive clean closes each redial (red); a request sent during the reconnect
gap is delivered on the redialed socket (red); `closeWebsocket()` does not
redial; re-opening with another unit during the reconnect gap cancels the
pending redial and opens only the new socket.
- **Mutation check**, each restored afterwards: deleting the `concatWith`,
moving it after `retryWhen`, or dropping the unsubscribe in `closeWebsocket()`
each turns named tests red.
- **Spec file**: 23 passed. **Full frontend suite** (`ng test
--watch=false`): 231 files, 6452 passed, 1 skipped (pre-existing), 0 failed.
`prettier --check` clean on both files.
- **Bug reproduced in a running local instance** (screenshots above): Chrome
against the Angular dev server and the local computing-unit master, with a
page-level `WebSocket` wrapper to observe socket creation and close events.
Server-side variant for reviewers: open a workflow with a connected unit, stop
the computing-unit master gracefully (IDE Stop button) and start it again;
before this PR the socket closes and never reopens while the badge stays green.
- **Fixed build exercised in the same local instance** (screenshots in the
"After" section): four consecutive clean closes, each followed by the badge
turning red, the "websocket connection lost, reconnecting in 3 seconds" console
line, a new socket about 3 to 4 s later, and the badge turning green once the
new socket was up.
### Was this PR authored or co-authored using generative AI tooling?
Yes. Generated-by: Claude Code (claude-fable-5-1). Co-authored with Claude;
the author reviewed the change before submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
