aicam commented on issue #6239: URL: https://github.com/apache/texera/issues/6239#issuecomment-4919500004
### Regression timeline — why this surfaced now (Ingress → Envoy Gateway) The client-side leak (**Leak B**) has existed since the compile-on-edit validation path started building a throwaway `WorkflowGraph` per edit (introduced in **#3501**). But it stayed **invisible for months** because of how the gateway in front of the shared-editing server was configured — and only became a user-visible outage after the gateway was migrated. **Old gateway: nginx `Ingress` — no per-backend connection cap.** The previous chart exposed traffic via a Kubernetes `Ingress` (`ingressClassName` + `nginx.ingress.kubernetes.io/*`) whose only annotations were `use-regex` and `ssl-redirect`. nginx-ingress has **no circuit-breaker / per-upstream `max_connections` concept**; the effective ceilings are `worker-connections` (default **16384 per worker**) × worker processes ≈ tens of thousands, and file-descriptor limits. So the leaked `/rtc` sockets accumulated against a very high ceiling and were **never rejected** — the leak grew silently and produced no visible failure. **New gateway: Envoy Gateway — default `max_connections: 1024` per cluster.** The migration to the Gateway API / Envoy Gateway (**#4191**) put Envoy in front of the y-websocket cluster. Envoy applies a **per-cluster `max_connections` circuit breaker that defaults to 1024** and **actively rejects** new upstream connections once tripped — roughly **16× lower** than nginx's effective ceiling. Once the accumulated leak crossed 1024 concurrent upstream connections, new `/rtc` connections were refused → **cursors/awareness stopped syncing**. This is the symptom described above. **Not a factor: the `y-websocket` version bump.** `y-websocket` went `1.4.0 → 1.5.4` (frontend Angular 20 upgrade, **#4528**), and `yjs` stayed at `13.5.41`. The reconnect health-check (`_checkInterval`, 3s), exponential backoff, and `destroy()` interval-clearing logic are **identical** between 1.4.0 and 1.5.4, so this bump neither created nor worsened the leak. **Summary** | Change | Role | | --- | --- | | #3501 — validation builds a throwaway `WorkflowGraph` (→ `SharedModel` → `/rtc` socket) per edit | **Root leak** (dormant, masked) | | #4191 — nginx `Ingress` → Envoy Gateway (default `max_connections: 1024`) | **What made it visible** — low, actively-enforced ceiling | | #4528 — `y-websocket` 1.4.0 → 1.5.4 | Not related — behavior identical | **Implication for the fix:** stopping the leak at the source (don't open a `/rtc` socket for throwaway/validation graphs, and use `wsProvider.destroy()` in teardown) is the primary fix. Raising the Envoy `max_connections` and reaping idle upstream connections at the gateway is complementary defense-in-depth — worth keeping so a future client-side regression degrades gracefully instead of hard-failing at 1024. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
