[
https://issues.apache.org/jira/browse/GUACAMOLE-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097461#comment-18097461
]
Ciro Iriarte commented on GUACAMOLE-2303:
-----------------------------------------
h2. Test evidence
Posting concrete results so the behaviour is easy to judge. Everything below
comes from an automated harness on a single host; the setup and one full trial
log are included so it can be reproduced.
h3. Rig
* One host running guacd, the guacamole webapp (Tomcat, :8080), and nginx as a
reverse proxy (:8081) configured per the WAN-tuning docs (proxy_buffering off,
WebSocket upgrade, long read timeout).
* The test browser (headless Chromium driven by Puppeteer) runs in a network
namespace joined to the host by a veth pair. {{tc netem}} is applied only to
that veth, so *only the browser-to-server hop is shaped* -- guacd:4822 and the
SSH/VNC target stay on unimpaired loopback. This matches production, where only
browser-to-proxy crosses the WAN.
* Outage = 100% packet loss in both directions for 20s (longer than the
client's 15000ms receiveTimeout, so the tunnel genuinely closes), then the link
is restored.
* guacd started with {{GUACD_RESUME_GRACE=90}}; per-connection resume enabled
on the client.
h3. What a passing trial looks like (one full run, via the nginx proxy)
The browser stays on one {{Guacamole.Client}} the whole time. Timeline (t=0 is
when the outage starts), token = the tunnel UUID:
{noformat}
+0.0s state=CONNECTED unstable=false token=85347556 <- steady state
+2.5s state=CONNECTED unstable=true token=85347556 <- outage begins,
pings stop getting through
+15.0s state=CONNECTED unstable=true token=85347556
+17.0s state=WAITING wsReady=0 token=85347556 <- receiveTimeout
fires, WebSocket #1 closes
+21.1s state=WAITING token=ca45d1c2 rx=8 <- fresh tunnel
opened, receiving again
+22.1s state=CONNECTED token=ca45d1c2 <- resumed, same
client, work intact
FINAL state=CONNECTED uuid_final=ca45d1c2-208e-4ba2-91b3-0097977d5574
WENT_UNSTABLE=true HIT_DISCONNECTED=false CLIENT_RECOVERED=true
LARGEST GAP between resetTimers calls: 16.0s (the outage stall)
WebSockets: ws#1 msgs=19 ready=3(CLOSED) ws#2 msgs=154 ready=1(OPEN)
{noformat}
Two things to note:
* {{HIT_DISCONNECTED=false}} -- the drop never surfaced as a terminal
{{TUNNEL_ERROR}}, so the stock 15s reconnect-countdown never armed and did not
replace the resumed client.
* One old WebSocket closed (ready=3), one new WebSocket open (ready=1): a clean
tunnel swap on a single client, not a new session.
Correlated guacd log for the same trial:
{noformat}
Creating new client for protocol "ssh"
Connection ID is "$4a97a780-67ea-436c-8624-481327da0df7"
Joining existing connection "$4a97a780-67ea-436c-8624-481327da0df7"
{noformat}
{{Creating new client}} appears once (the initial connect); on resume the
client *joins the existing connection* by ID rather than creating a second one.
That is the whole mechanism, visible in the log.
h3. Repeated trials
Same 20s outage, back to back. Each row: did the client recover, the measured
stall, the old->new tunnel UUIDs (they differ, so it is a genuine new socket,
not a survivor), the two WebSocket states, and the guacd rejoin line.
{noformat}
mode recovered stall old ws -> new ws rejoined guacd connection
proxy #1 yes 15.8s closed -> open (151)
$a6591bbd-7cd4-40d3-9b05-8abf9bd8a9f1
proxy #2 yes 16.0s closed -> open (151)
$832cfe4d-dbb2-4c8b-bd7f-a07ad00baf9c
proxy #3 yes 15.9s closed -> open (151)
$8d722ce5-fba7-4110-9d53-f8ab6301a07b
proxy #4 yes 16.1s closed -> open (153)
$1f2875f5-b61f-4d3e-9734-507b03a614c7
direct #1 yes 15.8s closed -> open (153)
$cb8064ec-a125-4297-8f81-33807e0886e7
direct #2 yes 15.9s closed -> open (151)
$a489b405-6111-408d-8f5f-14e0e1aa5f5c
{noformat}
6 of 6 recovered -- 4 through the nginx reverse proxy, 2 direct. In every trial
the old WebSocket was closed and a new one opened on the same client, and guacd
logged a join onto the existing connection rather than a second {{Creating new
client}}. The stall clusters at ~16s (the 15s receiveTimeout plus the
reconnect/rejoin round trip); the user sees a brief freeze, then their session
as they left it.
h3. Other checks
* Verified on SSH and VNC. VNC replays its framebuffer on rejoin, so the screen
is intact after resume.
* Steady-state input latency is unchanged -- it is round-trip-bound and is not
what this touches.
* guacamole-common-js karma suite passes, including added coverage for the
reconnect tunnel lifecycle (old tunnel detached and disconnected, late
instructions on the old tunnel ignored).
The harness scripts (netem shaper, Puppeteer driver, orchestrator) can be
shared if useful for reproducing or for a regression test.
> [RFC] Optional session resume so a short network drop does not end the session
> ------------------------------------------------------------------------------
>
> Key: GUACAMOLE-2303
> URL: https://issues.apache.org/jira/browse/GUACAMOLE-2303
> Project: Guacamole
> Issue Type: New Feature
> Reporter: Ciro Iriarte
> Priority: Major
>
> h2. Summary
> Add an optional session-resume capability so a short network drop doesn't end
> a Guacamole session. Today a stall longer than ~15s closes the tunnel, and
> the built-in reconnect starts a new session, so the user loses their work.
> This lets the client rejoin the same live session instead. It's off by
> default.
> h2. Problem
> When the browser-to-server link stalls, the client tunnel's receiveTimeout
> (default 15000ms) fires and the tunnel closes. The built-in reconnect then
> opens a new connection, so the previous guacd session is gone and the user is
> back at a login/parameter prompt with their work lost. There's currently no
> way to recover a session after a drop.
> The 15s threshold is easier to hit than it sounds:
> * TCP retransmission backoff stretches short drops. On a netem test rig, an
> ~8s outage left the tunnel frozen for ~14s, and a 20s outage closed the
> session.
> * Ordinary WAN/mobile events go past it: Wi-Fi handoff and DHCP renewal,
> cellular dead zones, VPN reconnects, laptop sleep/resume, NAT rebinding, and
> background-tab throttling (the client's pings pause and a proxy idle timeout
> then closes the socket).
> h2. Proposal
> Let a reconnecting client rejoin its existing guacd connection instead of
> starting a new one. Three parts:
> # guacd: an optional grace window on connection-sharing teardown. When the
> last user of a connection disconnects, hold it for a configurable number of
> seconds (GUACD_RESUME_GRACE, default 0 = off) before tearing it down, so a
> reconnecting client can rejoin by connection ID. This uses the existing
> {{$<uuid>}} join path.
> # guacamole-client webapp: a session-scoped registry mapping a resume token
> to the guacd connection ID. A reconnect that presents the token rejoins via
> {{select $uuid}} (owner-checked, time-limited), and the old tunnel is closed.
> # guacamole-common-js and the Angular client: a reconnect controller that,
> after a drop on a connection that had reached OPEN, opens a fresh
> WebSocketTunnel on the same Guacamole.Client (keeping display, input, and
> streams), sends the resume token, and redisplays the last frame during the
> gap.
> This also fixes a related problem: a dropped tunnel was reported as a
> terminal TUNNEL_ERROR, which started the existing 15s reconnect-countdown
> notification and replaced the client that had already resumed.
> h2. Configuration and scope
> * Off by default. GUACD_RESUME_GRACE defaults to 0 and the client controller
> is disabled, so existing installs behave exactly as before.
> * Suggested control: enable it per connection, with a server-side maximum
> grace. Calling it "session resume" rather than a broad "WAN mode" seems
> clearer.
> * It should be possible to turn off, and it should be off for share/anonymous
> connections and anywhere a hard drop on disconnect is required.
> h2. Evidence
> * Recovers a 20s outage in every trial (8/8 through an nginx reverse proxy,
> 2/2 direct), rejoining the same guacd session (guacd logs {{Joining existing
> connection $<uuid>}}) with work intact.
> * Tested on SSH and VNC (VNC replays its framebuffer on rejoin).
> * No change to steady-state input latency, which is bound by round-trip time
> and isn't the target here.
> * guacamole-common-js karma tests pass, including new coverage for the
> reconnect tunnel lifecycle.
> h2. Non-goals
> * Not HA or session migration. The registry is per-JVM and in-memory, and
> relies on the same session affinity the existing in-memory auth-token map
> already needs. Rejoining across nodes is out of scope.
> * Not a bandwidth or latency optimization; it's about recovering from a drop.
> h2. Compatibility and risk
> * The grace window holds a guacd session briefly after its last user leaves,
> which changes teardown timing and uses resources, so it's opt-in and bounded.
> * The resume token is currently the client-facing tunnel UUID; an opaque
> single-use token would be a good follow-up.
> * A "reconnecting" overlay with a manual retry is worth adding before
> enabling this widely, since users tend to hit refresh, which throws away the
> client state and prevents the resume.
> h2. Patch
> Implemented on forks, branch wan-optimization:
> * github.com/ciroiriarte/guacamole-server -- guacd grace window
> * github.com/ciroiriarte/guacamole-client -- webapp rebind, client reconnect
> controller, diagnostics
> * github.com/ciroiriarte/guacamole-manual -- reverse-proxy WAN-tuning docs
> I can submit these as separate PRs and adjust the configuration to whatever
> you'd prefer. A reasonable order:
> # diagnostics/telemetry (off by default)
> # client reconnect-lifecycle hardening (ignore stale tunnels, don't raise a
> terminal error during an active retry, clean tunnel swap)
> # the resume feature (guacd grace + webapp rebind + client controller)
> # the reconnecting overlay and manual retry
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
