viirya opened a new pull request, #24: URL: https://github.com/apache/spark-connect-gateway/pull/24
The e2e workflow now covers baseline routing, JWT identity, the backend trust boundary, and per-tenant pool isolation, but nothing in CI stands the gateway up as more than one replica. The multi-replica walkthrough under deploy/examples/e2e-multi-replica-redis has never run automatically, so the chart's Redis topology and the gateway's behaviour when a replica dies are only verified by hand. The cross-replica semantics themselves are already covered in-process by the ha_smoke example, which runs in the integration job against a real Redis. What is missing is the Kubernetes shape around it: the bundled Redis StatefulSet, the synthesized redis:// URL, two gateway replicas behind one Service, and a pod actually being deleted. This adds an e2e-multi-replica-redis job that asserts: (1) Both gateway replicas report affinity_store=redis on startup. The pod count is checked first, so a selector that matched nothing could not let the check pass silently. (2) A session driven through the Service writes its binding to Redis, under the documented key shape, with a value that is one of the real backend pod IPs and a TTL within the configured hour. Redis is asserted empty beforehand so a stale key cannot satisfy this. (3) After deleting one replica, the binding is unchanged, and a client arriving at the replacement pod -- port-forwarded directly, so the surviving replica cannot serve it -- reaches the same backend. The replacement's own log must name that backend, an address it could only have read from Redis. (4) Fresh sessions still spread across both backends. Without this, assertion 3 would also pass on a gateway that always picked the same backend regardless of affinity. Verified locally against a kind cluster first. The replacement replica returned the same serverSideSessionId as the original, confirming the same Spark driver answered, and eight fresh sessions split 4/3 across the two backends while the pinned session stayed put. --- Verified locally against a kind cluster (`scg-redis`) before writing the workflow. The decisive evidence for assertion 3: after deleting the replica that had served the session, a client port-forwarded **directly to the replacement pod** got back ``` serverSideSessionId: 28870e4e-f886-42b9-b686-04a095e0abb9 ``` byte-identical to what the original replica returned. That id is minted by the Spark driver, so the same driver answered — and the replacement pod, which had never seen the session, could only have found that backend in Redis. The negative control (assertion 4) split 8 fresh sessions 4/3 across the two backends while the pinned session stayed on its own, so the pinning is a property of the binding rather than of the pool. Cross-replica *semantics* are already covered by `ha_smoke` in the integration job; this job covers the Kubernetes topology around it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
