viirya opened a new pull request, #20:
URL: https://github.com/apache/spark-connect-gateway/pull/20
### What changes were proposed in this pull request?
The `deploy/examples/e2e-smoke` walkthrough exercises the whole deployment
path —
gateway image, Helm chart, Kubernetes Endpoints discovery, session affinity,
audit
and metrics — but only by hand. Nothing in CI covered it, so a break in the
chart,
the Dockerfile or the K8s pool would only surface when someone next ran the
walkthrough manually.
This adds an `E2E` workflow that automates it:
1. build the gateway image
2. create a kind cluster, load the image into it
3. deploy two `apache/spark:4.0.0` Spark Connect servers
4. install the gateway with the Helm chart + the example values
5. assert the gateway discovered **both** backends (K8s Endpoints watch)
6. run the repo's own `test/integration/client_smoke.py` through a
port-forward
7. assert `scg_backend_pool_size`, the `ExecutePlan` counters, and that the
audit
log recorded `ExecutePlan`
`kind` and `helm` are installed with plain `curl` at pinned versions rather
than
third-party actions — both to stay clear of the ASF GitHub Actions allowlist
and to
keep the versions explicit. The only action used is `actions/checkout`. On
failure
the job dumps pod state and gateway/Spark logs; the kind cluster is always
deleted.
**Note on where this runs:** it is wired to `pull_request` here so we can
measure
what it actually costs on a hosted runner. Locally the walkthrough takes
~7.5 min
(image build 5m23s, Spark image pull + pod readiness 54s, the rest seconds);
on a
2-core runner it will be slower. If the measured time turns out too high to
gate
PRs on, the trigger can move to a nightly `schedule` + `workflow_dispatch`
instead —
that is a two-line change.
### Why are the changes needed?
This is the only coverage of the real deployment path: the Helm chart, the
Kubernetes Endpoints-watch pool, and session affinity as observed by an
actual
Spark client. The in-process integration tests cover the proxy logic, but
nothing
else verifies that the chart deploys a working gateway.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Ran the entire walkthrough locally before writing the workflow. The PySpark
client
returned correct results including a TempView query — the meaningful check
that
session affinity held, since a TempView lives in one driver's memory and a
misrouted follow-up RPC would fail it. `scg_backend_pool_size 2`, the RPC
counters
and the audit records were all as expected. This PR's own run exercises the
workflow on a hosted runner.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
