[
https://issues.apache.org/jira/browse/SPARK-59224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh resolved SPARK-59224.
---------------------------------
Fix Version/s: connect-gateway-0.1.0
Resolution: Fixed
Issue resolved by pull request 20
[https://github.com/apache/spark-connect-gateway/pull/20]
> Add an end-to-end workflow running the gateway on kind with a real PySpark
> client
> ---------------------------------------------------------------------------------
>
> Key: SPARK-59224
> URL: https://issues.apache.org/jira/browse/SPARK-59224
> Project: Spark
> Issue Type: Sub-task
> Components: Connect
> Affects Versions: connect-gateway-0.1.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-gateway-0.1.0
>
>
> The deploy/examples/e2e-smoke walkthrough exercises the whole deployment path
> —
> gateway image, Helm chart, Kubernetes Endpoints discovery, session affinity,
> audit
> and metrics — but only by hand. Nothing in CI covered it, so a break in the
> chart,
> the Dockerfile or the K8s pool would only surface when someone next ran the
> walkthrough manually.
> This adds an E2E workflow that automates it: build the gateway image, create
> a kind
> cluster and load the image into it, deploy two apache/spark:4.0.0 Spark
> Connect
> servers, install the gateway with the Helm chart and the e2e-smoke values,
> assert
> the gateway discovered both backends, run the repo's own
> test/integration/client_smoke.py through a port-forward, then assert
> scg_backend_pool_size, the ExecutePlan counters, and that the audit log
> recorded
> ExecutePlan. On failure the job dumps pod state and gateway/Spark logs; the
> kind
> cluster is always deleted.
> Cost on a GitHub-hosted runner, measured rather than estimated: 6m22s — docker
> build 208s (55% of the total), PySpark install plus client 57s, kind create
> cluster
> 41s, Spark backends ready including the ~700 MiB image pull 43s, Helm install
> and
> gateway readiness 11s, tool install and assertions about 14s. It runs in
> parallel
> with the existing build job (2m19s), so it adds roughly four minutes to a
> PR's wall
> release build, there is headroom later if it needs trimming.
> Two implementation notes:
>
> - kind and helm are installed with plain curl at pinned versions rather than
> third-party actions, both to stay clear of the ASF GitHub Actions
> allowlist and to
> keep the versions explicit. They go into ~/.local/bin, which is writable
> without
> sudo on the hosted runners (/usr/local/bin is not). The only action used
> is
> actions/checkout.
> - The PySpark client is pinned to 4.0.0 to match the backend image. An
> unpinned
> install picks up 4.2, whose createDataFrame reads the SQL config
> spark.sql.session.localRelationSizeLimit; Spark 4.0 does not have it and
> the
> server answers SQL_CONF_NOT_FOUND.
> Verified by running the whole walkthrough locally first, then on the runner:
> the
> PySpark client returns correct results for range().count(), an explicit-schema
> createDataFrame round-trip, and five follow-up queries on the same session —
> the
> last being the meaningful session-affinity check, since a misrouted follow-up
> RPC
> would land on a different driver.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
