Hello Zoltan Borok-Nagy, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/25043
to look at the new patch set (#4).
Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster
......................................................................
IMPALA-15506: Add utility to test Spark with impala minicluster
Add convenience scripts to test Spark (4.1.3 by default) with
the Impala minicluster. Based on similar scripts for Trino:
start/stop/run sql with run-spark.sh/kill-spark.sh/spark-sql.sh
The official Spark docker image is used unmodified, so there is no
image to build. run-spark.sh bind mounts the Iceberg runtime jar and
a conf directory assembled from the minicluster configuration, and
passes everything the image would otherwise bake in as environment
variables. The jar is downloaded from Maven Central on the first run.
Both mount sources live under testdata/bin/minicluster_spark and are
gitignored.
No tests added at this point, the goal is only to help with testing
Spark<->Impala interop.
The main motivation was GEOMETRY testing, so newest (1.12.0) Iceberg
is used in Spark instead of Impala's 1.11.0. Versions used can be
overridden with env vars read by run-spark.sh.
Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Assisted-by: Claude Opus 5
---
A testdata/bin/SPARK-README.md
A testdata/bin/kill-spark.sh
A testdata/bin/minicluster_spark/.gitignore
A testdata/bin/minicluster_spark/spark-defaults-iceberg.conf
A testdata/bin/minicluster_spark/spark-defaults.conf
A testdata/bin/run-spark.sh
A testdata/bin/spark-sql.sh
7 files changed, 383 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/25043/4
--
To view, visit http://gerrit.cloudera.org:8080/25043
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6
Gerrit-Change-Number: 25043
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
