dhruv-pratap opened a new pull request, #57383: URL: https://github.com/apache/spark/pull/57383
### What changes were proposed in this pull request? This is a **[WIP] reference implementation** posted to accompany a dev@ discussion (see thread link below) — it is intended to make the proposal concrete for reviewers, not (yet) as a merge-ready change. It adds Spark Connect support to the `spark-sql` CLI: `spark-sql --remote sc://...` starts a SQL shell backed by a remote Spark Connect session, mirroring the existing `spark-shell --remote` (SPARK-48936). A new `SparkConnectSQLCLIDriver` is added in `sql/connect/client/jvm`, selected when `--remote` / `spark.api.mode=connect` is set (the same dispatch `spark-shell` uses). The driver: - builds a remote `SparkSession` against the `sc://` URL (as `ConnectRepl` does), - executes statements via `spark.sql(...)`, - prints basic tabular output client-side, - supports `-e` / `-f` and an interactive loop. The existing `SparkSQLCLIDriver` (Catalyst / hive-thriftserver-coupled) is unchanged and remains the default; there is no `sql/hive-thriftserver` dependency and no change to existing public API. **MVP scope** — deliberately minimal to seed the discussion. Deferred to follow-ups: full `hiveResultString` output parity (types / nulls / complex / timestamps), remaining CLI flags, init files, prompt / `USE db` semantics. Design doc: https://docs.google.com/document/d/14iOZAZFt5BN9whbbjCzZX1mVuhaKq2LEkfP6Ts4prHo/edit dev@ thread: https://lists.apache.org/thread/cw897hvjqlmrtdo7byr10b86wtls819l ### Why are the changes needed? `spark-sql` is the last interactive entry point without Spark Connect support — `spark-shell`, `spark-submit`, and `pyspark` all have `--remote` under the SPARK-49194 umbrella — and the only one still requiring a local Catalyst driver and the Hive Thrift Server module. This completes the umbrella and lets the SQL CLI run against remote, service-hosted compute. ### Does this PR introduce _any_ user-facing change? Yes — a new `spark-sql --remote sc://...` mode. No change to existing `spark-sql` behavior; the classic local driver remains the default. ### How was this patch tested? - **Unit** (server-free, `ConnectFunSuite`): `SparkConnectSQLCLIDriverSuite` — 5 tests covering the `-e`/`-f` split from the Connect client args, statement splitting, and row formatting. All pass. - **Integration / E2E** (`SparkConnectSQLCLIDriverE2ESuite`, on `RemoteSparkSession`): boots a local Spark Connect server and drives the real remote path — `spark-sql --remote <sc://localhost> -e "select ..."`, multiple `;`-separated statements, and an `-f` script — asserting the rendered tab-separated output. Runs in the build. - `./build/sbt "launcher/compile" "connect-client-jvm/Test/compile"` — compiles clean; `./dev/scalafmt` — clean. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 (Claude Code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
