andygrove opened a new pull request, #326:
URL: https://github.com/apache/datafusion-spark/pull/326
## Which issue does this PR close?
Part of #56. It covers the `datafusion-cli --spark` references in the
"Update the imported docs" item, since those READMEs now point to the new CLI.
## Rationale for this change
apache/datafusion#27337 removed `datafusion-cli --spark`, which enabled the
Spark SQL dialect and registered the Spark-compatible functions. Since then
there is no way to try the functions interactively, and
`crates/spark/README.md` and `crates/sqllogictest/test_files/README.md` still
tell people to run `datafusion-cli --spark`.
## What changes are included in this PR?
- **`crates/cli`, a new `datafusion-spark-cli` binary.** It uses the
`datafusion-cli` library for the REPL, for running files and commands, for
output formats, and for querying files by path or URL (`SELECT * FROM
'data.parquet'`). Its `main.rs` is about 200 lines, following
`datafusion-cli`'s.
- **Always Spark:** the Spark SQL dialect and the Spark functions are
always enabled. The Spark functions override DataFusion's functions of the same
name.
- **ANSI mode on by default,** as in Spark 4 (`spark.sql.ansi.enabled =
true`). `--ansi false` gives Spark's legacy behavior. The startup banner says
which mode is on.
- **Options kept from `datafusion-cli`:** `-c`, `-f`, `-r`, `-p`, `-b`,
`--format`, `-q`, `--maxrows` and `--color`. The memory and disk tuning options
are left out for now.
- **Not published:** `publish = false`.
- **Not a default workspace member:** `datafusion-cli` brings in the object
store and AWS dependencies, so a plain `cargo build` or `cargo test` at the
root skips this crate. The clippy job uses `--workspace`, so it still lints the
crate.
- **New workflow, `.github/workflows/cli.yml`.** It builds the CLI and runs
a smoke test:
- `sha2` returns Spark's result.
- `abs(cast(-2147483648 AS int))` fails by default, because ANSI mode is
on.
- The same query returns `-2147483648` with `--ansi false`.
- **Docs:** `crates/spark/README.md`, `test_files/README.md`, the root
`README.md` and `AGENTS.md` describe the new CLI.
`datafusion-cli` on DataFusion's `branch-56` still depends on that branch's
own `datafusion-spark`, so the build compiles that copy too. Only this
repository's functions are registered. The extra copy goes away with DataFusion
57, where `datafusion-cli` no longer depends on `datafusion-spark`.
## Are these changes tested?
- The smoke test's three queries pass locally.
- I also tried the REPL reading from stdin, a backquoted identifier (Spark
dialect), `--help` and the banner.
- `cargo clippy --all-targets --all-features --workspace -- -D warnings` is
clean.
- `cargo test -p datafusion-spark` and `cargo test -p
datafusion-spark-sqllogictest` still pass.
- `actionlint` reports no problems with the new workflow.
## Are there any user-facing changes?
There is a new, unpublished `datafusion-spark-cli` binary. You run it with
`cargo run -p datafusion-spark-cli`.
_This PR was written by an LLM (Claude) and reviewed before opening._
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]