nchammas opened a new pull request, #57313:
URL: https://github.com/apache/spark/pull/57313
### What changes were proposed in this pull request?
This PR adds a new command line tool named `spark` that unifies Spark's
command line interface.
It's a pure Python script that sticks to the standard library and acts
simply (at least for now) as a dispatcher to various scripts that already exist
under `bin/` and `sbin/`.
Here are the top-level commands:
```
$ ./bin/spark
usage: spark [-h] COMMAND ...
Unified CLI for Apache Spark.
options:
-h, --help show this help message and exit
standalone cluster commands:
cluster Manage the standalone cluster (master + workers).
master Manage the standalone cluster master.
worker Manage the standalone cluster workers.
shells:
scala Start the Scala shell.
python Start the PySpark shell.
sql Start the SQL shell.
commands:
connect Manage the Connect server.
history Manage the History server.
thrift Manage the Thrift server.
pipelines Run Spark Declarative Pipelines.
submit Submit an application.
```
Each command has its own `--help` and may also have its own sub-commands.
For example:
```
$ ./bin/spark connect
usage: spark connect [-h] COMMAND ...
options:
-h, --help show this help message and exit
commands:
start Start the server.
stop Stop the server.
status Show server status.
```
### Why are the changes needed?
Today, Spark's CLI is spread across 20+ scripts under `bin/` and `sbin/`.
This fragmented interface makes discovery more difficult for users and
maintenance more difficult for contributors.
A unified CLI gives users a single entry point with a clear help menu that
makes it easy to discover and use what's available. This follows a pattern that
other projects have successfully adopted already for their CLIs, like
`kubectl`, `docker`, and `ray`.
### Does this PR introduce _any_ user-facing change?
Yes, it introduces a major new user interface.
### How was this patch tested?
Local manuals tests + new automated tests.
### Was this patch authored or co-authored using generative AI tooling?
I authored this patch with assistance from GitHub Copilot.
### TODO
- [ ] Secure committer buy-in for this idea + design. (Does it need an SPIP?)
- [ ] Add new tests to CI build.
- [ ] Figure out how this new CLI should be packaged.
- [ ] Determine what interface stability guarantees the new CLI should have.
- [ ] Add user-facing documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
