To give a more concrete feel to what I’m proposing, here is the help output of the prototype unified CLI on my local branch:
$ ./bin/spark usage: spark [-h] COMMAND ... Unified CLI for Apache Spark. options: -h, --help show this help message and exit standalone cluster commands: cluster Manage the standalone cluster (master + workers). master Manage the standalone cluster master. worker Manage the standalone cluster workers. shells: scala Start the Scala shell. python Start the PySpark shell. sql Start the SQL shell. commands: connect Manage the Connect server. history Manage the History server. thrift Manage the Thrift server. pipelines Run Spark Declarative Pipelines. submit Submit an application. Are any committers interested in this idea? Nick > On Jun 22, 2026, at 10:27 PM, Nicholas Chammas <[email protected]> > wrote: > > Today, Spark's CLI is spread across 20+ scripts in bin/ and sbin/. This > fragmented interface makes discovery more difficult for users and maintenance > more difficult for contributors. > > I would like to propose adding a new command line tool named spark that > unifies Spark's command line interface. It would be a pure Python script that > sticks to the standard library and acts simply (at least at first) as a > dispatcher. It would offer users a single entry point with a clear --help > that makes it easy to discover and understand what's available. > > The commands I would like to implement are as follows: > > spark > connect > start > status > stop > history > start > status > stop > master > start > status > stop > pipelines > python > scala > sql > submit > thrift > start > status > stop > worker[s] > decommission > run > start > status > stop > These commands all dispatch to existing scripts. The unified Python CLI would > only implement sub-commands itself when the underlying script does not > already offer its own handling of CLI arguments. For example, pipelines > accepts various commands, but those are handled already by its own CLI so the > unified CLI doesn't need to implement them. connect, on the other hand, > implements start and stop as scripts, so we need to implement those in our > unified CLI and route the arguments to the appropriate script. > > There are several things still to hash out, like: > > Packaging of this unified CLI so it's placed on the PATH correctly. > The exact naming and structure of the commands. > Whether we eventually want to fold any Bash scripts directly into this new > Python CLI. > But before we get into that, I first want to see if this idea is attractive > to the community. > > Shall I flesh this out into a ticket and publish a working prototype, or does > the idea have some critical flaw? > > Nick >
