Spark already supports `bin/pyspark --remote local` now but I think it doesn't work with the interpreter way Nicholas mentioned. This is probably related to https://lists.apache.org/thread/j382to15zgy8mr2pvrtcod9c02zj1org as well.
On Tue, 23 Jun 2026 at 05:05, Tian Gao via dev <[email protected]> wrote: > Yes I think that's totally fine. > > I even support `.remote("local").getOrCreate()` as long as `"local"` has > no other ambiguity (is it used now? if not I'm fine to have this as a > special value). What I don't support is that we change the behavior for > something we are supporting now. For example, > .remote("sc://localhost:15002").getOrCreate() should not magically create > the server if there's no existing server. > > Tian > > On Mon, Jun 22, 2026 at 12:01 PM Nicholas Chammas < > [email protected]> wrote: > >> >> > On Jun 22, 2026, at 2:45 PM, Tian Gao <[email protected]> wrote: >> > >> > All I'm saying is that this is a tradeoff, it has benefits in some >> cases, but it's not a free lunch. I support having this as a feature when >> users know what they are doing, aka explicit consent. I'm against making >> this an implicit default to replace the existing behavior. >> >> OK, so are you in support of the weaker solution I outlined then? That >> is, we add a `spark connect` CLI with some accompanying user guide >> documentation, and make any existing Connect server discoverable by >> `.remote(“local”)`. The user still has to explicitly start up a persistent >> server. >> >> I think down the line we want to consider making this part of the default >> local workflow, but we can revisit that after this first step. > >
