> but I think it doesn't work with the interpreter way Nicholas mentioned.
Oh oops sorry I forgot that I added that support as well. Anyway it's still related to https://lists.apache.org/thread/j382to15zgy8mr2pvrtcod9c02zj1org On Tue, 23 Jun 2026 at 07:57, Hyukjin Kwon <[email protected]> wrote: > Spark already supports `bin/pyspark --remote local` now but I think it > doesn't work with the interpreter way Nicholas mentioned. > This is probably related to > https://lists.apache.org/thread/j382to15zgy8mr2pvrtcod9c02zj1org as well. > > > On Tue, 23 Jun 2026 at 05:05, Tian Gao via dev <[email protected]> > wrote: > >> Yes I think that's totally fine. >> >> I even support `.remote("local").getOrCreate()` as long as `"local"` has >> no other ambiguity (is it used now? if not I'm fine to have this as a >> special value). What I don't support is that we change the behavior for >> something we are supporting now. For example, >> .remote("sc://localhost:15002").getOrCreate() should not magically create >> the server if there's no existing server. >> >> Tian >> >> On Mon, Jun 22, 2026 at 12:01 PM Nicholas Chammas < >> [email protected]> wrote: >> >>> >>> > On Jun 22, 2026, at 2:45 PM, Tian Gao <[email protected]> wrote: >>> > >>> > All I'm saying is that this is a tradeoff, it has benefits in some >>> cases, but it's not a free lunch. I support having this as a feature when >>> users know what they are doing, aka explicit consent. I'm against making >>> this an implicit default to replace the existing behavior. >>> >>> OK, so are you in support of the weaker solution I outlined then? That >>> is, we add a `spark connect` CLI with some accompanying user guide >>> documentation, and make any existing Connect server discoverable by >>> `.remote(“local”)`. The user still has to explicitly start up a persistent >>> server. >>> >>> I think down the line we want to consider making this part of the >>> default local workflow, but we can revisit that after this first step. >> >>
