+1 for the concept. Correct me if I'm wrong, but at a high level this is proposing adding a new user API (which is language agnostic) and the proposal is to start with something like the Logical Plan, with the addition of being able to remotely call this API.
+0 on architecture/design as it's not clear from the doc how much impact this truly has. But that is a problem with SPIPs which I have voiced my concern about in the past. I can see how this could be flushed out and keep the overall impact minimal (vs blow up the world architecture change) since it's not just a drop in replacement for all existing APIs. For instance, conceptually this is just a version of the Spark thriftserver which uses grpc and passes the new API and internally we add a new API runPlan(LogicalPlan). You could potentially also not use the internal version of the catalyst Logical Plan API but have some conversion still to allow changes to catalyst internals, not sure if that is needed but a possibility. With any API addition it will have to be kept stable and require more testing and likely more dev work, so weighing that vs usefulness is the question for me. Tom On Mon, Jun 13, 2022 at 1:04 PM Herman van Hovell <her...@databricks.com.invalid> wrote: > > Hi all, > > I’d like to start a vote for SPIP: "Spark Connect" > > The goal of the SPIP is to introduce a Dataframe based client/server API for > Spark > > Please also refer to: > > - Previous discussion in dev mailing list: [DISCUSS] SPIP: Spark Connect - A > client and server interface for Apache Spark. > - Design doc: Spark Connect - A client and server interface for Apache Spark. > - JIRA: SPARK-39375 > > Please vote on the SPIP for the next 72 hours: > > [ ] +1: Accept the proposal as an official SPIP > [ ] +0 > [ ] -1: I don’t think this is a good idea because … > > Kind Regards, > Herman --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
