+1 On Thu, Jun 16, 2022 at 7:17 AM Thomas Graves <tgraves...@gmail.com> wrote:
> +1 for the concept. > Correct me if I'm wrong, but at a high level this is proposing adding > a new user API (which is language agnostic) and the proposal is to > start with something like the Logical Plan, with the addition of being > able to remotely call this API. > > +0 on architecture/design as it's not clear from the doc how much > impact this truly has. But that is a problem with SPIPs which I have > voiced my concern about in the past. > I can see how this could be flushed out and keep the overall impact > minimal (vs blow up the world architecture change) since it's not just > a drop in replacement for all existing APIs. For instance, > conceptually this is just a version of the Spark thriftserver which > uses grpc and passes the new API and internally we add a new API > runPlan(LogicalPlan). You could potentially also not use the internal > version of the catalyst Logical Plan API but have some conversion > still to allow changes to catalyst internals, not sure if that is > needed but a possibility. > With any API addition it will have to be kept stable and require more > testing and likely more dev work, so weighing that vs usefulness is > the question for me. > > Tom > > On Mon, Jun 13, 2022 at 1:04 PM Herman van Hovell > <her...@databricks.com.invalid> wrote: > > > > Hi all, > > > > I’d like to start a vote for SPIP: "Spark Connect" > > > > The goal of the SPIP is to introduce a Dataframe based client/server API > for Spark > > > > Please also refer to: > > > > - Previous discussion in dev mailing list: [DISCUSS] SPIP: Spark Connect > - A client and server interface for Apache Spark. > > - Design doc: Spark Connect - A client and server interface for Apache > Spark. > > - JIRA: SPARK-39375 > > > > Please vote on the SPIP for the next 72 hours: > > > > [ ] +1: Accept the proposal as an official SPIP > > [ ] +0 > > [ ] -1: I don’t think this is a good idea because … > > > > Kind Regards, > > Herman > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
