Bjørn, that is the project of "spark connect" which dissociates client and server from Spark driver. Not related to data source (which is also known as connector).
Mitch, as I understand correctly, unfortunately we don't have dedicated documentation for implementing data source/connectors. It's encouraged to look at reference implementations like Kafka and understand interfaces. Each interface has its own documentation so it will guide you to implement your own. Please post any question on dev@ mailing list if you have doubts or are stuck with implementing it. Thanks, Jungtaek Lim (HeartSaVioR) On Thu, Nov 24, 2022 at 7:06 AM Bjørn Jørgensen <bjornjorgen...@gmail.com> wrote: > This is from the vote for spark connector. Is this you are looking for? > > The goal of the SPIP is to introduce a Dataframe based client/server API > for Spark > > Please also refer to: > > - Previous discussion in dev mailing list: [DISCUSS] SPIP: Spark Connect > - A client and server interface for Apache Spark. > <https://lists.apache.org/thread/3fd2n34hlyg872nr55rylbv5cg8m1556> > - Design doc: Spark Connect - A client and server interface for Apache > Spark. > <https://docs.google.com/document/d/1Mnl6jmGszixLW4KcJU5j9IgpG9-UabS0dcM6PM2XGDc/edit?usp=sharing> > - JIRA: SPARK-39375 <https://issues.apache.org/jira/browse/SPARK-39375> > > ons. 23. nov. 2022 kl. 17:36 skrev Mitch Shepherd < > mitch.sheph...@marklogic.com>: > >> Hello, >> >> >> >> I’m wondering if anyone can point me in the right direction for a Spark >> connector developer guide. >> >> >> >> I’m looking for information on writing a new connector for Spark to move >> data between Apache Spark and other systems. >> >> >> >> Any information would be helpful. I found a similar thing for Kafka >> <https://docs.confluent.io/platform/current/connect/devguide.html> but >> haven’t been able to track down documentation for Spark. >> >> >> >> Best, >> >> Mitch >> >> This message and any attached documents contain information of MarkLogic >> and/or its customers that may be confidential and/or privileged. If you are >> not the intended recipient, you may not read, copy, distribute, or use this >> information. If you have received this transmission in error, please notify >> the sender immediately by reply e-mail and then delete this message. This >> email may contain pricing or other suggested contract terms related to >> MarkLogic software or services. Any such terms are not binding on MarkLogic >> unless and until they are included in a definitive agreement executed by >> MarkLogic. >> > > > -- > Bjørn Jørgensen > Vestre Aspehaug 4, 6010 Ålesund > Norge > > +47 480 94 297 >
