Given the probable need for IP clearance, getting it into Arrow would also be a Process(TM) unfortunately. We also don't really have a great place for "not quite in tree" projects; there have been discussions of a 'contrib' repo in the past, but nothing has materialized.
That said - have you shown this to Spark users? I'd guess there'd be more enthusiasm there, especially if there are particular data source(s) you anticipate this would make available to them. (Though again, Flight SQL or ADBC over plain Flight RPC would might be a more attractive target for such a Spark plugin.) -David On Tue, Oct 18, 2022, at 16:50, Matt Phelps wrote: > Hi David and Antoine, > > Thanks for your input. On past experience talking to some other Arrow / > Spark developers, we anticipate that it would take a long time to get > into Spark. Our plan was to build up a user base in the Arrow community > before submitting for inclusion to Spark. Is there a place the code can > live in the mean time? > > Matt Phelps > > > From: Antoine Pitrou <[email protected]> > Date: Monday, October 17, 2022 at 2:48 PM > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] Integrate existing Spark connector for Flight > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and > know the content is safe. > > Le 17/10/2022 à 21:27, David Li a écrit : >> Hey Matt, >> >> This is cool to see. To be clear, this is an implementation of Spark >> DataSourceV2 using Arrow Flight? >> >> I think the questions I have are: >> >> - Does this belong under Arrow, or under Spark - I lean towards it being >> closer to Spark than Arrow; > > FWIW, that is my feeling as well. > > Regards > > Antoine.
