Hi Ryan, Great job on this! Shall we call a vote for the plan standardization SPIP? I think this is a good idea and we should do it.
Notes: We definitely need new user-facing APIs to produce these new logical plans like DeleteData. But we need a design doc for these new APIs after the SPIP passed. We definitely need the data source to provide the ability to create/drop/alter/lookup tables, but that belongs to the other SPIP and should be voted separately. Thanks, Wenchen On Fri, Apr 20, 2018 at 5:01 AM Ryan Blue <rb...@netflix.com.invalid> wrote: > Hi everyone, > > A few weeks ago, I wrote up a proposal to standardize SQL logical plans > <https://docs.google.com/document/d/1gYm5Ji2Mge3QBdOliFV5gSPTKlX4q1DCBXIkiyMv62A/edit?ts=5ace0718#heading=h.m45webtwxf2d> > and > a supporting design doc for data source catalog APIs > <https://docs.google.com/document/d/1zLFiA1VuaWeVxeTDXNg8bL6GP3BVoOZBkewFtEnjEoo/edit#heading=h.m45webtwxf2d>. > From the comments on those docs, it looks like we mostly have agreement > around standardizing plans and around the data source catalog API. > > We still need to work out details, like the transactional API extension, > but I'd like to get started implementing those proposals so we have > something working for the 2.4.0 release. I'm starting this thread because I > think we're about ready to vote on the proposal > <https://spark.apache.org/improvement-proposals.html#discussing-an-spip> > and I'd like to get any remaining discussion going or get anyone that > missed this to read through the docs. > > Thanks! > > rb > > -- > Ryan Blue > Software Engineer > Netflix >
