I'm slightly confused about this discussion. I worked on all of the aforementioned PRs: the module PR that has been merged, the current PR that introduces the Graph API and the PoC PR that contains the full implementation. The issues around shading were addressed and the module PR eventually got merged. Two PMC members including the SPIP shepherd are working with me (and others) on the current API PR. The SPIP to bring Spark Graph into Apache Spark itself has been successfully voted on earlier this year. I presented this work at Spark Summit in San Fransisco in May and was asked by the organizers to present the topic at the European Spark Summit. I'm currently sitting in the speakers room of that conference preparing for the talk and reading this thread. I hope you understand my confusion.
I admit - and Xiangrui pointed this out in the other thread, too - that we made the early mistake of not bringing more Spark committers on board which lead to a stagnation period during summer when Xiangrui wasn't around to help review and bring progress to the project. Sean, if your concern is the lack of maintainers of that module, I personally would like to volunteer to maintain Spark Graph. I'm also a contributor to the Okapi stack and am able to work on whatever issue might come up on that end including updating dependencies etc. FWIW, Okapi is actively maintained by a team at Neo4j. Best, Martin On Wed, 16 Oct 2019, 4:35 AM Sean Owen <sro...@gmail.com> wrote: > I do not have a very informed opinion here, so take this with a grain of > salt. > > I'd say that we need to either commit a coherent version of this for > Spark 3, or not at all. If it doesn't have support, I'd back out the > existing changes. > I was initially skeptical about how much this needs to be in Spark vs > a third-party package, and that still stands. > > The addition of another dependency isn't that big a deal IMHO, but, > yes, it does add something to the maintenance overhead. But that's all > the more true of a new module. > > I don't feel strongly about it, but if this isn't obviously getting > support from any committers, can we keep it as a third party library > for now? > > > On Tue, Oct 15, 2019 at 8:53 PM Weichen Xu <weichen...@databricks.com> > wrote: > > > > Hi Mats Rydberg, > > > > Although this dependency "org.opencypher:okapi-shade.okapi" was added > into spark, but Xiangrui raised two concerns (see above mail) about it, so > we'd better rethink on this and consider whether this is a good choice, so > I call this vote. > > > > Thanks! > > > > On Tue, Oct 15, 2019 at 10:56 PM Mats Rydberg <m...@neo4j.org.invalid> > wrote: > >> > >> Hello Weichen, community > >> > >> I'm sorry, I'm feeling a little bit confused about this vote. Is this > about the PR (https://github.com/apache/spark/pull/24490) that was merged > in early June and introduced the spark-graph module including the > okapi-shade dependency? > >> > >> Regarding the okapi-shade dependency which was developed as part of the > above PR work, some advice was offered by Scala experts at TripleQuote > which helped find a satisfactory solution. The shading mechanism used is > standard and very comparable to a Java library shading solution. > >> > >> The PR you link (https://github.com/apache/spark/pull/24297) is not > meant for merging. It is just a proof-of-concept branch containing a full > implementation of the system, which is kept up-to-date with the API > discussion on the currently proposed PR: > https://github.com/apache/spark/pull/24851. > >> > >> Thank you > >> Mats > >> > >> > >> On Tue, Oct 15, 2019 at 10:38 AM Weichen Xu <weichen...@databricks.com> > wrote: > >>> > >>> Hi everyone, > >>> > >>> I'd like to call a new vote on the issue: should we add dependency > "org.opencypher:okapi-shade.okapi" into spark ? The issue background is: > >>> > >>> Spark is going to add a big feature "Spark Graph", the prototypical > implementation is here > >>> https://github.com/apache/spark/pull/24297 > >>> which will introduce dependency org.opencypher:okapi-shade.okapi > >>> > >>> Xiangrui already mentioned 2 concerns on this dependency change: > >>>> > >>>> On the technical side, my main concern is the runtime dependency on > org.opencypher:okapi-shade.okapi depends on several Scala libraries. We > came out with the solution to shade a few Scala libraries to avoid > pollution. However, I'm not super confident that the approach is > sustainable for two reasons: a) there exists no proper shading libraries > for Scala, 2) We will have to wait for upgrades from those Scala libraries > before we can upgrade Spark to use a newer Scala version. So it would be > great if some Scala experts can help review the current implementation and > help assess the risk. > >>> > >>> > >>> So let's discuss and vote whether this is a good choice. > >>> Before this spark graph feature to get into spark ASAP, this issue > should be resolved first. > >>> > >>> This vote is open until next Tuseday (Oct. 22). > >>> > >>> [ ] +1: Accept the proposal > >>> [ ] +0 > >>> [ ] -1: I don't think this is a good idea because ... > >>> > >>> Thank you! > >>> > >>> Weichen > >>> > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
