After some collaboration with other community members, we have created a initial draft for Spark Extras which is available for review at
https://docs.google.com/document/d/1zRFGG4414LhbKlGbYncZ13nyX34Rw4sfWhZRA5YBtIE/edit?usp=sharing We would like to invite other community members to participate in the project, particularly the Spark Committers and PMC (feel free to express interest and I will update the proposal). Another option here is just to give ALL Spark committers write access to "Spark Extras". We also have couple asks from the Spark PMC : - Permission to use "Spark Extras" as the project name. We already checked this with Apache Brand Management, and the recommendation was to discuss and reach consensus with the Spark PMC. - We would also want to check with the Spark PMC that, in case of successfully creation of "Spark Extras", if the PMC would be willing to continue the development of the remaining connectors that stayed in Spark 2.0 codebase in the "Spark Extras" project. Thanks in advance, and we welcome any feedback around this proposal before we present to the Apache Board for consideration. On Sat, Mar 26, 2016 at 10:07 AM, Luciano Resende <luckbr1...@gmail.com> wrote: > I believe some of this has been resolved in the context of some parts that > had interest in one extra connector, but we still have a few removed, and > as you mentioned, we still don't have a simple way or willingness to manage > and be current on new packages like kafka. And based on the fact that this > thread is still alive, I believe that other community members might have > other concerns as well. > > After some thought, I believe having a separate project (what was > mentioned here as Spark Extras) to handle Spark Connectors and Spark > add-ons in general could be very beneficial to Spark and the overall Spark > community, which would have a central place in Apache to collaborate around > related Spark components. > > Some of the benefits on this approach > > - Enables maintaining the connectors inside Apache, following the Apache > governance and release rules, while allowing Spark proper to focus on the > core runtime. > - Provides more flexibility in controlling the direction (currency) of the > existing connectors (e.g. willing to find a solution and maintain multiple > versions of same connectors like kafka 0.8x and 0.9x) > - Becomes a home for other types of Spark related connectors helping > expanding the community around Spark (e.g. Zeppelin see most of it's > current contribution around new/enhanced connectors) > > What are some requirements for Spark Extras to be successful: > > - Be up to date with Spark Trunk APIs (based on daily CIs against SNAPSHOT) > - Adhere to Spark release cycles (have a very little window compared to > Spark release) > - Be more open and flexible to the set of connectors it will accept and > maintain (e.g. also handle multiple versions like the kafka 0.9 issue we > have today) > > Where to start Spark Extras > > Depending on the interest here, we could follow the steps of (Apache > Arrow) and start this directly as a TLP, or start as an incubator project. > I would consider the first option first. > > Who would participate > > Have thought about this for a bit, and if we go to the direction of TLP, I > would say Spark Committers and Apache Members can request to participate as > PMC members, while other committers can request to become committers. Non > committers would be added based on meritocracy after the start of the > project. > > Project Name > > It would be ideal if we could have a project name that shows close ties to > Spark (e.g. Spark Extras or Spark Connectors) but we will need permission > and support from whoever is going to evaluate the project proposal (e.g. > Apache Board) > > > Thoughts ? > > Does anyone have any big disagreement or objection to moving into this > direction ? > > Otherwise, who would be interested in joining the project, so I can start > working on some concrete proposal ? > > > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
