Hi JB,
[X ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only [ ] 0 (I don't care ;)) [ ] -1, I would like to still support Spark 1.x, and so having support of both Spark 1.x and 2.x (please provide specific comment) On 13 November 2017 at 10:32, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Beamers, > > I'm forwarding this discussion & vote from the dev mailing list to the > user mailing list. > The goal is to have your feedback as user. > > Basically, we have two options: > 1. Right now, in the PR, we support both Spark 1.x and 2.x using three > artifacts (common, spark1, spark2). You, as users, pick up spark1 or spark2 > in your dependencies set depending the Spark target version you want. > 2. The other option is to upgrade and focus on Spark 2.x in Beam 2.3.0. If > you still want to use Spark 1.x, then, you will be stuck up to Beam 2.2.0. > > Thoughts ? > > Thanks ! > Regards > JB > > > -------- Forwarded Message -------- > Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x > Date: Wed, 8 Nov 2017 08:27:58 +0100 > From: Jean-Baptiste Onofré <j...@nanthrax.net> > Reply-To: d...@beam.apache.org > To: d...@beam.apache.org > > Hi all, > > as you might know, we are working on Spark 2.x support in the Spark runner. > > I'm working on a PR about that: > > https://github.com/apache/beam/pull/3808 > > Today, we have something working with both Spark 1.x and 2.x from a code > standpoint, but I have to deal with dependencies. It's the first step of > the update as I'm still using RDD, the second step would be to support > dataframe (but for that, I would need PCollection elements with schemas, > that's another topic on which Eugene, Reuven and I are discussing). > > However, as all major distributions now ship Spark 2.x, I don't think it's > required anymore to support Spark 1.x. > > If we agree, I will update and cleanup the PR to only support and focus on > Spark 2.x. > > So, that's why I'm calling for a vote: > > [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only > [ ] 0 (I don't care ;)) > [ ] -1, I would like to still support Spark 1.x, and so having support > of both Spark 1.x and 2.x (please provide specific comment) > > This vote is open for 48 hours (I have the commits ready, just waiting the > end of the vote to push on the PR). > > Thanks ! > Regards > JB > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
