[ ] Use Spark 1 & Spark 2 Support Branch [X] Use Spark 2 Only Branch On Sun, Nov 19, 2017 at 1:46 PM Amit Sela <amitsel...@gmail.com> wrote:
> [X] Use Spark 2 Only Branch > > On Sun, Nov 19, 2017, 02:46 Reuven Lax <re...@google.com.invalid> wrote: > > > [ ] Use Spark 1 & Spark 2 Support Branch > > [X] Use Spark 2 Only Branch > > > > On Sat, Nov 18, 2017 at 1:54 AM, Ben Sidhom <sid...@google.com.invalid> > > wrote: > > > > > [ ] Use Spark 1 & Spark 2 Support Branch > > > [X] Use Spark 2 Only Branch > > > > > > On Fri, Nov 17, 2017 at 9:46 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > > > [ ] Use Spark 1 & Spark 2 Support Branch > > > > [X] Use Spark 2 Only Branch > > > > > > > > On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré < > j...@nanthrax.net > > > > > > > wrote: > > > > > > > > > Hi guys, > > > > > > > > > > To illustrate the current discussion about Spark versions support, > > you > > > > can > > > > > take a look on: > > > > > > > > > > -- > > > > > Spark 1 & Spark 2 Support Branch > > > > > > > > > > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES > > > > > > > > > > This branch contains a Spark runner common module compatible with > > both > > > > > Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2 > > > > > modules/artifacts containing just a pom.xml to define the > > dependencies > > > > set. > > > > > > > > > > -- > > > > > Spark 2 Only Branch > > > > > > > > > > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY > > > > > > > > > > This branch is an upgrade to Spark 2.x and "drop" support of Spark > > 1.x. > > > > > > > > > > As I'm ready to merge one of the other in the PR, I would like to > > > > complete > > > > > the vote/discussion pretty soon. > > > > > > > > > > Correct me if I'm wrong, but it seems that the preference is to > drop > > > > Spark > > > > > 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch). > > > > > > > > > > I would like to call a final vote to act the merge I will do: > > > > > > > > > > [ ] Use Spark 1 & Spark 2 Support Branch > > > > > [ ] Use Spark 2 Only Branch > > > > > > > > > > This informal vote is open for 48 hours. > > > > > > > > > > Please, let me know what your preference is. > > > > > > > > > > Thanks ! > > > > > Regards > > > > > JB > > > > > > > > > > On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote: > > > > > > > > > >> Hi Beamers, > > > > >> > > > > >> I'm forwarding this discussion & vote from the dev mailing list to > > the > > > > >> user mailing list. > > > > >> The goal is to have your feedback as user. > > > > >> > > > > >> Basically, we have two options: > > > > >> 1. Right now, in the PR, we support both Spark 1.x and 2.x using > > three > > > > >> artifacts (common, spark1, spark2). You, as users, pick up spark1 > or > > > > spark2 > > > > >> in your dependencies set depending the Spark target version you > > want. > > > > >> 2. The other option is to upgrade and focus on Spark 2.x in Beam > > > 2.3.0. > > > > >> If you still want to use Spark 1.x, then, you will be stuck up to > > Beam > > > > >> 2.2.0. > > > > >> > > > > >> Thoughts ? > > > > >> > > > > >> Thanks ! > > > > >> Regards > > > > >> JB > > > > >> > > > > >> > > > > >> -------- Forwarded Message -------- > > > > >> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x > > > > >> Date: Wed, 8 Nov 2017 08:27:58 +0100 > > > > >> From: Jean-Baptiste Onofré <j...@nanthrax.net> > > > > >> Reply-To: dev@beam.apache.org > > > > >> To: dev@beam.apache.org > > > > >> > > > > >> Hi all, > > > > >> > > > > >> as you might know, we are working on Spark 2.x support in the > Spark > > > > >> runner. > > > > >> > > > > >> I'm working on a PR about that: > > > > >> > > > > >> https://github.com/apache/beam/pull/3808 > > > > >> > > > > >> Today, we have something working with both Spark 1.x and 2.x from > a > > > code > > > > >> standpoint, but I have to deal with dependencies. It's the first > > step > > > of > > > > >> the update as I'm still using RDD, the second step would be to > > support > > > > >> dataframe (but for that, I would need PCollection elements with > > > schemas, > > > > >> that's another topic on which Eugene, Reuven and I are > discussing). > > > > >> > > > > >> However, as all major distributions now ship Spark 2.x, I don't > > think > > > > >> it's required anymore to support Spark 1.x. > > > > >> > > > > >> If we agree, I will update and cleanup the PR to only support and > > > focus > > > > >> on Spark 2.x. > > > > >> > > > > >> So, that's why I'm calling for a vote: > > > > >> > > > > >> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only > > > > >> [ ] 0 (I don't care ;)) > > > > >> [ ] -1, I would like to still support Spark 1.x, and so having > > > > support > > > > >> of both Spark 1.x and 2.x (please provide specific comment) > > > > >> > > > > >> This vote is open for 48 hours (I have the commits ready, just > > waiting > > > > >> the end of the vote to push on the PR). > > > > >> > > > > >> Thanks ! > > > > >> Regards > > > > >> JB > > > > >> > > > > > > > > > > -- > > > > > Jean-Baptiste Onofré > > > > > jbono...@apache.org > > > > > http://blog.nanthrax.net > > > > > Talend - http://www.talend.com > > > > > > > > > > > > > > > > > > > > > -- > > > -Ben > > > > > >
