I don't know if ditching Spark 1 out right right now would be a great move given that a lot of the main support applications around spark haven't yet fully moved to Spark 2 yet. Yet alone have support for having a cluster with both. Oozie for example is still pre stable release for their Spark 1 and can't support a cluster with mixed Spark version. I think maybe doing as suggested above with the common, spark1, spark2 packaging might be best during this carry over phase. Maybe even just flag spark 1 as deprecated and just being maintained might be enough.
On Wed, Nov 8, 2017 at 10:25 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > Also, upgrading Spark 1 to 2 is generally easier than changing JVM > versions. For folks using YARN or the hosted environments it pretty much > trivial since you can effectively have distinct Spark clusters for each > job. > > On Wed, Nov 8, 2017 at 9:19 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > > > I'm +1 on dropping Spark 1. There are a lot of exciting improvements in > > Spark 2, and trying to write efficient code that runs between Spark 1 and > > Spark 2 is super painful in the long term. It would be one thing if there > > were a lot of people available to work on the Spark runners, but it seems > > like we'd be better spent focusing our energy on the future. > > > > I don't know a lot of folks who are stuck on Spark 1, and the few that I > > know are planning to migrate in the next few months anyways. > > > > Note: this is a non-binding vote as I'm not a committer or PMC member. > > > > On Wed, Nov 8, 2017 at 3:43 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > > >> Having both Spark1 and Spark2 modules would benefit wider user base. > >> > >> I would vote for that. > >> > >> Cheers > >> > >> On Wed, Nov 8, 2017 at 12:51 AM, Jean-Baptiste Onofré <j...@nanthrax.net> > >> wrote: > >> > >> > Hi Robert, > >> > > >> > Thanks for your feedback ! > >> > > >> > From an user perspective, with the current state of the PR, the same > >> > pipelines can run on both Spark 1.x and 2.x: the only difference is > the > >> > dependencies set. > >> > > >> > I'm calling the vote to get suck kind of feedback: if we consider > Spark > >> > 1.x still need to be supported, no problem, I will improve the PR to > >> have > >> > three modules (common, spark1, spark2) and let users pick the desired > >> > version. > >> > > >> > Let's wait a bit other feedbacks, I will update the PR accordingly. > >> > > >> > Regards > >> > JB > >> > > >> > > >> > On 11/08/2017 09:47 AM, Robert Bradshaw wrote: > >> > > >> >> I'm generally a -0.5 on this change, or at least doing so hastily. > >> >> > >> >> As with dropping Java 7 support, I think this should at least be > >> >> announced in release notes that we're considering dropping support in > >> >> the subsequent release, as this dev list likely does not reach a > >> >> substantial portion of the userbase. > >> >> > >> >> How much work is it to move from a Spark 1.x cluster to a Spark 2.x > >> >> cluster? I get the feeling it's not nearly as transparent as > upgrading > >> >> Java versions. Can Spark 1.x pipelines be run on Spark 2.x clusters, > >> >> or is a new cluster (and/or upgrading all pipelines) required (e.g. > >> >> for those who operate spark clusters shared among their many users)? > >> >> > >> >> Looks like the latest release of Spark 1.x was about a year ago, > >> >> overlapping a bit with the 2.x series which is coming up on 1.5 years > >> >> old, so I could see a lot of people still using 1.x even if 2.x is > >> >> clearly the future. But it sure doesn't seem very backwards > >> >> compatible. > >> >> > >> >> Mostly I'm not comfortable with dropping 1.x in the same release as > >> >> adding support for 2.x, giving no transition period, but could be > >> >> convinced if this transition is mostly a no-op or no one's still > using > >> >> 1.x. If there's non-trivial code complexity issues, I would perhaps > >> >> revisit the issue of having a single Spark Runner that does chooses > >> >> the backend implicitly in favor of simply having two runners which > >> >> share the code that's easy to share and diverge otherwise (which > seems > >> >> it would be much simpler both to implement and explain to users). I > >> >> would be OK with even letting the Spark 1.x runner be somewhat > >> >> stagnant (e.g. few or no new features) until we decide we can kill it > >> >> off. > >> >> > >> >> On Tue, Nov 7, 2017 at 11:27 PM, Jean-Baptiste Onofré < > j...@nanthrax.net > >> > > >> >> wrote: > >> >> > >> >>> Hi all, > >> >>> > >> >>> as you might know, we are working on Spark 2.x support in the Spark > >> >>> runner. > >> >>> > >> >>> I'm working on a PR about that: > >> >>> > >> >>> https://github.com/apache/beam/pull/3808 > >> >>> > >> >>> Today, we have something working with both Spark 1.x and 2.x from a > >> code > >> >>> standpoint, but I have to deal with dependencies. It's the first > step > >> of > >> >>> the > >> >>> update as I'm still using RDD, the second step would be to support > >> >>> dataframe > >> >>> (but for that, I would need PCollection elements with schemas, > that's > >> >>> another topic on which Eugene, Reuven and I are discussing). > >> >>> > >> >>> However, as all major distributions now ship Spark 2.x, I don't > think > >> >>> it's > >> >>> required anymore to support Spark 1.x. > >> >>> > >> >>> If we agree, I will update and cleanup the PR to only support and > >> focus > >> >>> on > >> >>> Spark 2.x. > >> >>> > >> >>> So, that's why I'm calling for a vote: > >> >>> > >> >>> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only > >> >>> [ ] 0 (I don't care ;)) > >> >>> [ ] -1, I would like to still support Spark 1.x, and so having > >> >>> support of > >> >>> both Spark 1.x and 2.x (please provide specific comment) > >> >>> > >> >>> This vote is open for 48 hours (I have the commits ready, just > waiting > >> >>> the > >> >>> end of the vote to push on the PR). > >> >>> > >> >>> Thanks ! > >> >>> Regards > >> >>> JB > >> >>> -- > >> >>> Jean-Baptiste Onofré > >> >>> jbono...@apache.org > >> >>> http://blog.nanthrax.net > >> >>> Talend - http://www.talend.com > >> >>> > >> >> > >> > -- > >> > Jean-Baptiste Onofré > >> > jbono...@apache.org > >> > http://blog.nanthrax.net > >> > Talend - http://www.talend.com > >> > > >> > > > > > > > > -- > > Twitter: https://twitter.com/holdenkarau > > > > > > -- > Twitter: https://twitter.com/holdenkarau > -- Nick Verbeck - NerdyNick ---------------------------------------------------- NerdyNick.com TrailsOffroad.com NoKnownBoundaries.com
