+1 to option #2.
On Fri, Apr 25, 2014 at 3:45 PM, Josh Wills <[email protected]> wrote: > Hey all, > > I've been working on doing a new release candidate, and I think that we're > going to need to upgrade to Spark 0.9.1 in order for Crunch-on-Spark to > work in Spark standalone/Spark-on-YARN on top of Hadoop, as opposed to the > local mode that works now. There are a couple of implications to this, but > the big one is that Spark 0.9.1 is only developed against Scala 2.10, which > would mean that we would need to switch over to 2.10 for at least > crunch-spark, and it would seem much easier to me for us to upgrade Scrunch > to 2.10 as well. > > Couple of options here off the top of my head: > 1) Do the upgrade to Scala 2.10/Spark 0.9.1 in 0.10.0, but not 0.8.3. > 2) Do the upgrade in both 0.8.3 and 0.10.0. > 3) Only upgrade Spark to 2.10 (in either 0.8.3 or 0.10.0), but still have > Scrunch build against 2.9.3. > > I'm in favor of #2 myself, but am open to suggestion here, and would > especially like to know if anyone feels strongly about staying on 2.9.3 for > any reason. > > Thanks! > Josh > > Director of Data Science > Cloudera <http://www.cloudera.com> > Twitter: @josh_wills <http://twitter.com/josh_wills> >
