Yes. Awesome. Great job guys. Sam Bessalah
> On Dec 14, 2013, at 9:59 AM, Patrick Wendell <pwend...@gmail.com> wrote: > > Alright I just merged this in - so Spark is officially "Scala 2.10" > from here forward. > > For reference I cut a new branch called scala-2.9 with the commit > immediately prior to the merge: > https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=shortlog;h=refs/heads/scala-2.9 > > - Patrick > >> On Thu, Dec 12, 2013 at 8:26 PM, Patrick Wendell <pwend...@gmail.com> wrote: >> Hey Reymond, >> >> Let's move this discussion out of this thread and into the associated JIRA. >> I'll write up our current approach over there. >> >> https://spark-project.atlassian.net/browse/SPARK-995 >> >> - Patrick >> >> >>> On Thu, Dec 12, 2013 at 5:56 PM, Liu, Raymond <raymond....@intel.com> wrote: >>> >>> Hi Patrick >>> >>> So what's the plan for support Yarn 2.2 in 0.9? As far as I can >>> see, if you want to support both 2.2 and 2.0 , due to protobuf version >>> incompatible issue. You need two version of akka anyway. >>> >>> Akka 2.3-M1 looks like have a little bit change in API, we >>> probably could isolate the code like what we did on yarn part API. I >>> remember that it is mentioned that to use reflection for different API is >>> preferred. So the purpose to use reflection is to use one release bin jar to >>> support both version of Hadoop/Yarn on runtime, instead of build different >>> bin jar on compile time? >>> >>> Then all code related to hadoop will also be built in separate >>> modules for loading on demand? This sounds to me involve a lot of works. And >>> you still need to have shim layer and separate code for different version >>> API and depends on different version Akka etc. Sounds like and even strict >>> demands versus our current approaching on master, and with dynamic class >>> loader in addition, And the problem we are facing now are still there? >>> >>> Best Regards, >>> Raymond Liu >>> >>> -----Original Message----- >>> From: Patrick Wendell [mailto:pwend...@gmail.com] >>> Sent: Thursday, December 12, 2013 5:13 PM >>> To: dev@spark.incubator.apache.org >>> Subject: Re: Scala 2.10 Merge >>> >>> Also - the code is still there because of a recent merge that took in some >>> newer changes... we'll be removing it for the final merge. >>> >>> >>> On Thu, Dec 12, 2013 at 1:12 AM, Patrick Wendell <pwend...@gmail.com> >>> wrote: >>> >>>> Hey Raymond, >>>> >>>> This won't work because AFAIK akka 2.3-M1 is not binary compatible >>>> with akka 2.2.3 (right?). For all of the non-yarn 2.2 versions we need >>>> to still use the older protobuf library, so we'd need to support both. >>>> >>>> I'd also be concerned about having a reference to a non-released >>>> version of akka. Akka is the source of our hardest-to-find bugs and >>>> simultaneously trying to support 2.2.3 and 2.3-M1 is a bit daunting. >>>> Of course, if you are building off of master you can maintain a fork >>>> that uses this. >>>> >>>> - Patrick >>>> >>>> >>>> On Thu, Dec 12, 2013 at 12:42 AM, Liu, Raymond >>>> <raymond....@intel.com>wrote: >>>> >>>>> Hi Patrick >>>>> >>>>> What does that means for drop YARN 2.2? seems codes are still >>>>> there. You mean if build upon 2.2 it will break, and won't and work >>>>> right? >>>>> Since the home made akka build on scala 2.10 are not there. While, if >>>>> for this case, can we just use akka 2.3-M1 which run on protobuf 2.5 >>>>> for replacement? >>>>> >>>>> Best Regards, >>>>> Raymond Liu >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Patrick Wendell [mailto:pwend...@gmail.com] >>>>> Sent: Thursday, December 12, 2013 4:21 PM >>>>> To: dev@spark.incubator.apache.org >>>>> Subject: Scala 2.10 Merge >>>>> >>>>> Hi Developers, >>>>> >>>>> In the next few days we are planning to merge Scala 2.10 support into >>>>> Spark. For those that haven't been following this, Prashant Sharma >>>>> has been maintaining the scala-2.10 branch of Spark for several >>>>> months. This branch is current with master and has been reviewed for >>>>> merging: >>>>> >>>>> https://github.com/apache/incubator-spark/tree/scala-2.10 >>>>> >>>>> Scala 2.10 support is one of the most requested features for Spark - >>>>> it will be great to get this into Spark 0.9! Please note that *Scala >>>>> 2.10 is not binary compatible with Scala 2.9*. With that in mind, I >>>>> wanted to give a few heads-up/requests to developers: >>>>> >>>>> If you are developing applications on top of Spark's master branch, >>>>> those will need to migrate to Scala 2.10. You may want to download >>>>> and test the current scala-2.10 branch in order to make sure you will >>>>> be okay as Spark developments move forward. Of course, you can always >>>>> stick with the current master commit and be fine (I'll cut a tag when >>>>> we do the merge in order to delineate where the version changes). >>>>> Please open new threads on the dev list to report and discuss any >>>>> issues. >>>>> >>>>> This merge will temporarily drop support for YARN 2.2 on the master >>>>> branch. >>>>> This is because the workaround we used was only compiled for Scala 2.9. >>>>> We are going to come up with a more robust solution to YARN 2.2 >>>>> support before releasing 0.9. >>>>> >>>>> Going forward, we will continue to make maintenance releases on >>>>> branch-0.8 which will remain compatible with Scala 2.9. >>>>> >>>>> For those interested, the primary code changes in this merge are >>>>> upgrading the akka version, changing the use of Scala 2.9's >>>>> ClassManifest construct to Scala 2.10's ClassTag, and updating the >>>>> spark shell to work with Scala 2.10's repl. >>>>> >>>>> - Patrick >> >>
