Whoohoo! Great job everyone especially Prashant!
— Sent from Mailbox for iPhone On Sat, Dec 14, 2013 at 10:59 AM, Patrick Wendell <[email protected]> wrote: > Alright I just merged this in - so Spark is officially "Scala 2.10" > from here forward. > For reference I cut a new branch called scala-2.9 with the commit > immediately prior to the merge: > https://git-wip-us.apache.org/repos/asf/incubator-spark/repo?p=incubator-spark.git;a=shortlog;h=refs/heads/scala-2.9 > - Patrick > On Thu, Dec 12, 2013 at 8:26 PM, Patrick Wendell <[email protected]> wrote: >> Hey Reymond, >> >> Let's move this discussion out of this thread and into the associated JIRA. >> I'll write up our current approach over there. >> >> https://spark-project.atlassian.net/browse/SPARK-995 >> >> - Patrick >> >> >> On Thu, Dec 12, 2013 at 5:56 PM, Liu, Raymond <[email protected]> wrote: >>> >>> Hi Patrick >>> >>> So what's the plan for support Yarn 2.2 in 0.9? As far as I can >>> see, if you want to support both 2.2 and 2.0 , due to protobuf version >>> incompatible issue. You need two version of akka anyway. >>> >>> Akka 2.3-M1 looks like have a little bit change in API, we >>> probably could isolate the code like what we did on yarn part API. I >>> remember that it is mentioned that to use reflection for different API is >>> preferred. So the purpose to use reflection is to use one release bin jar to >>> support both version of Hadoop/Yarn on runtime, instead of build different >>> bin jar on compile time? >>> >>> Then all code related to hadoop will also be built in separate >>> modules for loading on demand? This sounds to me involve a lot of works. And >>> you still need to have shim layer and separate code for different version >>> API and depends on different version Akka etc. Sounds like and even strict >>> demands versus our current approaching on master, and with dynamic class >>> loader in addition, And the problem we are facing now are still there? >>> >>> Best Regards, >>> Raymond Liu >>> >>> -----Original Message----- >>> From: Patrick Wendell [mailto:[email protected]] >>> Sent: Thursday, December 12, 2013 5:13 PM >>> To: [email protected] >>> Subject: Re: Scala 2.10 Merge >>> >>> Also - the code is still there because of a recent merge that took in some >>> newer changes... we'll be removing it for the final merge. >>> >>> >>> On Thu, Dec 12, 2013 at 1:12 AM, Patrick Wendell <[email protected]> >>> wrote: >>> >>> > Hey Raymond, >>> > >>> > This won't work because AFAIK akka 2.3-M1 is not binary compatible >>> > with akka 2.2.3 (right?). For all of the non-yarn 2.2 versions we need >>> > to still use the older protobuf library, so we'd need to support both. >>> > >>> > I'd also be concerned about having a reference to a non-released >>> > version of akka. Akka is the source of our hardest-to-find bugs and >>> > simultaneously trying to support 2.2.3 and 2.3-M1 is a bit daunting. >>> > Of course, if you are building off of master you can maintain a fork >>> > that uses this. >>> > >>> > - Patrick >>> > >>> > >>> > On Thu, Dec 12, 2013 at 12:42 AM, Liu, Raymond >>> > <[email protected]>wrote: >>> > >>> >> Hi Patrick >>> >> >>> >> What does that means for drop YARN 2.2? seems codes are still >>> >> there. You mean if build upon 2.2 it will break, and won't and work >>> >> right? >>> >> Since the home made akka build on scala 2.10 are not there. While, if >>> >> for this case, can we just use akka 2.3-M1 which run on protobuf 2.5 >>> >> for replacement? >>> >> >>> >> Best Regards, >>> >> Raymond Liu >>> >> >>> >> >>> >> -----Original Message----- >>> >> From: Patrick Wendell [mailto:[email protected]] >>> >> Sent: Thursday, December 12, 2013 4:21 PM >>> >> To: [email protected] >>> >> Subject: Scala 2.10 Merge >>> >> >>> >> Hi Developers, >>> >> >>> >> In the next few days we are planning to merge Scala 2.10 support into >>> >> Spark. For those that haven't been following this, Prashant Sharma >>> >> has been maintaining the scala-2.10 branch of Spark for several >>> >> months. This branch is current with master and has been reviewed for >>> >> merging: >>> >> >>> >> https://github.com/apache/incubator-spark/tree/scala-2.10 >>> >> >>> >> Scala 2.10 support is one of the most requested features for Spark - >>> >> it will be great to get this into Spark 0.9! Please note that *Scala >>> >> 2.10 is not binary compatible with Scala 2.9*. With that in mind, I >>> >> wanted to give a few heads-up/requests to developers: >>> >> >>> >> If you are developing applications on top of Spark's master branch, >>> >> those will need to migrate to Scala 2.10. You may want to download >>> >> and test the current scala-2.10 branch in order to make sure you will >>> >> be okay as Spark developments move forward. Of course, you can always >>> >> stick with the current master commit and be fine (I'll cut a tag when >>> >> we do the merge in order to delineate where the version changes). >>> >> Please open new threads on the dev list to report and discuss any >>> >> issues. >>> >> >>> >> This merge will temporarily drop support for YARN 2.2 on the master >>> >> branch. >>> >> This is because the workaround we used was only compiled for Scala 2.9. >>> >> We are going to come up with a more robust solution to YARN 2.2 >>> >> support before releasing 0.9. >>> >> >>> >> Going forward, we will continue to make maintenance releases on >>> >> branch-0.8 which will remain compatible with Scala 2.9. >>> >> >>> >> For those interested, the primary code changes in this merge are >>> >> upgrading the akka version, changing the use of Scala 2.9's >>> >> ClassManifest construct to Scala 2.10's ClassTag, and updating the >>> >> spark shell to work with Scala 2.10's repl. >>> >> >>> >> - Patrick >>> >> >>> > >>> > >> >>
