Hey Reymond, Let's move this discussion out of this thread and into the associated JIRA. I'll write up our current approach over there.
https://spark-project.atlassian.net/browse/SPARK-995 - Patrick On Thu, Dec 12, 2013 at 5:56 PM, Liu, Raymond <[email protected]> wrote: > Hi Patrick > > So what's the plan for support Yarn 2.2 in 0.9? As far as I can > see, if you want to support both 2.2 and 2.0 , due to protobuf version > incompatible issue. You need two version of akka anyway. > > Akka 2.3-M1 looks like have a little bit change in API, we > probably could isolate the code like what we did on yarn part API. I > remember that it is mentioned that to use reflection for different API is > preferred. So the purpose to use reflection is to use one release bin jar > to support both version of Hadoop/Yarn on runtime, instead of build > different bin jar on compile time? > > Then all code related to hadoop will also be built in separate > modules for loading on demand? This sounds to me involve a lot of works. > And you still need to have shim layer and separate code for different > version API and depends on different version Akka etc. Sounds like and even > strict demands versus our current approaching on master, and with dynamic > class loader in addition, And the problem we are facing now are still there? > > Best Regards, > Raymond Liu > > -----Original Message----- > From: Patrick Wendell [mailto:[email protected]] > Sent: Thursday, December 12, 2013 5:13 PM > To: [email protected] > Subject: Re: Scala 2.10 Merge > > Also - the code is still there because of a recent merge that took in some > newer changes... we'll be removing it for the final merge. > > > On Thu, Dec 12, 2013 at 1:12 AM, Patrick Wendell <[email protected]> > wrote: > > > Hey Raymond, > > > > This won't work because AFAIK akka 2.3-M1 is not binary compatible > > with akka 2.2.3 (right?). For all of the non-yarn 2.2 versions we need > > to still use the older protobuf library, so we'd need to support both. > > > > I'd also be concerned about having a reference to a non-released > > version of akka. Akka is the source of our hardest-to-find bugs and > > simultaneously trying to support 2.2.3 and 2.3-M1 is a bit daunting. > > Of course, if you are building off of master you can maintain a fork > that uses this. > > > > - Patrick > > > > > > On Thu, Dec 12, 2013 at 12:42 AM, Liu, Raymond <[email protected] > >wrote: > > > >> Hi Patrick > >> > >> What does that means for drop YARN 2.2? seems codes are still > >> there. You mean if build upon 2.2 it will break, and won't and work > right? > >> Since the home made akka build on scala 2.10 are not there. While, if > >> for this case, can we just use akka 2.3-M1 which run on protobuf 2.5 > >> for replacement? > >> > >> Best Regards, > >> Raymond Liu > >> > >> > >> -----Original Message----- > >> From: Patrick Wendell [mailto:[email protected]] > >> Sent: Thursday, December 12, 2013 4:21 PM > >> To: [email protected] > >> Subject: Scala 2.10 Merge > >> > >> Hi Developers, > >> > >> In the next few days we are planning to merge Scala 2.10 support into > >> Spark. For those that haven't been following this, Prashant Sharma > >> has been maintaining the scala-2.10 branch of Spark for several > >> months. This branch is current with master and has been reviewed for > merging: > >> > >> https://github.com/apache/incubator-spark/tree/scala-2.10 > >> > >> Scala 2.10 support is one of the most requested features for Spark - > >> it will be great to get this into Spark 0.9! Please note that *Scala > >> 2.10 is not binary compatible with Scala 2.9*. With that in mind, I > >> wanted to give a few heads-up/requests to developers: > >> > >> If you are developing applications on top of Spark's master branch, > >> those will need to migrate to Scala 2.10. You may want to download > >> and test the current scala-2.10 branch in order to make sure you will > >> be okay as Spark developments move forward. Of course, you can always > >> stick with the current master commit and be fine (I'll cut a tag when > >> we do the merge in order to delineate where the version changes). > >> Please open new threads on the dev list to report and discuss any > issues. > >> > >> This merge will temporarily drop support for YARN 2.2 on the master > >> branch. > >> This is because the workaround we used was only compiled for Scala 2.9. > >> We are going to come up with a more robust solution to YARN 2.2 > >> support before releasing 0.9. > >> > >> Going forward, we will continue to make maintenance releases on > >> branch-0.8 which will remain compatible with Scala 2.9. > >> > >> For those interested, the primary code changes in this merge are > >> upgrading the akka version, changing the use of Scala 2.9's > >> ClassManifest construct to Scala 2.10's ClassTag, and updating the > >> spark shell to work with Scala 2.10's repl. > >> > >> - Patrick > >> > > > > >
