I’m curious, Viren, do you have a patch you could post to build this against YARN 2.1 / 2.2? It would be nice to see how big the changes are.
Matei On Sep 30, 2013, at 10:14 AM, viren kumar <vire...@gmail.com> wrote: > I was able to get Spark 0.8.0 to compile with Hadoop/Yarn 2.1.0-beta, by > following some of the changes described here: > http://hortonworks.com/blog/stabilizing-yarn-apis-for-apache-hadoop-2-beta-and-beyond/ > > That should help you build most of it. One change not covered there is the > change from ProtoUtils.convertFromProtoFormat(containerToken, cmAddress) to > ConverterUtils.convertFromYarn(containerToken, cmAddress). > > Not 100% sure that my changes are correct. > > Hope that helps, > Viren > > > On Sun, Sep 29, 2013 at 8:59 AM, Matei Zaharia <matei.zaha...@gmail.com> > wrote: > Hi Terence, > > YARN's API changed in an incompatible way in Hadoop 2.1.0, so I'd suggest > sticking with 2.0.x for now. We may create a different branch for this > version. Unfortunately due to the API change it may not be possible to > support this version while also supporting other widely-used versions like > 0.23.x. > > Matei > > On Sep 29, 2013, at 11:00 AM, Terance Dias <terance.d...@gmail.com> wrote: > > > > > Hi, I'm trying to build spark-0.8.0 with hadoop-2.1.0-beta. > > I have changed the following properties in SparkBuild.scala file. > > > > val DEFAULT_HADOOP_VERSION = "2.1.0-beta" > > val DEFAULT_YARN = true > > > > when i do sbt clean compile, I get an error saying > > > > [error] > > /usr/local/spark-0.8.0-incubating/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:42: > > not found: type AMRMProtocol > > [error] private var resourceManager: AMRMProtocol = null > > > > Thanks, > > Terance. > > > >
