@Sean and @Sandy
Thanks for the reply. I used to be able to see yarn-alpha and yarn
directories which corresponding to the modules.
I guess due to the recent SparkBuild.scala changes, I did not see
yarn-alpha (by default) and I thought yarn-alpha is renamed to "yarn" and
"yarn-stable" is the old yarn. So I compiled "yarn" against the
hadoop.version = 2.0.5-alpha. My mistake.
I tried
export SPARK_HADOOP_VERSION=2.0.5-alpha
sbt/sbt -Pyarn-alpha yarn-alpha/test
the compilation errors are all gone.
sbt/sbt -Pyarn-alpha projects
does show the yarn-alpha project, I did not realize this is dynamically
enabled based on yarn flag. Thanks Sean for pointing that out.
To Sandy's point, I am not trying to use alpha version of Yarn. I am
experimenting some changes in Yarn Client and refactoring code and just
want to make sure I am passing tests for both yarn-alpha and yarn-stable.
The yarn-alpha tests actually failing due to the yarn API changes in
MRJobConfig class.
as I mentioned in earlier email
The field
DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH
returns String in yarn-stable, but returns String array in yarn-alpha API.
So the method in ClientBaseSuite.scala
val knownDefMRAppCP: Seq[String] =
getFieldValue[String, Seq[String]](classOf[MRJobConfig],
"DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
Seq[String]())(a => a.split(","))
will fail for yarn-alpha.
sbt/sbt -Pyarn-alpha -Dhadoop.version=2.0.5-alpha yarn-alpha/test
...
4/07/17 07:07:16 INFO ClientBase: Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
[info] - default Yarn application classpath *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$1.apply$mcV$sp(ClientBaseSuite.scala:47)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$1.apply(ClientBaseSuite.scala:47)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$1.apply(ClientBaseSuite.scala:47)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - default MR application classpath *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$2.apply$mcV$sp(ClientBaseSuite.scala:51)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$2.apply(ClientBaseSuite.scala:51)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$2.apply(ClientBaseSuite.scala:51)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - resultant classpath for an application that defines a classpath
for YARN *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$3.apply$mcV$sp(ClientBaseSuite.scala:55)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$3.apply(ClientBaseSuite.scala:55)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$3.apply(ClientBaseSuite.scala:55)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - resultant classpath for an application that defines a classpath
for MR *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$4.apply$mcV$sp(ClientBaseSuite.scala:64)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$4.apply(ClientBaseSuite.scala:64)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$4.apply(ClientBaseSuite.scala:64)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - resultant classpath for an application that defines both
classpaths, YARN and MR *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$5.apply$mcV$sp(ClientBaseSuite.scala:73)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$5.apply(ClientBaseSuite.scala:73)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$5.apply(ClientBaseSuite.scala:73)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - Local jar URIs
On Thu, Jul 17, 2014 at 12:44 AM, Sandy Ryza <sandy.r...@cloudera.com>
wrote:
> To add, we've made some effort to yarn-alpha to work with the 2.0.x line,
> but this was a time when YARN went through wild API changes. The only line
> that the yarn-alpha profile is guaranteed to work against is the 0.23 line.
>
>
> On Thu, Jul 17, 2014 at 12:40 AM, Sean Owen <so...@cloudera.com> wrote:
>
> > Are you setting -Pyarn-alpha? ./sbt/sbt -Pyarn-alpha, followed by
> > "projects", shows it as a module. You should only build yarn-stable
> > *or* yarn-alpha at any given time.
> >
> > I don't remember the modules changing in a while. 'yarn-alpha' is for
> > YARN before it stabilized, circa early Hadoop 2.0.x. 'yarn-stable' is
> > for beta and stable YARN, circa late Hadoop 2.0.x and onwards. 'yarn'
> > is code common to both, so should compile with yarn-alpha.
> >
> > What's the compile error, and are you setting yarn.version? the
> > default is to use hadoop.version, but that defaults to 1.0.4 and there
> > is no such YARN.
> >
> > Unless I missed it, I only see compile errors in yarn-stable, and you
> > are trying to compile vs YARN alpha versions no?
> >
> > On Thu, Jul 17, 2014 at 5:39 AM, Chester Chen <ches...@alpinenow.com>
> > wrote:
> > > Looking further, the yarn and yarn-stable are both for the stable
> version
> > > of Yarn, that explains the compilation errors when using 2.0.5-alpha
> > > version of hadoop.
> > >
> > > the module yarn-alpha ( although is still on SparkBuild.scala), is no
> > > longer there in sbt console.
> > >
> > >
> > >> projects
> > >
> > > [info] In file:/Users/chester/projects/spark/
> > >
> > > [info] assembly
> > >
> > > [info] bagel
> > >
> > > [info] catalyst
> > >
> > > [info] core
> > >
> > > [info] examples
> > >
> > > [info] graphx
> > >
> > > [info] hive
> > >
> > > [info] mllib
> > >
> > > [info] oldDeps
> > >
> > > [info] repl
> > >
> > > [info] spark
> > >
> > > [info] sql
> > >
> > > [info] streaming
> > >
> > > [info] streaming-flume
> > >
> > > [info] streaming-kafka
> > >
> > > [info] streaming-mqtt
> > >
> > > [info] streaming-twitter
> > >
> > > [info] streaming-zeromq
> > >
> > > [info] tools
> > >
> > > [info] yarn
> > >
> > > [info] * yarn-stable
> > >
> > >
> > > On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ches...@alpinenow.com>
> > wrote:
> > >
> > >> Hmm
> > >> looks like a Build script issue:
> > >>
> > >> I run the command with :
> > >>
> > >> sbt/sbt clean *yarn/*test:compile
> > >>
> > >> but errors came from
> > >>
> > >> [error] 40 errors found
> > >>
> > >> [error] (*yarn-stable*/compile:compile) Compilation failed
> > >>
> > >>
> > >> Chester
> > >>
> > >>
> > >> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ches...@alpinenow.com>
> > >> wrote:
> > >>
> > >>> Hi, Sandy
> > >>>
> > >>> We do have some issue with this. The difference is in Yarn-Alpha
> > and
> > >>> Yarn Stable ( I noticed that in the latest build, the module name has
> > >>> changed,
> > >>> yarn-alpha --> yarn
> > >>> yarn --> yarn-stable
> > >>> )
> > >>>
> > >>> For example: MRJobConfig.class
> > >>> the field:
> > >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
> > >>>
> > >>>
> > >>> In Yarn-Alpha : the field returns java.lang.String[]
> > >>>
> > >>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> > >>>
> > >>> while in Yarn-Stable, it returns a String
> > >>>
> > >>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> > >>>
> > >>> So in ClientBaseSuite.scala
> > >>>
> > >>> The following code:
> > >>>
> > >>> val knownDefMRAppCP: Seq[String] =
> > >>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
> > >>>
> > >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> > >>> Seq[String]())(a =>
> > >>> *a.split(",")*)
> > >>>
> > >>>
> > >>> works for yarn-stable, but doesn't work for yarn-alpha.
> > >>>
> > >>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago.
> I
> > >>> believe this can be refactored to yarn-alpha module and make
> different
> > >>> tests according different API signatures.
> > >>>
> > >>> I just update the master branch and build doesn't even compile for
> > >>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test
> > passed.
> > >>>
> > >>>
> > >>> Does the Spark Jenkins job run against yarn-alpha ?
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Here is output from yarn-alpha compilation:
> > >>>
> > >>> I got the 40 compilation errors.
> > >>>
> > >>> sbt/sbt clean yarn/test:compile
> > >>>
> > >>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home
> > as
> > >>> default JAVA_HOME.
> > >>>
> > >>> Note, this will be overridden by -java-home if it is set.
> > >>>
> > >>> [info] Loading project definition from
> > >>> /Users/chester/projects/spark/project/project
> > >>>
> > >>> [info] Loading project definition from
> > >>>
> >
> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
> > >>>
> > >>> [warn] Multiple resolvers having different access mechanism
> configured
> > >>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove
> > duplicate
> > >>> project resolvers (`resolvers`) or rename publishing resolver
> > (`publishTo`).
> > >>>
> > >>> [info] Loading project definition from
> > >>> /Users/chester/projects/spark/project
> > >>>
> > >>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
> > >>> -Dhadoop.version=2.0.5-alpha
> > >>>
> > >>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
> > >>>
> > >>> [info] Set current project to spark-parent (in build
> > >>> file:/Users/chester/projects/spark/)
> > >>>
> > >>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
> > >>>
> > >>> [info] Updating {file:/Users/chester/projects/spark/}core...
> > >>>
> > >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> > >>>
> > >>> [info] Done updating.
> > >>>
> > >>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
> > >>>
> > >>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
> > >>>
> > >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> > >>>
> > >>> [info] Done updating.
> > >>>
> > >>> [info] Resolving commons-net#commons-net;3.1 ...
> > >>>
> > >>> [info] Compiling 358 Scala sources and 34 Java sources to
> > >>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
> > >>>
> > >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> > >>>
> > >>> [info] Done updating.
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
> > >>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
> > >>> corresponding Javadoc for more information.
> > >>>
> > >>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
> > >>> attemptId)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewHadoopJob(conf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
> > >>> constructor TaskID in class TaskID is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] new TaskAttemptID(new TaskID(jID.value, true,
> splitID),
> > >>> attemptID))
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
> > >>> method makeQualified in class Path is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] outputPath.makeQualified(fs)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] if (!fs.getFileStatus(path).isDir) {
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] val logDirs = if (logStatus != null)
> > >>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
> > >>>
> > >>> [warn]
> ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] if (file.isDir) 0L else file.getLen
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
> > >>> method getDefaultReplication in class FileSystem is deprecated: see
> > >>> corresponding Javadoc for more information.
> > >>>
> > >>> [warn] fs.create(tempOutputPath, false, bufferSize,
> > >>> fs.getDefaultReplication, blockSize)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
> > >>> constructor TaskID in class TaskID is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
> > >>> splitId), attemptId)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new Job(conf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn] 14 warnings found
> > >>>
> > >>> [warn] Note:
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
> > >>> uses unchecked or unsafe operations.
> > >>>
> > >>> [warn] Note: Recompile with -Xlint:unchecked for details.
> > >>>
> > >>> [info] Compiling 15 Scala sources to
> > >>>
> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
> > >>> not found: value YarnClient
> > >>>
> > >>> [error] val yarnClient = YarnClient.createYarnClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import
> > >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
> > >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
> > >>> value RM_AM_MAX_ATTEMPTS is not a member of object
> > >>> org.apache.hadoop.yarn.conf.YarnConfiguration
> > >>>
> > >>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> > >>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
> > >>> not found: value AMRMClient
> > >>>
> > >>> [error] amClient = AMRMClient.createAMRMClient()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
> > >>> not found: value WebAppUtils
> > >>>
> > >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
> > >>> value CONTAINER_ID is not a member of object
> > >>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
> > >>>
> > >>> [error] val containerIdString = System.getenv(
> > >>> ApplicationConstants.Environment.CONTAINER_ID.name())
> > >>>
> > >>> [error]
> > >>> ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
> > >>> value setTokens is not a member of
> > >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> > >>>
> > >>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import
> > >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
> > >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
> > >>> not found: value AMRMClient
> > >>>
> > >>> [error] amClient = AMRMClient.createAMRMClient()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
> > >>> not found: value WebAppUtils
> > >>>
> > >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
> > >>> object ProtoUtils is not a member of package
> > >>> org.apache.hadoop.yarn.api.records.impl.pb
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.NMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
> > >>> not found: type NMClient
> > >>>
> > >>> [error] var nmClient: NMClient = _
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
> > >>> not found: value NMClient
> > >>>
> > >>> [error] nmClient = NMClient.createNMClient()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
> > >>> value setTokens is not a member of
> > >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> > >>>
> > >>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
> > >>> object ApplicationMasterProtocol is not a member of package
> > >>> org.apache.hadoop.yarn.api
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import
> > >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] val amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] ): ArrayBuffer[ContainerRequest] = {
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] hostContainers: ArrayBuffer[ContainerRequest]
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val requestedContainers = new
> > >>> ArrayBuffer[ContainerRequest](rackToCounts.size)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val containerRequests: List[ContainerRequest] =
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] ): ArrayBuffer[ContainerRequest] = {
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val hostContainerRequests = new
> > >>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val rackContainerRequests: List[ContainerRequest] =
> > >>> createRackResourceRequests(
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val containerRequestBuffer = new
> > >>> ArrayBuffer[ContainerRequest](
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] ): ArrayBuffer[ContainerRequest] = {
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
> > >>> value newInstance is not a member of object
> > >>> org.apache.hadoop.yarn.api.records.Resource
> > >>>
> > >>> [error] val resource = Resource.newInstance(memoryRequest,
> > >>> executorCores)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val requests = new ArrayBuffer[ContainerRequest]()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error] 40 errors found
> > >>>
> > >>> [error] (yarn-stable/compile:compile) Compilation failed
> > >>>
> > >>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sandy.r...@cloudera.com
> >
> > >>> wrote:
> > >>>
> > >>>> Hi Ron,
> > >>>>
> > >>>> I just checked and this bug is fixed in recent releases of Spark.
> > >>>>
> > >>>> -Sandy
> > >>>>
> > >>>>
> > >>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <
> ches...@alpinenow.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Ron,
> > >>>>> Which distribution and Version of Hadoop are you using ?
> > >>>>>
> > >>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
> > >>>>> 2.3.0-cdh5.0.0),
> > >>>>>
> > >>>>> MRJobConfig does have the field :
> > >>>>>
> > >>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> > >>>>>
> > >>>>> Chester
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <
> zlgonza...@yahoo.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi,
> > >>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
> > >>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
> > >>>>>>
> > >>>>>> val field =
> > >>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
> > >>>>>> MRJobConfig doesn't have this field so the created launch env is
> > >>>>>> incomplete. Workaround is to set yarn.application.classpath with
> > the value
> > >>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
> > >>>>>>
> > >>>>>> This results in having the spark job hang if the submission config
> > is
> > >>>>>> different from the default config. For example, if my resource
> > manager port
> > >>>>>> is 8050 instead of 8030, then the spark app is not able to
> register
> > itself
> > >>>>>> and stays in ACCEPTED state.
> > >>>>>>
> > >>>>>> I can easily fix this by changing this to YarnConfiguration
> instead
> > of
> > >>>>>> MRJobConfig but was wondering what the steps are for submitting a
> > fix.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Ron
> > >>>>>>
> > >>>>>> Sent from my iPhone
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>
