Hmm
looks like a Build script issue:
I run the command with :
sbt/sbt clean *yarn/*test:compile
but errors came from
[error] 40 errors found
[error] (*yarn-stable*/compile:compile) Compilation failed
Chester
On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ches...@alpinenow.com> wrote:
> Hi, Sandy
>
> We do have some issue with this. The difference is in Yarn-Alpha and
> Yarn Stable ( I noticed that in the latest build, the module name has
> changed,
> yarn-alpha --> yarn
> yarn --> yarn-stable
> )
>
> For example: MRJobConfig.class
> the field:
> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
>
>
> In Yarn-Alpha : the field returns java.lang.String[]
>
> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>
> while in Yarn-Stable, it returns a String
>
> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>
> So in ClientBaseSuite.scala
>
> The following code:
>
> val knownDefMRAppCP: Seq[String] =
> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
>
> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> Seq[String]())(a =>
> *a.split(",")*)
>
>
> works for yarn-stable, but doesn't work for yarn-alpha.
>
> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
> believe this can be refactored to yarn-alpha module and make different
> tests according different API signatures.
>
> I just update the master branch and build doesn't even compile for
> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test passed.
>
>
> Does the Spark Jenkins job run against yarn-alpha ?
>
>
>
>
>
> Here is output from yarn-alpha compilation:
>
> I got the 40 compilation errors.
>
> sbt/sbt clean yarn/test:compile
>
> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home as
> default JAVA_HOME.
>
> Note, this will be overridden by -java-home if it is set.
>
> [info] Loading project definition from
> /Users/chester/projects/spark/project/project
>
> [info] Loading project definition from
> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
>
> [warn] Multiple resolvers having different access mechanism configured
> with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate
> project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
>
> [info] Loading project definition from
> /Users/chester/projects/spark/project
>
> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
> -Dhadoop.version=2.0.5-alpha
>
> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
>
> [info] Set current project to spark-parent (in build
> file:/Users/chester/projects/spark/)
>
> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
>
> [info] Updating {file:/Users/chester/projects/spark/}core...
>
> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>
> [info] Done updating.
>
> [info] Updating {file:/Users/chester/projects/spark/}yarn...
>
> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
>
> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>
> [info] Done updating.
>
> [info] Resolving commons-net#commons-net;3.1 ...
>
> [info] Compiling 358 Scala sources and 34 Java sources to
> /Users/chester/projects/spark/core/target/scala-2.10/classes...
>
> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>
> [info] Done updating.
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
> corresponding Javadoc for more information.
>
> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
> attemptId)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewHadoopJob(hadoopConfiguration)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewHadoopJob(conf)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
> attemptID))
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
> method makeQualified in class Path is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] outputPath.makeQualified(fs)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] if (!fs.getFileStatus(path).isDir) {
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] val logDirs = if (logStatus != null)
> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] if (file.isDir) 0L else file.getLen
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
> method getDefaultReplication in class FileSystem is deprecated: see
> corresponding Javadoc for more information.
>
> [warn] fs.create(tempOutputPath, false, bufferSize,
> fs.getDefaultReplication, blockSize)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true, splitId),
> attemptId)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new Job(conf)
>
> [warn] ^
>
> [warn] 14 warnings found
>
> [warn] Note:
> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
> uses unchecked or unsafe operations.
>
> [warn] Note: Recompile with -Xlint:unchecked for details.
>
> [info] Compiling 15 Scala sources to
> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.YarnClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
> not found: value YarnClient
>
> [error] val yarnClient = YarnClient.createYarnClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import
> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
> object util is not a member of package org.apache.hadoop.yarn.webapp
>
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
> value RM_AM_MAX_ATTEMPTS is not a member of object
> org.apache.hadoop.yarn.conf.YarnConfiguration
>
> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
> not found: type AMRMClient
>
> [error] private var amClient: AMRMClient[ContainerRequest] = _
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
> not found: value AMRMClient
>
> [error] amClient = AMRMClient.createAMRMClient()
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
> not found: value WebAppUtils
>
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
> not found: type AMRMClient
>
> [error] amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
> not found: type AMRMClient
>
> [error] amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
> not found: type AMRMClient
>
> [error] amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
> value CONTAINER_ID is not a member of object
> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
>
> [error] val containerIdString = System.getenv(
> ApplicationConstants.Environment.CONTAINER_ID.name())
>
> [error]
> ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
> value setTokens is not a member of
> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>
> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import
> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
> object util is not a member of package org.apache.hadoop.yarn.webapp
>
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
> not found: type AMRMClient
>
> [error] private var amClient: AMRMClient[ContainerRequest] = _
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
> not found: value AMRMClient
>
> [error] amClient = AMRMClient.createAMRMClient()
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
> not found: value WebAppUtils
>
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
> object ProtoUtils is not a member of package
> org.apache.hadoop.yarn.api.records.impl.pb
>
> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.NMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
> not found: type NMClient
>
> [error] var nmClient: NMClient = _
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
> not found: value NMClient
>
> [error] nmClient = NMClient.createNMClient()
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
> value setTokens is not a member of
> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>
> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
> object ApplicationMasterProtocol is not a member of package
> org.apache.hadoop.yarn.api
>
> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import
> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
> not found: type AMRMClient
>
> [error] val amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
> not found: type ContainerRequest
>
> [error] ): ArrayBuffer[ContainerRequest] = {
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
> not found: type ContainerRequest
>
> [error] hostContainers: ArrayBuffer[ContainerRequest]
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
> not found: type ContainerRequest
>
> [error] val requestedContainers = new
> ArrayBuffer[ContainerRequest](rackToCounts.size)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
> not found: type ContainerRequest
>
> [error] val containerRequests: List[ContainerRequest] =
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
> not found: type ContainerRequest
>
> [error] ): ArrayBuffer[ContainerRequest] = {
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
> not found: type ContainerRequest
>
> [error] val hostContainerRequests = new
> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
> not found: type ContainerRequest
>
> [error] val rackContainerRequests: List[ContainerRequest] =
> createRackResourceRequests(
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
> not found: type ContainerRequest
>
> [error] val containerRequestBuffer = new
> ArrayBuffer[ContainerRequest](
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
> not found: type ContainerRequest
>
> [error] ): ArrayBuffer[ContainerRequest] = {
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
> value newInstance is not a member of object
> org.apache.hadoop.yarn.api.records.Resource
>
> [error] val resource = Resource.newInstance(memoryRequest,
> executorCores)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
> not found: type ContainerRequest
>
> [error] val requests = new ArrayBuffer[ContainerRequest]()
>
> [error] ^
>
> [error] 40 errors found
>
> [error] (yarn-stable/compile:compile) Compilation failed
>
> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sandy.r...@cloudera.com>
> wrote:
>
>> Hi Ron,
>>
>> I just checked and this bug is fixed in recent releases of Spark.
>>
>> -Sandy
>>
>>
>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ches...@alpinenow.com>
>> wrote:
>>
>>> Ron,
>>> Which distribution and Version of Hadoop are you using ?
>>>
>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
>>> 2.3.0-cdh5.0.0),
>>>
>>> MRJobConfig does have the field :
>>>
>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>
>>> Chester
>>>
>>>
>>>
>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zlgonza...@yahoo.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I was doing programmatic submission of Spark yarn jobs and I saw code
>>>> in ClientBase.getDefaultYarnApplicationClasspath():
>>>>
>>>> val field =
>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>>>> MRJobConfig doesn't have this field so the created launch env is
>>>> incomplete. Workaround is to set yarn.application.classpath with the value
>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>>>
>>>> This results in having the spark job hang if the submission config is
>>>> different from the default config. For example, if my resource manager port
>>>> is 8050 instead of 8030, then the spark app is not able to register itself
>>>> and stays in ACCEPTED state.
>>>>
>>>> I can easily fix this by changing this to YarnConfiguration instead of
>>>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>>>
>>>> Thanks,
>>>> Ron
>>>>
>>>> Sent from my iPhone
>>>
>>>
>>>
>>
>
