I think you might be conflating two things. The first error you posted was because YARN didn't standardize the shuffle API in alpha versions so our spark-network-yarn module won't compile. We should just disable that module if yarn alpha is used. spark-network-yarn is a leaf in the intra-module dependency graph, and core doesn't depend on it.
This second error is something else. Maybe you are excluding network-shuffle instead of spark-network-yarn? On Fri, Nov 7, 2014 at 11:50 PM, Sean Owen <[email protected]> wrote: > Hm. Problem is, core depends directly on it: > > [error] > /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/SecurityManager.scala:25: > object sasl is not a member of package org.apache.spark.network > [error] import org.apache.spark.network.sasl.SecretKeyHolder > [error] ^ > [error] > /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/SecurityManager.scala:147: > not found: type SecretKeyHolder > [error] private[spark] class SecurityManager(sparkConf: SparkConf) > extends Logging with SecretKeyHolder { > [error] > ^ > [error] > /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala:29: > object RetryingBlockFetcher is not a member of package > org.apache.spark.network.shuffle > [error] import org.apache.spark.network.shuffle.{RetryingBlockFetcher, > BlockFetchingListener, OneForOneBlockFetcher} > [error] ^ > [error] > /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/deploy/worker/StandaloneWorkerShuffleService.scala:23: > object sasl is not a member of package org.apache.spark.network > [error] import org.apache.spark.network.sasl.SaslRpcHandler > [error] > > ... > > [error] > /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/storage/BlockManager.scala:124: > too many arguments for constructor ExternalShuffleClient: (x$1: > org.apache.spark.network.util.TransportConf, x$2: > String)org.apache.spark.network.shuffle.ExternalShuffleClient > [error] new > ExternalShuffleClient(SparkTransportConf.fromSparkConf(conf), > securityManager, > [error] ^ > [error] > /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/storage/BlockManager.scala:39: > object protocol is not a member of package > org.apache.spark.network.shuffle > [error] import org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo > [error] ^ > [error] > /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/storage/BlockManager.scala:214: > not found: type ExecutorShuffleInfo > [error] val shuffleConfig = new ExecutorShuffleInfo( > [error] > ... > > > More refactoring needed? Either to support YARN alpha as a separate > shuffle module, or sever this dependency? > > Of course this goes away when yarn-alpha goes away too. > > > On Sat, Nov 8, 2014 at 7:45 AM, Patrick Wendell <[email protected]> wrote: >> I bet it doesn't work. +1 on isolating it's inclusion to only the >> newer YARN API's. >> >> - Patrick >> >> On Fri, Nov 7, 2014 at 11:43 PM, Sean Owen <[email protected]> wrote: >>> I noticed that this doesn't compile: >>> >>> mvn -Pyarn-alpha -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean >>> package >>> >>> [error] warning: [options] bootstrap class path not set in conjunction >>> with -source 1.6 >>> [error] >>> /Users/srowen/Documents/spark/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:26: >>> error: cannot find symbol >>> [error] import org.apache.hadoop.yarn.server.api.AuxiliaryService; >>> [error] ^ >>> [error] symbol: class AuxiliaryService >>> [error] location: package org.apache.hadoop.yarn.server.api >>> [error] >>> /Users/srowen/Documents/spark/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:27: >>> error: cannot find symbol >>> [error] import >>> org.apache.hadoop.yarn.server.api.ApplicationInitializationContext; >>> [error] ^ >>> ... >>> >>> Should it work? if not shall I propose to enable the service only with >>> -Pyarn? >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
