Hm. Problem is, core depends directly on it: [error] /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/SecurityManager.scala:25: object sasl is not a member of package org.apache.spark.network [error] import org.apache.spark.network.sasl.SecretKeyHolder [error] ^ [error] /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/SecurityManager.scala:147: not found: type SecretKeyHolder [error] private[spark] class SecurityManager(sparkConf: SparkConf) extends Logging with SecretKeyHolder { [error] ^ [error] /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala:29: object RetryingBlockFetcher is not a member of package org.apache.spark.network.shuffle [error] import org.apache.spark.network.shuffle.{RetryingBlockFetcher, BlockFetchingListener, OneForOneBlockFetcher} [error] ^ [error] /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/deploy/worker/StandaloneWorkerShuffleService.scala:23: object sasl is not a member of package org.apache.spark.network [error] import org.apache.spark.network.sasl.SaslRpcHandler [error]
... [error] /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/storage/BlockManager.scala:124: too many arguments for constructor ExternalShuffleClient: (x$1: org.apache.spark.network.util.TransportConf, x$2: String)org.apache.spark.network.shuffle.ExternalShuffleClient [error] new ExternalShuffleClient(SparkTransportConf.fromSparkConf(conf), securityManager, [error] ^ [error] /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/storage/BlockManager.scala:39: object protocol is not a member of package org.apache.spark.network.shuffle [error] import org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo [error] ^ [error] /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/storage/BlockManager.scala:214: not found: type ExecutorShuffleInfo [error] val shuffleConfig = new ExecutorShuffleInfo( [error] ... More refactoring needed? Either to support YARN alpha as a separate shuffle module, or sever this dependency? Of course this goes away when yarn-alpha goes away too. On Sat, Nov 8, 2014 at 7:45 AM, Patrick Wendell <pwend...@gmail.com> wrote: > I bet it doesn't work. +1 on isolating it's inclusion to only the > newer YARN API's. > > - Patrick > > On Fri, Nov 7, 2014 at 11:43 PM, Sean Owen <so...@cloudera.com> wrote: >> I noticed that this doesn't compile: >> >> mvn -Pyarn-alpha -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean >> package >> >> [error] warning: [options] bootstrap class path not set in conjunction >> with -source 1.6 >> [error] >> /Users/srowen/Documents/spark/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:26: >> error: cannot find symbol >> [error] import org.apache.hadoop.yarn.server.api.AuxiliaryService; >> [error] ^ >> [error] symbol: class AuxiliaryService >> [error] location: package org.apache.hadoop.yarn.server.api >> [error] >> /Users/srowen/Documents/spark/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:27: >> error: cannot find symbol >> [error] import >> org.apache.hadoop.yarn.server.api.ApplicationInitializationContext; >> [error] ^ >> ... >> >> Should it work? if not shall I propose to enable the service only with >> -Pyarn? >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org