I think you might be conflating two things. The first error you posted
was because YARN didn't standardize the shuffle API in alpha versions
so our spark-network-yarn module won't compile. We should just disable
that module if yarn alpha is used. spark-network-yarn is a leaf in the
intra-module dependency graph, and core doesn't depend on it.
This second error is something else. Maybe you are excluding
network-shuffle instead of spark-network-yarn?
On Fri, Nov 7, 2014 at 11:50 PM, Sean Owen <so...@cloudera.com> wrote:
> Hm. Problem is, core depends directly on it:
>
> [error]
> /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/SecurityManager.scala:25:
> object sasl is not a member of package org.apache.spark.network
> [error] import org.apache.spark.network.sasl.SecretKeyHolder
> [error] ^
> [error]
> /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/SecurityManager.scala:147:
> not found: type SecretKeyHolder
> [error] private[spark] class SecurityManager(sparkConf: SparkConf)
> extends Logging with SecretKeyHolder {
> [error]
> ^
> [error]
> /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala:29:
> object RetryingBlockFetcher is not a member of package
> org.apache.spark.network.shuffle
> [error] import org.apache.spark.network.shuffle.{RetryingBlockFetcher,
> BlockFetchingListener, OneForOneBlockFetcher}
> [error] ^
> [error]
> /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/deploy/worker/StandaloneWorkerShuffleService.scala:23:
> object sasl is not a member of package org.apache.spark.network
> [error] import org.apache.spark.network.sasl.SaslRpcHandler
> [error]
>
> ...
>
> [error]
> /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/storage/BlockManager.scala:124:
> too many arguments for constructor ExternalShuffleClient: (x$1:
> org.apache.spark.network.util.TransportConf, x$2:
> String)org.apache.spark.network.shuffle.ExternalShuffleClient
> [error] new
> ExternalShuffleClient(SparkTransportConf.fromSparkConf(conf),
> securityManager,
> [error] ^
> [error]
> /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/storage/BlockManager.scala:39:
> object protocol is not a member of package
> org.apache.spark.network.shuffle
> [error] import org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo
> [error] ^
> [error]
> /Users/srowen/Documents/spark/core/src/main/scala/org/apache/spark/storage/BlockManager.scala:214:
> not found: type ExecutorShuffleInfo
> [error] val shuffleConfig = new ExecutorShuffleInfo(
> [error]
> ...
>
>
> More refactoring needed? Either to support YARN alpha as a separate
> shuffle module, or sever this dependency?
>
> Of course this goes away when yarn-alpha goes away too.
>
>
> On Sat, Nov 8, 2014 at 7:45 AM, Patrick Wendell <pwend...@gmail.com> wrote:
>> I bet it doesn't work. +1 on isolating it's inclusion to only the
>> newer YARN API's.
>>
>> - Patrick
>>
>> On Fri, Nov 7, 2014 at 11:43 PM, Sean Owen <so...@cloudera.com> wrote:
>>> I noticed that this doesn't compile:
>>>
>>> mvn -Pyarn-alpha -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean
>>> package
>>>
>>> [error] warning: [options] bootstrap class path not set in conjunction
>>> with -source 1.6
>>> [error]
>>> /Users/srowen/Documents/spark/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:26:
>>> error: cannot find symbol
>>> [error] import org.apache.hadoop.yarn.server.api.AuxiliaryService;
>>> [error] ^
>>> [error] symbol: class AuxiliaryService
>>> [error] location: package org.apache.hadoop.yarn.server.api
>>> [error]
>>> /Users/srowen/Documents/spark/network/yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java:27:
>>> error: cannot find symbol
>>> [error] import
>>> org.apache.hadoop.yarn.server.api.ApplicationInitializationContext;
>>> [error] ^
>>> ...
>>>
>>> Should it work? if not shall I propose to enable the service only with
>>> -Pyarn?
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: dev-h...@spark.apache.org
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
