Never mind, just figured out my problem, I was running: *deploy.ExternalShuffleService* instead of *deploy.mesos.MesosExternalShuffleService*
- Jo Voordeckers On Fri, Apr 15, 2016 at 2:29 PM, Jo Voordeckers <[email protected]> wrote: > Forgot to mention we're running Spark (Streaming) 1.5.1 > > - Jo Voordeckers > > > On Fri, Apr 15, 2016 at 12:21 PM, Jo Voordeckers <[email protected] > > wrote: > >> Hi all, >> >> I've got mesos in coarse grained mode with dyn alloc, shuffle service >> enabled and am running the shuffle service on every mesos slave. >> >> I'm assuming I misconfigured something on the scheduler service, any >> ideas? >> >> On my driver is see a few of these, I guess it's one for every executor : >> >> 19:12:29 WARN [shuffle-client-1] MesosExternalShuffleClient - Unable to >>> register app 78a944c9-3a89-4334-bca3-7108aadb1798-0000 with external >>> shuffle service. Please manually remove shuffle data after driver exit. >>> Error: java.lang.RuntimeException: java.lang.UnsupportedOperationException: >>> Unexpected message: >>> org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a >>> at >>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92) >>> at >>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68) >>> at >>> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114) >>> at >>> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87) >>> at >>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101) >>> at >>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) >>> at >>> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) >>> at >>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) >>> at >>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) >>> >> [...] >> >> In the scheduler service I see logs like this on every box: >> >> log4j:WARN No appenders could be found for logger >>> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). >>> log4j:WARN Please initialize the log4j system properly. >>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig >>> for more info. >>> Using Spark's repl log4j profile: >>> org/apache/spark/log4j-defaults-repl.properties >>> To adjust logging level use sc.setLogLevel("INFO") >>> 16/04/14 19:12:29 ERROR TransportRequestHandler: Error while invoking >>> RpcHandler#receive() on RPC id 7280403447531815366 >>> java.lang.UnsupportedOperationException: Unexpected message: >>> org.apache.spark.network.shuffle.protocol.mesos.RegisterDriver@c399c59a >>> at >>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:92) >>> at >>> org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:68) >>> at >>> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:114) >>> at >>> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:87) >>> at >>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:101) >>> at >>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) >>> at >>> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) >>> at >>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) >>> >> [...] >> >> Thanks! >> >> - Jo Voordeckers >> >> >
