Shuffle service fails to register driver - Spark - Mesos

er.channelRead0(TransportChannelHandler.java:51) > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) > [...] Thanks! - Jo Voordeckers

Re: Shuffle service fails to register driver - Spark - Mesos

Forgot to mention we're running Spark (Streaming) 1.5.1 - Jo Voordeckers On Fri, Apr 15, 2016 at 12:21 PM, Jo Voordeckers wrote: > Hi all, > > I've got mesos in coarse grained mode with dyn alloc, shuffle service > enabled and am running the shuffle service on ever

Re: Shuffle service fails to register driver - Spark - Mesos

Never mind, just figured out my problem, I was running: *deploy.ExternalShuffleService* instead of *deploy.mesos.MesosExternalShuffleService* - Jo Voordeckers On Fri, Apr 15, 2016 at 2:29 PM, Jo Voordeckers wrote: > Forgot to mention we're running Spark (Streaming) 1.5.1 > > -

Spark 1.5.1 on Mesos NO Executor Java Options

Hi everyone, I'm trying to setup Spark 1.5.1 with mesos and the Cluster Dispatcher that I'm currently running on one of the slaves. We're migrating from a 1.3 standalone cluster and we're hoping to benefit from dynamic resource allocation with fine grained mesos for a better distribution of resour

Mesos cluster dispatcher doesn't respect most args from the submit req

a look: https://github.com/apache/spark/pull/9752 Thanks - Jo Voordeckers

Re: Spark-shell connecting to Mesos stuck at sched.cpp

I've seen this issue when the mesos cluster couldn't figure out my IP address correctly, have you tried setting the ENV var with your IP address when launching spark or mesos cluster dispatcher like: LIBPROCESS_IP="172.16.0.180" - Jo Voordeckers On Sun, Nov 15, 2015 at 6:

Re: Conf Settings in Mesos

I have run in a related issue I think, args passed to spark-submit to my cluster dispatcher get lost in translation when lauching the driver from mesos, I'm suggesting this patch: https://github.com/jayv/spark/commit/b2025ddc1d565d1cc3036200fc3b3046578f4b02 - Jo Voordeckers On Thu, N

Re: Mesos cluster dispatcher doesn't respect most args from the submit req

.apache.org/jira/browse/SPARK-11327> is exactly my problem, but I don't run docker. - Jo On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers > wrote: > >> >> Hi all, >> >> I'm running the mesos cluster dispatcher, however when I submit jobs with >>

Re: Mesos cluster dispatcher doesn't respect most args from the submit req

Hi Tim, I've done more forensics on this bug, see my comment here: https://issues.apache.org/jira/browse/SPARK-11327?focusedCommentId=15009843&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15009843 - Jo Voordeckers On Tue, Nov 17, 2015 at 12:01 PM