Re: Weird Spark Dispatcher Offers?

Do you have jobs enqueued? And if none of the jobs matches any offer it will just decline it. What's your job resource specifications? Tim On Fri, Oct 2, 2015 at 11:34 AM, Alan Braithwaite wrote: > Hey All, > > Using spark with mesos and docker. > > I'm wondering if

Weird Spark Dispatcher Offers?

Hey All, Using spark with mesos and docker. I'm wondering if anybody's seen the behavior of spark dispatcher where it just continually requests resources and immediately declines the offer. https://gist.github.com/anonymous/41e7c91899b0122b91a7 I'm trying to debug some issues with spark and

Re: Weird Spark Dispatcher Offers?

> > So if there is no jobs to run the dispatcher will decline all offers by > default. > So would this be a bug in mesos then? I'm not sure I understand how this offer is appearing in the first place. It only shows up in the master logs when I start the dispatcher. > Also we list all the jobs

Re: Weird Spark Dispatcher Offers?

This happened right after blowing away /var/lib/mesos zk://mesos and zk://spark_mesos_dispatcher and before I've submitted anything new to it so I _shouldn't_ have anything enqueued. Unless there's state being stored somewhere besides those places that I don't know about. I'm not sure what the

Re: Weird Spark Dispatcher Offers?

Hi Alan, The dispatcher is a Mesos framework and all frameworks in Mesos receives offers from the master. Mesos is different than most schedulers where we don't issue containers based on requests, but we offer available resources to all frameworks and they in turn decide if they want to use these

Re: Weird Spark Dispatcher Offers?

So if there is no jobs to run the dispatcher will decline all offers by default. Also we list all the jobs enqueued and it's specifications in the Spark dispatcher UI, you should see the port in the dispatcher logs itself. Tim On Fri, Oct 2, 2015 at 11:46 AM, Alan Braithwaite