Re: Removing the Mesos fine-grained mode

That'd be great, thanks Adam! On Tue, Jan 19, 2016 at 5:41 PM, Adam McElwee wrote: > Sorry, I never got a chance to circle back with the master logs for this. > I definitely can't share the job code, since it's used to build a pretty > core dataset for my company, but let me

Re: Removing the Mesos fine-grained mode

It would be good to get to the bottom of this. Adam, could you share the Spark app that you're using to test this? iulian On Mon, Nov 30, 2015 at 10:10 PM, Timothy Chen wrote: > Hi Adam, > > Thanks for the graphs and the tests, definitely interested to dig a > bit deeper to

Re: Removing the Mesos fine-grained mode

Sorry, I never got a chance to circle back with the master logs for this. I definitely can't share the job code, since it's used to build a pretty core dataset for my company, but let me see if I can pull some logs together in the next couple days. On Tue, Jan 19, 2016 at 10:08 AM, Iulian Dragoș

Re: Removing the Mesos fine-grained mode

Hi Adam, Thanks for the graphs and the tests, definitely interested to dig a bit deeper to find out what's could be the cause of this. Do you have the spark driver logs for both runs? Tim On Mon, Nov 30, 2015 at 9:06 AM, Adam McElwee wrote: > To eliminate any skepticism

Re: Removing the Mesos fine-grained mode

To eliminate any skepticism around whether cpu is a good performance metric for this workload, I did a couple comparison runs of an example job to demonstrate a more universal change in performance metrics (stage/job time) between coarse and fine-grained mode on mesos. The workload is identical

Re: Removing the Mesos fine-grained mode

@Jerry Lam Can someone confirm if it is true that dynamic allocation on mesos "is > designed to run one executor per slave with the configured amount of > resources." I copied this sentence from the documentation. Does this mean > there is at most 1 executor per node? Therefore, if you have a

Re: Removing the Mesos fine-grained mode

Hi Andrew, Thank you for confirming this. I’m referring to this because I used fine-grained mode before and it was a headache because of the memory issue. Therefore, I switched to Yarn with dynamic allocation. I was thinking if I can switch back to Mesos with coarse-grained mode + dynamic

Re: Removing the Mesos fine-grained mode

@Andrew Or I assume you are referring to this ticket [SPARK-5095]: https://issues.apache.org/jira/browse/SPARK-5095 Thank you! Best Regards, Jerry > On Nov 23, 2015, at 2:41 PM, Andrew Or wrote: > > @Jerry Lam > >

Re: Removing the Mesos fine-grained mode

On Sat, Nov 21, 2015 at 3:37 AM, Adam McElwee wrote: > I've used fine-grained mode on our mesos spark clusters until this week, > mostly because it was the default. I started trying coarse-grained because > of the recent chatter on the mailing list about wanting to move the

Re: Removing the Mesos fine-grained mode

Hi guys, Can someone confirm if it is true that dynamic allocation on mesos "is designed to run one executor per slave with the configured amount of resources." I copied this sentence from the documentation. Does this mean there is at most 1 executor per node? Therefore, if you have a big

Re: Removing the Mesos fine-grained mode

On Mon, Nov 23, 2015 at 7:36 AM, Iulian Dragoș wrote: > > > On Sat, Nov 21, 2015 at 3:37 AM, Adam McElwee wrote: > >> I've used fine-grained mode on our mesos spark clusters until this week, >> mostly because it was the default. I started trying

Re: Removing the Mesos fine-grained mode

gt;> having the two code paths is just noise. >> >> -Chris >> >> From: Iulian Dragoș <iulian.dra...@typesafe.com> >> Date: Thursday, November 19, 2015 at 6:42 AM >> To: "dev@spark.apache.org" <dev@spark.apache.org> >> Subject: Removing th

Re: Removing the Mesos fine-grained mode

ld define a Mesos framework. That said, with dyn-allocation and Mesos >>> support for both resource reservation, oversubscription and revocation, I >>> think the direction is clear that the coarse mode is the proper way >>> forward, and having the two code paths is just noi

Re: Removing the Mesos fine-grained mode

forward, and > having the two code paths is just noise. > > -Chris > > From: Iulian Dragoș <iulian.dra...@typesafe.com> > Date: Thursday, November 19, 2015 at 6:42 AM > To: "dev@spark.apache.org" <dev@spark.apache.org> > Subject: Removing the Mesos

Removing the Mesos fine-grained mode

Hi all, Mesos is the only cluster manager that has a fine-grained mode, but it's more often than not problematic, and it's a maintenance burden. I'd like to suggest removing it in the 2.0 release. A few reasons: - code/maintenance complexity. The two modes duplicate a lot of functionality (and

Re: Removing the Mesos fine-grained mode

Sounds like the right move. Simplifies things in important ways. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition (O'Reilly) Typesafe @deanwampler http://polyglotprogramming.com On

Re: Removing the Mesos fine-grained mode

he.org<mailto:dev@spark.apache.org>" <dev@spark.apache.org<mailto:dev@spark.apache.org>> Subject: Removing the Mesos fine-grained mode Hi all, Mesos is the only cluster manager that has a fine-grained mode, but it's more often than not problematic, and it's a maintenan