Thanks,
Travis
From: Reynold Xin
Sent: Thursday, December 15, 2016 14:07
To: Hegner, Travis
Cc: Jörn Franke; Apache Spark Dev
Subject: Re: SPARK-18689: A proposal for priority based app scheduling
utilizing linux cgroups.
In general this falls directly i
> Travis
>
> ----------
> *From:* Jörn Franke
> *Sent:* Thursday, December 15, 2016 12:48
> *To:* Hegner, Travis
> *Cc:* Apache Spark Dev
>
> *Subject:* Re: SPARK-18689: A proposal for priority based app scheduling
> utilizing linux cgroups.
>
>
:48
To: Hegner, Travis
Cc: Apache Spark Dev
Subject: Re: SPARK-18689: A proposal for priority based app scheduling
utilizing linux cgroups.
Hi,
What about yarn or mesos used in combination with Spark. The have also cgroups.
Or a kubernetes etc deployment.
On 15 Dec 2016, at 17:37, He
believe that everyone can get better throughput out of their
> hardware, at any cluster size. But, alas, *that* is not a problem I'm trying
> to solve currently.
>
>
>
>
> Sorry that the patch is pretty rough still, as I'm still getting my head
> wrapped around spa
launched, so that I can more
intuitively and more optimally utilize my small spark cluster.
Thanks,
Travis
From: Steve Loughran
Sent: Tuesday, December 6, 2016 06:54
To: Hegner, Travis
Cc: Shuai Lin; Apache Spark Dev
Subject: Re: SPARK-18689: A proposal for prior
hran
Sent: Tuesday, December 6, 2016 06:54
To: Hegner, Travis
Cc: Shuai Lin; Apache Spark Dev
Subject: Re: SPARK-18689: A proposal for priority based app scheduling
utilizing linux cgroups.
This is essentially what the cluster schedulers do: allow different people to
submit work with different cre
This is essentially what the cluster schedulers do: allow different people to
submit work with different credentials and priority; cgroups & equivalent to
limit granted resources to requested ones. If you have pre-emption enabled, you
can even have one job kill work off the others. Spark does re
better now. Does anyone else can see value in it?
Travis
*From:* Shuai Lin
*Sent:* Saturday, December 3, 2016 06:52
*To:* Hegner, Travis
*Cc:* dev@spark.apache.org
*Subject:* Re: SPARK-18689: A proposal for priority based app
scheduling utilizing linux cgroups.
Sorry but I don't g
__
From: Shuai Lin
Sent: Saturday, December 3, 2016 06:52
To: Hegner, Travis
Cc: dev@spark.apache.org
Subject: Re: SPARK-18689: A proposal for priority based app scheduling
utilizing linux cgroups.
Sorry but I don't get the scope of the problem from your description. Seems
it'
Sorry but I don't get the scope of the problem from your description. Seems
it's an improvement for spark standalone scheduler (i.e. not for yarn or
mesos)?
On Sat, Dec 3, 2016 at 4:27 AM, Hegner, Travis
wrote:
> Hello,
>
>
> I've just created a JIRA to open up discussion of a new feature that I
Hello,
I've just created a JIRA to open up discussion of a new feature that I'd like
to propose.
https://issues.apache.org/jira/browse/SPARK-18689
I'd love to get some feedback on the idea. I know that normally anything
related to scheduling or queuing automatically throws up the "hard to
11 matches
Mail list logo
