Re: Dynamic resource allocation to Spark on Mesos

[Michael Gummelt]

> by specifying a larger heap size than default on each worker node.

I don't follow.  Which heap?  Are you specifying a large heap size on the
executors?  If so, do you mean you somehow launch the shuffle service when
you launch executors?  Or something else?

On Wed, Feb 8, 2017 at 5:50 PM, Sun Rui  wrote:

> Michael,
> No. We directly launch the external shuffle service by specifying a larger
> heap size than default on each worker node. It is observed that the
> processes are quite stable.
>
> On Feb 9, 2017, at 05:21, Michael Gummelt  wrote:
>
> Sun, are you using marathon to run the shuffle service?
>
> On Tue, Feb 7, 2017 at 7:36 PM, Sun Rui  wrote:
>
>> Yi Jan,
>>
>> We have been using Spark on Mesos with dynamic allocation enabled, which
>> works and improves the overall cluster utilization.
>>
>> In terms of job, do you mean jobs inside a Spark application or jobs
>> among different applications? Maybe you can read
>> http://spark.apache.org/docs/latest/job-scheduling.html for help.
>>
>> On Jan 31, 2017, at 03:34, Michael Gummelt 
>> wrote:
>>
>>
>>
>> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:
>>
>>> Tasks begin scheduling as soon as the first executor comes up
>>>
>>>
>>> Thanks all for the clarification. Is this the default behavior of Spark
>>> on Mesos today? I think this is what we are looking for because sometimes a
>>> job can take up lots of resources and later jobs could not get all the
>>> resources that it asks for. If a Spark job starts with only a subset of
>>> resources that it asks for, does it know to expand its resources later when
>>> more resources become available?
>>>
>>
>> Yes.
>>
>>
>>>
>>> Launch each executor with at least 1GB RAM, but if mesos offers 2GB at
 some moment, then launch an executor with 2GB RAM
>>>
>>>
>>> This is less useful in our use case. But I am also quite interested in
>>> cases in which this could be helpful. I think this will also help with
>>> overall resource utilization on the cluster if when another job starts up
>>> that has a hard requirement on resources, the extra resources to the first
>>> job can be flexibly re-allocated to the second job.
>>>
>>> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt >> > wrote:
>>>
 We've talked about that, but it hasn't become a priority because we
 haven't had a driving use case.  If anyone has a good argument for
 "variable" resource allocation like this, please let me know.

 On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin  w
 rote:

> An alternative behavior is to launch the job with the best resource
>> offer Mesos is able to give
>
>
> Michael has just made an excellent explanation about dynamic
> allocation support in mesos. But IIUC, what you want to achieve is
> something like (using RAM as an example) : "Launch each executor with at
> least 1GB RAM, but if mesos offers 2GB at some moment, then launch an
> executor with 2GB RAM".
>
> I wonder what's benefit of that? To reduce the "resource
> fragmentation"?
>
> Anyway, that is not supported at this moment. In all the supported
> cluster managers of spark (mesos, yarn, standalone, and the up-to-coming
> spark on kubernetes), you have to specify the cores and memory of each
> executor.
>
> It may not be supported in the future, because only mesos has the
> concepts of offers because of its two-level scheduling model.
>
>
> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:
>
>> Dear Spark Users,
>>
>> Currently is there a way to dynamically allocate resources to Spark
>> on Mesos? Within Spark we can specify the CPU cores, memory before 
>> running
>> job. The way I understand is that the Spark job will not run if the 
>> CPU/Mem
>> requirement is not met. This may lead to decrease in overall utilization 
>> of
>> the cluster. An alternative behavior is to launch the job with the best
>> resource offer Mesos is able to give. Is this possible with the current
>> implementation?
>>
>> Thanks
>> Ji
>>
>> The information in this email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this email
>> by anyone else is unauthorized. If you are not the intended recipient, 
>> any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful.
>>
>
>


 --
 Michael Gummelt
 Software Engineer
 Mesosphere

>>>
>>>
>>> The information in this email is confidential and may be legally
>>> privileged. It is intended solely for the addressee. Access to this email
>>> by anyone else is unauthorized. If you are not the intended 

Re: Dynamic resource allocation to Spark on Mesos

[Sun Rui]

Michael,
No. We directly launch the external shuffle service by specifying a larger heap 
size than default on each worker node. It is observed that the processes are 
quite stable.
> On Feb 9, 2017, at 05:21, Michael Gummelt  wrote:
> 
> Sun, are you using marathon to run the shuffle service?
> 
> On Tue, Feb 7, 2017 at 7:36 PM, Sun Rui  > wrote:
> Yi Jan,
> 
> We have been using Spark on Mesos with dynamic allocation enabled, which 
> works and improves the overall cluster utilization.
> 
> In terms of job, do you mean jobs inside a Spark application or jobs among 
> different applications? Maybe you can read 
> http://spark.apache.org/docs/latest/job-scheduling.html 
>  for help.
> 
>> On Jan 31, 2017, at 03:34, Michael Gummelt > > wrote:
>> 
>> 
>> 
>> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan > > wrote:
>> Tasks begin scheduling as soon as the first executor comes up
>> 
>> Thanks all for the clarification. Is this the default behavior of Spark on 
>> Mesos today? I think this is what we are looking for because sometimes a job 
>> can take up lots of resources and later jobs could not get all the resources 
>> that it asks for. If a Spark job starts with only a subset of resources that 
>> it asks for, does it know to expand its resources later when more resources 
>> become available?
>> 
>> Yes.
>>  
>> 
>> Launch each executor with at least 1GB RAM, but if mesos offers 2GB at some 
>> moment, then launch an executor with 2GB RAM
>> 
>> This is less useful in our use case. But I am also quite interested in cases 
>> in which this could be helpful. I think this will also help with overall 
>> resource utilization on the cluster if when another job starts up that has a 
>> hard requirement on resources, the extra resources to the first job can be 
>> flexibly re-allocated to the second job. 
>> 
>> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt > > wrote:
>> We've talked about that, but it hasn't become a priority because we haven't 
>> had a driving use case.  If anyone has a good argument for "variable" 
>> resource allocation like this, please let me know.
>> 
>> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin > > wrote:
>> An alternative behavior is to launch the job with the best resource offer 
>> Mesos is able to give
>> 
>> Michael has just made an excellent explanation about dynamic allocation 
>> support in mesos. But IIUC, what you want to achieve is something like 
>> (using RAM as an example) : "Launch each executor with at least 1GB RAM, but 
>> if mesos offers 2GB at some moment, then launch an executor with 2GB RAM".
>> 
>> I wonder what's benefit of that? To reduce the "resource fragmentation"?
>> 
>> Anyway, that is not supported at this moment. In all the supported cluster 
>> managers of spark (mesos, yarn, standalone, and the up-to-coming spark on 
>> kubernetes), you have to specify the cores and memory of each executor.
>> 
>> It may not be supported in the future, because only mesos has the concepts 
>> of offers because of its two-level scheduling model.
>> 
>> 
>> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan > > wrote:
>> Dear Spark Users,
>> 
>> Currently is there a way to dynamically allocate resources to Spark on 
>> Mesos? Within Spark we can specify the CPU cores, memory before running job. 
>> The way I understand is that the Spark job will not run if the CPU/Mem 
>> requirement is not met. This may lead to decrease in overall utilization of 
>> the cluster. An alternative behavior is to launch the job with the best 
>> resource offer Mesos is able to give. Is this possible with the current 
>> implementation?
>> 
>> Thanks
>> Ji
>> 
>> The information in this email is confidential and may be legally privileged. 
>> It is intended solely for the addressee. Access to this email by anyone else 
>> is unauthorized. If you are not the intended recipient, any disclosure, 
>> copying, distribution or any action taken or omitted to be taken in reliance 
>> on it, is prohibited and may be unlawful.
>> 
>> 
>> 
>> 
>> 
>> -- 
>> Michael Gummelt
>> Software Engineer
>> Mesosphere
>> 
>> 
>> The information in this email is confidential and may be legally privileged. 
>> It is intended solely for the addressee. Access to this email by anyone else 
>> is unauthorized. If you are not the intended recipient, any disclosure, 
>> copying, distribution or any action taken or omitted to be taken in reliance 
>> on it, is prohibited and may be unlawful.
>> 
>> 
>> 
>> 
>> -- 
>> Michael Gummelt
>> Software Engineer
>> Mesosphere
> 
> 
> 
> 
> -- 
> Michael Gummelt
> Software Engineer
> Mesosphere

Re: Dynamic resource allocation to Spark on Mesos

[Michael Gummelt]

Sun, are you using marathon to run the shuffle service?

On Tue, Feb 7, 2017 at 7:36 PM, Sun Rui  wrote:

> Yi Jan,
>
> We have been using Spark on Mesos with dynamic allocation enabled, which
> works and improves the overall cluster utilization.
>
> In terms of job, do you mean jobs inside a Spark application or jobs among
> different applications? Maybe you can read http://spark.apache.org/
> docs/latest/job-scheduling.html for help.
>
> On Jan 31, 2017, at 03:34, Michael Gummelt  wrote:
>
>
>
> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:
>
>> Tasks begin scheduling as soon as the first executor comes up
>>
>>
>> Thanks all for the clarification. Is this the default behavior of Spark
>> on Mesos today? I think this is what we are looking for because sometimes a
>> job can take up lots of resources and later jobs could not get all the
>> resources that it asks for. If a Spark job starts with only a subset of
>> resources that it asks for, does it know to expand its resources later when
>> more resources become available?
>>
>
> Yes.
>
>
>>
>> Launch each executor with at least 1GB RAM, but if mesos offers 2GB at
>>> some moment, then launch an executor with 2GB RAM
>>
>>
>> This is less useful in our use case. But I am also quite interested in
>> cases in which this could be helpful. I think this will also help with
>> overall resource utilization on the cluster if when another job starts up
>> that has a hard requirement on resources, the extra resources to the first
>> job can be flexibly re-allocated to the second job.
>>
>> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt 
>>  wrote:
>>
>>> We've talked about that, but it hasn't become a priority because we
>>> haven't had a driving use case.  If anyone has a good argument for
>>> "variable" resource allocation like this, please let me know.
>>>
>>> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin  w
>>> rote:
>>>
 An alternative behavior is to launch the job with the best resource
> offer Mesos is able to give


 Michael has just made an excellent explanation about dynamic allocation
 support in mesos. But IIUC, what you want to achieve is something like
 (using RAM as an example) : "Launch each executor with at least 1GB RAM,
 but if mesos offers 2GB at some moment, then launch an executor with 2GB
 RAM".

 I wonder what's benefit of that? To reduce the "resource fragmentation"?

 Anyway, that is not supported at this moment. In all the supported
 cluster managers of spark (mesos, yarn, standalone, and the up-to-coming
 spark on kubernetes), you have to specify the cores and memory of each
 executor.

 It may not be supported in the future, because only mesos has the
 concepts of offers because of its two-level scheduling model.


 On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:

> Dear Spark Users,
>
> Currently is there a way to dynamically allocate resources to Spark on
> Mesos? Within Spark we can specify the CPU cores, memory before running
> job. The way I understand is that the Spark job will not run if the 
> CPU/Mem
> requirement is not met. This may lead to decrease in overall utilization 
> of
> the cluster. An alternative behavior is to launch the job with the best
> resource offer Mesos is able to give. Is this possible with the current
> implementation?
>
> Thanks
> Ji
>
> The information in this email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
>


>>>
>>>
>>> --
>>> Michael Gummelt
>>> Software Engineer
>>> Mesosphere
>>>
>>
>>
>> The information in this email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this email
>> by anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful.
>>
>
>
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere
>
>
>


-- 
Michael Gummelt
Software Engineer
Mesosphere

Re: Dynamic resource allocation to Spark on Mesos

[Sun Rui]

Yi Jan,

We have been using Spark on Mesos with dynamic allocation enabled, which works 
and improves the overall cluster utilization.

In terms of job, do you mean jobs inside a Spark application or jobs among 
different applications? Maybe you can read 
http://spark.apache.org/docs/latest/job-scheduling.html 
 for help.

> On Jan 31, 2017, at 03:34, Michael Gummelt  wrote:
> 
> 
> 
> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  > wrote:
> Tasks begin scheduling as soon as the first executor comes up
> 
> Thanks all for the clarification. Is this the default behavior of Spark on 
> Mesos today? I think this is what we are looking for because sometimes a job 
> can take up lots of resources and later jobs could not get all the resources 
> that it asks for. If a Spark job starts with only a subset of resources that 
> it asks for, does it know to expand its resources later when more resources 
> become available?
> 
> Yes.
>  
> 
> Launch each executor with at least 1GB RAM, but if mesos offers 2GB at some 
> moment, then launch an executor with 2GB RAM
> 
> This is less useful in our use case. But I am also quite interested in cases 
> in which this could be helpful. I think this will also help with overall 
> resource utilization on the cluster if when another job starts up that has a 
> hard requirement on resources, the extra resources to the first job can be 
> flexibly re-allocated to the second job. 
> 
> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt  > wrote:
> We've talked about that, but it hasn't become a priority because we haven't 
> had a driving use case.  If anyone has a good argument for "variable" 
> resource allocation like this, please let me know.
> 
> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin  > wrote:
> An alternative behavior is to launch the job with the best resource offer 
> Mesos is able to give
> 
> Michael has just made an excellent explanation about dynamic allocation 
> support in mesos. But IIUC, what you want to achieve is something like (using 
> RAM as an example) : "Launch each executor with at least 1GB RAM, but if 
> mesos offers 2GB at some moment, then launch an executor with 2GB RAM".
> 
> I wonder what's benefit of that? To reduce the "resource fragmentation"?
> 
> Anyway, that is not supported at this moment. In all the supported cluster 
> managers of spark (mesos, yarn, standalone, and the up-to-coming spark on 
> kubernetes), you have to specify the cores and memory of each executor.
> 
> It may not be supported in the future, because only mesos has the concepts of 
> offers because of its two-level scheduling model.
> 
> 
> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  > wrote:
> Dear Spark Users,
> 
> Currently is there a way to dynamically allocate resources to Spark on Mesos? 
> Within Spark we can specify the CPU cores, memory before running job. The way 
> I understand is that the Spark job will not run if the CPU/Mem requirement is 
> not met. This may lead to decrease in overall utilization of the cluster. An 
> alternative behavior is to launch the job with the best resource offer Mesos 
> is able to give. Is this possible with the current implementation?
> 
> Thanks
> Ji
> 
> The information in this email is confidential and may be legally privileged. 
> It is intended solely for the addressee. Access to this email by anyone else 
> is unauthorized. If you are not the intended recipient, any disclosure, 
> copying, distribution or any action taken or omitted to be taken in reliance 
> on it, is prohibited and may be unlawful.
> 
> 
> 
> 
> 
> -- 
> Michael Gummelt
> Software Engineer
> Mesosphere
> 
> 
> The information in this email is confidential and may be legally privileged. 
> It is intended solely for the addressee. Access to this email by anyone else 
> is unauthorized. If you are not the intended recipient, any disclosure, 
> copying, distribution or any action taken or omitted to be taken in reliance 
> on it, is prohibited and may be unlawful.
> 
> 
> 
> 
> -- 
> Michael Gummelt
> Software Engineer
> Mesosphere

Re: Dynamic resource allocation to Spark on Mesos

[ji yan]

got it, thanks for clarifying!

On Thu, Feb 2, 2017 at 2:57 PM, Michael Gummelt 
wrote:

> Yes, that's expected.  spark.executor.cores sizes a single executor.  It
> doesn't limit the number of executors.  For that, you need spark.cores.max
> (--total-executor-cores).
>
> And rdd.parallelize does not specify the number of executors.  It
> specifies the number of partitions, which relates to the number of tasks,
> not executors.  Unless you're running with dynamic allocation enabled, the
> number of executors for your job is static, and determined at start time.
> It's not influenced by your job itself.
>
>
> On Thu, Feb 2, 2017 at 2:42 PM, Ji Yan  wrote:
>
>> I tried setting spark.executor.cores per executor, but Spark seems to be
>> spinning up as many executors as possible up to spark.cores.max or however
>> many cpu cores available on the cluster, and this may be undesirable
>> because the number of executors in rdd.parallelize(collection, # of
>> partitions) is being overriden
>>
>> On Thu, Feb 2, 2017 at 1:30 PM, Michael Gummelt 
>> wrote:
>>
>>> As of Spark 2.0, Mesos mode does support setting cores on the executor
>>> level, but you might need to set the property directly (--conf
>>> spark.executor.cores=).  I've written about this here:
>>> https://docs.mesosphere.com/1.8/usage/service-guides/spark/j
>>> ob-scheduling/.  That doc is for DC/OS, but the configuration is the
>>> same.
>>>
>>> On Thu, Feb 2, 2017 at 1:06 PM, Ji Yan  wrote:
>>>
 I was mainly confused why this is the case with memory, but with cpu
 cores, it is not specified on per executor level

 On Thu, Feb 2, 2017 at 1:02 PM, Michael Gummelt  wrote:

> It sounds like you've answered your own question, right?
> --executor-memory means the memory per executor.  If you have no executor
> w/ 200GB memory, then the driver will accept no offers.
>
> On Thu, Feb 2, 2017 at 1:01 PM, Ji Yan  wrote:
>
>> sorry, to clarify, i was using --executor-memory for memory,
>> and --total-executor-cores for cpu cores
>>
>> On Thu, Feb 2, 2017 at 12:56 PM, Michael Gummelt <
>> mgumm...@mesosphere.io> wrote:
>>
>>> What CLI args are your referring to?  I'm aware of spark-submit's
>>> arguments (--executor-memory, --total-executor-cores, and 
>>> --executor-cores)
>>>
>>> On Thu, Feb 2, 2017 at 12:41 PM, Ji Yan  wrote:
>>>
 I have done a experiment on this today. It shows that only CPUs are
 tolerant of insufficient cluster size when a job starts. On my 
 cluster, I
 have 180Gb of memory and 64 cores, when I run spark-submit ( on mesos )
 with --cpu_cores set to 1000, the job starts up with 64 cores. but 
 when I
 set --memory to 200Gb, the job fails to start with "Initial job
 has not accepted any resources; check your cluster UI to ensure that
 workers are registered and have sufficient resources"

 Also it is confusing to me that --cpu_cores specifies the number of
 cpu cores across all executors, but --memory specifies per executor 
 memory
 requirement.

 On Mon, Jan 30, 2017 at 11:34 AM, Michael Gummelt <
 mgumm...@mesosphere.io> wrote:

>
>
> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:
>
>> Tasks begin scheduling as soon as the first executor comes up
>>
>>
>> Thanks all for the clarification. Is this the default behavior of
>> Spark on Mesos today? I think this is what we are looking for because
>> sometimes a job can take up lots of resources and later jobs could 
>> not get
>> all the resources that it asks for. If a Spark job starts with only a
>> subset of resources that it asks for, does it know to expand its 
>> resources
>> later when more resources become available?
>>
>
> Yes.
>
>
>>
>> Launch each executor with at least 1GB RAM, but if mesos offers
>>> 2GB at some moment, then launch an executor with 2GB RAM
>>
>>
>> This is less useful in our use case. But I am also quite
>> interested in cases in which this could be helpful. I think this 
>> will also
>> help with overall resource utilization on the cluster if when 
>> another job
>> starts up that has a hard requirement on resources, the extra 
>> resources to
>> the first job can be flexibly re-allocated to the second job.
>>
>> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt <
>> mgumm...@mesosphere.io> wrote:
>>
>>> We've talked about that, but it hasn't become a priority because
>>> 

Re: Dynamic resource allocation to Spark on Mesos

[Michael Gummelt]

Yes, that's expected.  spark.executor.cores sizes a single executor.  It
doesn't limit the number of executors.  For that, you need spark.cores.max
(--total-executor-cores).

And rdd.parallelize does not specify the number of executors.  It specifies
the number of partitions, which relates to the number of tasks, not
executors.  Unless you're running with dynamic allocation enabled, the
number of executors for your job is static, and determined at start time.
It's not influenced by your job itself.

On Thu, Feb 2, 2017 at 2:42 PM, Ji Yan  wrote:

> I tried setting spark.executor.cores per executor, but Spark seems to be
> spinning up as many executors as possible up to spark.cores.max or however
> many cpu cores available on the cluster, and this may be undesirable
> because the number of executors in rdd.parallelize(collection, # of
> partitions) is being overriden
>
> On Thu, Feb 2, 2017 at 1:30 PM, Michael Gummelt 
> wrote:
>
>> As of Spark 2.0, Mesos mode does support setting cores on the executor
>> level, but you might need to set the property directly (--conf
>> spark.executor.cores=).  I've written about this here:
>> https://docs.mesosphere.com/1.8/usage/service-guides/spark/j
>> ob-scheduling/.  That doc is for DC/OS, but the configuration is the
>> same.
>>
>> On Thu, Feb 2, 2017 at 1:06 PM, Ji Yan  wrote:
>>
>>> I was mainly confused why this is the case with memory, but with cpu
>>> cores, it is not specified on per executor level
>>>
>>> On Thu, Feb 2, 2017 at 1:02 PM, Michael Gummelt 
>>> wrote:
>>>
 It sounds like you've answered your own question, right?
 --executor-memory means the memory per executor.  If you have no executor
 w/ 200GB memory, then the driver will accept no offers.

 On Thu, Feb 2, 2017 at 1:01 PM, Ji Yan  wrote:

> sorry, to clarify, i was using --executor-memory for memory,
> and --total-executor-cores for cpu cores
>
> On Thu, Feb 2, 2017 at 12:56 PM, Michael Gummelt <
> mgumm...@mesosphere.io> wrote:
>
>> What CLI args are your referring to?  I'm aware of spark-submit's
>> arguments (--executor-memory, --total-executor-cores, and 
>> --executor-cores)
>>
>> On Thu, Feb 2, 2017 at 12:41 PM, Ji Yan  wrote:
>>
>>> I have done a experiment on this today. It shows that only CPUs are
>>> tolerant of insufficient cluster size when a job starts. On my cluster, 
>>> I
>>> have 180Gb of memory and 64 cores, when I run spark-submit ( on mesos )
>>> with --cpu_cores set to 1000, the job starts up with 64 cores. but when 
>>> I
>>> set --memory to 200Gb, the job fails to start with "Initial job has
>>> not accepted any resources; check your cluster UI to ensure that workers
>>> are registered and have sufficient resources"
>>>
>>> Also it is confusing to me that --cpu_cores specifies the number of
>>> cpu cores across all executors, but --memory specifies per executor 
>>> memory
>>> requirement.
>>>
>>> On Mon, Jan 30, 2017 at 11:34 AM, Michael Gummelt <
>>> mgumm...@mesosphere.io> wrote:
>>>


 On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:

> Tasks begin scheduling as soon as the first executor comes up
>
>
> Thanks all for the clarification. Is this the default behavior of
> Spark on Mesos today? I think this is what we are looking for because
> sometimes a job can take up lots of resources and later jobs could 
> not get
> all the resources that it asks for. If a Spark job starts with only a
> subset of resources that it asks for, does it know to expand its 
> resources
> later when more resources become available?
>

 Yes.


>
> Launch each executor with at least 1GB RAM, but if mesos offers
>> 2GB at some moment, then launch an executor with 2GB RAM
>
>
> This is less useful in our use case. But I am also quite
> interested in cases in which this could be helpful. I think this will 
> also
> help with overall resource utilization on the cluster if when another 
> job
> starts up that has a hard requirement on resources, the extra 
> resources to
> the first job can be flexibly re-allocated to the second job.
>
> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt <
> mgumm...@mesosphere.io> wrote:
>
>> We've talked about that, but it hasn't become a priority because
>> we haven't had a driving use case.  If anyone has a good argument for
>> "variable" resource allocation like this, please let me know.
>>
>> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin <
>> 

Re: Dynamic resource allocation to Spark on Mesos

[Ji Yan]

I tried setting spark.executor.cores per executor, but Spark seems to be
spinning up as many executors as possible up to spark.cores.max or however
many cpu cores available on the cluster, and this may be undesirable
because the number of executors in rdd.parallelize(collection, # of
partitions) is being overriden

On Thu, Feb 2, 2017 at 1:30 PM, Michael Gummelt 
wrote:

> As of Spark 2.0, Mesos mode does support setting cores on the executor
> level, but you might need to set the property directly (--conf
> spark.executor.cores=).  I've written about this here:
> https://docs.mesosphere.com/1.8/usage/service-guides/spark/job-scheduling/.
> That doc is for DC/OS, but the configuration is the same.
>
> On Thu, Feb 2, 2017 at 1:06 PM, Ji Yan  wrote:
>
>> I was mainly confused why this is the case with memory, but with cpu
>> cores, it is not specified on per executor level
>>
>> On Thu, Feb 2, 2017 at 1:02 PM, Michael Gummelt 
>> wrote:
>>
>>> It sounds like you've answered your own question, right?
>>> --executor-memory means the memory per executor.  If you have no executor
>>> w/ 200GB memory, then the driver will accept no offers.
>>>
>>> On Thu, Feb 2, 2017 at 1:01 PM, Ji Yan  wrote:
>>>
 sorry, to clarify, i was using --executor-memory for memory,
 and --total-executor-cores for cpu cores

 On Thu, Feb 2, 2017 at 12:56 PM, Michael Gummelt <
 mgumm...@mesosphere.io> wrote:

> What CLI args are your referring to?  I'm aware of spark-submit's
> arguments (--executor-memory, --total-executor-cores, and 
> --executor-cores)
>
> On Thu, Feb 2, 2017 at 12:41 PM, Ji Yan  wrote:
>
>> I have done a experiment on this today. It shows that only CPUs are
>> tolerant of insufficient cluster size when a job starts. On my cluster, I
>> have 180Gb of memory and 64 cores, when I run spark-submit ( on mesos )
>> with --cpu_cores set to 1000, the job starts up with 64 cores. but when I
>> set --memory to 200Gb, the job fails to start with "Initial job has
>> not accepted any resources; check your cluster UI to ensure that workers
>> are registered and have sufficient resources"
>>
>> Also it is confusing to me that --cpu_cores specifies the number of
>> cpu cores across all executors, but --memory specifies per executor 
>> memory
>> requirement.
>>
>> On Mon, Jan 30, 2017 at 11:34 AM, Michael Gummelt <
>> mgumm...@mesosphere.io> wrote:
>>
>>>
>>>
>>> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:
>>>
 Tasks begin scheduling as soon as the first executor comes up


 Thanks all for the clarification. Is this the default behavior of
 Spark on Mesos today? I think this is what we are looking for because
 sometimes a job can take up lots of resources and later jobs could not 
 get
 all the resources that it asks for. If a Spark job starts with only a
 subset of resources that it asks for, does it know to expand its 
 resources
 later when more resources become available?

>>>
>>> Yes.
>>>
>>>

 Launch each executor with at least 1GB RAM, but if mesos offers 2GB
> at some moment, then launch an executor with 2GB RAM


 This is less useful in our use case. But I am also quite interested
 in cases in which this could be helpful. I think this will also help 
 with
 overall resource utilization on the cluster if when another job starts 
 up
 that has a hard requirement on resources, the extra resources to the 
 first
 job can be flexibly re-allocated to the second job.

 On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt <
 mgumm...@mesosphere.io> wrote:

> We've talked about that, but it hasn't become a priority because
> we haven't had a driving use case.  If anyone has a good argument for
> "variable" resource allocation like this, please let me know.
>
> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin  > wrote:
>
>> An alternative behavior is to launch the job with the best
>>> resource offer Mesos is able to give
>>
>>
>> Michael has just made an excellent explanation about dynamic
>> allocation support in mesos. But IIUC, what you want to achieve is
>> something like (using RAM as an example) : "Launch each executor 
>> with at
>> least 1GB RAM, but if mesos offers 2GB at some moment, then launch an
>> executor with 2GB RAM".
>>
>> I wonder what's benefit of that? To reduce the "resource
>> fragmentation"?
>>
>> Anyway, that is not supported at this 

Re: Dynamic resource allocation to Spark on Mesos

[Michael Gummelt]

As of Spark 2.0, Mesos mode does support setting cores on the executor
level, but you might need to set the property directly (--conf
spark.executor.cores=).  I've written about this here:
https://docs.mesosphere.com/1.8/usage/service-guides/spark/job-scheduling/.
That doc is for DC/OS, but the configuration is the same.

On Thu, Feb 2, 2017 at 1:06 PM, Ji Yan  wrote:

> I was mainly confused why this is the case with memory, but with cpu
> cores, it is not specified on per executor level
>
> On Thu, Feb 2, 2017 at 1:02 PM, Michael Gummelt 
> wrote:
>
>> It sounds like you've answered your own question, right?
>> --executor-memory means the memory per executor.  If you have no executor
>> w/ 200GB memory, then the driver will accept no offers.
>>
>> On Thu, Feb 2, 2017 at 1:01 PM, Ji Yan  wrote:
>>
>>> sorry, to clarify, i was using --executor-memory for memory,
>>> and --total-executor-cores for cpu cores
>>>
>>> On Thu, Feb 2, 2017 at 12:56 PM, Michael Gummelt >> > wrote:
>>>
 What CLI args are your referring to?  I'm aware of spark-submit's
 arguments (--executor-memory, --total-executor-cores, and --executor-cores)

 On Thu, Feb 2, 2017 at 12:41 PM, Ji Yan  wrote:

> I have done a experiment on this today. It shows that only CPUs are
> tolerant of insufficient cluster size when a job starts. On my cluster, I
> have 180Gb of memory and 64 cores, when I run spark-submit ( on mesos )
> with --cpu_cores set to 1000, the job starts up with 64 cores. but when I
> set --memory to 200Gb, the job fails to start with "Initial job has
> not accepted any resources; check your cluster UI to ensure that workers
> are registered and have sufficient resources"
>
> Also it is confusing to me that --cpu_cores specifies the number of
> cpu cores across all executors, but --memory specifies per executor memory
> requirement.
>
> On Mon, Jan 30, 2017 at 11:34 AM, Michael Gummelt <
> mgumm...@mesosphere.io> wrote:
>
>>
>>
>> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:
>>
>>> Tasks begin scheduling as soon as the first executor comes up
>>>
>>>
>>> Thanks all for the clarification. Is this the default behavior of
>>> Spark on Mesos today? I think this is what we are looking for because
>>> sometimes a job can take up lots of resources and later jobs could not 
>>> get
>>> all the resources that it asks for. If a Spark job starts with only a
>>> subset of resources that it asks for, does it know to expand its 
>>> resources
>>> later when more resources become available?
>>>
>>
>> Yes.
>>
>>
>>>
>>> Launch each executor with at least 1GB RAM, but if mesos offers 2GB
 at some moment, then launch an executor with 2GB RAM
>>>
>>>
>>> This is less useful in our use case. But I am also quite interested
>>> in cases in which this could be helpful. I think this will also help 
>>> with
>>> overall resource utilization on the cluster if when another job starts 
>>> up
>>> that has a hard requirement on resources, the extra resources to the 
>>> first
>>> job can be flexibly re-allocated to the second job.
>>>
>>> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt <
>>> mgumm...@mesosphere.io> wrote:
>>>
 We've talked about that, but it hasn't become a priority because we
 haven't had a driving use case.  If anyone has a good argument for
 "variable" resource allocation like this, please let me know.

 On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin 
 wrote:

> An alternative behavior is to launch the job with the best
>> resource offer Mesos is able to give
>
>
> Michael has just made an excellent explanation about dynamic
> allocation support in mesos. But IIUC, what you want to achieve is
> something like (using RAM as an example) : "Launch each executor with 
> at
> least 1GB RAM, but if mesos offers 2GB at some moment, then launch an
> executor with 2GB RAM".
>
> I wonder what's benefit of that? To reduce the "resource
> fragmentation"?
>
> Anyway, that is not supported at this moment. In all the supported
> cluster managers of spark (mesos, yarn, standalone, and the 
> up-to-coming
> spark on kubernetes), you have to specify the cores and memory of each
> executor.
>
> It may not be supported in the future, because only mesos has the
> concepts of offers because of its two-level scheduling model.
>
>
> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:
>
>> Dear Spark Users,
>>

Re: Dynamic resource allocation to Spark on Mesos

[Ji Yan]

I was mainly confused why this is the case with memory, but with cpu cores,
it is not specified on per executor level

On Thu, Feb 2, 2017 at 1:02 PM, Michael Gummelt 
wrote:

> It sounds like you've answered your own question, right?
> --executor-memory means the memory per executor.  If you have no executor
> w/ 200GB memory, then the driver will accept no offers.
>
> On Thu, Feb 2, 2017 at 1:01 PM, Ji Yan  wrote:
>
>> sorry, to clarify, i was using --executor-memory for memory,
>> and --total-executor-cores for cpu cores
>>
>> On Thu, Feb 2, 2017 at 12:56 PM, Michael Gummelt 
>> wrote:
>>
>>> What CLI args are your referring to?  I'm aware of spark-submit's
>>> arguments (--executor-memory, --total-executor-cores, and --executor-cores)
>>>
>>> On Thu, Feb 2, 2017 at 12:41 PM, Ji Yan  wrote:
>>>
 I have done a experiment on this today. It shows that only CPUs are
 tolerant of insufficient cluster size when a job starts. On my cluster, I
 have 180Gb of memory and 64 cores, when I run spark-submit ( on mesos )
 with --cpu_cores set to 1000, the job starts up with 64 cores. but when I
 set --memory to 200Gb, the job fails to start with "Initial job has
 not accepted any resources; check your cluster UI to ensure that workers
 are registered and have sufficient resources"

 Also it is confusing to me that --cpu_cores specifies the number of cpu
 cores across all executors, but --memory specifies per executor memory
 requirement.

 On Mon, Jan 30, 2017 at 11:34 AM, Michael Gummelt <
 mgumm...@mesosphere.io> wrote:

>
>
> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:
>
>> Tasks begin scheduling as soon as the first executor comes up
>>
>>
>> Thanks all for the clarification. Is this the default behavior of
>> Spark on Mesos today? I think this is what we are looking for because
>> sometimes a job can take up lots of resources and later jobs could not 
>> get
>> all the resources that it asks for. If a Spark job starts with only a
>> subset of resources that it asks for, does it know to expand its 
>> resources
>> later when more resources become available?
>>
>
> Yes.
>
>
>>
>> Launch each executor with at least 1GB RAM, but if mesos offers 2GB
>>> at some moment, then launch an executor with 2GB RAM
>>
>>
>> This is less useful in our use case. But I am also quite interested
>> in cases in which this could be helpful. I think this will also help with
>> overall resource utilization on the cluster if when another job starts up
>> that has a hard requirement on resources, the extra resources to the 
>> first
>> job can be flexibly re-allocated to the second job.
>>
>> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt <
>> mgumm...@mesosphere.io> wrote:
>>
>>> We've talked about that, but it hasn't become a priority because we
>>> haven't had a driving use case.  If anyone has a good argument for
>>> "variable" resource allocation like this, please let me know.
>>>
>>> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin 
>>> wrote:
>>>
 An alternative behavior is to launch the job with the best resource
> offer Mesos is able to give


 Michael has just made an excellent explanation about dynamic
 allocation support in mesos. But IIUC, what you want to achieve is
 something like (using RAM as an example) : "Launch each executor with 
 at
 least 1GB RAM, but if mesos offers 2GB at some moment, then launch an
 executor with 2GB RAM".

 I wonder what's benefit of that? To reduce the "resource
 fragmentation"?

 Anyway, that is not supported at this moment. In all the supported
 cluster managers of spark (mesos, yarn, standalone, and the 
 up-to-coming
 spark on kubernetes), you have to specify the cores and memory of each
 executor.

 It may not be supported in the future, because only mesos has the
 concepts of offers because of its two-level scheduling model.


 On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:

> Dear Spark Users,
>
> Currently is there a way to dynamically allocate resources to
> Spark on Mesos? Within Spark we can specify the CPU cores, memory 
> before
> running job. The way I understand is that the Spark job will not run 
> if the
> CPU/Mem requirement is not met. This may lead to decrease in overall
> utilization of the cluster. An alternative behavior is to launch the 
> job
> with the best resource offer Mesos is able to give. Is this possible 

Re: Dynamic resource allocation to Spark on Mesos

[Michael Gummelt]

It sounds like you've answered your own question, right?  --executor-memory
means the memory per executor.  If you have no executor w/ 200GB memory,
then the driver will accept no offers.

On Thu, Feb 2, 2017 at 1:01 PM, Ji Yan  wrote:

> sorry, to clarify, i was using --executor-memory for memory,
> and --total-executor-cores for cpu cores
>
> On Thu, Feb 2, 2017 at 12:56 PM, Michael Gummelt 
> wrote:
>
>> What CLI args are your referring to?  I'm aware of spark-submit's
>> arguments (--executor-memory, --total-executor-cores, and --executor-cores)
>>
>> On Thu, Feb 2, 2017 at 12:41 PM, Ji Yan  wrote:
>>
>>> I have done a experiment on this today. It shows that only CPUs are
>>> tolerant of insufficient cluster size when a job starts. On my cluster, I
>>> have 180Gb of memory and 64 cores, when I run spark-submit ( on mesos )
>>> with --cpu_cores set to 1000, the job starts up with 64 cores. but when I
>>> set --memory to 200Gb, the job fails to start with "Initial job has not
>>> accepted any resources; check your cluster UI to ensure that workers are
>>> registered and have sufficient resources"
>>>
>>> Also it is confusing to me that --cpu_cores specifies the number of cpu
>>> cores across all executors, but --memory specifies per executor memory
>>> requirement.
>>>
>>> On Mon, Jan 30, 2017 at 11:34 AM, Michael Gummelt <
>>> mgumm...@mesosphere.io> wrote:
>>>


 On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:

> Tasks begin scheduling as soon as the first executor comes up
>
>
> Thanks all for the clarification. Is this the default behavior of
> Spark on Mesos today? I think this is what we are looking for because
> sometimes a job can take up lots of resources and later jobs could not get
> all the resources that it asks for. If a Spark job starts with only a
> subset of resources that it asks for, does it know to expand its resources
> later when more resources become available?
>

 Yes.


>
> Launch each executor with at least 1GB RAM, but if mesos offers 2GB at
>> some moment, then launch an executor with 2GB RAM
>
>
> This is less useful in our use case. But I am also quite interested in
> cases in which this could be helpful. I think this will also help with
> overall resource utilization on the cluster if when another job starts up
> that has a hard requirement on resources, the extra resources to the first
> job can be flexibly re-allocated to the second job.
>
> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt <
> mgumm...@mesosphere.io> wrote:
>
>> We've talked about that, but it hasn't become a priority because we
>> haven't had a driving use case.  If anyone has a good argument for
>> "variable" resource allocation like this, please let me know.
>>
>> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin 
>> wrote:
>>
>>> An alternative behavior is to launch the job with the best resource
 offer Mesos is able to give
>>>
>>>
>>> Michael has just made an excellent explanation about dynamic
>>> allocation support in mesos. But IIUC, what you want to achieve is
>>> something like (using RAM as an example) : "Launch each executor with at
>>> least 1GB RAM, but if mesos offers 2GB at some moment, then launch an
>>> executor with 2GB RAM".
>>>
>>> I wonder what's benefit of that? To reduce the "resource
>>> fragmentation"?
>>>
>>> Anyway, that is not supported at this moment. In all the supported
>>> cluster managers of spark (mesos, yarn, standalone, and the up-to-coming
>>> spark on kubernetes), you have to specify the cores and memory of each
>>> executor.
>>>
>>> It may not be supported in the future, because only mesos has the
>>> concepts of offers because of its two-level scheduling model.
>>>
>>>
>>> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:
>>>
 Dear Spark Users,

 Currently is there a way to dynamically allocate resources to Spark
 on Mesos? Within Spark we can specify the CPU cores, memory before 
 running
 job. The way I understand is that the Spark job will not run if the 
 CPU/Mem
 requirement is not met. This may lead to decrease in overall 
 utilization of
 the cluster. An alternative behavior is to launch the job with the best
 resource offer Mesos is able to give. Is this possible with the current
 implementation?

 Thanks
 Ji

 The information in this email is confidential and may be legally
 privileged. It is intended solely for the addressee. Access to this 
 email
 by anyone else is unauthorized. If you are not the intended recipient, 
 any
 

Re: Dynamic resource allocation to Spark on Mesos

[Ji Yan]

sorry, to clarify, i was using --executor-memory for memory,
and --total-executor-cores for cpu cores

On Thu, Feb 2, 2017 at 12:56 PM, Michael Gummelt 
wrote:

> What CLI args are your referring to?  I'm aware of spark-submit's
> arguments (--executor-memory, --total-executor-cores, and --executor-cores)
>
> On Thu, Feb 2, 2017 at 12:41 PM, Ji Yan  wrote:
>
>> I have done a experiment on this today. It shows that only CPUs are
>> tolerant of insufficient cluster size when a job starts. On my cluster, I
>> have 180Gb of memory and 64 cores, when I run spark-submit ( on mesos )
>> with --cpu_cores set to 1000, the job starts up with 64 cores. but when I
>> set --memory to 200Gb, the job fails to start with "Initial job has not
>> accepted any resources; check your cluster UI to ensure that workers are
>> registered and have sufficient resources"
>>
>> Also it is confusing to me that --cpu_cores specifies the number of cpu
>> cores across all executors, but --memory specifies per executor memory
>> requirement.
>>
>> On Mon, Jan 30, 2017 at 11:34 AM, Michael Gummelt > > wrote:
>>
>>>
>>>
>>> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:
>>>
 Tasks begin scheduling as soon as the first executor comes up


 Thanks all for the clarification. Is this the default behavior of Spark
 on Mesos today? I think this is what we are looking for because sometimes a
 job can take up lots of resources and later jobs could not get all the
 resources that it asks for. If a Spark job starts with only a subset of
 resources that it asks for, does it know to expand its resources later when
 more resources become available?

>>>
>>> Yes.
>>>
>>>

 Launch each executor with at least 1GB RAM, but if mesos offers 2GB at
> some moment, then launch an executor with 2GB RAM


 This is less useful in our use case. But I am also quite interested in
 cases in which this could be helpful. I think this will also help with
 overall resource utilization on the cluster if when another job starts up
 that has a hard requirement on resources, the extra resources to the first
 job can be flexibly re-allocated to the second job.

 On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt <
 mgumm...@mesosphere.io> wrote:

> We've talked about that, but it hasn't become a priority because we
> haven't had a driving use case.  If anyone has a good argument for
> "variable" resource allocation like this, please let me know.
>
> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin 
> wrote:
>
>> An alternative behavior is to launch the job with the best resource
>>> offer Mesos is able to give
>>
>>
>> Michael has just made an excellent explanation about dynamic
>> allocation support in mesos. But IIUC, what you want to achieve is
>> something like (using RAM as an example) : "Launch each executor with at
>> least 1GB RAM, but if mesos offers 2GB at some moment, then launch an
>> executor with 2GB RAM".
>>
>> I wonder what's benefit of that? To reduce the "resource
>> fragmentation"?
>>
>> Anyway, that is not supported at this moment. In all the supported
>> cluster managers of spark (mesos, yarn, standalone, and the up-to-coming
>> spark on kubernetes), you have to specify the cores and memory of each
>> executor.
>>
>> It may not be supported in the future, because only mesos has the
>> concepts of offers because of its two-level scheduling model.
>>
>>
>> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:
>>
>>> Dear Spark Users,
>>>
>>> Currently is there a way to dynamically allocate resources to Spark
>>> on Mesos? Within Spark we can specify the CPU cores, memory before 
>>> running
>>> job. The way I understand is that the Spark job will not run if the 
>>> CPU/Mem
>>> requirement is not met. This may lead to decrease in overall 
>>> utilization of
>>> the cluster. An alternative behavior is to launch the job with the best
>>> resource offer Mesos is able to give. Is this possible with the current
>>> implementation?
>>>
>>> Thanks
>>> Ji
>>>
>>> The information in this email is confidential and may be legally
>>> privileged. It is intended solely for the addressee. Access to this 
>>> email
>>> by anyone else is unauthorized. If you are not the intended recipient, 
>>> any
>>> disclosure, copying, distribution or any action taken or omitted to be
>>> taken in reliance on it, is prohibited and may be unlawful.
>>>
>>
>>
>
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere
>


 The information in this email is confidential and may be legally
 privileged. It is intended solely 

Re: Dynamic resource allocation to Spark on Mesos

[Michael Gummelt]

What CLI args are your referring to?  I'm aware of spark-submit's arguments
(--executor-memory, --total-executor-cores, and --executor-cores)

On Thu, Feb 2, 2017 at 12:41 PM, Ji Yan  wrote:

> I have done a experiment on this today. It shows that only CPUs are
> tolerant of insufficient cluster size when a job starts. On my cluster, I
> have 180Gb of memory and 64 cores, when I run spark-submit ( on mesos )
> with --cpu_cores set to 1000, the job starts up with 64 cores. but when I
> set --memory to 200Gb, the job fails to start with "Initial job has not
> accepted any resources; check your cluster UI to ensure that workers are
> registered and have sufficient resources"
>
> Also it is confusing to me that --cpu_cores specifies the number of cpu
> cores across all executors, but --memory specifies per executor memory
> requirement.
>
> On Mon, Jan 30, 2017 at 11:34 AM, Michael Gummelt 
> wrote:
>
>>
>>
>> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:
>>
>>> Tasks begin scheduling as soon as the first executor comes up
>>>
>>>
>>> Thanks all for the clarification. Is this the default behavior of Spark
>>> on Mesos today? I think this is what we are looking for because sometimes a
>>> job can take up lots of resources and later jobs could not get all the
>>> resources that it asks for. If a Spark job starts with only a subset of
>>> resources that it asks for, does it know to expand its resources later when
>>> more resources become available?
>>>
>>
>> Yes.
>>
>>
>>>
>>> Launch each executor with at least 1GB RAM, but if mesos offers 2GB at
 some moment, then launch an executor with 2GB RAM
>>>
>>>
>>> This is less useful in our use case. But I am also quite interested in
>>> cases in which this could be helpful. I think this will also help with
>>> overall resource utilization on the cluster if when another job starts up
>>> that has a hard requirement on resources, the extra resources to the first
>>> job can be flexibly re-allocated to the second job.
>>>
>>> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt >> > wrote:
>>>
 We've talked about that, but it hasn't become a priority because we
 haven't had a driving use case.  If anyone has a good argument for
 "variable" resource allocation like this, please let me know.

 On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin 
 wrote:

> An alternative behavior is to launch the job with the best resource
>> offer Mesos is able to give
>
>
> Michael has just made an excellent explanation about dynamic
> allocation support in mesos. But IIUC, what you want to achieve is
> something like (using RAM as an example) : "Launch each executor with at
> least 1GB RAM, but if mesos offers 2GB at some moment, then launch an
> executor with 2GB RAM".
>
> I wonder what's benefit of that? To reduce the "resource
> fragmentation"?
>
> Anyway, that is not supported at this moment. In all the supported
> cluster managers of spark (mesos, yarn, standalone, and the up-to-coming
> spark on kubernetes), you have to specify the cores and memory of each
> executor.
>
> It may not be supported in the future, because only mesos has the
> concepts of offers because of its two-level scheduling model.
>
>
> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:
>
>> Dear Spark Users,
>>
>> Currently is there a way to dynamically allocate resources to Spark
>> on Mesos? Within Spark we can specify the CPU cores, memory before 
>> running
>> job. The way I understand is that the Spark job will not run if the 
>> CPU/Mem
>> requirement is not met. This may lead to decrease in overall utilization 
>> of
>> the cluster. An alternative behavior is to launch the job with the best
>> resource offer Mesos is able to give. Is this possible with the current
>> implementation?
>>
>> Thanks
>> Ji
>>
>> The information in this email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this email
>> by anyone else is unauthorized. If you are not the intended recipient, 
>> any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful.
>>
>
>


 --
 Michael Gummelt
 Software Engineer
 Mesosphere

>>>
>>>
>>> The information in this email is confidential and may be legally
>>> privileged. It is intended solely for the addressee. Access to this email
>>> by anyone else is unauthorized. If you are not the intended recipient, any
>>> disclosure, copying, distribution or any action taken or omitted to be
>>> taken in reliance on it, is prohibited and may be unlawful.
>>>
>>
>>
>>
>> --
>> Michael Gummelt
>> Software Engineer

Re: Dynamic resource allocation to Spark on Mesos

[Ji Yan]

I have done a experiment on this today. It shows that only CPUs are
tolerant of insufficient cluster size when a job starts. On my cluster, I
have 180Gb of memory and 64 cores, when I run spark-submit ( on mesos )
with --cpu_cores set to 1000, the job starts up with 64 cores. but when I
set --memory to 200Gb, the job fails to start with "Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient resources"

Also it is confusing to me that --cpu_cores specifies the number of cpu
cores across all executors, but --memory specifies per executor memory
requirement.

On Mon, Jan 30, 2017 at 11:34 AM, Michael Gummelt 
wrote:

>
>
> On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:
>
>> Tasks begin scheduling as soon as the first executor comes up
>>
>>
>> Thanks all for the clarification. Is this the default behavior of Spark
>> on Mesos today? I think this is what we are looking for because sometimes a
>> job can take up lots of resources and later jobs could not get all the
>> resources that it asks for. If a Spark job starts with only a subset of
>> resources that it asks for, does it know to expand its resources later when
>> more resources become available?
>>
>
> Yes.
>
>
>>
>> Launch each executor with at least 1GB RAM, but if mesos offers 2GB at
>>> some moment, then launch an executor with 2GB RAM
>>
>>
>> This is less useful in our use case. But I am also quite interested in
>> cases in which this could be helpful. I think this will also help with
>> overall resource utilization on the cluster if when another job starts up
>> that has a hard requirement on resources, the extra resources to the first
>> job can be flexibly re-allocated to the second job.
>>
>> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt 
>> wrote:
>>
>>> We've talked about that, but it hasn't become a priority because we
>>> haven't had a driving use case.  If anyone has a good argument for
>>> "variable" resource allocation like this, please let me know.
>>>
>>> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin 
>>> wrote:
>>>
 An alternative behavior is to launch the job with the best resource
> offer Mesos is able to give


 Michael has just made an excellent explanation about dynamic allocation
 support in mesos. But IIUC, what you want to achieve is something like
 (using RAM as an example) : "Launch each executor with at least 1GB RAM,
 but if mesos offers 2GB at some moment, then launch an executor with 2GB
 RAM".

 I wonder what's benefit of that? To reduce the "resource fragmentation"?

 Anyway, that is not supported at this moment. In all the supported
 cluster managers of spark (mesos, yarn, standalone, and the up-to-coming
 spark on kubernetes), you have to specify the cores and memory of each
 executor.

 It may not be supported in the future, because only mesos has the
 concepts of offers because of its two-level scheduling model.


 On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:

> Dear Spark Users,
>
> Currently is there a way to dynamically allocate resources to Spark on
> Mesos? Within Spark we can specify the CPU cores, memory before running
> job. The way I understand is that the Spark job will not run if the 
> CPU/Mem
> requirement is not met. This may lead to decrease in overall utilization 
> of
> the cluster. An alternative behavior is to launch the job with the best
> resource offer Mesos is able to give. Is this possible with the current
> implementation?
>
> Thanks
> Ji
>
> The information in this email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
>


>>>
>>>
>>> --
>>> Michael Gummelt
>>> Software Engineer
>>> Mesosphere
>>>
>>
>>
>> The information in this email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this email
>> by anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful.
>>
>
>
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere
>

-- 
 

The information in this email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this email 
by anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be 
taken in reliance on it, is prohibited and may be 

Re: Dynamic resource allocation to Spark on Mesos

[Michael Gummelt]

On Mon, Jan 30, 2017 at 9:47 AM, Ji Yan  wrote:

> Tasks begin scheduling as soon as the first executor comes up
>
>
> Thanks all for the clarification. Is this the default behavior of Spark on
> Mesos today? I think this is what we are looking for because sometimes a
> job can take up lots of resources and later jobs could not get all the
> resources that it asks for. If a Spark job starts with only a subset of
> resources that it asks for, does it know to expand its resources later when
> more resources become available?
>

Yes.


>
> Launch each executor with at least 1GB RAM, but if mesos offers 2GB at
>> some moment, then launch an executor with 2GB RAM
>
>
> This is less useful in our use case. But I am also quite interested in
> cases in which this could be helpful. I think this will also help with
> overall resource utilization on the cluster if when another job starts up
> that has a hard requirement on resources, the extra resources to the first
> job can be flexibly re-allocated to the second job.
>
> On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt 
> wrote:
>
>> We've talked about that, but it hasn't become a priority because we
>> haven't had a driving use case.  If anyone has a good argument for
>> "variable" resource allocation like this, please let me know.
>>
>> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin 
>> wrote:
>>
>>> An alternative behavior is to launch the job with the best resource
 offer Mesos is able to give
>>>
>>>
>>> Michael has just made an excellent explanation about dynamic allocation
>>> support in mesos. But IIUC, what you want to achieve is something like
>>> (using RAM as an example) : "Launch each executor with at least 1GB RAM,
>>> but if mesos offers 2GB at some moment, then launch an executor with 2GB
>>> RAM".
>>>
>>> I wonder what's benefit of that? To reduce the "resource fragmentation"?
>>>
>>> Anyway, that is not supported at this moment. In all the supported
>>> cluster managers of spark (mesos, yarn, standalone, and the up-to-coming
>>> spark on kubernetes), you have to specify the cores and memory of each
>>> executor.
>>>
>>> It may not be supported in the future, because only mesos has the
>>> concepts of offers because of its two-level scheduling model.
>>>
>>>
>>> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:
>>>
 Dear Spark Users,

 Currently is there a way to dynamically allocate resources to Spark on
 Mesos? Within Spark we can specify the CPU cores, memory before running
 job. The way I understand is that the Spark job will not run if the CPU/Mem
 requirement is not met. This may lead to decrease in overall utilization of
 the cluster. An alternative behavior is to launch the job with the best
 resource offer Mesos is able to give. Is this possible with the current
 implementation?

 Thanks
 Ji

 The information in this email is confidential and may be legally
 privileged. It is intended solely for the addressee. Access to this email
 by anyone else is unauthorized. If you are not the intended recipient, any
 disclosure, copying, distribution or any action taken or omitted to be
 taken in reliance on it, is prohibited and may be unlawful.

>>>
>>>
>>
>>
>> --
>> Michael Gummelt
>> Software Engineer
>> Mesosphere
>>
>
>
> The information in this email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
>



-- 
Michael Gummelt
Software Engineer
Mesosphere

Re: Dynamic resource allocation to Spark on Mesos

[Ji Yan]

>
> Tasks begin scheduling as soon as the first executor comes up


Thanks all for the clarification. Is this the default behavior of Spark on
Mesos today? I think this is what we are looking for because sometimes a
job can take up lots of resources and later jobs could not get all the
resources that it asks for. If a Spark job starts with only a subset of
resources that it asks for, does it know to expand its resources later when
more resources become available?

Launch each executor with at least 1GB RAM, but if mesos offers 2GB at some
> moment, then launch an executor with 2GB RAM


This is less useful in our use case. But I am also quite interested in
cases in which this could be helpful. I think this will also help with
overall resource utilization on the cluster if when another job starts up
that has a hard requirement on resources, the extra resources to the first
job can be flexibly re-allocated to the second job.

On Sat, Jan 28, 2017 at 2:32 PM, Michael Gummelt 
wrote:

> We've talked about that, but it hasn't become a priority because we
> haven't had a driving use case.  If anyone has a good argument for
> "variable" resource allocation like this, please let me know.
>
> On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin  wrote:
>
>> An alternative behavior is to launch the job with the best resource offer
>>> Mesos is able to give
>>
>>
>> Michael has just made an excellent explanation about dynamic allocation
>> support in mesos. But IIUC, what you want to achieve is something like
>> (using RAM as an example) : "Launch each executor with at least 1GB RAM,
>> but if mesos offers 2GB at some moment, then launch an executor with 2GB
>> RAM".
>>
>> I wonder what's benefit of that? To reduce the "resource fragmentation"?
>>
>> Anyway, that is not supported at this moment. In all the supported
>> cluster managers of spark (mesos, yarn, standalone, and the up-to-coming
>> spark on kubernetes), you have to specify the cores and memory of each
>> executor.
>>
>> It may not be supported in the future, because only mesos has the
>> concepts of offers because of its two-level scheduling model.
>>
>>
>> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:
>>
>>> Dear Spark Users,
>>>
>>> Currently is there a way to dynamically allocate resources to Spark on
>>> Mesos? Within Spark we can specify the CPU cores, memory before running
>>> job. The way I understand is that the Spark job will not run if the CPU/Mem
>>> requirement is not met. This may lead to decrease in overall utilization of
>>> the cluster. An alternative behavior is to launch the job with the best
>>> resource offer Mesos is able to give. Is this possible with the current
>>> implementation?
>>>
>>> Thanks
>>> Ji
>>>
>>> The information in this email is confidential and may be legally
>>> privileged. It is intended solely for the addressee. Access to this email
>>> by anyone else is unauthorized. If you are not the intended recipient, any
>>> disclosure, copying, distribution or any action taken or omitted to be
>>> taken in reliance on it, is prohibited and may be unlawful.
>>>
>>
>>
>
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere
>

-- 
 

The information in this email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this email 
by anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be 
taken in reliance on it, is prohibited and may be unlawful.

Re: Dynamic resource allocation to Spark on Mesos

[Michael Gummelt]

We've talked about that, but it hasn't become a priority because we haven't
had a driving use case.  If anyone has a good argument for "variable"
resource allocation like this, please let me know.

On Sat, Jan 28, 2017 at 9:17 AM, Shuai Lin  wrote:

> An alternative behavior is to launch the job with the best resource offer
>> Mesos is able to give
>
>
> Michael has just made an excellent explanation about dynamic allocation
> support in mesos. But IIUC, what you want to achieve is something like
> (using RAM as an example) : "Launch each executor with at least 1GB RAM,
> but if mesos offers 2GB at some moment, then launch an executor with 2GB
> RAM".
>
> I wonder what's benefit of that? To reduce the "resource fragmentation"?
>
> Anyway, that is not supported at this moment. In all the supported cluster
> managers of spark (mesos, yarn, standalone, and the up-to-coming spark on
> kubernetes), you have to specify the cores and memory of each executor.
>
> It may not be supported in the future, because only mesos has the concepts
> of offers because of its two-level scheduling model.
>
>
> On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:
>
>> Dear Spark Users,
>>
>> Currently is there a way to dynamically allocate resources to Spark on
>> Mesos? Within Spark we can specify the CPU cores, memory before running
>> job. The way I understand is that the Spark job will not run if the CPU/Mem
>> requirement is not met. This may lead to decrease in overall utilization of
>> the cluster. An alternative behavior is to launch the job with the best
>> resource offer Mesos is able to give. Is this possible with the current
>> implementation?
>>
>> Thanks
>> Ji
>>
>> The information in this email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this email
>> by anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful.
>>
>
>


-- 
Michael Gummelt
Software Engineer
Mesosphere

Re: Dynamic resource allocation to Spark on Mesos

[Shuai Lin]

>
> An alternative behavior is to launch the job with the best resource offer
> Mesos is able to give


Michael has just made an excellent explanation about dynamic allocation
support in mesos. But IIUC, what you want to achieve is something like
(using RAM as an example) : "Launch each executor with at least 1GB RAM,
but if mesos offers 2GB at some moment, then launch an executor with 2GB
RAM".

I wonder what's benefit of that? To reduce the "resource fragmentation"?

Anyway, that is not supported at this moment. In all the supported cluster
managers of spark (mesos, yarn, standalone, and the up-to-coming spark on
kubernetes), you have to specify the cores and memory of each executor.

It may not be supported in the future, because only mesos has the concepts
of offers because of its two-level scheduling model.


On Sat, Jan 28, 2017 at 1:35 AM, Ji Yan  wrote:

> Dear Spark Users,
>
> Currently is there a way to dynamically allocate resources to Spark on
> Mesos? Within Spark we can specify the CPU cores, memory before running
> job. The way I understand is that the Spark job will not run if the CPU/Mem
> requirement is not met. This may lead to decrease in overall utilization of
> the cluster. An alternative behavior is to launch the job with the best
> resource offer Mesos is able to give. Is this possible with the current
> implementation?
>
> Thanks
> Ji
>
> The information in this email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
>

Re: Dynamic resource allocation to Spark on Mesos

[Mihai Iacob]

What about Spark on Kubernetes, is there a way to manage dynamic resource allocation?
 
Regards, 

Mihai Iacob 
 
- Original message -From: Michael Gummelt To: Ji Yan Cc: user Subject: Re: Dynamic resource allocation to Spark on MesosDate: Fri, Jan 27, 2017 2:14 PM 
> The way I understand is that the Spark job will not run if the CPU/Mem requirement is not met. Spark jobs will still run if they only have a subset of the requested resources.  Tasks begin scheduling as soon as the first executor comes up.  Dynamic allocation yields increased utilization by only allocating as many executors as a job needs, rather than a single static amount set up front. Dynamic Allocation is supported in Spark on Mesos, but we here at Mesosphere haven't been testing it much, and I'm not sure what the community adoption is.  So I can't yet speak to its robustness, but we will be investing in it soon.  Many users want it.
 
On Fri, Jan 27, 2017 at 9:35 AM, Ji Yan  wrote:

Dear Spark Users,
 
Currently is there a way to dynamically allocate resources to Spark on Mesos? Within Spark we can specify the CPU cores, memory before running job. The way I understand is that the Spark job will not run if the CPU/Mem requirement is not met. This may lead to decrease in overall utilization of the cluster. An alternative behavior is to launch the job with the best resource offer Mesos is able to give. Is this possible with the current implementation?
 
Thanks
Ji 

The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful.--
 Michael GummeltSoftware EngineerMesosphere
 


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Dynamic resource allocation to Spark on Mesos

[Michael Gummelt]

> The way I understand is that the Spark job will not run if the CPU/Mem
requirement is not met.

Spark jobs will still run if they only have a subset of the requested
resources.  Tasks begin scheduling as soon as the first executor comes up.
Dynamic allocation yields increased utilization by only allocating as many
executors as a job needs, rather than a single static amount set up front.

Dynamic Allocation is supported in Spark on Mesos, but we here at
Mesosphere haven't been testing it much, and I'm not sure what the
community adoption is.  So I can't yet speak to its robustness, but we will
be investing in it soon.  Many users want it.

On Fri, Jan 27, 2017 at 9:35 AM, Ji Yan  wrote:

> Dear Spark Users,
>
> Currently is there a way to dynamically allocate resources to Spark on
> Mesos? Within Spark we can specify the CPU cores, memory before running
> job. The way I understand is that the Spark job will not run if the CPU/Mem
> requirement is not met. This may lead to decrease in overall utilization of
> the cluster. An alternative behavior is to launch the job with the best
> resource offer Mesos is able to give. Is this possible with the current
> implementation?
>
> Thanks
> Ji
>
> The information in this email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
>



-- 
Michael Gummelt
Software Engineer
Mesosphere