[jira] [Created] (SPARK-26588) Idle executor should properly be killed when no job is submitted

2019-01-10 Thread Qingxin Wu (JIRA) 
Qingxin Wu created SPARK-26588:
--

 Summary: Idle executor should properly be killed when no job is 
submitted
 Key: SPARK-26588
 URL: https://issues.apache.org/jira/browse/SPARK-26588
 Project: Spark
  Issue Type: Bug
  Components: Scheduler, Spark Core
Affects Versions: 2.4.0
Reporter: Qingxin Wu


I enable dynamic allocation feature with *spark-shell* and do not submit any 
task. After *spark.dynamicAllocation.executorIdleTimeout* seconds(default 60s), 
there is still one active executor, which is abnormal. All idle executors are 
timeout and should be removed.(default 
*spark.dynamicAllocation.minExecutors*=0). The spark-shell command show below:
{code:java}
spark-shell --master=yarn --conf spark.ui.port=8040 --conf 
spark.dynamicAllocation.enabled=true --conf 
spark.dynamicAllocation.maxExecutors=8 --conf 
spark.dynamicAllocation.initialExecutors=4 --conf 
spark.shuffle.service.enabled=true
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-26446) Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager

2018-12-27 Thread Qingxin Wu (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qingxin Wu updated SPARK-26446:
---
Description: 
Add docs to describe how remove policy act while considering the property 
_*{{spark.dynamicAllocation.cachedExecutorIdleTimeout}}*_ in 
ExecutorAllocationManager.

 

 

  was:
 

Add docs to describe how remove policy act while considering the property
{code:java}
spark.dynamicAllocation.cachedExecutorIdleTimeout
{code}
  in ExecutorAllocationManager.

 

 

 


> Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager
> ---
>
> Key: SPARK-26446
> URL: https://issues.apache.org/jira/browse/SPARK-26446
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Qingxin Wu
>Priority: Minor
>
> Add docs to describe how remove policy act while considering the property 
> _*{{spark.dynamicAllocation.cachedExecutorIdleTimeout}}*_ in 
> ExecutorAllocationManager.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-26446) Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager

2018-12-27 Thread Qingxin Wu (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qingxin Wu updated SPARK-26446:
---
Description: 
 

Add docs to describe how remove policy act while considering the property
{code:java}
spark.dynamicAllocation.cachedExecutorIdleTimeout
{code}
  in ExecutorAllocationManager.

 

 

 

  was:
Add docs to describe how remove policy act while considering the property 
{{spark.dynamicAllocation.cachedExecutorIdleTimeout}} in 
ExecutorAllocationManager.

 

 


> Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager
> ---
>
> Key: SPARK-26446
> URL: https://issues.apache.org/jira/browse/SPARK-26446
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Qingxin Wu
>Priority: Minor
>
>  
> Add docs to describe how remove policy act while considering the property
> {code:java}
> spark.dynamicAllocation.cachedExecutorIdleTimeout
> {code}
>   in ExecutorAllocationManager.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-26446) Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager

2018-12-27 Thread Qingxin Wu (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qingxin Wu updated SPARK-26446:
---
Description: 
Add docs to describe how remove policy act while considering the property 
{{spark.dynamicAllocation.cachedExecutorIdleTimeout}} in 
ExecutorAllocationManager.

 

 

> Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager
> ---
>
> Key: SPARK-26446
> URL: https://issues.apache.org/jira/browse/SPARK-26446
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Qingxin Wu
>Priority: Minor
>
> Add docs to describe how remove policy act while considering the property 
> {{spark.dynamicAllocation.cachedExecutorIdleTimeout}} in 
> ExecutorAllocationManager.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-26446) Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager

2018-12-27 Thread Qingxin Wu (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qingxin Wu updated SPARK-26446:
---
Summary: Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager  
(was: improve doc on ExecutorAllocationManager)

> Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager
> ---
>
> Key: SPARK-26446
> URL: https://issues.apache.org/jira/browse/SPARK-26446
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Qingxin Wu
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-26446) improve doc on ExecutorAllocationManager

2018-12-26 Thread Qingxin Wu (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qingxin Wu updated SPARK-26446:
---
Component/s: (was: Scheduler)
 Spark Core

> improve doc on ExecutorAllocationManager
> 
>
> Key: SPARK-26446
> URL: https://issues.apache.org/jira/browse/SPARK-26446
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Qingxin Wu
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-26446) improve doc on ExecutorAllocationManager

2018-12-26 Thread Qingxin Wu (JIRA) 
Qingxin Wu created SPARK-26446:
--

 Summary: improve doc on ExecutorAllocationManager
 Key: SPARK-26446
 URL: https://issues.apache.org/jira/browse/SPARK-26446
 Project: Spark
  Issue Type: Improvement
  Components: Scheduler
Affects Versions: 2.4.0
Reporter: Qingxin Wu






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org