[
https://issues.apache.org/jira/browse/SPARK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934544#comment-14934544
]
Balagopal Nair commented on SPARK-10644:
----------------------------------------
Let me try to explain this one last time..
7 machines - 4 cores, 8GB RAM (Physical hardware)
Number of worker processes - 3
Number of executors per worker processes - 3
Total number of workers = 21
Total number of executors = 63
Per worker memory limit = 512m
Per executor memory limit = 512m
Scenario 1:
Submit one job requesting 21 cores => Number of remaining cores = 43
Submit another job requesting 20 cores - This WAITS
Scenario 2:
Submit one job requesting 20 cores => Number of remaining cores = 43
Submit one more job requesting 20 cores - This RUNS => Number of remaining
cores = 23.
Submit one more job requesting 20 cores - This WAITS
Comparing scenario 1 and 2, the speculation/theory based on lack of memory do
not hold.
What I'm trying to say here is that if at at least one worker is not free,
executors don't get allocated. This is the behavior that I see while using
spark. If you would still like to close it, please go ahead. I don't have
anymore details to provide.
> Applications wait even if free executors are available
> ------------------------------------------------------
>
> Key: SPARK-10644
> URL: https://issues.apache.org/jira/browse/SPARK-10644
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.5.0
> Environment: RHEL 6.5 64 bit
> Reporter: Balagopal Nair
> Priority: Minor
>
> Number of workers: 21
> Number of executors: 63
> Steps to reproduce:
> 1. Run 4 jobs each with max cores set to 10
> 2. The first 3 jobs run with 10 each. (30 executors consumed so far)
> 3. The 4 th job waits even though there are 33 idle executors.
> The reason is that a job will not get executors unless
> the total number of EXECUTORS in use < the number of WORKERS
> If there are executors available, resources should be allocated to the
> pending job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
