t oo created SPARK-27750:
----------------------------
Summary: Standalone scheduler - ability to prioritize applications
over drivers, many drivers act like Denial of Service
Key: SPARK-27750
URL: https://issues.apache.org/jira/browse/SPARK-27750
Project: Spark
Issue Type: New Feature
Components: Scheduler
Affects Versions: 2.4.3, 2.3.3
Reporter: t oo
If I submit 1000 spark submit drivers then they consume all the cores on my
cluster (essentially it acts like a Denial of Service) and no spark
'application' gets to run since the cores are all consumed by the 'drivers'.
This feature is about having the ability to prioritize applications over
drivers so that at least some 'applications' can start running. I guess it
would be like: If (driver.state = 'submitted' and (exists some app.state =
'submitted')) then set app.state = 'running'
if all apps have app.state = 'running' then set driver.state = 'submitted'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
