[jira] [Updated] (SPARK-27104) Spark Fair scheduler across applications in standalone mode
[ https://issues.apache.org/jira/browse/SPARK-27104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Zhang updated SPARK-27104: -- Description: Spark in standalone mode currently only supports FIFO (first-in-first-out) scheduler across applications. It will be great that a fair scheduler is supported. +A fair scheduler across applications, not in a application.+ Use case (for example with the integration of zeppelin) At certain moment, user A submits a heavy application and consumes all the resources of the spark cluster. At a later moment, user B submits a second application. No matter how many work nodes you added now, all the resources go to user A due to the FIFO. User B will never get any resource until user A release its allocated resources. A fair scheduler should distribute extra resources in a fair way on all running applications, which demands resources. was: Spark in standalone mode currently only supports FIFO (first-in-first-out) scheduler across applications. It will be great that a fair scheduler is supported. +A fair scheduler across applications, not in a application.+ Use case (for example with the integration of zeppelin) At certain moment, user A submits an heavy application and consumes all the resources of the spark cluster. At a later moment, user B submits a second application. No matter how many work nodes you added now, all the resources go to user A due to the FIFO. User B will never get any resource until user A release its allocated resources. A fair scheduler should distribute extra resources in a fair way on all running applications, which demands resources. > Spark Fair scheduler across applications in standalone mode > --- > > Key: SPARK-27104 > URL: https://issues.apache.org/jira/browse/SPARK-27104 > Project: Spark > Issue Type: Wish > Components: Scheduler >Affects Versions: 2.2.3, 2.3.3, 2.4.0 >Reporter: Hua Zhang >Priority: Minor > > Spark in standalone mode currently only supports FIFO (first-in-first-out) > scheduler across applications. > It will be great that a fair scheduler is supported. +A fair scheduler across > applications, not in a application.+ > > Use case (for example with the integration of zeppelin) > At certain moment, user A submits a heavy application and consumes all the > resources of the spark cluster. > At a later moment, user B submits a second application. > No matter how many work nodes you added now, all the resources go to user A > due to the FIFO. User B will never get any resource until user A release its > allocated resources. > > A fair scheduler should distribute extra resources in a fair way on all > running applications, which demands resources. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27104) Spark Fair scheduler across applications in standalone mode
[ https://issues.apache.org/jira/browse/SPARK-27104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Zhang updated SPARK-27104: -- Description: Spark in standalone mode currently only supports FIFO (first-in-first-out) scheduler across applications. It will be great that a fair scheduler is supported. +A fair scheduler across applications, not in a application.+ Use case (for example with the integration of zeppelin) At certain moment, user A submits an heavy application and consumes all the resources of the spark cluster. At a later moment, user B submits a second application. No matter how many work nodes you added now, all the resources go to user A due to the FIFO. User B will never get any resource until user A release its allocated resources. A fair scheduler should distribute extra resources in a fair way on all running applications, which demands resources. was: Spark in standalone mode currently only supports FIFO (first-in-first-out) scheduler across applications. It will be great that a fair scheduler is supported. +A fair scheduler across applications, not in a application.+ Use case (for example with the integration of zeppelin) At certain moment, user A submits an heavy application en consumes all the resources of the spark cluster. At a later moment, user B submits a second application. No matter how many work nodes you added now, all the resources go to user A due to the FIFO. User B will never get any resource until user A release its allocated resources. A fair scheduler should distribute extra resources in a fair way on all running applications, which demands resources. > Spark Fair scheduler across applications in standalone mode > --- > > Key: SPARK-27104 > URL: https://issues.apache.org/jira/browse/SPARK-27104 > Project: Spark > Issue Type: Wish > Components: Scheduler >Affects Versions: 2.2.3, 2.3.3, 2.4.0 >Reporter: Hua Zhang >Priority: Minor > > Spark in standalone mode currently only supports FIFO (first-in-first-out) > scheduler across applications. > It will be great that a fair scheduler is supported. +A fair scheduler across > applications, not in a application.+ > > Use case (for example with the integration of zeppelin) > At certain moment, user A submits an heavy application and consumes all the > resources of the spark cluster. > At a later moment, user B submits a second application. > No matter how many work nodes you added now, all the resources go to user A > due to the FIFO. User B will never get any resource until user A release its > allocated resources. > > A fair scheduler should distribute extra resources in a fair way on all > running applications, which demands resources. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
