[jira] [Updated] (SPARK-27104) Spark Fair scheduler across applications in standalone mode
[ https://issues.apache.org/jira/browse/SPARK-27104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Zhang updated SPARK-27104: -- Description: Spark in standalone mode currently only supports FIFO (first-in-first-out) scheduler across applications. It will be great that a fair scheduler is supported. +A fair scheduler across applications, not in a application.+ Use case (for example with the integration of zeppelin) At certain moment, user A submits a heavy application and consumes all the resources of the spark cluster. At a later moment, user B submits a second application. No matter how many work nodes you added now, all the resources go to user A due to the FIFO. User B will never get any resource until user A release its allocated resources. A fair scheduler should distribute extra resources in a fair way on all running applications, which demands resources. was: Spark in standalone mode currently only supports FIFO (first-in-first-out) scheduler across applications. It will be great that a fair scheduler is supported. +A fair scheduler across applications, not in a application.+ Use case (for example with the integration of zeppelin) At certain moment, user A submits an heavy application and consumes all the resources of the spark cluster. At a later moment, user B submits a second application. No matter how many work nodes you added now, all the resources go to user A due to the FIFO. User B will never get any resource until user A release its allocated resources. A fair scheduler should distribute extra resources in a fair way on all running applications, which demands resources. > Spark Fair scheduler across applications in standalone mode > --- > > Key: SPARK-27104 > URL: https://issues.apache.org/jira/browse/SPARK-27104 > Project: Spark > Issue Type: Wish > Components: Scheduler >Affects Versions: 2.2.3, 2.3.3, 2.4.0 >Reporter: Hua Zhang >Priority: Minor > > Spark in standalone mode currently only supports FIFO (first-in-first-out) > scheduler across applications. > It will be great that a fair scheduler is supported. +A fair scheduler across > applications, not in a application.+ > > Use case (for example with the integration of zeppelin) > At certain moment, user A submits a heavy application and consumes all the > resources of the spark cluster. > At a later moment, user B submits a second application. > No matter how many work nodes you added now, all the resources go to user A > due to the FIFO. User B will never get any resource until user A release its > allocated resources. > > A fair scheduler should distribute extra resources in a fair way on all > running applications, which demands resources. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27104) Spark Fair scheduler across applications in standalone mode
[ https://issues.apache.org/jira/browse/SPARK-27104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Zhang updated SPARK-27104: -- Description: Spark in standalone mode currently only supports FIFO (first-in-first-out) scheduler across applications. It will be great that a fair scheduler is supported. +A fair scheduler across applications, not in a application.+ Use case (for example with the integration of zeppelin) At certain moment, user A submits an heavy application and consumes all the resources of the spark cluster. At a later moment, user B submits a second application. No matter how many work nodes you added now, all the resources go to user A due to the FIFO. User B will never get any resource until user A release its allocated resources. A fair scheduler should distribute extra resources in a fair way on all running applications, which demands resources. was: Spark in standalone mode currently only supports FIFO (first-in-first-out) scheduler across applications. It will be great that a fair scheduler is supported. +A fair scheduler across applications, not in a application.+ Use case (for example with the integration of zeppelin) At certain moment, user A submits an heavy application en consumes all the resources of the spark cluster. At a later moment, user B submits a second application. No matter how many work nodes you added now, all the resources go to user A due to the FIFO. User B will never get any resource until user A release its allocated resources. A fair scheduler should distribute extra resources in a fair way on all running applications, which demands resources. > Spark Fair scheduler across applications in standalone mode > --- > > Key: SPARK-27104 > URL: https://issues.apache.org/jira/browse/SPARK-27104 > Project: Spark > Issue Type: Wish > Components: Scheduler >Affects Versions: 2.2.3, 2.3.3, 2.4.0 >Reporter: Hua Zhang >Priority: Minor > > Spark in standalone mode currently only supports FIFO (first-in-first-out) > scheduler across applications. > It will be great that a fair scheduler is supported. +A fair scheduler across > applications, not in a application.+ > > Use case (for example with the integration of zeppelin) > At certain moment, user A submits an heavy application and consumes all the > resources of the spark cluster. > At a later moment, user B submits a second application. > No matter how many work nodes you added now, all the resources go to user A > due to the FIFO. User B will never get any resource until user A release its > allocated resources. > > A fair scheduler should distribute extra resources in a fair way on all > running applications, which demands resources. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-27104) Spark Fair scheduler across applications in standalone mode
Hua Zhang created SPARK-27104: - Summary: Spark Fair scheduler across applications in standalone mode Key: SPARK-27104 URL: https://issues.apache.org/jira/browse/SPARK-27104 Project: Spark Issue Type: Wish Components: Scheduler Affects Versions: 2.4.0, 2.3.3, 2.2.3 Reporter: Hua Zhang Spark in standalone mode currently only supports FIFO (first-in-first-out) scheduler across applications. It will be great that a fair scheduler is supported. +A fair scheduler across applications, not in a application.+ Use case (for example with the integration of zeppelin) At certain moment, user A submits an heavy application en consumes all the resources of the spark cluster. At a later moment, user B submits a second application. No matter how many work nodes you added now, all the resources go to user A due to the FIFO. User B will never get any resource until user A release its allocated resources. A fair scheduler should distribute extra resources in a fair way on all running applications, which demands resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-26850) Make EventLoggingListener LOG_FILE_PERMISSIONS configurable
Hua Zhang created SPARK-26850: - Summary: Make EventLoggingListener LOG_FILE_PERMISSIONS configurable Key: SPARK-26850 URL: https://issues.apache.org/jira/browse/SPARK-26850 Project: Spark Issue Type: Wish Components: Scheduler Affects Versions: 2.4.0, 2.3.2, 2.2.3 Reporter: Hua Zhang private[spark] object EventLoggingListener extends Logging { ... private val LOG_FILE_PERMISSIONS = new FsPermission(Integer.parseInt("770", 8).toShort) ... } Currently the event log files are hard-coded with permission 770. It would be fine if this permission is +configurable+. User case: The spark application is submitted by user A but the spark history server is started by user B. Currently user B cannot access the history event files created by user A. When permission is set to 775, this will be possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org