[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-207946763 I've just rebase this branch with master branch. So it should be built and use. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-207943102 @fathersson this branch is what we implement for preemptive scheduling for standalone mode. I have deployed it on production Spark cluster in my previous company. But as you see that it doesn't catch the interesting from community, I didn't continue the deployment. You may clone this code to have a try. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user fathersson commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-191537656 I'd be interested in having preemptive scheduler for standalone deployment. I'd really appreciate it if you could share your code if it's been implemented. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-146283094 Because there seems be no interesting to this patch, I would close this PR first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user viirya closed the pull request at: https://github.com/apache/spark/pull/8129 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user thegiive commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-135786660 hi @JoshRosen , @andrewor14 , @srowen Viirya and I are in the same team. Currently Spark standalone has only one FIFO scheduler outside the application and that what we want to change. Base on our team's simulation and observe, a advance scheduler will boost the spark standalone cluster's CPU/memory resource usage A LOT. We are working on implement a better scheduler than current version. This is our team's result and we have already push to prod. Any idea or comment will be very helpful. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130551156 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130551177 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130551460 [Test build #40745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40745/consoleFull) for PR 8129 at commit [`9298fa0`](https://github.com/apache/spark/commit/9298fa0577dae7018c4d2aaa58301ce5e340251a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130551698 [Test build #40745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40745/console) for PR 8129 at commit [`9298fa0`](https://github.com/apache/spark/commit/9298fa0577dae7018c4d2aaa58301ce5e340251a). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ApplicationSubmission(val appInfo: ApplicationInfo, val submittedTime: Date)` * ` class Pool(val poolName: String, val priority: Int, val cores: Int) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130551699 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130550969 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130553847 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130553715 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130555864 [Test build #40746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40746/consoleFull) for PR 8129 at commit [`867c941`](https://github.com/apache/spark/commit/867c9417d3119f53f055bb5064a8977b9ddd8304). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130605258 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130605274 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130591634 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130591580 **[Test build #40746 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40746/console)** for PR 8129 at commit [`867c941`](https://github.com/apache/spark/commit/867c9417d3119f53f055bb5064a8977b9ddd8304) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130605212 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130605632 [Test build #40757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40757/consoleFull) for PR 8129 at commit [`867c941`](https://github.com/apache/spark/commit/867c9417d3119f53f055bb5064a8977b9ddd8304). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130549724 [Test build #40722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40722/console) for PR 8129 at commit [`553c47f`](https://github.com/apache/spark/commit/553c47fd3f2cbde604e0a961224a71321612dbe7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ApplicationSubmission(val appInfo: ApplicationInfo, val submittedTime: Date)` * ` class Pool(val poolName: String, val priority: Int, val cores: Int) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130549775 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130663858 [Test build #40757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40757/console) for PR 8129 at commit [`867c941`](https://github.com/apache/spark/commit/867c9417d3119f53f055bb5064a8977b9ddd8304). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ApplicationSubmission(val appInfo: ApplicationInfo, val submittedTime: Date)` * ` class Pool(val poolName: String, val priority: Int, val cores: Int) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130663980 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130961086 /cc @JoshRosen @andrewor14 @srowen Would you have time to look at this? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130473133 [Test build #40686 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40686/console) for PR 8129 at commit [`7317783`](https://github.com/apache/spark/commit/7317783dd325dbea217b8f90627157ef73df894c). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ApplicationSubmission(val appInfo: ApplicationInfo, val submittedTime: Date)` * ` class Pool(val poolName: String, val priority: Int, val cores: Int) ` * `case class TaskSetFailed(taskSet: TaskSet, reason: String, exception: Option[Throwable])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130473135 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130485842 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130485829 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130471986 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130471965 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130472818 [Test build #40686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40686/consoleFull) for PR 8129 at commit [`7317783`](https://github.com/apache/spark/commit/7317783dd325dbea217b8f90627157ef73df894c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130486485 [Test build #40696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40696/consoleFull) for PR 8129 at commit [`0531b86`](https://github.com/apache/spark/commit/0531b86d1a340fae512f5ea951e61a239d02a05a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130513636 **[Test build #40696 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40696/console)** for PR 8129 at commit [`0531b86`](https://github.com/apache/spark/commit/0531b86d1a340fae512f5ea951e61a239d02a05a) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130515792 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130515801 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130517227 [Test build #40722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40722/consoleFull) for PR 8129 at commit [`553c47f`](https://github.com/apache/spark/commit/553c47fd3f2cbde604e0a961224a71321612dbe7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130535563 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130513785 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130535577 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130536464 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/8129 [SPARK-9882][Core] Priority-based scheduling for Spark applications JIRA: https://issues.apache.org/jira/browse/SPARK-9882 This is a priority-based scheduling based on the interface proposed in #7958. It uses a XML configuration file to specify the submission pools for applications. This patch adds a new parameter `--pool` to SparkSubmit under standalone mode to specify which pool the submitted application should be assigned to. The priority of the submitted application is defined by the assigned pool. It also defines the cores it can acquire. You can merge this pull request into a Git repository by running: $ git pull https://github.com/viirya/spark-1 priority_scheduler_pr Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8129.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8129 commit 25a3d7580bd6f3934f52b48faef1eed825bea4fb Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-04T10:53:44Z Add interface for SchedulingAlgorithm. commit 36db2c384e15cebee4cee95bb1fd31a97459a74b Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-05T09:34:51Z Add class PrioritySchedulingAlgorithm. commit 340bb1251b8ddda1ab3577752f23c220036747fc Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-06T06:31:16Z Fix MasterSuite. commit 79ed50138b3b287e4aed0116ceea656b403c0107 Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-11T09:08:38Z Fix MasterSuite for newly added test. commit 6b443827a15e31159af3fde3e06a84080c0a5e4f Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-11T09:27:08Z Add Pool and Queue. commit 12519ebccdcb654f89b0039791174ab3426b6730 Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-11T09:36:12Z Add wrongly deleted file. commit a0789805f0cfbe9f3ff703f9161aab3b2ea9cedd Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-11T11:02:22Z Add test for parsing pool setting. commit c978ddfbc170c21231f3b4a6206e9f6c3ce7a134 Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-12T02:05:17Z Add test. commit 112573fcf4fdff588d1099acacea3cf46b12bbd7 Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-12T04:49:00Z Add basic scheduling for applications in pools. commit cc68c84a18ec32e32467c6ab13942a653bb59636 Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-12T07:53:20Z Add --pool parameter to SparkSubmit. commit f7ec9cce2b38f8c4f1a67f362de484dedcae7c0d Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-12T08:16:01Z Add removeApplication into SchedulingAlgorithm. commit 607450a9d1b126dfdeb708b34a3a611dc7dbd1d9 Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-12T10:12:29Z Fix asInstanceOf bug. commit d09d027d94bbd38c46896a0282bed35527726e1b Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-12T11:04:57Z Load default pool setting if no configuration file specified. Use pool.cores - app.coresGranted instead of app.coresLeft. commit 3a4e4cb20bee618eddeb2e40a6e7efbff3ced3bc Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-12T15:42:16Z Use Date instead of LocalDateTime (only in Java 8). commit 3b48cfa4781b2c386ff93da57d601eeb7abd10fe Author: Liang-Chi Hsieh vii...@appier.com Date: 2015-08-12T15:59:35Z Add configuration file template. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130364043 [Test build #40639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40639/console) for PR 8129 at commit [`3b48cfa`](https://github.com/apache/spark/commit/3b48cfa4781b2c386ff93da57d601eeb7abd10fe). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ApplicationSubmission(val appInfo: ApplicationInfo, val submittedTime: Date)` * ` class Pool(val poolName: String, val priority: Int, val cores: Int) ` * `case class TaskSetFailed(taskSet: TaskSet, reason: String, exception: Option[Throwable])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130356031 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130355993 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130362157 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130357213 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130362672 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130362651 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130364047 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130364014 [Test build #40639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40639/consoleFull) for PR 8129 at commit [`3b48cfa`](https://github.com/apache/spark/commit/3b48cfa4781b2c386ff93da57d601eeb7abd10fe). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
