[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20032 Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20032 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20032 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85214/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20032 **[Test build #85214 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85214/testReport)** for PR 20032 at commit [`7427204`](https://github.com/apache/spark/commit/7427204d0ec2c2f3cbc9a456ac55b3accbb0ee0c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20032 **[Test build #85214 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85214/testReport)** for PR 20032 at commit [`7427204`](https://github.com/apache/spark/commit/7427204d0ec2c2f3cbc9a456ac55b3accbb0ee0c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20032 Fixed, thanks for the reviews. @mridulm, the questions about performance are super helpful for us also to keep track of how we're doing wrt other cluster managers and wrt user expectations. Thanks! :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20032 > (And what happens if set to 0/-ve ?) We have a check preventing that in the option itself. The value should be strictly greater than 0 ms. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20032 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85208/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20032 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20032 Within the allocator's control loop, it's all asynchronous requests being made for executor pods from the k8s API, so, each loop doesn't take very long. If a user were to set a very low value for the delay, it wouldn't necessarily make more requests because the creation is still [rate limited](https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala#L111) by the time it takes for the executors launched in the previous round to become ready (which is around 1s typically, if not higher). So, in the worst case, we may end up polling the state of our internal data structures quickly adding to the driver's CPU load, but the data structures are updated async by watches, so, it doesn't increase the load on the K8s API server or have to fetch any state over the network. I can add a caveat to the documentation that setting it to a very low number of ms may increase load on th e driver. If the intent with reducing batch internal is for people looking to spin up N executors as quickly as possible, if the resources are present, I'd recommend that they use the `batch.size` parameter instead and set that equal to num_executors. That would realize the same effect, and schedule all executors in a single loop. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20032 **[Test build #85208 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85208/testReport)** for PR 20032 at commit [`4adb04b`](https://github.com/apache/spark/commit/4adb04bc9d1e3a27e64f7b92b83e6885ba3a04dc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20032 **[Test build #85208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85208/testReport)** for PR 20032 at commit [`4adb04b`](https://github.com/apache/spark/commit/4adb04bc9d1e3a27e64f7b92b83e6885ba3a04dc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20032 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85185/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20032 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20032 **[Test build #85185 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85185/testReport)** for PR 20032 at commit [`48a3326`](https://github.com/apache/spark/commit/48a3326faaea69bf74d97d028bffdd0552777ffe). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20032: [SPARK-22845] [Scheduler] Modify spark.kubernetes.alloca...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20032 **[Test build #85185 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85185/testReport)** for PR 20032 at commit [`48a3326`](https://github.com/apache/spark/commit/48a3326faaea69bf74d97d028bffdd0552777ffe). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
