[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 merged to master. thanks @attilapiros --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86490/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #86490 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86490/testReport)** for PR 20203 at commit [`882bb9e`](https://github.com/apache/spark/commit/882bb9ecd0de0ee916e6b1f75eda48c5a27f5ab6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 @attilapiros can you please update the PR description to also mention node blacklisting (at least briefly) and file a jira for the followup work, and ping me & tom on it? lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user squito commented on the issue:
https://github.com/apache/spark/pull/20203
I tried this out on a cluster and seemed fine. I also tried with a bad
app, where all tasks fail, its not just because of the host, and all the
executors show up as blacklisted, but I guess that is fine.
```scala
sc.parallelize(1 to 1000, 10).map {x => throw new RuntimeException("bad
application")}.count()
```
https://user-images.githubusercontent.com/71240/35247305-88bceb4c-ff8f-11e7-9e98-285ffd604f46.png";>
or about the same if there is just one bad task:
```scala
sc.parallelize(1 to 1000, 10).map {x =>
if (TaskContext.get().partitionId() == 0) throw new
RuntimeException("bad task")
else 0
}.count()
```
of course if you have more executors, then a whole bunch of them show up as
blacklisted, but still seems fine.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #86490 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86490/testReport)** for PR 20203 at commit [`882bb9e`](https://github.com/apache/spark/commit/882bb9ecd0de0ee916e6b1f75eda48c5a27f5ab6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/20203 Jenkins, test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20203 Test failure seems to be unrelated as it it is in org.apache.spark.sql.hive.client.HiveClientSuites. @squito, @tgravescs could you please restart testing --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86482/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #86482 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86482/testReport)** for PR 20203 at commit [`882bb9e`](https://github.com/apache/spark/commit/882bb9ecd0de0ee916e6b1f75eda48c5a27f5ab6). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #86482 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86482/testReport)** for PR 20203 at commit [`882bb9e`](https://github.com/apache/spark/commit/882bb9ecd0de0ee916e6b1f75eda48c5a27f5ab6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20203 One more reason to run tests in sbt / maven. In intelliJ somehow the complete suite was successful. But the current failure seems to me unrelated, as org.apache.spark.deploy.history has 0 failures. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86408/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #86408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86408/testReport)** for PR 20203 at commit [`f388c45`](https://github.com/apache/spark/commit/f388c45ee56c17f48d393240f29901f73865bb74). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #86408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86408/testReport)** for PR 20203 at commit [`f388c45`](https://github.com/apache/spark/commit/f388c45ee56c17f48d393240f29901f73865bb74). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86400/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #86400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86400/testReport)** for PR 20203 at commit [`cf6e0c9`](https://github.com/apache/spark/commit/cf6e0c919e151c26772ec78a10abc6d2454f7dd5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 @attilapiros test failures look real (you probably just need to regenerate some of those expectations). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86398/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #86398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86398/testReport)** for PR 20203 at commit [`41dd7bb`](https://github.com/apache/spark/commit/41dd7bbc1f62e093738e730bf3f5bfeb3dff16fb). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerNodeBlacklistedForStage(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #86400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86400/testReport)** for PR 20203 at commit [`cf6e0c9`](https://github.com/apache/spark/commit/cf6e0c919e151c26772ec78a10abc6d2454f7dd5). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user squito commented on the issue:
https://github.com/apache/spark/pull/20203
btw another way you could test out having a bad host would be something
like this (untested):
```scala
import org.apache.spark.SparkEnv
val hosts = sc.parallelize(1 to 1, 100).map { _ =>
InetAddress.getHostName()}.collect().toSet
val badHost = hosts.head
sc.parallelize(1 to 1, 10).map { x =>
if (InetAddress.getHostName() == badHost) throw new RuntimeException("Bad
host")
else (x % 3, x)
}.reduceByKey((a, b) => a + b).collect()
```
that way you make sure the failures are consistently on one host, not
dependent on higher executor ids getting concentrated on one host.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/20203
The node blacklisting is tested by unit tests:
- HistoryServerSuite
- TaskSetBlacklistSuite
- AppStatusListenerSuite
And manually with a 2 node cluster:
https://issues.apache.org/jira/secure/attachment/12906833/node_blacklisting_for_stage.png
Here you can see apiros3.gce.test.com was node blacklisted for the stage
because of failures on executor 4 and 5. As expected executor 3 is also
blacklisted even it has no failures itself but sharing the node with 4 and 5.
Spark was started as:
``` bash
./bin/spark-shell --master yarn --deploy-mode client --executor-memory=2G
--num-executors=8 --conf "spark.blacklist.enabled=true" --conf
"spark.blacklist.stage.maxFailedTasksPerExecutor=1" --conf
"spark.blacklist.stage.maxFailedExecutorsPerNode=1" --conf
"spark.blacklist.application.maxFailedTasksPerExecutor=10" --conf
"spark.eventLog.enabled=true"
```
And the job was:
``` scala
import org.apache.spark.SparkEnv
sc.parallelize(1 to 1, 10).map { x =>
if (SparkEnv.get.executorId.toInt >= 4) throw new RuntimeException("Bad
executor")
else (x % 3, x)
}.reduceByKey((a, b) => a + b).collect()
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #86398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86398/testReport)** for PR 20203 at commit [`41dd7bb`](https://github.com/apache/spark/commit/41dd7bbc1f62e093738e730bf3f5bfeb3dff16fb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 that sounds fine with me --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20203 @tgravescs @squito: But what about doing it in a separate PR as this is already quite big? So node backlisting would be still within the scope but executor page changes would be done for a separate issue? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20203 Of course I can update the executors page too. @tgravescs I think via handling the onStageCompleted I can do what you would like to see here. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/20203 I think you only want the stages currently running to show up. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 yeah I think its probably fine to update the executors page that way -- lets at least see how it looks. Would the list include all stages ever blacklisted? Only those stages still running? The most recent 3 blacklisted stages? @attilapiros how about you take a shot at updating the executors page as well, we can see what that looks like? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/20203 I think it would be nice to add something on the executors page as well. Perthaps we can put something like "for stage[, ,...]" if its only blacklisted for those tasksets. But I'm also fine with leaving it like this for now and see how it goes like @squito mentioned on the jira. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/20203 @squito thanks, I missed the img link and misread ExecutorTable as ExecutorPage. On that note the UI portion of this change LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 attila attached some screenshots to the jira (showing both what happens with stage blacklisting and full application blacklisting). The only change here is to the page for a specific stage, so it seems clear that its saying the executor is blacklisted for the same stage. OTOH, if we were to change the executors page as well, then you would need to put something to indicate which page ... I'm not sure what you would put, to be both useful and succinct. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/20203 From a code pov the UI change looks fine, but could you upload a few screenshots of the change? Also the UI simply says if the exec is blacklisted for the whole app or just a stage, but doesn't specify which stage. Is knowing the stage which blacklisted the node important? If so we should try to raise that to the UI as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 @ajbozarth maybe you have some thoughts on the UI, and whether it makes sense to put anything on the executors page? @CodingCat you also often have good UI suggestions :) thanks --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85991/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #85991 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85991/testReport)** for PR 20203 at commit [`0120955`](https://github.com/apache/spark/commit/0120955e6f8382e912cdfbe1d8c13640b9bd1e93). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #85991 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85991/testReport)** for PR 20203 at commit [`0120955`](https://github.com/apache/spark/commit/0120955e6f8382e912cdfbe1d8c13640b9bd1e93). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20203 @tgravescs: You are welcome. Running Spark as (with lower application.maxFailedTasksPerExecutor than stage.maxFailedTasksPerExecutor): ``` $ bin/spark-shell --master "local-cluster[2,1,1024]" --conf "spark.blacklist.enabled=true" --conf "spark.blacklist.stage.maxFailedTasksPerExecutor=10" --conf "spark.blacklist.application.maxFailedTasksPerExecutor=1" --conf "spark.eventLog.enabled=true" ``` The stage is: [application level backlisting screenshot](https://issues.apache.org/jira/secure/attachment/12905713/app_blacklisting.png) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/20203 thanks for working on this, I'm going to try this out and do further review. Did you test for application failures and on the history server? cc @squito since he had some comments on the jira. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20203 cc @tgravescs --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85872/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #85872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85872/testReport)** for PR 20203 at commit [`d8c214b`](https://github.com/apache/spark/commit/d8c214b33f4b014f5a2c0644074f9b7668364799). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #85872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85872/testReport)** for PR 20203 at commit [`d8c214b`](https://github.com/apache/spark/commit/d8c214b33f4b014f5a2c0644074f9b7668364799). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85869/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #85869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85869/testReport)** for PR 20203 at commit [`8d736c1`](https://github.com/apache/spark/commit/8d736c1cd56e341d4d7da88bae01ac3a47649f80). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerExecutorBlacklistedForStage(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20203 **[Test build #85869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85869/testReport)** for PR 20203 at commit [`8d736c1`](https://github.com/apache/spark/commit/8d736c1cd56e341d4d7da88bae01ac3a47649f80). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20203 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20203 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
