[ https://issues.apache.org/jira/browse/SPARK-33029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-33029: ---------------------------------- Fix Version/s: 3.0.2 > Standalone mode blacklist executors page UI marks driver as blacklisted > ----------------------------------------------------------------------- > > Key: SPARK-33029 > URL: https://issues.apache.org/jira/browse/SPARK-33029 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.0.0 > Reporter: Thomas Graves > Assignee: Baohe Zhang > Priority: Major > Fix For: 3.0.2, 3.1.0 > > Attachments: Screen Shot 2020-09-29 at 1.52.09 PM.png, Screen Shot > 2020-09-29 at 1.53.37 PM.png > > > I am running a spark shell on a 1 node standalone cluster. I noticed that > the executors page ui was marking the driver as blacklisted for the stage > that is running. Attached a screen shot. > Also, in my case one of the executors died and it doesn't seem like the > schedule rpicked up the new one. It doesn't show up on the stages page and > just shows it as active but none of the tasks ran there. > > You can reproduce this by starting a master and slave on a single node, then > launch a shell like where you will get multiple executors (in this case I got > 3) > $SPARK_HOME/bin/spark-shell --master spark://yourhost:7077 --executor-cores 4 > --conf spark.blacklist.enabled=true > > From shell run: > {code:java} > import org.apache.spark.TaskContext > val rdd = sc.makeRDD(1 to 1000, 5).mapPartitions { it => > val context = TaskContext.get() > if (context.attemptNumber() < 2) { > throw new Exception("test attempt num") > } > it > }{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org