[ https://issues.apache.org/jira/browse/SPARK-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-20205. ------------------------------------ Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.3.0 > DAGScheduler posts SparkListenerStageSubmitted before updating stage > -------------------------------------------------------------------- > > Key: SPARK-20205 > URL: https://issues.apache.org/jira/browse/SPARK-20205 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.2.0 > Reporter: Marcelo Vanzin > Assignee: Marcelo Vanzin > Fix For: 2.3.0 > > > Probably affects other versions, haven't checked. > The code that submits the event to the bus is around line 991: > {code} > stage.makeNewStageAttempt(partitionsToCompute.size, > taskIdToLocations.values.toSeq) > listenerBus.post(SparkListenerStageSubmitted(stage.latestInfo, > properties)) > {code} > Later in the same method, the stage information is updated (around line 1057): > {code} > if (tasks.size > 0) { > logInfo(s"Submitting ${tasks.size} missing tasks from $stage > (${stage.rdd}) (first 15 " + > s"tasks are for partitions ${tasks.take(15).map(_.partitionId)})") > taskScheduler.submitTasks(new TaskSet( > tasks.toArray, stage.id, stage.latestInfo.attemptId, jobId, > properties)) > stage.latestInfo.submissionTime = Some(clock.getTimeMillis()) > {code} > That means an event handler might get a stage submitted event with an unset > submission time. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org