deshanxiao commented on issue #23637: [SPARK-26714][CORE][WEBUI] Show 0 partition job in WebUI URL: https://github.com/apache/spark/pull/23637#issuecomment-458787898 @gengliangwang @srowen Sorry, maybe I didn't make it clear. The job.name will be ` (Unknown Stage Name)` in this case whatever changing PR or not. I change the original PR is to fit with the code logic. We can see the default job.name will be sent in: ``` // AppStatusListener.scala val lastStageInfo = event.stageInfos.sortBy(_.stageId).lastOption val lastStageName = lastStageInfo.map(_.name).getOrElse("(Unknown Stage Name)") val jobGroup = Option(event.properties) .flatMap { p => Option(p.getProperty(SparkContext.SPARK_JOB_GROUP_ID)) } val sqlExecutionId = Option(event.properties) .flatMap(p => Option(p.getProperty(SQL_EXECUTION_ID_KEY)).map(_.toLong)) val job = new LiveJob( event.jobId, lastStageName, if (event.time > 0) Some(new Date(event.time)) else None, event.stageIds, jobGroup, numTasks, sqlExecutionId) liveJobs.put(event.jobId, job) liveUpdate(job, now) ``` ``` private class LiveJob( val jobId: Int, name: String, val submissionTime: Option[Date], val stageIds: Seq[Int], jobGroup: Option[String], numTasks: Int, sqlExecutionId: Option[Long]) extends LiveEntity { ``` So, if necessary， we can set job.name in two methods: > 1.Add more info in SparkListenerJobStart > 2.Constructing a empty stage contains a callsite in StageInfo? But it will be more complex and get some compatibility problem. In the end, I change it to original. Mybe the UI is kind of confusing. This is acceptable because the job description in here places the last stage info originally. In this case, we have no stage. So showing "Unknown Stage Name" is acceptable I think.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
