This is an automated email from the ASF dual-hosted git repository.
mridulm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 90fce00c984d [SPARK-54879][CORE] Add final status to Spark History
Server for failed executions
90fce00c984d is described below
commit 90fce00c984d47141c52ea62302b9b5d072a3335
Author: Rob Reeves <[email protected]>
AuthorDate: Sat Jun 27 00:32:51 2026 -0500
[SPARK-54879][CORE] Add final status to Spark History Server for failed
executions
### What changes were proposed in this pull request?
This adds the final application status to SHS for failures with non-zero
exit codes to improve debugging.
#### Decisions made
At first I wanted to always include the final exit code. When using
spark-submit in local mode I found that throwing driver exceptions can result
in exit code 0. This would be confusing because the status would show as a
success. Instead, I decided to scope this to only showing the final status for
known failures. Ideally, a future enhancement can be made to not use exit code
0 for failed executions for all cases, but this is a riskier breaking change
that I did not want to make here.
#### Possible future extensions
Right now I only show the exit code for explicit failures (non-zero exit
code). There are some cases I wanted to solve, but require more foundational
work.
1. Help users understand when a driver crashed. Right now SHS will show the
app as still running. This is another reason I didn't want to show the status
outside of known failures. This is tricky and I wanted to infer it based on the
time gap between the last event in the event log and current time, but that
information is not there now (and I'm not sure on the approach).
2. Enhance exit codes to never use 0 during a failure so it can help users
debug more scenarios.
3. Instead of just showing an exit code, map it to its meaning. This
requires more work since it is resource manager specific.
### Why are the changes needed?
If a Spark job fails due to an issue on the driver side (e.g. during the
commit phase, custom driver code) this is not surfaced anywhere in SHS. This
becomes even more confusing when a query/job has run successfully. SHS will
show all jobs as completed successfully and the only way to know the job
failed is to look in the logs. Before this change this is what a failed driver
side application would look like in SHS. It looks like a success.
<img width="961" height="417" alt="before"
src="https://github.com/user-attachments/assets/8d233666-b5cf-4268-ad2e-69184a7eb04d";
/>
After the change this is what it looks like:
<img width="1041" height="514" alt="after"
src="https://github.com/user-attachments/assets/44b8aa13-2c85-4438-bb21-85d6b9f0c33c";
/>
### Does this PR introduce _any_ user-facing change?
Yes, it adds the application final status to the SHS overview when an
execution fails and has a non-zero exit code.
### How was this patch tested?
Unit tests and manually tested SHS.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code 2.0.76
Closes #53657 from robreeves/shs_final_status.
Authored-by: Rob Reeves <[email protected]>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
.../apache/spark/status/protobuf/store_types.proto | 1 +
.../spark/deploy/history/FsHistoryProvider.scala | 3 ++
.../apache/spark/status/AppStatusListener.scala | 3 +-
.../scala/org/apache/spark/status/api/v1/api.scala | 1 +
.../ApplicationInfoWrapperSerializer.scala | 4 +-
.../org/apache/spark/ui/jobs/AllJobsPage.scala | 29 ++++++++++++
.../application_list_json_expectation.json | 19 ++++++++
.../completed_app_list_json_expectation.json | 19 ++++++++
.../limit_app_list_json_expectation.json | 36 ++++++++-------
.../minDate_app_list_json_expectation.json | 19 ++++++++
.../minEndDate_app_list_json_expectation.json | 19 ++++++++
...p_with_non_zero_exit_code_json_expectation.json | 19 ++++++++
.../resources/spark-events/local-1430917381536 | 53 ++++++++++++++++++++++
.../deploy/history/FsHistoryProviderSuite.scala | 9 ++--
.../spark/deploy/history/HistoryServerSuite.scala | 1 +
.../spark/status/AppStatusListenerSuite.scala | 4 +-
.../protobuf/KVStoreProtobufSerializerSuite.scala | 3 +-
dev/.rat-excludes | 1 +
project/MimaExcludes.scala | 3 ++
19 files changed, 221 insertions(+), 25 deletions(-)
diff --git
a/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
b/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
index 2b6ac763cc3e..ae020e74f8c6 100644
--- a/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
+++ b/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
@@ -213,6 +213,7 @@ message ApplicationAttemptInfo {
optional string spark_user = 6;
bool completed = 7;
optional string app_spark_version = 8;
+ optional int32 exit_code = 9;
}
message ApplicationInfo {
diff --git
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index 79687bae0a10..a87db3557553 100644
---
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -1859,6 +1859,7 @@ private[history] class AppListingListener(
attempt.lastUpdated = new Date(reader.modificationTime)
attempt.duration = event.time - attempt.startTime.getTime()
attempt.completed = true
+ attempt.exitCode = event.exitCode
}
override def onEnvironmentUpdate(event: SparkListenerEnvironmentUpdate):
Unit = {
@@ -1926,6 +1927,7 @@ private[history] class AppListingListener(
var duration = 0L
var sparkUser: String = null
var completed = false
+ var exitCode = Option.empty[Int]
var appSparkVersion = ""
var adminAcls: Option[String] = None
@@ -1944,6 +1946,7 @@ private[history] class AppListingListener(
sparkUser,
completed,
appSparkVersion,
+ exitCode,
Some(logSourceName),
Some(logSourceFullPath))
new AttemptInfoWrapper(
diff --git
a/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
b/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
index 5674798f0209..ee887a4cda73 100644
--- a/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
+++ b/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
@@ -199,7 +199,8 @@ private[spark] class AppStatusListener(
event.time - old.startTime.getTime(),
old.sparkUser,
true,
- old.appSparkVersion)
+ old.appSparkVersion,
+ event.exitCode)
appInfo = v1.ApplicationInfo(
appInfo.id,
diff --git a/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
b/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
index f3fb916f0820..3e7f5e6d2ec3 100644
--- a/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
+++ b/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
@@ -54,6 +54,7 @@ case class ApplicationAttemptInfo private[spark](
sparkUser: String,
completed: Boolean = false,
appSparkVersion: String,
+ exitCode: Option[Int] = None,
logSourceName: Option[String] = None,
logSourceFullPath: Option[String] = None) {
diff --git
a/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
b/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
index cf3159c7b481..8526e9459472 100644
---
a/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
+++
b/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
@@ -95,6 +95,7 @@ private[protobuf] class ApplicationInfoWrapperSerializer
info.attemptId.foreach{ id =>
builder.setAttemptId(id)
}
+ info.exitCode.foreach(builder.setExitCode)
builder.build()
}
@@ -110,7 +111,8 @@ private[protobuf] class ApplicationInfoWrapperSerializer
duration = info.getDuration,
sparkUser = getStringField(info.hasSparkUser, () => info.getSparkUser),
completed = info.getCompleted,
- appSparkVersion = getStringField(info.hasAppSparkVersion, () =>
info.getAppSparkVersion)
+ appSparkVersion = getStringField(info.hasAppSparkVersion, () =>
info.getAppSparkVersion),
+ exitCode = if (info.hasExitCode) Some(info.getExitCode) else None
)
}
}
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
index 75115ed4366e..6b9cb41633d1 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
@@ -278,6 +278,7 @@ private[ui] class AllJobsPage(parent: JobsTab, store:
AppStatusStore) extends We
val startDate = appInfo.attempts.head.startTime
val startTime = startDate.getTime()
val endTime = appInfo.attempts.head.endTime.getTime()
+ val exitCode = appInfo.attempts.head.exitCode
val activeJobs = new ListBuffer[v1.JobData]()
val completedJobs = new ListBuffer[v1.JobData]()
@@ -321,6 +322,34 @@ private[ui] class AllJobsPage(parent: JobsTab, store:
AppStatusStore) extends We
val summary: NodeSeq =
<div>
<ul class="list-unstyled">
+ <li>
+ <strong>User:</strong>
+ {parent.getSparkUser}
+ </li>
+ <li>
+ <strong>Started At:</strong>
+ {UIUtils.formatDate(startDate)}
+ </li>
+ <li>
+ <strong>Total Uptime:</strong>
+ {
+ if (endTime < 0 && parent.sc.isDefined) {
+ UIUtils.formatDuration(System.currentTimeMillis() - startTime)
+ } else if (endTime > 0) {
+ UIUtils.formatDuration(endTime - startTime)
+ }
+ }
+ </li>
+ {
+ exitCode match {
+ case Some(code) if code != 0 =>
+ <li>
+ <strong>Exit Code:</strong>
+ {code.toString}
+ </li>
+ case _ => <!-- -->
+ }
+ }
<li>
<strong>Scheduling Mode: </strong>
{schedulingMode}
diff --git
a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
index ac6e5c63c0b6..9da4c43affe8 100644
---
a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
+++
b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
@@ -1,4 +1,22 @@
[ {
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}, {
"attempts" : [ {
"appSparkVersion" : "4.2.0-SNAPSHOT",
"logSourceFullPath" : "",
@@ -7,6 +25,7 @@
"duration" : 2006,
"endTime" : "2025-12-27T14:15:12.221GMT",
"endTimeEpoch" : 1766844912221,
+ "exitCode" : 0,
"lastUpdated" : "",
"lastUpdatedEpoch" : 0,
"sparkUser" : "<script>alert('XSS')</script>",
diff --git
a/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
index ac6e5c63c0b6..9da4c43affe8 100644
---
a/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
+++
b/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
@@ -1,4 +1,22 @@
[ {
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}, {
"attempts" : [ {
"appSparkVersion" : "4.2.0-SNAPSHOT",
"logSourceFullPath" : "",
@@ -7,6 +25,7 @@
"duration" : 2006,
"endTime" : "2025-12-27T14:15:12.221GMT",
"endTimeEpoch" : 1766844912221,
+ "exitCode" : 0,
"lastUpdated" : "",
"lastUpdatedEpoch" : 0,
"sparkUser" : "<script>alert('XSS')</script>",
diff --git
a/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
index 06ef3a5ecc9f..fef54a1faa3f 100644
---
a/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
+++
b/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
@@ -1,4 +1,22 @@
[ {
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}, {
"attempts" : [ {
"appSparkVersion" : "4.2.0-SNAPSHOT",
"logSourceFullPath" : "",
@@ -7,6 +25,7 @@
"duration" : 2006,
"endTime" : "2025-12-27T14:15:12.221GMT",
"endTimeEpoch" : 1766844912221,
+ "exitCode" : 0,
"lastUpdated" : "",
"lastUpdatedEpoch" : 0,
"sparkUser" : "<script>alert('XSS')</script>",
@@ -32,21 +51,4 @@
} ],
"id" : "local-1642039451826",
"name" : "Spark shell"
-}, {
- "attempts" : [ {
- "appSparkVersion" : "3.1.1.119",
- "logSourceFullPath" : "",
- "logSourceName" : "",
- "completed" : true,
- "duration" : 274875,
- "endTime" : "2021-08-10T23:29:30.208GMT",
- "endTimeEpoch" : 1628638170208,
- "lastUpdated" : "",
- "lastUpdatedEpoch" : 0,
- "sparkUser" : "john",
- "startTime" : "2021-08-10T23:24:55.333GMT",
- "startTimeEpoch" : 1628637895333
- } ],
- "id" : "application_1628109047826_1317105",
- "name" : "Spark shell"
} ]
diff --git
a/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
index a1d2f9e9ef4d..885045f2faf5 100644
---
a/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
+++
b/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
@@ -1,4 +1,22 @@
[ {
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}, {
"attempts" : [ {
"appSparkVersion" : "4.2.0-SNAPSHOT",
"logSourceFullPath" : "",
@@ -7,6 +25,7 @@
"duration" : 2006,
"endTime" : "2025-12-27T14:15:12.221GMT",
"endTimeEpoch" : 1766844912221,
+ "exitCode" : 0,
"lastUpdated" : "",
"lastUpdatedEpoch" : 0,
"sparkUser" : "<script>alert('XSS')</script>",
diff --git
a/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
index 413660c4b689..ead957a5fbbb 100644
---
a/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
+++
b/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
@@ -1,4 +1,22 @@
[ {
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}, {
"attempts" : [ {
"appSparkVersion" : "4.2.0-SNAPSHOT",
"logSourceFullPath" : "",
@@ -7,6 +25,7 @@
"duration" : 2006,
"endTime" : "2025-12-27T14:15:12.221GMT",
"endTimeEpoch" : 1766844912221,
+ "exitCode" : 0,
"lastUpdated" : "",
"lastUpdatedEpoch" : 0,
"sparkUser" : "<script>alert('XSS')</script>",
diff --git
a/core/src/test/resources/HistoryServerExpectations/one_app_with_non_zero_exit_code_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/one_app_with_non_zero_exit_code_json_expectation.json
new file mode 100644
index 000000000000..9b9ee4c7ed9d
--- /dev/null
+++
b/core/src/test/resources/HistoryServerExpectations/one_app_with_non_zero_exit_code_json_expectation.json
@@ -0,0 +1,19 @@
+{
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}
diff --git a/core/src/test/resources/spark-events/local-1430917381536
b/core/src/test/resources/spark-events/local-1430917381536
new file mode 100644
index 000000000000..c356d307895e
--- /dev/null
+++ b/core/src/test/resources/spark-events/local-1430917381536
@@ -0,0 +1,53 @@
+{"Event":"SparkListenerLogStart","Spark Version":"3.5.2"}
+{"Event":"SparkListenerResourceProfileAdded","Resource Profile Id":0,"Executor
Resource Requests":{"cores":{"Resource Name":"cores","Amount":1,"Discovery
Script":"","Vendor":""},"memory":{"Resource
Name":"memory","Amount":2048,"Discovery
Script":"","Vendor":""},"offHeap":{"Resource
Name":"offHeap","Amount":0,"Discovery Script":"","Vendor":""}},"Task Resource
Requests":{"cpus":{"Resource Name":"cpus","Amount":1.0}}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":587552240,"JVMOffHeapMemory":169149216,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":41075,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9117978624,"ProcessTreeJVMRSSMemory":1213
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":587552240,"JVMOffHeapMemory":169436864,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":41075,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9117978624,"ProcessTreeJVMRSSMemory":1213
[...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor
ID":"driver","Host":"localhost","Port":24790},"Maximum
Memory":3676726886,"Timestamp":1782235880552,"Maximum Onheap
Memory":3676726886,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerEnvironmentUpdate","JVM Information":{"Java
Home":"/usr/lib/jvm/java","Java Version":"17.0.5 (Microsoft)","Scala
Version":"version 2.12.18"},"Spark Properties":{},"Hadoop
Properties":{},"System Properties":{},"Metrics Properties":{},"Classpath
Entries":{}}
+{"Event":"SparkListenerApplicationStart","App Name":"Spark shell","App
ID":"local-1430917381536","Timestamp":1782235854003,"User":"spark"}
+{"Event":"org.apache.spark.scheduler.SparkListenerMiscellaneousProcessAdded","time":1782235881468,"processId":"yarn-am","info":{"hostPort":"localhost:8042","cores":1,"logUrlInfo":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000001/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000001/spark/stderr?start=-4096"}}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":663049712,"JVMOffHeapMemory":181725520,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":4311030,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9135951872,"ProcessTreeJVMRSSMemory":12
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":663049712,"JVMOffHeapMemory":182043856,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":4311030,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9135951872,"ProcessTreeJVMRSSMemory":12
[...]
+{"Event":"SparkListenerExecutorAdded","Timestamp":1782235902754,"Executor
ID":"1","Executor Info":{"Host":"localhost","Total Cores":1,"Log
Urls":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000002/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000002/spark/stderr?start=-4096"},"Attributes":{"NM_HTTP_ADDRESS":"localhost:8042","USER":"spark","LOG_FILES":"stderr,stdout","NM_HTTP_PORT":"8042";
[...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor
ID":"1","Host":"localhost","Port":11366},"Maximum
Memory":1099746508,"Timestamp":1782235902912,"Maximum Onheap
Memory":1099746508,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":684021232,"JVMOffHeapMemory":183676472,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":12723107,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9146445824,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorAdded","Timestamp":1782235908831,"Executor
ID":"3","Executor Info":{"Host":"localhost","Total Cores":1,"Log
Urls":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000003/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000003/spark/stderr?start=-4096"},"Attributes":{"NM_HTTP_ADDRESS":"localhost:8042","USER":"spark","LOG_FILES":"stderr,stdout","NM_HTTP_PORT":"8042";
[...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor
ID":"3","Host":"localhost","Port":11298},"Maximum
Memory":1099746508,"Timestamp":1782235908994,"Maximum Onheap
Memory":1099746508,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerExecutorAdded","Timestamp":1782235910397,"Executor
ID":"2","Executor Info":{"Host":"localhost","Total Cores":1,"Log
Urls":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000004/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000004/spark/stderr?start=-4096"},"Attributes":{"NM_HTTP_ADDRESS":"localhost:8042","USER":"spark","LOG_FILES":"stderr,stdout","NM_HTTP_PORT":"8042";
[...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor
ID":"2","Host":"localhost","Port":16966},"Maximum
Memory":1099746508,"Timestamp":1782235910574,"Maximum Onheap
Memory":1099746508,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":500075000,"JVMOffHeapMemory":200249712,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29547261,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9211654144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":562989560,"JVMOffHeapMemory":207438768,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29659885,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9231601664,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":528762488,"JVMOffHeapMemory":230684216,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29659885,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9231601664,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":532956792,"JVMOffHeapMemory":230282752,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29668077,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9231601664,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":537151096,"JVMOffHeapMemory":232446888,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29668077,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9239990272,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":709117560,"JVMOffHeapMemory":244835032,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29816850,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9239990272,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":713311864,"JVMOffHeapMemory":244563024,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29816850,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9240129536,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235983810,"Executor
ID":"1","Removed Reason":"Executor decommission: spark scale down"}
+{"Event":"SparkListenerBlockManagerRemoved","Block Manager ID":{"Executor
ID":"1","Host":"localhost","Port":11366},"Timestamp":1782235983814}
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235985799,"Executor
ID":"1","Removed Reason":"Executor for container
container_e121_1430917381536_01_000002 exited because of a YARN event (e.g.,
preemption) and not because of an error in the running job."}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":793003640,"JVMOffHeapMemory":248284608,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29808921,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235989914,"Executor
ID":"3","Removed Reason":"Executor decommission: spark scale down"}
+{"Event":"SparkListenerBlockManagerRemoved","Block Manager ID":{"Executor
ID":"3","Host":"localhost","Port":11298},"Timestamp":1782235989915}
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235991801,"Executor
ID":"3","Removed Reason":"Executor for container
container_e121_1430917381536_01_000003 exited because of a YARN event (e.g.,
preemption) and not because of an error in the running job."}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":801392248,"JVMOffHeapMemory":249805880,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29808922,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":805586552,"JVMOffHeapMemory":249421632,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29800730,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":805586552,"JVMOffHeapMemory":249945392,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29800730,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":809780856,"JVMOffHeapMemory":250125488,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29780056,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerJobStart","Job ID":0,"Submission
Time":1782236034538,"Stage Infos":[{"Stage ID":0,"Stage Attempt ID":0,"Stage
Name":"writeDataFrame at <console>:84","Number of Tasks":3,"RDD Info":[{"RDD
ID":3,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"0\",\"name\":\"WriteFiles\"}","Callsite":"writeDataFrame
at <console>:84","Parent IDs":[2],"Storage Level":{"Use Disk":false,"Use
Memory":false,"Use Off
Heap":false,"Deserialized":false,"Replication":1},"Barrier":false,"Determinist
[...]
+{"Event":"SparkListenerStageSubmitted","Stage Info":{"Stage ID":0,"Stage
Attempt ID":0,"Stage Name":"writeDataFrame at <console>:84","Number of
Tasks":3,"RDD Info":[{"RDD
ID":3,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"0\",\"name\":\"WriteFiles\"}","Callsite":"writeDataFrame
at <console>:84","Parent IDs":[2],"Storage Level":{"Use Disk":false,"Use
Memory":false,"Use Off
Heap":false,"Deserialized":false,"Replication":1},"Barrier":false,"DeterministicLevel":"DETERMINATE","Number
of Parti [...]
+{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task
Info":{"Task ID":0,"Index":0,"Attempt":0,"Partition ID":0,"Launch
Time":1782236034680,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[]}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":673064904,"JVMOffHeapMemory":281199496,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33969867,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9349148672,"ProcessTreeJVMRS
[...]
+{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task
Info":{"Task ID":1,"Index":1,"Attempt":0,"Partition ID":1,"Launch
Time":1782236037542,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task
ID":0,"Index":0,"Attempt":0,"Partition ID":0,"Launch
Time":1782236034680,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish
Time":1782236037555,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[{"ID":4,"Name":"task commit
time","Update":"50","Value":"50","In [...]
+{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task
Info":{"Task ID":2,"Index":2,"Attempt":0,"Partition ID":2,"Launch
Time":1782236037693,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task
ID":1,"Index":1,"Attempt":0,"Partition ID":1,"Launch
Time":1782236037542,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish
Time":1782236037694,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[{"ID":4,"Name":"task commit
time","Update":"37","Value":"87","In [...]
+{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task
ID":2,"Index":2,"Attempt":0,"Partition ID":2,"Launch
Time":1782236037693,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish
Time":1782236037801,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[{"ID":4,"Name":"task commit
time","Update":"40","Value":"127","I [...]
+{"Event":"SparkListenerStageExecutorMetrics","Executor ID":"2","Stage
ID":0,"Stage Attempt ID":0,"Executor
Metrics":{"JVMHeapMemory":186850048,"JVMOffHeapMemory":116149944,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":16807195,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":4315377664,"ProcessTreeJVMRSSMemory":628396032,"ProcessTreePythonVMemory":0,"ProcessTr
[...]
+{"Event":"SparkListenerStageExecutorMetrics","Executor ID":"driver","Stage
ID":0,"Stage Attempt ID":0,"Executor
Metrics":{"JVMHeapMemory":673064904,"JVMOffHeapMemory":281199496,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33969867,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9349148672,"ProcessTreeJVMRSSMemory":1489293312,"ProcessTreePythonVMemory":0,"Pro
[...]
+{"Event":"SparkListenerStageCompleted","Stage Info":{"Stage ID":0,"Stage
Attempt ID":0,"Stage Name":"writeDataFrame at <console>:84","Number of
Tasks":3,"RDD Info":[{"RDD
ID":3,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"0\",\"name\":\"WriteFiles\"}","Callsite":"writeDataFrame
at <console>:84","Parent IDs":[2],"Storage Level":{"Use Disk":false,"Use
Memory":false,"Use Off
Heap":false,"Deserialized":false,"Replication":1},"Barrier":false,"DeterministicLevel":"DETERMINATE","Number
of Parti [...]
+{"Event":"SparkListenerJobEnd","Job ID":0,"Completion Time":1782236037805,"Job
Result":{"Result":"JobSucceeded"}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":798894024,"JVMOffHeapMemory":302532712,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33973602,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9373675520,"ProcessTreeJVMRS
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":807282632,"JVMOffHeapMemory":304663000,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33991159,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9373675520,"ProcessTreeJVMRS
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":811476936,"JVMOffHeapMemory":306466816,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33999351,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9373675520,"ProcessTreeJVMRS
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":542426624,"JVMOffHeapMemory":305486288,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33999351,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9391607808,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerApplicationEnd","Timestamp":1782236075148,"ExitCode":1}
diff --git
a/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
index 16f9a4300db0..873431d7942e 100644
---
a/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
+++
b/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
@@ -121,7 +121,7 @@ abstract class FsHistoryProviderSuite extends SparkFunSuite
with Matchers with P
writeFile(newAppComplete, None,
SparkListenerApplicationStart(newAppComplete.getName(),
Some("new-app-complete"), 1L, "test",
None),
- SparkListenerApplicationEnd(5L)
+ SparkListenerApplicationEnd(5L, Some(1))
)
// Write a new-style application log.
@@ -153,7 +153,8 @@ abstract class FsHistoryProviderSuite extends SparkFunSuite
with Matchers with P
end: Long,
lastMod: Long,
user: String,
- completed: Boolean): ApplicationInfo = {
+ completed: Boolean,
+ exitCode: Option[Int] = None): ApplicationInfo = {
val duration = if (end > 0) end - start else 0
// Get log dir info from the provider
@@ -161,13 +162,13 @@ abstract class FsHistoryProviderSuite extends
SparkFunSuite with Matchers with P
val (logSourceName, logSourceFullPath) =
provider.getLogDirInfo(logPath)
new ApplicationInfo(id, name, None, None, None, None,
List(ApplicationAttemptInfo(None, new Date(start),
- new Date(end), new Date(lastMod), duration, user, completed,
SPARK_VERSION,
+ new Date(end), new Date(lastMod), duration, user, completed,
SPARK_VERSION, exitCode,
Some(logSourceName), Some(logSourceFullPath))))
}
// For completed files, lastUpdated would be lastModified time.
list(0) should be (makeAppInfo("new-app-complete",
newAppComplete.getName(), 1L, 5L,
- newAppComplete.lastModified(), "test", true))
+ newAppComplete.lastModified(), "test", true, Some(1)))
list(1) should be (makeAppInfo("new-complete-lzf",
newAppCompressedComplete.getName(),
1L, 4L, newAppCompressedComplete.lastModified(), "test", true))
diff --git
a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
index 13432b6ed9fc..79e0001fa22f 100644
---
a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
+++
b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
@@ -140,6 +140,7 @@ abstract class HistoryServerSuite extends SparkFunSuite
with BeforeAndAfter with
"limit app list json" -> "applications?limit=3",
"one app json" -> "applications/local-1422981780767",
"one app multi-attempt json" -> "applications/local-1426533911241",
+ "one app with non-zero exit code json" ->
"applications/local-1430917381536",
"job list json" -> "applications/local-1422981780767/jobs",
"job list from multi-attempt app json(1)" ->
"applications/local-1426533911241/1/jobs",
"job list from multi-attempt app json(2)" ->
"applications/local-1426533911241/2/jobs",
diff --git
a/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
b/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
index 4d75f5d7a1fc..bbcc245b2ee4 100644
--- a/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
@@ -134,6 +134,7 @@ abstract class AppStatusListenerSuite extends SparkFunSuite
with BeforeAndAfter
assert(attempt.endTime.getTime() === -1L)
assert(attempt.sparkUser === "user")
assert(!attempt.completed)
+ assert(attempt.exitCode === None)
assert(attempt.appSparkVersion === "TestSparkVersion")
}
@@ -691,7 +692,7 @@ abstract class AppStatusListenerSuite extends SparkFunSuite
with BeforeAndAfter
}
// End the application.
- listener.onApplicationEnd(SparkListenerApplicationEnd(42L))
+ listener.onApplicationEnd(SparkListenerApplicationEnd(42L, Some(1)))
check[ApplicationInfoWrapper]("id") { app =>
assert(app.info.name === "name")
@@ -706,6 +707,7 @@ abstract class AppStatusListenerSuite extends SparkFunSuite
with BeforeAndAfter
assert(attempt.duration === 41L)
assert(attempt.sparkUser === "user")
assert(attempt.completed)
+ assert(attempt.exitCode === Some(1))
}
}
diff --git
a/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
b/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
index ac568fee1ad4..68e302628bc4 100644
---
a/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
+++
b/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
@@ -395,7 +395,8 @@ class KVStoreProtobufSerializerSuite extends SparkFunSuite {
duration = 100,
sparkUser = null,
completed = true,
- appSparkVersion = null
+ appSparkVersion = null,
+ exitCode = Some(1)
))
val input = new ApplicationInfoWrapper(
ApplicationInfo(
diff --git a/dev/.rat-excludes b/dev/.rat-excludes
index a61fe35f2729..81f05cd01175 100644
--- a/dev/.rat-excludes
+++ b/dev/.rat-excludes
@@ -84,6 +84,7 @@ local-1426633911242
local-1430917381534
local-1430917381535_1
local-1430917381535_2
+local-1430917381536
local-1642039451826
eventlog_v2_local-1766844910796
DESCRIPTION
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index ec962eb173b6..df2b51f3b732 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -41,6 +41,9 @@ object MimaExcludes {
// Exclude rules for 4.2.x from 4.1.0
lazy val v42excludes = v41excludes ++ Seq(
+ // [SPARK-54879] Add exitCode field to ApplicationAttemptInfo
+
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.ApplicationAttemptInfo.tupled"),
+
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.ApplicationAttemptInfo.curried"),
// [SQL] SafeJsonSerializer.safeMapToJValue: second parameter widened from
Function1 to
// Function2 so the key is passed to the value serializer
(progress.scala). Binary-incompatible
// vs spark-sql-api 4.0.0; not part of the public supported API
(private[streaming] package).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
