david-mollitor-db opened a new pull request, #59921:
URL: https://github.com/apache/spark/pull/59921
### What changes were proposed in this pull request?
Switch the timestamp in Spark's default plain-text log layouts from
`%d{yy/MM/dd HH:mm:ss}` (JVM local time zone) to
`%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX}{UTC}`, an ISO-8601 timestamp in UTC:
| Layout | Before | After |
|---|---|---|
| Bundled default `log4j2-defaults.properties` (used when no
`log4j2.properties` is provided) | `26/10/29 08:05:34` |
`2026-10-29T15:05:34.394Z` |
| `conf/log4j2.properties.template` | `26/10/29 08:05:34` |
`2026-10-29T15:05:34.394Z` |
| `DriverLogger` default layout (`spark.driver.log.layout` unset) |
`26/10/29 08:05:34.394` | `2026-10-29T15:05:34.394Z` |
The rest of each layout is unchanged. The PR also updates the documented
default of `spark.driver.log.layout` in `docs/configuration.md` and adds a note
to the core migration guide.
### Why are the changes needed?
The current format has a two-digit year, no time zone, and (on the console)
no milliseconds. That makes logs hard to correlate across hosts in different
time zones, and it doesn't match Spark's JSON structured-logging layout
(`SparkLayout.json`), which already emits ISO-8601 UTC
(`yyyy-MM-dd'T'HH:mm:ss.SSS'Z'`). Log aggregation tools parse ISO-8601 natively.
### Does this PR introduce _any_ user-facing change?
Yes. Plain-text log lines from Spark's default configuration, from a
`log4j2.properties` copied from the template, and from persisted driver logs
now start with an ISO-8601 UTC timestamp:
```
# before
26/10/29 08:05:34 INFO SparkContext: Running Spark version ...
# after
2026-10-29T15:05:34.394Z INFO SparkContext: Running Spark version ...
```
Users who already have their own `log4j2.properties` are not affected. To
restore the legacy format, provide a `log4j2.properties` whose pattern uses
`%d{yy/MM/dd HH:mm:ss}`, and set `spark.driver.log.layout` to `%d{yy/MM/dd
HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex`. This is documented in the core migration
guide.
### How was this patch tested?
Manually, against log4j 2.26.1, Spark's current version:
- Loaded the edited `log4j2-defaults.properties` and
`conf/log4j2.properties.template` (copied to `log4j2.properties`) through
`-Dlog4j2.configurationFile` in a JVM running with `TZ=America/Los_Angeles`,
and logged a line. Output: `2026-10-29T15:11:43.082Z INFO SparkContext: ...`,
which is UTC with a `Z` suffix.
- Rendered the new `DriverLogger` default layout with `PatternLayout`.
Output: `2026-10-29T15:15:31.646Z main INFO SparkContext: ...`.
No existing test asserts on the default timestamp format.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
