surawut-jirasaktavee opened a new pull request, #59989:
URL: https://github.com/apache/spark/pull/59989
### What changes were proposed in this pull request?
This PR reduces the list calls that `FsHistoryProvider` makes on rolling
event log directories (`eventlog_v2_*`) in two ways.
1. **Remove the duplicate listing.** `EventLogFileReader.apply` lists the
directory to check that it is valid, and then `RollingEventLogFilesFileReader`
lists it again. The first listing is now passed to the reader, so each scan
lists a directory once instead of twice.
2. **Skip completed applications whose directory has not changed.**
`checkForLogsInDir` no longer lists a directory when all of these are true:
- the `LogInfo` entry says the application is complete
- the directory's modification time matches the value stored in `LogInfo`
- that modification time is greater than 0
`lastProcessed` is still updated for skipped entries, so they are not
treated as stale.
To support the second change:
- `LogInfo` gets an optional `rootModificationTime` field. Entries written
before this change read it as `None`, so they are listed and then record the
time.
- `EventLogFileReader` gets `rootModificationTime`, with a concrete default
of `None`.
- The new config `spark.history.fs.skipCompletedRollingLogListing.enabled`
(default `true`) turns the skipping off.
"Completed" alone is not enough to skip a directory. With
`spark.eventLog.overwrite=true` and a fixed application ID, the directory can
be deleted and recreated, which changes its modification time, so the log is
reloaded. File systems that report a directory modification time of 0 are
always listed, so they get the first improvement only.
SPARK-48575 also proposes other changes that are not part of this PR.
Changing the default of `spark.history.fs.cleaner.maxNum` will go in a separate
PR, because it deletes logs on upgrade and deserves its own discussion. The UI
proposal is not included.
### Why are the changes needed?
The History Server scans the log directory every 10 seconds by default, and
it lists every rolling event log directory on each scan, twice. List calls on
cloud object stores are billed and slow, so the cost grows with the number of
retained applications, and most of those applications are completed and will
never change.
### Does this PR introduce _any_ user-facing change?
Yes, there is a new config,
`spark.history.fs.skipCompletedRollingLogListing.enabled`, enabled by default.
It only changes how many list calls the History Server makes. The applications
shown are expected to stay the same.
### How was this patch tested?
New tests were added:
- `EventLogFileReadersSuite`: a rolling event log directory is listed only
once.
- `FsHistoryProviderSuite` uses a list-counting `FileSystem` to check the
number of list calls per scan:
- a directory is listed once per scan
- a completed, unchanged directory is not listed
- an overwritten log is detected and reloaded
- a directory with modification time 0 is always listed
- setting the config to `false` keeps listing the directory
- The existing `LogInfo` SerDe test now covers `rootModificationTime`.
Ran locally with JDK 17:
```
build/sbt 'core/testOnly org.apache.spark.deploy.history.*'
```
359 tests passed and none failed. Scalastyle reported 0 errors on main and
test code.
`core/mimaReportBinaryIssues` fails locally with 19 problems, but a tree
without this change reports the identical 19, and none are in the history
package.
This was not tested against a real cloud object store.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
