viirya opened a new pull request, #59454: URL: https://github.com/apache/spark/pull/59454
### What changes were proposed in this pull request? Backport #59328 (SPARK-60110, master commit bb462a1700eaee72d26e9d9b507cf7bc8559c6f0) to `branch-3.5`. The master patch does not apply cleanly, so this backport adapts it to `branch-3.5`: - The two new compaction warnings in `EventLogFileCompactor` use the string-based logging API of `branch-3.5` instead of structured logging, with the same messages. - The skipped-line count is added to the bounded line reader of `branch-3.5` in `ReplayListenerBus`, which does not have the line origin diagnostics added by SPARK-60109. The rest of the patch, including the docs, the config doc and the tests, is the same as in #59328. ### Why are the changes needed? `branch-3.5` contains SPARK-59407 (8defa3c9418), so event log compaction has the same problem fixed by #59328: when replay skips an over-long line, such as the start of a live job or a live SQL execution without jobs, the filters do not know that it is live, the rewrite drops it, and the original files are deleted. The same happens when replay stops in the middle of a file on an error. ### Does this PR introduce _any_ user-facing change? Yes, the same change as #59328. When event log compaction is enabled and the History Server cannot fully replay the files to compact, it skips compaction of that application and keeps the original files, instead of losing events. With an event log line over `spark.history.fs.eventLog.maxLineLength`, compaction stays skipped until the limit is increased and a new event log file is rolled. ### How was this patch tested? On `branch-3.5` with Java 17.0.16 and Java 1.8.0_462 (Scala 2.12.18): ``` build/sbt 'core/testOnly *EventLogFileCompactorSuite *ReplayListenerSuite *EventFilterSuite *BasicEventFilterBuilderSuite *RocksDBBackendFsHistoryProviderSuite *EventLogFileWritersSuite *EventLogFileReadersSuite' 'sql/testOnly *SQLEventFilterBuilderSuite *SQLLiveEntitiesEventFilterSuite' build/sbt core/scalastyle core/Test/scalastyle sql/Test/scalastyle ``` All 87 core tests and 3 SQL tests passed on both JDKs, including the four new SPARK-60110 tests, and scalastyle passed (run with Java 17). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
