viirya commented on PR #59327: URL: https://github.com/apache/spark/pull/59327#issuecomment-6133172932
#59360 (SPARK-60158) is merged to master and to branch-4.x, 4.3, 4.2, 4.1, 4.0 and 3.5, so I merged the latest master into this branch. There were no conflicts. With both changes, the end-event reparse on the same 210 MB logs takes 0.20 s (lz4) and 0.19 s (zstd), compared with 0.18 s and 0.19 s for the previous `Source.getLines()` path, so the listing slowdown is gone. `ReplayListenerSuite`, `FsHistoryProviderSuite` and `EventLogFileCompactorSuite` pass on JDK 17. This PR is ready to merge once CI passes. It should be backported to the same branches as #59360. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
