vanzin commented on a change in pull request #27208: [SPARK-30481][CORE]
Integrate event log compactor into Spark History Server
URL: https://github.com/apache/spark/pull/27208#discussion_r368086161
##########
File path:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -661,26 +691,33 @@ private[history] class FsHistoryProvider(conf:
SparkConf, clock: Clock)
reader: EventLogFileReader,
scanTime: Long,
enableOptimizations: Boolean): Unit = {
+ val rootPath = reader.rootPath
try {
+ val (shouldReload, lastCompactionIndex) = compact(reader)
Review comment:
I was thinking that instead of doing this inline here, you could just submit
a task to `replayExecutor` after the listing data is updated, so that you give
other tasks fetching listing data a chance to run before you try compaction.
(You'd call `endProcessing()` at the end of that separate task.)
Any reason why that would not work?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
