HeartSaVioR commented on a change in pull request #31495:
URL: https://github.com/apache/spark/pull/31495#discussion_r571832414
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala
##########
@@ -239,18 +239,35 @@ class HDFSMetadataLog[T <: AnyRef :
ClassTag](sparkSession: SparkSession, path:
.reverse
}
+ private var lastPurgedBatchId: Long = -1L
+
/**
* Removes all the log entry earlier than thresholdBatchId (exclusive).
*/
override def purge(thresholdBatchId: Long): Unit = {
- val batchIds = fileManager.list(metadataPath, batchFilesFilter)
- .map(f => pathToBatchId(f.getPath))
-
- for (batchId <- batchIds if batchId < thresholdBatchId) {
- val path = batchIdToPath(batchId)
- fileManager.delete(path)
- logTrace(s"Removed metadata log file: $path")
+ val possibleTargetBatchIds = (lastPurgedBatchId + 1 until thresholdBatchId)
+ if (possibleTargetBatchIds.length <= 3) {
+ // avoid using list if we only need to purge at most 3 elements
+ possibleTargetBatchIds.foreach { batchId =>
+ val path = batchIdToPath(batchId)
+ if (fileManager.exists(path)) {
Review comment:
(Just wanted to mention; the case what I'm considering is when the file
doesn't exist - then the case would be changed to exist vs delete. I'm going to
evaluate this because I don't know the cost comparison between exist and delete
on non-exist - if the cost difference is significant and exist is faster, it's
going to be some sort of probability/heuristic. If not, we should simply try
calling delete.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
