[GitHub] [spark] HeartSaVioR commented on a change in pull request #27664: [SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the metadata log file when finding the latest batch ID

HeartSaVioR commented on a change in pull request #27664: URL: https://github.com/apache/spark/pull/27664#discussion_r429313023 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/FileStreamSinkLogSuite.scala ## @@ -267,4 +308,38 @@ class FileStreamS

[GitHub] [spark] HeartSaVioR commented on a change in pull request #27664: [SPARK-30915][SS] CompactibleFileStreamLog: Avoid reading the metadata log file when finding the latest batch ID

HeartSaVioR commented on a change in pull request #27664: URL: https://github.com/apache/spark/pull/27664#discussion_r428668027 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala ## @@ -182,19 +182,34 @@ class HDFSMetadataLog[T