dongjoon-hyun commented on a change in pull request #23850: [SPARK-26949][SS]
Prevent 'purge' to remove needed batch files in CompactibleFileStreamLog
URL: https://github.com/apache/spark/pull/23850#discussion_r291825976
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala
##########
@@ -163,6 +163,16 @@ abstract class CompactibleFileStreamLog[T <: AnyRef :
ClassTag](
batchAdded
}
+ /**
+ * CompactibleFileStreamLog maintains logs by itself, and manual purging
might break internal
+ * state, specifically which latest compaction batch is purged.
+ *
+ * To simplify the situation, this method just throws
UnsupportedOperationException regardless
+ * of given parameter, and let CompactibleFileStreamLog handles purging by
itself.
+ */
+ override def purge(thresholdBatchId: Long): Unit = throw new
UnsupportedOperationException(
+ s"'purge' might break internal state of CompactibleFileStreamLog hence not
supported")
Review comment:
- nit. `s"` -> `"`?
- `CompactibleFileStreamLog hence not supported` seems to need some revision.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
