Jungtaek Lim created SPARK-26949:
------------------------------------
Summary: Prevent "purge" to remove needed batch files in
CompactibleFileStreamLog
Key: SPARK-26949
URL: https://issues.apache.org/jira/browse/SPARK-26949
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.0.0
Reporter: Jungtaek Lim
I've seen couple of trials (in opened PRs, even I've also tried) which calls
purge() in CompactibleFileStreamLog, but after looking at the codebase of
CompactibleFileStreamLog, I've realized that purging latest compaction batch
would break internal of CompactibleFileStreamLog and it throws
IllegalStateException.
Given that CompactibleFileStreamLog maintains the batches and purges according
to its configuration, it would be safer to just rely on
CompactibleFileStreamLog to purge and prevent calling `purge` outside of
CompactibleFileStreamLog.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
