Shruti Gumma created SPARK-28784:
------------------------------------
Summary: StreamExecution and StreamingQueryManager should utilize
CheckpointFileManager to interact with checkpoint directories
Key: SPARK-28784
URL: https://issues.apache.org/jira/browse/SPARK-28784
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.0
Reporter: Shruti Gumma
After PR [https://github.com/apache/spark/pull/21048], the
CheckpointFileManager interface was created to handle all structured streaming
checkpointing operations and helps users to choose how they wish to write
checkpointing files atomically.
StreamExecution and StreamingQueryManager still uses some FileSystem operations
without using the CheckpointFileManager.
For instance,
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L137]
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L392]
Instead, StreamExecution and StreamingQueryManager should use
CheckpointFileManager for these operations.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
