Jungtaek Lim created SPARK-27254:
------------------------------------
Summary: Cleanup complete but becoming invalid output files in
ManifestFileCommitProtocol if job is aborted
Key: SPARK-27254
URL: https://issues.apache.org/jira/browse/SPARK-27254
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.0.0
Reporter: Jungtaek Lim
ManifestFileCommitProtocol doesn't clean up complete (but will become invalid)
output files when job is aborted.
ManifestFileCommitProtocol doesn't do anything for cleaning up when job is
aborted but just maintains the metadata which list of complete output files are
written. SPARK-27210 addressed for task level cleanup, but it still doesn't
clean up it as job level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
