[
https://issues.apache.org/jira/browse/FLINK-11116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas updated FLINK-11116:
-----------------------------------
Summary: Overwrite outdated in-progress files in StreamingFileSink. (was:
Clean-up temporary files that upon recovery, they belong to no checkpoint.)
> Overwrite outdated in-progress files in StreamingFileSink.
> ----------------------------------------------------------
>
> Key: FLINK-11116
> URL: https://issues.apache.org/jira/browse/FLINK-11116
> Project: Flink
> Issue Type: Improvement
> Components: filesystem-connector
> Affects Versions: 1.7.0
> Reporter: Kostas Kloudas
> Assignee: Kostas Kloudas
> Priority: Major
> Fix For: 1.7.2
>
>
> In order to guarantee exactly-once semantics, the streaming file sink is
> implementing a two-phase commit protocol when writing files to the filesystem.
> Initially data is written to in-progress files. These files are then put into
> "pending" state when they are completed (based on the rolling policy), and
> they are finally committed when the checkpoint that put them in the "pending"
> state is acknowledged as complete.
> The above shows that in the case that we have:
> 1) checkpoints A, B, C coming
> 2) checkpoint A being acknowledged and
> 3) failure
> Then we may have files that do not belong to any checkpoint (because B and C
> were not considered successful). These files are currently not cleaned up.
> This issue aims at cleaning up these files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
