[
https://issues.apache.org/jira/browse/FLINK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475438#comment-16475438
]
ASF GitHub Bot commented on FLINK-9325:
---------------------------------------
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5982
Hi, I met a problem here, for the Hadoop FileSystem when the
`schema='hdfs'`, we can't rename a file to a existing file. This means that it
hard(I'm not sure whether it's possible) to support the `WriteModel.OVERWRITE`
for `createAtomically()` with `schema='hdfs'`, if we work-around this by
firstly rename the existing file to a tmp file and then rename the pre-commit
file to the target file, then this two step operations is not atomic, which may
lead to a inconsistent result if the JVM crashes between this operations. I'm
running out of myself on this now...Do you have any suggestions for the
`createAtomically()` when `schema='hdfs'` and
`writeMode='WriteMode.OVERWRITE'`? @StephanEwen
> generate the _meta file for checkpoint only when the writing is truly
> successful
> --------------------------------------------------------------------------------
>
> Key: FLINK-9325
> URL: https://issues.apache.org/jira/browse/FLINK-9325
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
> Affects Versions: 1.5.0
> Reporter: Sihua Zhou
> Assignee: Sihua Zhou
> Priority: Major
>
> We should generate the _meta file for checkpoint only when the writing is
> totally successful. We should write the metadata file first to a temp file
> and then atomically rename it (with an equivalent workaround for S3).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
