[
https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414262#comment-16414262
]
Steve Loughran commented on FLINK-8794:
---------------------------------------
it does if you turn s3guard on with Hadoop 3.0+ and its S3A connector, as it
(like amazon's EMRFS) uses dynamodb for that consistency.
Unless you write code on the explicit assumption that the store is eventually
consistent, treating S3 "just" like a filesystem is dangerous. It'll usually
work most of the time in tests, but at larger scale production deployments you
can get burned.
> When using BucketingSink, it happens that one of the files is always in the
> [.in-progress] state
> ------------------------------------------------------------------------------------------------
>
> Key: FLINK-8794
> URL: https://issues.apache.org/jira/browse/FLINK-8794
> Project: Flink
> Issue Type: Improvement
> Components: filesystem-connector
> Affects Versions: 1.4.0, 1.4.1
> Reporter: yanxiaobin
> Priority: Major
>
> When using BucketingSink, it happens that one of the files is always in the
> [.in-progress] state. And this state has never changed after that. The
> underlying use of S3 as storage.
>
> {code:java}
> // code placeholder
> {code}
> 2018-02-28 11:58:42 147341619 {color:#d04437}_part-28-0.in-progress{color}
> 2018-02-28 12:06:27 147315059 part-0-0
> 2018-02-28 12:06:27 147462359 part-1-0
> 2018-02-28 12:06:27 147316006 part-10-0
> 2018-02-28 12:06:28 147349854 part-100-0
> 2018-02-28 12:06:27 147421625 part-101-0
> 2018-02-28 12:06:27 147443830 part-102-0
> 2018-02-28 12:06:27 147372801 part-103-0
> 2018-02-28 12:06:27 147343670 part-104-0
> ......
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
