[ https://issues.apache.org/jira/browse/FLINK-6306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15970683#comment-15970683 ]
Tzu-Li (Gordon) Tai commented on FLINK-6306: -------------------------------------------- This definitely needs a closer look. Thanks for picking this up! Could you briefly describe what you have in mind for the implementation? > Sink for eventually consistent file systems > ------------------------------------------- > > Key: FLINK-6306 > URL: https://issues.apache.org/jira/browse/FLINK-6306 > Project: Flink > Issue Type: New Feature > Components: filesystem-connector > Reporter: Seth Wiesman > Assignee: Seth Wiesman > > Currently Flink provides the BucketingSink as an exactly once method for > writing out to a file system. It provides there guarantees by moving files > through several stages and deleting or truncating files that get into a bad > state. While this is a powerful abstraction, it causes issues with eventually > consistent file systems such as Amazon's S3 where must operations (ie rename, > delete, truncate) are not guaranteed to become consistent within a reasonable > amount of time. Flink should provide a sink that provides exactly once writes > to a file system where only PUT operations are considered consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)