[
https://issues.apache.org/jira/browse/FLINK-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669358#comment-16669358
]
ASF GitHub Bot commented on FLINK-9752:
---------------------------------------
addisonj commented on issue #6795: [FLINK-9752][s3-fs-connector] Add s3
recoverable writer.
URL: https://github.com/apache/flink/pull/6795#issuecomment-434477664
@kl0u the jira issue (https://issues.apache.org/jira/browse/FLINK-10383)
says this is fixed in 1.6.2, but it doesn't appear like this has been
backported to that release... Is this work somehow still reflected in 1.6.2?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add an S3 RecoverableWriter
> ---------------------------
>
> Key: FLINK-9752
> URL: https://issues.apache.org/jira/browse/FLINK-9752
> Project: Flink
> Issue Type: Sub-task
> Components: Streaming Connectors
> Reporter: Stephan Ewen
> Assignee: Kostas Kloudas
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.2, 1.7.0
>
>
> S3 offers persistence only when uploads are complete. That means at the end
> of simple uploads and uploads of parts of a MultiPartUpload.
> We should implement a RecoverableWriter for S3 that does a MultiPartUpload
> with a Part per checkpoint.
> Recovering the reader needs the MultiPartUploadID and the list of ETags of
> previous parts.
> We need additional staging of data in Flink state to work around the fact that
> - Parts in a MultiPartUpload must be at least 5MB
> - Part sizes must be known up front. (Note that data can still be streamed
> in the upload)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
