[
https://issues.apache.org/jira/browse/NIFI-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15002846#comment-15002846
]
Joe Skora commented on NIFI-1107:
---------------------------------
[~joewitt]
I made a separate processor for a couple of reasons.
* The AWS API methods differ with PutS3Object using the PutObject method and
PutS3ObjectMultipart using the InitiateMultipartUpload, UploadPart, and
CompleteMultipartUpload methods.
* PutS3Object uploads the file in a single API call, but has the 5GB size limit.
* PutS3ObjectMultipart does not have the size limit, but data is uploaded in
stateful fashion with the ability to stop and re-start the process without
having to repeat the upload of any part. So there's a good bit of state in the
multipart upload that is not needed in the single part upload.
I think it makes sense to keep them separate, much like the GetXXX and ListXXX
metaphors in other parts of the framework. The Get/List behaviors could also
be merged, but the cost is code complexity and greater chance of
misconfiguration by users.
Regards,
Joe
> Create new PutS3ObjectMultipart processor
> -----------------------------------------
>
> Key: NIFI-1107
> URL: https://issues.apache.org/jira/browse/NIFI-1107
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
> Reporter: Joe Skora
> Assignee: Joe Skora
> Labels: s3
> Fix For: 0.5.0
>
>
> A new `PutS3ObjectMultipart` processor using the AWS S3 API to upload files
> larger than those supported by `PutS3Object` which has a [5GB
> limit|http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html]
> limit.
> To support S3 compatible endpoints this will also add an `Endpoint Override
> URL` property to `AbstractAWSProcessor` to set the service
> [endpoint|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/AmazonWebServiceClient.html#setEndpoint(java.lang.String)]
> to override the endpoint URL normally selected based on the the Amazon
> region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
