Re: Default Retry mechanism for NIFI puts3Object processor
Thanks Peter for helping me out. Regards, Sanjeet On Tue, 12 Nov, 2019, 6:49 PM Peter Turcsanyi, wrote: > Hi Sanjeet, > > There is an open issue [1] about retry handling in AWS processors with a > pull request available [2] that might be interesting for you / solve your > problem. Unfortunately it has not been merged yet. > > This would be a more generic solution for all AWS processors which also > adds an option to configure the retry policy. > > Regards, > Peter > > [1] https://issues.apache.org/jira/browse/NIFI-6486 > [2] https://github.com/apache/nifi/pull/3612 > > On Mon, Nov 11, 2019 at 6:15 PM sanjeet rath > wrote: > >> Hi Team, >> >> I am using puts3Object processor of the nifi , to uploading object from >> onprem to AWS s3 bucket. i believe we have 2 types of uploading , single >> part upload and multipart upload as per the threshold value defined for >> multipart. >> >> for multipart , 3 steps are followed >> 1)s3.nitiateMultipartUpload , 2)s3.uploadPart 3)s3.completeMultipartUpload >> >> while checking the code i found , in s3.completeMultipartUpload method, >> if there is any server side exception(5**), then it is retrying 3 times (as >> in CompleteMultipartUploadRetryCondition class of AWS SDK, >> MAX_RETRY_ATTEMPTS is constant variable of value 3) using a do while loop . >> >> I have 2 questions >> >> a) This default retry mechanism (value is 3)is only used in >> s3.completeMultipartUpload method ? as i don't find any code for retry used >> in single object upload. >> >> b) if am going to changes MaxErrorRetry value AWS ClientConfiguration, >> does this will change it retry count if there is S3exception(5**) as per >> value i have set, as its a constant value of 3. Please confirm. >> >> c)If B answer is YES. Then only >> ClientConfiguration.MaxErrorRetry(myCostumValue) will work or >> >> I have to add bellow code for retry policy also. >> >> ClientConfiguration.setRetryPolicy(new >> RetryPolicy(config.getRetryPolicy().getRetryCondition(),config.getRetryPolicy().getBackoffStrategy(), >> myCostumValue, true). >> >> >> Thanks , >> >> Sanjeet >> >> >> >>
Re: Default Retry mechanism for NIFI puts3Object processor
Hi Sanjeet, There is an open issue [1] about retry handling in AWS processors with a pull request available [2] that might be interesting for you / solve your problem. Unfortunately it has not been merged yet. This would be a more generic solution for all AWS processors which also adds an option to configure the retry policy. Regards, Peter [1] https://issues.apache.org/jira/browse/NIFI-6486 [2] https://github.com/apache/nifi/pull/3612 On Mon, Nov 11, 2019 at 6:15 PM sanjeet rath wrote: > Hi Team, > > I am using puts3Object processor of the nifi , to uploading object from > onprem to AWS s3 bucket. i believe we have 2 types of uploading , single > part upload and multipart upload as per the threshold value defined for > multipart. > > for multipart , 3 steps are followed > 1)s3.nitiateMultipartUpload , 2)s3.uploadPart 3)s3.completeMultipartUpload > > while checking the code i found , in s3.completeMultipartUpload method, if > there is any server side exception(5**), then it is retrying 3 times (as in > CompleteMultipartUploadRetryCondition class of AWS SDK, MAX_RETRY_ATTEMPTS > is constant variable of value 3) using a do while loop . > > I have 2 questions > > a) This default retry mechanism (value is 3)is only used in > s3.completeMultipartUpload method ? as i don't find any code for retry used > in single object upload. > > b) if am going to changes MaxErrorRetry value AWS ClientConfiguration, > does this will change it retry count if there is S3exception(5**) as per > value i have set, as its a constant value of 3. Please confirm. > > c)If B answer is YES. Then only > ClientConfiguration.MaxErrorRetry(myCostumValue) will work or > > I have to add bellow code for retry policy also. > > ClientConfiguration.setRetryPolicy(new > RetryPolicy(config.getRetryPolicy().getRetryCondition(),config.getRetryPolicy().getBackoffStrategy(), > myCostumValue, true). > > > Thanks , > > Sanjeet > > > >
Default Retry mechanism for NIFI puts3Object processor
Hi Team, I am using puts3Object processor of the nifi , to uploading object from onprem to AWS s3 bucket. i believe we have 2 types of uploading , single part upload and multipart upload as per the threshold value defined for multipart. for multipart , 3 steps are followed 1)s3.nitiateMultipartUpload , 2)s3.uploadPart 3)s3.completeMultipartUpload while checking the code i found , in s3.completeMultipartUpload method, if there is any server side exception(5**), then it is retrying 3 times (as in CompleteMultipartUploadRetryCondition class of AWS SDK, MAX_RETRY_ATTEMPTS is constant variable of value 3) using a do while loop . I have 2 questions a) This default retry mechanism (value is 3)is only used in s3.completeMultipartUpload method ? as i don't find any code for retry used in single object upload. b) if am going to changes MaxErrorRetry value AWS ClientConfiguration, does this will change it retry count if there is S3exception(5**) as per value i have set, as its a constant value of 3. Please confirm. c)If B answer is YES. Then only ClientConfiguration.MaxErrorRetry(myCostumValue) will work or I have to add bellow code for retry policy also. ClientConfiguration.setRetryPolicy(new RetryPolicy(config.getRetryPolicy().getRetryCondition(),config.getRetryPolicy().getBackoffStrategy(), myCostumValue, true). Thanks , Sanjeet
