[
https://issues.apache.org/jira/browse/NIFI-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navneet Kaur updated NIFI-10112:
--------------------------------
Description:
While listing and fetching objects from S3, and have received this error
frequently:
{color:#ff0000}{{ListS3 Failed to list contents of bucket due to Please reduce
your request rate. (Service: Amazon S3; Status Code: 503; Error Code:
Slowdown;.....)}}{color}
{color:#ff0000}{{FetchS3 Failed to retrieve S3 object for FlowFile ...; routing
to failure: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce
your request rate. (Service: Amazon S3; Status Code: 503; Error Code:
Slowdown;.....)}}{color}
{color:#172b4d}Scenario: We were listing and fetching many files (1000+) on the
buckets which are unique on the path, and was getting this error as
transactions per second increased by 5.5k (I couldn't test if it actually
increased by this number). Solution to this was setting "Run Schedule" time for
ListS3 (5 mins) and FetchS3 (15sec).{color}
{color:#172b4d}[https://aws.amazon.com/premiumsupport/knowledge-center/s3-resolve-503-slowdown-throttling/]{color}
{color:#172b4d}
This solution didn't solve the problem throughly, as I still get this error but
sparsely.{color}
{color:#172b4d}Looking forward to get this handled within the processors so
that the transactions per second don't increase beyond the threshold.{color}
was:
While listing and fetching objects from S3, and have received this error
frequently:
{color:#ff0000}{{ListS3 Failed to list contents of bucket due to Please reduce
your request rate. (Service: Amazon S3; Status Code: 503; Error Code:
Slowdown;.....)}}{color}
{color:#ff0000}{{FetchS3 Failed to retrieve S3 object for FlowFile ...; routing
to failure: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce
your request rate. (Service: Amazon S3; Status Code: 503; Error Code:
Slowdown;.....)}}{color}
{color:#172b4d}Scenario: We were listing and fetching many files (1000+) on the
buckets which are unique on the path, and was getting this error as
transactions per second increased by 10k (I couldn't test if it actually
increased by this number). Solution to this was setting "Run Schedule" time for
ListS3 (5 mins) and FetchS3 (15sec).
This solution didn't solve the problem throughly, as I still get this error but
sparsely.{color}
{color:#172b4d}Looking forward to get this handled within the processors so
that the transactions per second don't increase beyond the threshold.{color}
> Error to slowdown for ListS3 and FetchS3
> ----------------------------------------
>
> Key: NIFI-10112
> URL: https://issues.apache.org/jira/browse/NIFI-10112
> Project: Apache NiFi
> Issue Type: Wish
> Components: Core Framework
> Affects Versions: 1.16.1
> Environment: Linux
> Reporter: Navneet Kaur
> Priority: Minor
>
> While listing and fetching objects from S3, and have received this error
> frequently:
> {color:#ff0000}{{ListS3 Failed to list contents of bucket due to Please
> reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code:
> Slowdown;.....)}}{color}
> {color:#ff0000}{{FetchS3 Failed to retrieve S3 object for FlowFile ...;
> routing to failure: com.amazonaws.services.s3.model.AmazonS3Exception: Please
> reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code:
> Slowdown;.....)}}{color}
>
> {color:#172b4d}Scenario: We were listing and fetching many files (1000+) on
> the buckets which are unique on the path, and was getting this error as
> transactions per second increased by 5.5k (I couldn't test if it actually
> increased by this number). Solution to this was setting "Run Schedule" time
> for ListS3 (5 mins) and FetchS3 (15sec).{color}
> {color:#172b4d}[https://aws.amazon.com/premiumsupport/knowledge-center/s3-resolve-503-slowdown-throttling/]{color}
> {color:#172b4d}
> This solution didn't solve the problem throughly, as I still get this error
> but sparsely.{color}
>
> {color:#172b4d}Looking forward to get this handled within the processors so
> that the transactions per second don't increase beyond the threshold.{color}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
