[
https://issues.apache.org/jira/browse/NIFI-11811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-11811.
-----------------------------------
Resolution: Feedback Received
Apache NiFi 1.x is no longer maintained and no new release is planned on the
1.x release line. Marking as resolved as part of a cleanup operation. Please
open a new one with an updated description if this is still relevant for NiFi
2.x.
> ListS3 Processor doesn't back pressure
> --------------------------------------
>
> Key: NIFI-11811
> URL: https://issues.apache.org/jira/browse/NIFI-11811
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.16.2
> Reporter: Daniel Hermon
> Priority: Critical
>
> Dear NiFi team,
> I have built a data flow for syncing two S3 buckets on two separate sites.
> I am using NiFi because the throughput is much higher than any other tool
> (aws-s3-cli and RClone)
> The flow is pretty straightforward:
> ListS3 on-site A > FetchS3 from site A -> FetchS3 from site B (check if file
> already exists)> (on failure) PutS3Object in site B
> Sometimes site B throws 504 timeout exceptions which starts the back pressure
> mechanism.
> Everything back pressures perfectly fine but not the ListS3 processor output
> queue, I have to manually terminate it in order to stop listing new files,
> even trying to stop it fails.
> I have tried limiting the size (even though it is 0 bytes flow files since
> the flow file only contains attributes) and also limiting the number of
> objects.
>
> p.s the bucket has 5 Billion objects
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
