Daniel Hermon created NIFI-11811:
------------------------------------
Summary: ListS3 Processor doesn't back pressure
Key: NIFI-11811
URL: https://issues.apache.org/jira/browse/NIFI-11811
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.16.2
Reporter: Daniel Hermon
Dear NiFi team,
I have built a data flow for syncing two S3 buckets on two separate sites.
I am using NiFi because the throughput is much higher than any other tool
(aws-s3-cli and RClone)
The flow is pretty straightforward:
ListS3 on-site A -> FetchS3 from site A -> FetchS3 from site B (check if file
already exists)-> (on failure) PutS3Object in site B
Sometimes site B throws 504 timeout exceptions which starts the back pressure
mechanism.
Everything back pressures perfectly fine but not the ListS3 processor output
queue, I have to manually terminate it in order to stop listing new files, even
trying to stop it fails.
I have tried limiting the size (even though it is 0 bytes flow files since the
flow file only contains attributes) and also limiting the number of objects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
