[
https://issues.apache.org/jira/browse/NIFI-11811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Hermon updated NIFI-11811:
---------------------------------
Description:
Dear NiFi team,
I have built a data flow for syncing two S3 buckets on two separate sites.
I am using NiFi because the throughput is much higher than any other tool
(aws-s3-cli and RClone)
The flow is pretty straightforward:
ListS3 on-site A > FetchS3 from site A -> FetchS3 from site B (check if file
already exists)> (on failure) PutS3Object in site B
Sometimes site B throws 504 timeout exceptions which starts the back pressure
mechanism.
Everything back pressures perfectly fine but not the ListS3 processor output
queue, I have to manually terminate it in order to stop listing new files, even
trying to stop it fails.
I have tried limiting the size (even though it is 0 bytes flow files since the
flow file only contains attributes) and also limiting the number of objects.
was:
Dear NiFi team,
I have built a data flow for syncing two S3 buckets on two separate sites.
I am using NiFi because the throughput is much higher than any other tool
(aws-s3-cli and RClone)
The flow is pretty straightforward:
ListS3 on-site A -> FetchS3 from site A -> FetchS3 from site B (check if file
already exists)-> (on failure) PutS3Object in site B
Sometimes site B throws 504 timeout exceptions which starts the back pressure
mechanism.
Everything back pressures perfectly fine but not the ListS3 processor output
queue, I have to manually terminate it in order to stop listing new files, even
trying to stop it fails.
I have tried limiting the size (even though it is 0 bytes flow files since the
flow file only contains attributes) and also limiting the number of objects.
> ListS3 Processor doesn't back pressure
> --------------------------------------
>
> Key: NIFI-11811
> URL: https://issues.apache.org/jira/browse/NIFI-11811
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.16.2
> Reporter: Daniel Hermon
> Priority: Critical
>
> Dear NiFi team,
> I have built a data flow for syncing two S3 buckets on two separate sites.
> I am using NiFi because the throughput is much higher than any other tool
> (aws-s3-cli and RClone)
> The flow is pretty straightforward:
> ListS3 on-site A > FetchS3 from site A -> FetchS3 from site B (check if file
> already exists)> (on failure) PutS3Object in site B
> Sometimes site B throws 504 timeout exceptions which starts the back pressure
> mechanism.
> Everything back pressures perfectly fine but not the ListS3 processor output
> queue, I have to manually terminate it in order to stop listing new files,
> even trying to stop it fails.
> I have tried limiting the size (even though it is 0 bytes flow files since
> the flow file only contains attributes) and also limiting the number of
> objects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
