[
https://issues.apache.org/jira/browse/NIFI-11811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743422#comment-17743422
]
Daniel Hermon commented on NIFI-11811:
--------------------------------------
[~EndzeitBegins]
Thank you for your answer, so as far as I understand there is no way other than
changing the flow to the following:
ListS3 on-site A > Publish (Kafka / RabbitMQ / another queue mech) -> Consume
-> FetchS3 from site A -> FetchS3 from site B (check if the file already
exists)> (on failure) PutS3Object in site B.
The main problem is that since the processor keeps emitting flow files, the RAM
consumption ramps up to the point that the main node is not responding
> ListS3 Processor doesn't back pressure
> --------------------------------------
>
> Key: NIFI-11811
> URL: https://issues.apache.org/jira/browse/NIFI-11811
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.16.2
> Reporter: Daniel Hermon
> Priority: Critical
>
> Dear NiFi team,
> I have built a data flow for syncing two S3 buckets on two separate sites.
> I am using NiFi because the throughput is much higher than any other tool
> (aws-s3-cli and RClone)
> The flow is pretty straightforward:
> ListS3 on-site A > FetchS3 from site A -> FetchS3 from site B (check if file
> already exists)> (on failure) PutS3Object in site B
> Sometimes site B throws 504 timeout exceptions which starts the back pressure
> mechanism.
> Everything back pressures perfectly fine but not the ListS3 processor output
> queue, I have to manually terminate it in order to stop listing new files,
> even trying to stop it fails.
> I have tried limiting the size (even though it is 0 bytes flow files since
> the flow file only contains attributes) and also limiting the number of
> objects.
>
> p.s the bucket has 5 Billion objects
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
