[
https://issues.apache.org/jira/browse/NIFI-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-15269.
-------------------------------------
Fix Version/s: 2.7.0
Resolution: Fixed
> Don't limit http concurrency in ConsumeKinesis
> ----------------------------------------------
>
> Key: NIFI-15269
> URL: https://issues.apache.org/jira/browse/NIFI-15269
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.6.0
> Reporter: Alaksiej Ščarbaty
> Assignee: Alaksiej Ščarbaty
> Priority: Major
> Fix For: 2.7.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> h2. The problem
> Currently a concurrency of ConsumeKinesis's underlying http client depends on
> a task count
> ([source|https://github.com/apache/nifi/blob/main/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-kinesis/src/main/java/org/apache/nifi/processors/aws/kinesis/ConsumeKinesis.java#L458]).
> When using the processor I observed that if the concurrency is lower than the
> number of shards, the following error occurs:
> _Acquire operation took longer than 10000 milliseconds._
> _Cancelling subscription, and marking self as failed. KCL will recreate the
> subscription as necessary to continue processing._
> _An acquire timeout occurred which usually indicates that the
> KinesisAsyncClient_
> _supplied has a low maximum streams limit._
> _Please use the software.amazon.kinesis.common.KinesisClientUtil to setup the
> client,_
> _or refer to the class to setup the client manually._
> h2. Cause
> This can be explained by the fact that during enhanced fan-out data is pushed
> to the processor via http/2. Meaning, that for each shard there must be an
> active stream.
> KCL library doesn't limit http concurrency at all
> ([source|https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/KinesisClientUtil.java#L46]).
> When using http/2 this *does not* represent a number of created connections
> ([source|https://github.com/aws/aws-sdk-java-v2/blob/master/http-clients/netty-nio-client/src/main/java/software/amazon/awssdk/http/nio/netty/NettyNioAsyncHttpClient.java#L287]),
> as multiplexing is used.
> It's recommended that the number of streams per connection is set to no less
> than 100 ([rfc
> doc|https://www.rfc-editor.org/rfc/rfc9113.html#section-6.5.2-2.6.1]),
> meaning that {*}the number of used connections is going to be relatively
> low{*}.
> h2. Proposal
> Only after setting quite a large task count (25 for a stream with 400 shards)
> the errors disappeared. This isn't sustainable as:
> 1. a customer must set unrealistically large task counts only to ensure the
> http concurrency is sufficient;
> 2. with dynamically changing number shards a working setup might start
> accidentally failing after adding new shards to a stream.
> Instead, we should not limit the http concurrency in the processor, like in
> KCL. This will resolve the underlying problem and will improve the
> processor's reliability.
> *This will not* create a large number of http connections, since http/2
> protocol is used.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
