gresockj commented on PR #7529: URL: https://github.com/apache/nifi/pull/7529#issuecomment-1730281684
> @gresockj Thanks for migrating the Kinesis processors to v2 client lib! The change generally looks good to me, I also tested it with a simple flow. > > We had issues with stopping worker threads using the v1 lib and v2 seems to have similar problems. The consumer client creates 20 `aws-java-sdk-NettyEventLoop-1-*` threads. Not sure we need that many, furthermore these threads remain alive when the processor is stopped. Could you please check if the thread count can be configurable and how to stop these threads? Looking into this -- > > When I stop the consume processor I get the following error: > > ``` > 2023-07-31 23:04:40,056 ERROR [prefetch-cache-shardId-000000000000-0000] s.a.k.r.polling.PrefetchRecordsPublisher turcsanyi-stream:shardId-000000000000 : Unexpected exception was thrown. This could probably be an issue or a bug. Please search for the exception/error online to check what is going on. If the issue persists or is a recurring problem, feel free to open an issue on, https://github.com/awslabs/amazon-kinesis-client. > java.lang.IllegalStateException: Shutdown has been called on the cache, can't accept new requests. > ``` > > Do you see the same? > @gresockj Thanks for migrating the Kinesis processors to v2 client lib! The change generally looks good to me, I also tested it with a simple flow. > > We had issues with stopping worker threads using the v1 lib and v2 seems to have similar problems. The consumer client creates 20 `aws-java-sdk-NettyEventLoop-1-*` threads. Not sure we need that many, furthermore these threads remain alive when the processor is stopped. Could you please check if the thread count can be configurable and how to stop these threads? It looks like the thread count is configurable, via the following code: ``` NettyNioAsyncHttpClient.builder() .eventLoopGroupBuilder(SdkEventLoopGroup.builder() .numberOfThreads(MAX_EVENT_LOOP_GROUP_THREADS)) ``` However, when I do this, it seems to spawn a new thread loop group every time I stop/start a processor, and also for each new ConsumeKinesisStream processor I create, regardless of whether I cache the `SdkClient`. On the other hand, when I leave the default `NettyNioAsyncHttpClient` builder in place, I get a constant 24 threads regardless of starting/stopping and adding new processors. As it is, I'm inclined to leave the configuration as is, and perhaps this could be a future improvement if we can determine how to properly set the event loop group thread count. > > When I stop the consume processor I get the following error: > > ``` > 2023-07-31 23:04:40,056 ERROR [prefetch-cache-shardId-000000000000-0000] s.a.k.r.polling.PrefetchRecordsPublisher turcsanyi-stream:shardId-000000000000 : Unexpected exception was thrown. This could probably be an issue or a bug. Please search for the exception/error online to check what is going on. If the issue persists or is a recurring problem, feel free to open an issue on, https://github.com/awslabs/amazon-kinesis-client. > java.lang.IllegalStateException: Shutdown has been called on the cache, can't accept new requests. > ``` > > Do you see the same? Yes, unfortunately this appears to be an open issue in the client: https://github.com/awslabs/amazon-kinesis-client/issues/914 > > Not related to this change: `AbstractAwsProcessor` has `client` and `region` fields but the `awsClientCache` is used instead. I think `client` / `region` should be removed. We got rid of those fields in the v1 abstract processor. I guess the client cache solution was ported from there. Done, good catch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
