[
https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509924#comment-17509924
]
David Handermann commented on NIFI-9820:
----------------------------------------
I agree that the NiFi use case is different than the default value set in the
Kudu Client library.
Using a small default value is probably a safer approach, and would follow a
similar pattern to other processors, as well as the default number of
concurrent tasks.
On the other hand, for flows that have existing PutKudu processors, the
challenge is to avoid introducing a negative impact. For example, if an
existing flow has a larger number of concurrent tasks, upgrading to a version
of NiFi that defaults to the Worker Client Count to 1 or 2 would have a
negative impact on performance after the upgrade. That was part of the reason
for setting the default value to match the internal value from the Kudu Client
library. For this reason, changing the default value to the number of CPU
cores, versus the number multiplied by 2, seems a middle way forward.
> Change PutKudu Property "Kudu Client Worker Count" Default Value
> ----------------------------------------------------------------
>
> Key: NIFI-9820
> URL: https://issues.apache.org/jira/browse/NIFI-9820
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 1.15.3
> Reporter: Josef Zahner
> Assignee: David Handermann
> Priority: Minor
>
> The PutKudu processor property "Kudu Client Worker Count" has a suboptimal
> value. Please don't use the current "number of CPUs multiplied by 2"
> behaviour as it leads to a massive amount of workers in our case with
> physical servers. We have a 8-node cluster where each server has 64 CPUs. We
> have about 30 PutKudu processors configured -> a lot of worker threads per
> default just for kudu.
> We have changed the number of worker threads in our case to the number of
> concurrent tasks. I don't know, maybe it would be great to set it a bit
> higher than that, but to be honest, I don't exactly understand the impact. It
> looks still fast with the current config.
> *To sum it up, please set a low default value (eg. 4 or 8) for the property
> "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu
> processor.*
> Btw. are there any suggestions how big the number should be?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
