Chris Toomey created KAFKA-9127:
-----------------------------------
Summary: Needless group coordination overhead for GlobalKTables
Key: KAFKA-9127
URL: https://issues.apache.org/jira/browse/KAFKA-9127
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 2.3.0
Reporter: Chris Toomey
When creating a simple stream topology to just populate a GlobalKTable, I
noticed from logging that the stream consumer was doing group coordination
requests (JoinGroup, SyncGroup, ...) to the server, which it had no reason to
do since the global consumer thread populating the table fetches from all
partitions and thus doesn't use the group requests. So this adds needless
overhead on the client, network, and server.
I tracked this down to the stream thread consumer, which is created regardless
of whether it's needed based solely on NUM_STREAM_THREADS_CONFIG which defaults
to 1 I guess.
I found that setting NUM_STREAM_THREADS_CONFIG to 0 will prevent this from
happening, but it'd be a worthwhile improvement to be able to override this
setting in cases of topologies like this that don't have any need for stream
threads. Hence this ticket.
I originally asked about this on the users mailing list where Bruno suggested I
file it as an improvement request.
Here's the Scala code that I'm using that exhibits this:
{code:scala}
val builder: StreamsBuilder = new StreamsBuilder()
val gTable = builder.globalTable[K, V](...)
val stream = new KafkaStreams(builder.build(), props)
stream.start(){code}
Not shown is the state store that I'm populating/using.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
