it would be great if someone provides input(s)/hint :) thanks! --Senthil
On Fri, Aug 21, 2020 at 3:28 PM SenthilKumar K <senthilec...@gmail.com> wrote: > Updating the Kafka broker version: > > Kafka Version: 2.4.1 > > On Fri, Aug 21, 2020 at 3:21 PM SenthilKumar K <senthilec...@gmail.com> > wrote: > >> Hi Team, We have deployed 150 node Kafka cluster on production for our >> use case. Recently I have seen issue(s) in Kafka Producer Client. >> >> Use Case: >> <Incoming-Topic> --> (Consume)Stream App (Multiple Topologies) >> (Transform) --> Kafka Producer Topology ( Produce it to Multiple Topics) >> >> Initially, the data is written to a common buffer topic. The stream >> topologies will consume from buffer topic and transform from one form to >> JSON. Finally transformed output goes to Kafka Producer Topology. The >> nature of the Kafka Producer Topology is to get input from Transform >> Topology and Write it to stream specific topics. >> >> No of Transformation Topologies: 200 >> No Of Kafka Producer Topologies: 100 >> >> Each Kafka Producer Topology will be producing on multiple topics( >> approx: 2000). Initially, it used to produce to 1000 topics, and now the >> number of topics has increased hence the problem observed. >> >> What is the recommended approach to produce data on multiple topics from >> multiple clients? >> >> Looking forward to your inputs to optimize the Kafka producer topology. >> Thanks in advance! >> >> Client Version: >> >> <kafka-client.version>2.3.0</kafka-client.version> >> >> >> Kafka Producer Configuration: >> >> >> retries=3 >> linger.ms=5 >> buffer.memory=67108864 >> batch.size=32768 >> compression.type=snappy >> >> >> --Senthil >> >
