Updating the Kafka broker version: Kafka Version: 2.4.1
On Fri, Aug 21, 2020 at 3:21 PM SenthilKumar K <senthilec...@gmail.com> wrote: > Hi Team, We have deployed 150 node Kafka cluster on production for our > use case. Recently I have seen issue(s) in Kafka Producer Client. > > Use Case: > <Incoming-Topic> --> (Consume)Stream App (Multiple Topologies) (Transform) > --> Kafka Producer Topology ( Produce it to Multiple Topics) > > Initially, the data is written to a common buffer topic. The stream > topologies will consume from buffer topic and transform from one form to > JSON. Finally transformed output goes to Kafka Producer Topology. The > nature of the Kafka Producer Topology is to get input from Transform > Topology and Write it to stream specific topics. > > No of Transformation Topologies: 200 > No Of Kafka Producer Topologies: 100 > > Each Kafka Producer Topology will be producing on multiple topics( approx: > 2000). Initially, it used to produce to 1000 topics, and now the number of > topics has increased hence the problem observed. > > What is the recommended approach to produce data on multiple topics from > multiple clients? > > Looking forward to your inputs to optimize the Kafka producer topology. > Thanks in advance! > > Client Version: > > <kafka-client.version>2.3.0</kafka-client.version> > > > Kafka Producer Configuration: > > > retries=3 > linger.ms=5 > buffer.memory=67108864 > batch.size=32768 > compression.type=snappy > > > --Senthil >
