What is happening imho is that when you have multiple partitions, each consumer
will fetch data from its partition and find only 1/64th the amount of data
(compared to the single partition case) to send every time it is its turn to
send stuff. Therefore you end up having a more chatty situation,
already feasible with the api set?
Thanks in advance for the help
Eric Owhadi
Esgyn Corporation
1875 message/s 3.7mb/s
-Original Message-
From: Eric Owhadi
Sent: Thursday, October 3, 2019 4:34 PM
To: users@kafka.apache.org
Subject: RE: poor producing performance with very low CPU utilization?
External
To test if my backend was good, I tried using the kafka-p
} : 600.000
19/10/03 14:24:16 INFO producer.KafkaProducer: [Producer clientId=producer-1]
Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
-Original Message-
From: Eric Owhadi
Sent: Thursday, October 3, 2019 3:45 PM
To: users@kafka.apache.org
Subject: RE: poor
time in the synchronous send, and therefore my producing
pump increases.
I wonder if some sort of contention happen when producer populate the 200
partition queues when the rate of production is high in the user thread?
Eric
-Original Message-
From: Eric Owhadi
Sent: Thursday, October
s log], and all of the assorted
>>responses to those calls.
given all is local (producer running on same node as broker), and the size of
my node (80 vcore), I hope I don t need 250ms to do that...
The equivalent workload on hbase2.0 is 10 to 20X faster (and that include same
replica
Message-
From: Eric Owhadi
To: users@kafka.apache.org
Sent: Wed, 2 Oct 2019 16:42
Subject: poor producing performance with very low CPU utilization?
Hi Kafka users,
I am new to Kafka and am struggling with getting acceptable producing rate.
I am using a cluster of 2 nodes, 40 CPU cores/ 80 if
as hoping for at least 10X more if not 100X better.
Was my expectations too high, or am I missing something in config that is
causing this performance numbers?
Some details: I produce using a jetty custom handler that I verified to be
super-fast when I am not producing (commenting out the send()
