We could not find producer msg rate from the matrices in the JConsole — give us
some pointers.
Also confirming that the reduction in data is due to Avro encoding and we are
calculating what we send to producer rather than the output of serializer
encoder.
Thanks,
Maung
On Jun 3, 2014, at
It should be sth like clientId-MessagesPerSec.
Thanks,
Jun
On Wed, Jun 4, 2014 at 9:35 AM, Maung Than maung_t...@apple.com wrote:
We could not find producer msg rate from the matrices in the JConsole —
give us some pointers.
Also confirming that the reduction in data is due to Avro
Hi,
We are seeing less data on the brokers than we send form the producers: 84 GB
to 58 GB.
What is the best way to ensure / detect if all data has been send properly to
the brokers from the producers.
Is there any logs that we can check on the producers?
Configuration is 5 Brokers, 2
Hi Maung,
If your required.acks is 1 then the producer only ensures that one
broker receives the data before it's sucessfully returned to the
client.
Therefore if the broker crashes and lost all the data then you lose
data, or similarly it can happen even before the data is fsynced.
To ensure
Thanks, Tim.
We are just trying to benchmark the kafka producers and there is no issue with
cluster or brokers being down in this case.
We are seeing way less data on the borers after calculating the sizes of the
logs on the brokers) and there is no compression.
We send 84 GB, but total
We have a metric on msg rate in both the producer and the broker. Could you
see if they match?
Thanks,
Jun
On Tue, Jun 3, 2014 at 2:13 PM, Maung Than maung_t...@apple.com wrote:
Hi,
We are seeing less data on the brokers than we send form the producers:
84 GB to 58 GB.
What is the best
By the way if you're using async producer how do you verify that you
sent all the data from the producer?
Do you shutdown the producer before you check?
Tim
On Tue, Jun 3, 2014 at 3:27 PM, Maung Than maung_t...@apple.com wrote:
Thanks, Tim.
We are just trying to benchmark the kafka producers
Yes. We did..some output of it..
2014-06-03 21:46:09 INFO Producer:68 - Shutting down producer
2014-06-03 21:46:09 INFO ProducerSendThread:68 - Begin shutting down
ProducerSendThread
2014-06-03 21:46:09 INFO ProducerSendThread:68 - Shutdown ProducerSendThread
complete
2014-06-03 21:46:09
