Revin,
We instrument our data pipeline and Kafka applications using both with wall
time within our consumers and stream processors, and in a separate
application that measures end-to-end latency in our data processing
pipelines. We report these metrics to a metric aggregator, which in our
case is
When building these kinds of tests I always just orchestrated my producers
and consumers to spit metrics out somewhere easy to collect. Never looked
for a ui/tool to do it before.
Assuming good NTP configs (sub ms accuracy), I would typically put timing
data into the key portion of the messages
Look,. I'm a huge fan of sending identical data and using plane old 'wall
time' and averaging a couple runs to make sure you remove any whoops.
You can use fancy tools for reporting but in the real world wall time still
is the most critical factor. And let's face it it's also simple to measure.
We are testing kafka’s performance with the real prod data and plan to test
things like the below. We would have producers publishing and consumers
processing production data on a separate non-prod kafka cluster.
* Impact of number of Partitions per Topic on throughput and latency on
