Re: performance test using real data - comparing throughput & latency

Revin, We instrument our data pipeline and Kafka applications using both with wall time within our consumers and stream processors, and in a separate application that measures end-to-end latency in our data processing pipelines. We report these metrics to a metric aggregator, which in our case is

Adding or removing input topics to a Kafka Consumer without downtime

Hi there, In our company, we are running multiple Kafka Streams app that are deployed in a red/black fashion. A deployment means starting a new set of hosts, run them in parallel with the old hosts until they pass the health check, and then scale down the old hosts. This approach works very well