Hi Liam, Thank you for you kind reply. I will check it out to see if this helps.
There are amost 5K topics in our cluster with 6 nodes. We collect the metrics by collectd (https://collectd.org/) and visualize with Grafana. If all these per topic metrics are collected, will it affect the performance of the cluster? Haijun Liam Clarke <[email protected]> 于2019年8月27日周二 下午4:15写道： > Hi, > > The broker exposes per topic error rates, that might help? > > https://docs.confluent.io/current/kafka/monitoring.html#per-topic-metrics > > Kind regards, > > Liam Clarke > > > On Tue, 27 Aug. 2019, 8:07 pm 李海军, <[email protected]> wrote: > > > Hi all, > > > > We have a kafka cluster consisted of 6 brokers with the version of > > kafka_2.11-0.10.2.1. These days we saw a continuous increasing on the > > metrics of "BytesRejectedPerSec" on broker 1. I already knew that this > > could happen when the message records were too large and raise a > > "RecordTooLargeException" on the client side, which could be seen in the > > client's log by using "Callback()" in the send method or just adapting > the > > log level to debug. > > > > However, from the perspective of the server side, there are no logs about > > the exceptions at all. Since there are too many producers involved in the > > clusters and checking all the client logs is out of the question, how > could > > I find these certain topics causing "BytesRejectedPerSec" in the server > > logs? > > > > Thanks in advance. > > > > Sinsecrely, > > Haijun > > >
