poll-idle-ratio-avg=1.0 doesn't immediately mean fetch throughput problem since if processing is very fast, the metric will always be near 1.0.
2023年12月4日(月) 13:09 Debraj Manna <subharaj.ma...@gmail.com>: > Thanks for the reply. > > I read KIP > < > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=127406453#KIP517:Addconsumermetricstoobserveuserpollbehavior-poll-idle-ratio-avg > >. > Can you let me know if I observe poll-idle-ration.avg equal to 1.0 then > does that mean my fetch.max.bytes or max.partition.fetch.bytes is not > enough and I have to increase them? If not what could be the reason that > may cause poll-idle-ratio-avg to approach 1.0? > > > Can you let me know what > > On Sat, 2 Dec, 2023, 07:05 Haruki Okada, <ocadar...@gmail.com> wrote: > > > Hi. > > > > `max.poll.records` does nothing with fetch requests (refs: > > > > > https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records > > ) > > > > Then, how many records will be returned for single fetch request depends > on > > the partition-leader assignment. (note: we assume follower-fetch is not > > used here) > > If all partition leaders are in the same broker, 40MB (2MB * 20 > partition) > > will be returned for a single fetch request. > > > > 2023年11月30日(木) 17:10 Debraj Manna <subharaj.ma...@gmail.com>: > > > > > The doc states that fetch.max.bytes & max.partition.fetch.bytes > > > > > > are not absolute maximum. If the first record batch in the first > > non-empty > > > > partition of the fetch is larger than this limit, the batch will > still > > be > > > > returned to ensure that the consumer can make progress. > > > > > > > > > I am getting a bit confused. > > > > > > Let's say I have a configuration like below with sufficient messages in > > > each partition > > > > > > > > > - Partitions in a topic 20 > > > - Single message size 2MB > > > - Consumers 5 > > > - max.poll.records 20 > > > - fetch.max.bytes 50 MB > > > - max.partition.fetch.bytes 1 MB. > > > > > > The broker config message.max.bytes and max.message.bytes is set to > > default > > > 100MB > > > > > > If the consumer does a poll will it receive 20 records? If yes then > there > > > is no significance of fetch.max.bytes & max.partition.fetch.bytes with > > > max.poll.records? > > > > > > > > > - Java Kafka Client - 3.5.1 > > > - Kafka Broker - 2.8.1 > > > > > > > > > -- > > ======================== > > Okada Haruki > > ocadar...@gmail.com > > ======================== > > > -- ======================== Okada Haruki ocadar...@gmail.com ========================
