Thanks again. Another follow-up question, since max.poll.records has nothing to do with fetch requests, then is there any gain on number of network calls being made between consumer & broker if max.poll.records is set to 1 as against let's say the default 500.
On Wed, Dec 6, 2023 at 7:21 PM Haruki Okada <ocadar...@gmail.com> wrote: > poll-idle-ratio-avg=1.0 doesn't immediately mean fetch throughput problem > since if processing is very fast, the metric will always be near 1.0. > > 2023年12月4日(月) 13:09 Debraj Manna <subharaj.ma...@gmail.com>: > > > Thanks for the reply. > > > > I read KIP > > < > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=127406453#KIP517:Addconsumermetricstoobserveuserpollbehavior-poll-idle-ratio-avg > > >. > > Can you let me know if I observe poll-idle-ration.avg equal to 1.0 then > > does that mean my fetch.max.bytes or max.partition.fetch.bytes is not > > enough and I have to increase them? If not what could be the reason that > > may cause poll-idle-ratio-avg to approach 1.0? > > > > > > Can you let me know what > > > > On Sat, 2 Dec, 2023, 07:05 Haruki Okada, <ocadar...@gmail.com> wrote: > > > > > Hi. > > > > > > `max.poll.records` does nothing with fetch requests (refs: > > > > > > > > > https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records > > > ) > > > > > > Then, how many records will be returned for single fetch request > depends > > on > > > the partition-leader assignment. (note: we assume follower-fetch is not > > > used here) > > > If all partition leaders are in the same broker, 40MB (2MB * 20 > > partition) > > > will be returned for a single fetch request. > > > > > > 2023年11月30日(木) 17:10 Debraj Manna <subharaj.ma...@gmail.com>: > > > > > > > The doc states that fetch.max.bytes & max.partition.fetch.bytes > > > > > > > > are not absolute maximum. If the first record batch in the first > > > non-empty > > > > > partition of the fetch is larger than this limit, the batch will > > still > > > be > > > > > returned to ensure that the consumer can make progress. > > > > > > > > > > > > I am getting a bit confused. > > > > > > > > Let's say I have a configuration like below with sufficient messages > in > > > > each partition > > > > > > > > > > > > - Partitions in a topic 20 > > > > - Single message size 2MB > > > > - Consumers 5 > > > > - max.poll.records 20 > > > > - fetch.max.bytes 50 MB > > > > - max.partition.fetch.bytes 1 MB. > > > > > > > > The broker config message.max.bytes and max.message.bytes is set to > > > default > > > > 100MB > > > > > > > > If the consumer does a poll will it receive 20 records? If yes then > > there > > > > is no significance of fetch.max.bytes & max.partition.fetch.bytes > with > > > > max.poll.records? > > > > > > > > > > > > - Java Kafka Client - 3.5.1 > > > > - Kafka Broker - 2.8.1 > > > > > > > > > > > > > -- > > > ======================== > > > Okada Haruki > > > ocadar...@gmail.com > > > ======================== > > > > > > > > -- > ======================== > Okada Haruki > ocadar...@gmail.com > ======================== >
