> is there any gain on number of network calls being made No basically. However, since next fetch requests are sent only when previous-fetched records are processed, setting max.poll.records to too low would negatively affect the network-call frequency depending on how you process records Because if every poll() returns only small number of records, batch-processing (which is known to be efficient) wouldn't be able so processing previously-fetched records could take time due to the insufficient throughput.
Also posted on SO: https://stackoverflow.com/a/77633494/3225746 2023年12月10日(日) 3:42 Debraj Manna <subharaj.ma...@gmail.com>: > Can someone please clarify my below doubt? The same has been asked on stack > overflow also. > > https://stackoverflow.com/q/77630586/785523 > > > On Fri, 8 Dec, 2023, 21:33 Debraj Manna, <subharaj.ma...@gmail.com> wrote: > > > Thanks again. > > > > Another follow-up question, since max.poll.records has nothing to do with > > fetch requests, then is there any gain on number of network calls being > > made between consumer & broker if max.poll.records is set to 1 as against > > let's say the default 500. > > > > On Wed, Dec 6, 2023 at 7:21 PM Haruki Okada <ocadar...@gmail.com> wrote: > > > >> poll-idle-ratio-avg=1.0 doesn't immediately mean fetch throughput > problem > >> since if processing is very fast, the metric will always be near 1.0. > >> > >> 2023年12月4日(月) 13:09 Debraj Manna <subharaj.ma...@gmail.com>: > >> > >> > Thanks for the reply. > >> > > >> > I read KIP > >> > < > >> > > >> > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=127406453#KIP517:Addconsumermetricstoobserveuserpollbehavior-poll-idle-ratio-avg > >> > >. > >> > Can you let me know if I observe poll-idle-ration.avg equal to 1.0 > then > >> > does that mean my fetch.max.bytes or max.partition.fetch.bytes is not > >> > enough and I have to increase them? If not what could be the reason > that > >> > may cause poll-idle-ratio-avg to approach 1.0? > >> > > >> > > >> > Can you let me know what > >> > > >> > On Sat, 2 Dec, 2023, 07:05 Haruki Okada, <ocadar...@gmail.com> wrote: > >> > > >> > > Hi. > >> > > > >> > > `max.poll.records` does nothing with fetch requests (refs: > >> > > > >> > > > >> > > >> > https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records > >> > > ) > >> > > > >> > > Then, how many records will be returned for single fetch request > >> depends > >> > on > >> > > the partition-leader assignment. (note: we assume follower-fetch is > >> not > >> > > used here) > >> > > If all partition leaders are in the same broker, 40MB (2MB * 20 > >> > partition) > >> > > will be returned for a single fetch request. > >> > > > >> > > 2023年11月30日(木) 17:10 Debraj Manna <subharaj.ma...@gmail.com>: > >> > > > >> > > > The doc states that fetch.max.bytes & max.partition.fetch.bytes > >> > > > > >> > > > are not absolute maximum. If the first record batch in the first > >> > > non-empty > >> > > > > partition of the fetch is larger than this limit, the batch will > >> > still > >> > > be > >> > > > > returned to ensure that the consumer can make progress. > >> > > > > >> > > > > >> > > > I am getting a bit confused. > >> > > > > >> > > > Let's say I have a configuration like below with sufficient > >> messages in > >> > > > each partition > >> > > > > >> > > > > >> > > > - Partitions in a topic 20 > >> > > > - Single message size 2MB > >> > > > - Consumers 5 > >> > > > - max.poll.records 20 > >> > > > - fetch.max.bytes 50 MB > >> > > > - max.partition.fetch.bytes 1 MB. > >> > > > > >> > > > The broker config message.max.bytes and max.message.bytes is set > to > >> > > default > >> > > > 100MB > >> > > > > >> > > > If the consumer does a poll will it receive 20 records? If yes > then > >> > there > >> > > > is no significance of fetch.max.bytes & max.partition.fetch.bytes > >> with > >> > > > max.poll.records? > >> > > > > >> > > > > >> > > > - Java Kafka Client - 3.5.1 > >> > > > - Kafka Broker - 2.8.1 > >> > > > > >> > > > >> > > > >> > > -- > >> > > ======================== > >> > > Okada Haruki > >> > > ocadar...@gmail.com > >> > > ======================== > >> > > > >> > > >> > >> > >> -- > >> ======================== > >> Okada Haruki > >> ocadar...@gmail.com > >> ======================== > >> > > > -- ======================== Okada Haruki ocadar...@gmail.com ========================
