> is there any gain on number of network calls being
made
No basically.
However, since next fetch requests are sent only when previous-fetched
records are processed, setting max.poll.records to too low would negatively
affect the network-call frequency depending on how you process records
Because if every poll() returns only small number of records,
batch-processing (which is known to be efficient) wouldn't be able so
processing previously-fetched records could take time due to the
insufficient throughput.
Also posted on SO: https://stackoverflow.com/a/77633494/3225746
2023年12月10日(日) 3:42 Debraj Manna :
> Can someone please clarify my below doubt? The same has been asked on stack
> overflow also.
>
> https://stackoverflow.com/q/77630586/785523
>
>
> On Fri, 8 Dec, 2023, 21:33 Debraj Manna, wrote:
>
> > Thanks again.
> >
> > Another follow-up question, since max.poll.records has nothing to do with
> > fetch requests, then is there any gain on number of network calls being
> > made between consumer & broker if max.poll.records is set to 1 as against
> > let's say the default 500.
> >
> > On Wed, Dec 6, 2023 at 7:21 PM Haruki Okada wrote:
> >
> >> poll-idle-ratio-avg=1.0 doesn't immediately mean fetch throughput
> problem
> >> since if processing is very fast, the metric will always be near 1.0.
> >>
> >> 2023年12月4日(月) 13:09 Debraj Manna :
> >>
> >> > Thanks for the reply.
> >> >
> >> > I read KIP
> >> > <
> >> >
> >>
> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=127406453#KIP517:Addconsumermetricstoobserveuserpollbehavior-poll-idle-ratio-avg
> >> > >.
> >> > Can you let me know if I observe poll-idle-ration.avg equal to 1.0
> then
> >> > does that mean my fetch.max.bytes or max.partition.fetch.bytes is not
> >> > enough and I have to increase them? If not what could be the reason
> that
> >> > may cause poll-idle-ratio-avg to approach 1.0?
> >> >
> >> >
> >> > Can you let me know what
> >> >
> >> > On Sat, 2 Dec, 2023, 07:05 Haruki Okada, wrote:
> >> >
> >> > > Hi.
> >> > >
> >> > > `max.poll.records` does nothing with fetch requests (refs:
> >> > >
> >> > >
> >> >
> >>
> https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records
> >> > > )
> >> > >
> >> > > Then, how many records will be returned for single fetch request
> >> depends
> >> > on
> >> > > the partition-leader assignment. (note: we assume follower-fetch is
> >> not
> >> > > used here)
> >> > > If all partition leaders are in the same broker, 40MB (2MB * 20
> >> > partition)
> >> > > will be returned for a single fetch request.
> >> > >
> >> > > 2023年11月30日(木) 17:10 Debraj Manna :
> >> > >
> >> > > > The doc states that fetch.max.bytes & max.partition.fetch.bytes
> >> > > >
> >> > > > are not absolute maximum. If the first record batch in the first
> >> > > non-empty
> >> > > > > partition of the fetch is larger than this limit, the batch will
> >> > still
> >> > > be
> >> > > > > returned to ensure that the consumer can make progress.
> >> > > >
> >> > > >
> >> > > > I am getting a bit confused.
> >> > > >
> >> > > > Let's say I have a configuration like below with sufficient
> >> messages in
> >> > > > each partition
> >> > > >
> >> > > >
> >> > > >- Partitions in a topic 20
> >> > > >- Single message size 2MB
> >> > > >- Consumers 5
> >> > > >- max.poll.records 20
> >> > > >- fetch.max.bytes 50 MB
> >> > > >- max.partition.fetch.bytes 1 MB.
> >> > > >
> >> > > > The broker config message.max.bytes and max.message.bytes is set
> to
> >> > > default
> >> > > > 100MB
> >> > > >
> >> > > > If the consumer does a poll will it receive 20 records? If yes
> then
> >> > there
> >> > > > is no significance of fetch.max.bytes & max.partition.fetch.bytes
> >> with
> >> > > > max.poll.records?
> >> > > >
> >> > > >
> >> > > >- Java Kafka Client - 3.5.1
> >> > > >- Kafka Broker - 2.8.1
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Okada Haruki
> >> > > ocadar...@gmail.com
> >> > >
> >> > >
> >> >
> >>
> >>
> >> --
> >>
> >> Okada Haruki
> >> ocadar...@gmail.com
> >>
> >>
> >
>
--
Okada Haruki
ocadar...@gmail.com
