[
https://issues.apache.org/jira/browse/KAFKA-7636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689465#comment-16689465
]
ASF GitHub Bot commented on KAFKA-7636:
---------------------------------------
kcirtap7 opened a new pull request #5919: KAFKA-7636: Allow consumer to update
maxPollRecords value
URL: https://github.com/apache/kafka/pull/5919
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow consumer to update maxPollRecords value
> ---------------------------------------------
>
> Key: KAFKA-7636
> URL: https://issues.apache.org/jira/browse/KAFKA-7636
> Project: Kafka
> Issue Type: Improvement
> Components: clients
> Affects Versions: 2.0.1
> Reporter: Kcirtap Seven
> Priority: Minor
>
> Hi,
> We have two use cases where we would need to change the max.poll.records
> parameter on the fly :
> 1. We offer a REST API to get 'feedbacks'. This API takes into account a
> parameter 'count'.
> The system was previously based on cassandra. It is now based on kafka and
> 'feedbacks' are stored into kafka topics.
> To be compliant with the legacy interface contract, we would like to be able
> to change the max.poll.records on the fly to take into account this 'count'
> parameter.
> 2. We receive 'notification requests' related to a 'sender' via a REST API.
> We store those requests into topics (by sender).
> Each sender is associated with a weight. Here is the algorithm that process
> the requests :
> 1. At each iteration, we process at max n records (n configurable) for
> the whole bunch of requests. For this example, let's say 100.
> 2. We compute the max poll records for each sender. Let's say we have 3
> senders with the following weight 2, 1, 1. Hence 50 records max for the first
> one, 25 for the others two.
> 3. We consume the topics one after the other. We would like to reallocate
> some capacity to remaining consumers if the max.poll.records is not reached
> for the current consumer. Let'say at each iteration we make the following
> synchronous calls :
> sender1Consumer.poll() with computed max.poll.records 50
> sender2Consumer.poll() with computed max.poll.records 25
> sender3Consumer.poll() with computed max.poll.records 25
> If the first call returns only 10 records, we would like to reallocate
> the 40 "spare" records to the other consumers, 20 for each for instance (or
> another strategy). We would make the following calls instead :
> sender2Consumer.poll() with updated max.poll.records 45
> sender3Consumer.poll() with updated max.poll.records 45
>
> For that requirement we also need to change the max.poll.records on the fly.
> Regards,
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
