Hi Kafkaians, Me and Jason Gustafson have been working on a KIP to be able to set the maximum number of messages that a poll() call will return in the new Consumer API. You can find the KIP here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records It contains a full motivation, as well as links to JIRAs and mailing list discussion.
We are fairly done with the motivation as to why this is a needed feature. However, it feels like there are two approaches to limit the number of messages returned from a poll() call: 1. Introduce a second parameter to poll() so that it is given the following signature: ConsumerRecords<K, V> poll(long timeout, int maxMessages). 2. Introduce a configuration option, say *max.poll.messages*, which is given at KafkaConsumer construction. Arguments for 1): - It would make the Java API more explicit. - One would avoid an additional configuration. A counter-argument to that is that there are already many configuration options already. Avoiding yet another would not make that much difference. - It would allow the poll loop to dynamically change the max number of messages returned. However, this sounds like a non-requirement. Arguments for 2): - 1) would introduce yet another public method to would have to be maintained for a long time in the future. - Introducing the *max.poll.messages* approach would allow us to optimizations to reduce max messages to be fetched at initialization instead of at runtime. It would thus make the code paths simpler in each poll() call since message limitting could be abstracted out/away. Based on Jason's input on mailing list, on JIRA and in wiki, it feels like he's been leaning towards 1). I am personally leaning towards 2). What do you think? Do you have any other input? Looking forward to your input - cheers, Jens -- Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook <https://www.facebook.com/#!/tink.se> Linkedin <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary> Twitter <https://twitter.com/tink>
