Hi Jack, Thanks for your interest in share groups. It's good to see the results that you have achieved with your performance measurements.
Whenever a complex new feature is created, there are always trade-offs. For example, I chose to make the KafkaShareConsumer similar to KafkaConsumer to give existing Kafka developers a simple on-ramp to using share groups. I also chose to inherit many of the existing configurations. Using max.poll.records=500 for a topic with a large backlog, maybe 5 consumers per partition and very short processing time does process the records nice and quickly. The intention was always to have the broker adapt to the workload to balance the number of records delivered with the rate at which the consumers are requesting them. We have not yet made that optimisation, but we are thinking about it. Because the configuration is max.poll.records and the protocol has MaxRecords, we could return fewer records than that to existing clients to achieve better sharing without any client changes. And thanks to you, we have a performance tool which we can use to thrash any potential improvements to make sure we do not regress anything. As mentioned in KIP-932, it would also be nice to balance the number of consumers fetching from the partitions based on the lag. Again, this is a potential future enhancement which we are considering. Because there are releases every 4 months, we have frequent opportunities for enhancing the code and we plan to do this. Thanks, Andrew On 2026/05/25 14:15:48 Jack Vanlightly wrote: > Hi all, > > I’ve been benchmarking share groups with simulated processing time in the > consumers, and wanted to raise a question about record allocation across > pending fetches. > > Blog post with details/results: > https://jack-vanlightly.com/blog/2026/5/25/kafka-share-groups-and-parallelizing-consumption-part-1-tuning-maxpollrecords > > The immediate lesson from the benchmark is that max.poll.records needs to > be set appropriately for share groups. In particular, it should be sized > relative to group.share.partition.max.record.locks and the number of > consumers per partition. With consumer groups, max.poll.records was not > nearly as important, and I think the default of 500 is too high for share > groups. Standard benchmarks don't highlight the issue, only when processing > time is included in the benchmark. > > With the defaults, I saw 300 consumers, 6 partitions, and 5 ms processing > time settle at around 4.8K msg/s instead of the theoretical 60K msg/s. The > reason was that a few consumers could acquire large batches, occupy most of > the per-partition inflight window, and leave the rest of the consumers > mostly idle. Reducing max.poll.records fixed the benchmark and allowed the > same setup to sustain 60K msg/s. > > So users must tune max.poll.records. But I think this is suboptimal from a > usability perspective. The right value depends on the inflight record > limit, consumers per partition, processing time, timing variance, partition > skew, etc. The correct value can change over time. Processing time may > change as application behavior changes. The number of consumers may change > if the application autoscales. > > My post’s rule of thumb is to divide group.share.partition.max.record.locks > by consumers per partition, then set max.poll.records somewhat lower to > leave room for variance. > > So I wonder whether share groups should consider a less greedy allocation > strategy when multiple consumers have pending fetches. Instead of filling > one fetch up to max.poll.records before serving others, the broker could > try to distribute available records more evenly across pending fetches > (improving the effective parallelism). > > I’m not proposing a specific algorithm here, and there are obvious > trade-offs around broker complexity, fetch efficiency, and configurability. > But it seems worth discussing because fairer allocation could make share > groups less sensitive to precise max.poll.records tuning and better > preserve the parallelism that share groups are intended to provide. > > Curious what others think. > > Jack Vanlightly >
