Hi Jay, I'm not sure I understand the "group commit". Can you elaborate a little bit?
The ISR change are actually on a per partition base. When a broker fails and come back, there could be thousands of ISR changes in say 30 seconds. In that case how would you group them? In terms of ISR shrinking, it is already sort of grouped because ISR shrinking check runs periodically. Not sure how ISR expansion would work here. Another approach might be just only use ISR shrinking check to propagate ISR change, so we basically reuse replica.lag.time.max.ms. But I feel reusing configuration usually causes more confusion than adding a new one. Thanks, Jiangjie (Becket) QIn On Fri, Aug 7, 2015 at 9:52 AM, Jay Kreps <j...@confluent.io> wrote: > Would it be possible to take a "group commit" approach rather than > configuring a delay. I.e. all changes that come in at once while another > operation is occurring automatically get batched. This can sometimes be > nicer and more automatic than another tunable parameter... > > -Jay > > On Fri, Aug 7, 2015 at 9:29 AM, Jiangjie Qin <j...@linkedin.com.invalid> > wrote: > > > Hi, > > > > I just created KIP-29 to add a new configuration of > IsrPropagateIntervalMs > > to KafkaConfig. The motivation is to batch and throttle the ISR > propagation > > in cases where large amount of ISR change occurs such as a broker gets > > bounced or failed. Otherwise it might overwhelm the controller. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-29+-+Add+an+IsrPropagateIntervalMs+configuration+to+KafkaConfig > > > > This is a config addition and is backward compatible. > > > > Thoughts? > > > > Thanks, > > > > Jiangjie (Becket) Qin > > >
