guozhangwang commented on pull request #11340: URL: https://github.com/apache/kafka/pull/11340#issuecomment-948851978
Thanks everyone for the valuable discussions, this is a tricky issue indeed and worth some more clarifications, on the status quo. In general there are a lot of issues we've observed with coupling the rebalance protocol with the `poll` calls, and in the long run we are thinking to move the rebalance as well as fetching records completely into the background thread, and hence caller `poll` would only be getting the buffered records --- I will submit a JIRA ticket for this soon --- but as for now, what I'm thinking is a short-term fix, and hence the goal is to fix it with as less scope and complexity as possible. The status quo is that, when we are about to enter the rebalance we try to commit sync with a `rebalance.timeout`. That means: 1) the time blocked on commit maybe LARGER than the timeout passed into `poll`. This is the main issue we'd like to address. 2) if the commit exhausted the timeout but still cannot succeed, we would "give up" with a warning log and still continue the rebalance protocol with the risk that the new host of those partitions may fetch stale committed offsets (or even no offsets at all, if there's no committed offsets before the rebalance happened). This is not ideal, but okay according to the at least once semantics --- note, with exactly once semantics we do not rely on consumer to commit offsets so EOS is not impacted by this behavior. So any solution we may propose, as long as it would not make further regression, would be fine. Now here are my thoughts: * it's risky to just always upper bound the commit timeout with the `poll` parameter value, since in practice many users may call `poll(0)` or with a very small millisecond parameter. This would mean that, we may be much more aggressive on giving up and continue, i.e. making 2) above more exaggerate. Imagine if there's a leader migration happening just at the same time of the rebalance, we would very easily give up the commit even with just one round-trip failure. * it's also risky to always upper bound the commit timeout with the configured rebalance regardless of the `poll` timeout, which we have been discussed many times above and we all know its impact now. So what's left seems to be that, we still make a reasonably "best effort" to commit offsets before continue the rebalance protocol, and that "best effort" should be somewhat irrelevant to the poll timeout, but also not blocking more than the timeout value itself. Hence it occurs to me that, the best effort for now could be that we still do not block for longer than the poll timeout --- i.e. we may even return early --- but we do not give up the commit either, and hence here async commit where could be potentially completed across multiple `poll` calls would be reasonable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
