Jason, thanks for the KIP. A few comments: * I think Dong's question about whether to use timestamp-based approach v.s. start-offset-of-first-larger-epoch is valid; more specifically, with timestamp-based approach we may still be reseting to an offset falling into the truncated interval, and hence we may still miss some data, i.e. not guaranteeing at-least-once still. With the start-offset-of-first-larger-epoch, I'm not sure if it will guarantee no valid data is missed when we have consecutive log truncations (maybe we need to look back into details of KIP-101 to figure it out). If the latter can indeed guarantee at least once, we could consider using that approach.
* My current understanding is that, with unclean leader election turned on, exactly-once is out of the window since we cannot guarantee that all committed message markers will not be lost. And hence there is no need to have special handling logic for LOG_TRUNCATED or OOR error codes with read.committed turned on. Is that right? * MINOR: "if the epoch is greater than the minimum expected epoch, that the new epoch does not begin at an earlier offset than the fetch offset. In the latter case, the leader can respond with a new LOG_TRUNCATION error code" should it be "does not begin at a later offset than the fetch offset"? Guozhang On Tue, Jun 26, 2018 at 6:51 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Jason, > > Thanks for the explanation. > > Please correct me if this is wrong. The "unknown truncation offset" > scenario happens when consumer does not have the full leaderEpoch -> offset > mapping. In this case we can still use the KIP-101-based approach to > truncate offset to "start offset of the first Leader Epoch larger than last > epoch of the consumer" but it may be inaccurate. So the KIP chooses to use > the timestamp-based approach which is also best-effort. > > If this understanding is correct, for "closest" offset reset policy and > "unknown truncation offset" scenario, I am wondering whether it maybe > better to replace timestamp-based approach with KIP-101 based approach. In > comparison to timestamp-based approach, the KIP-101-based approach seems to > simplify the API a bit since user does not need to understand timestamp. > Similar to the timestamp-based approach, both approaches are best-effort > and do not guarantee that consumer can consume all messages. It is not like > KIP-279 which guarantees that follower broker can consume all messages from > the leader. > > Then it seems that the remaining difference is mostly about accuracy, i.e. > how much message will be duplicated or missed in the "unknown truncation > offset" scenario. Not sure either one is clearly better than the other. > Note that there are two scenarios mentioned in KIP-279 which are not > addressed by KIP-101. Both scenarios require quick leadership change > between brokers, which seems to suggest that the offset based obtained > by "start > offset of the first Leader Epoch larger than last epoch of the consumer" > under these two scenarios may be very close to the offset obtained by the > message timestamp. Does this sound reasonable? > > Good point that users on v1 format can get benefit with timestamp based > approach. On the other hand it seems like a short term benefit for users > who have not migrated. I am just not sure whether it is more important than > designing a better API. > > Also, for both "latest" and "earliest" reset policy, do you think it would > make sense to also use the KIP-101 based approach to truncate offset for > the "unknown truncation offset" scenario? > > > Thanks, > Dong > -- -- Guozhang
