jiangzzhu commented on PR #40870: URL: https://github.com/apache/beam/pull/40870#issuecomment-6586138161
> Actually, I'm not sure this would lead to lost things since if we haven't claimed the offset then on restart we would resume from the last claimed position. @scwhittle That's true for checkpoints (`trySplit(0)`), where the residual starts at `lastClaimedPosition + 1ns`. The stranding case happens on a non-checkpoint dynamic split (`trySplit(fraction > 0)`) after an earlier batch was claimed: if `splitPosition` lands between `T1` and `T2` of a cross-timestamp batch (`T1 < T_split <= T2`), calling `tryClaim(T2)` fails while advancing `lastAttemptedPosition` past `T_split` (so `checkDone()` passes on primary `[from, T_split)` without outputting `T1`), and the residual `[T_split, MAX)` starts past `T1`. Addressed both in the latest commit by moving `updateBatchToScheduled` after `tryClaim` and refusing non-checkpoint splits (`fractionOfRemainder != 0`) in `DetectNewPartitionsRangeTracker` — PTAL! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
