shunping commented on PR #38612: URL: https://github.com/apache/beam/pull/38612#issuecomment-4576866233
Some rationale for commits https://github.com/apache/beam/pull/38612/commits/76e656e87332ba9ea1f375ab05ee82546647385c and https://github.com/apache/beam/pull/38612/commits/d5e0add1a547c5c0fc30042db4acb7d39df9a781: The first commit broke Splittable DoFn (SDF) checkpoints because they split on the active element's restriction itself rather than returning unprocessed channel elements. As a result, splitBundle added 0 elements to the pending queue, and because the rescheduling logic was removed from ReturnResiduals, the residual restriction was completely lost. Dynamically calculated the original remaining element count: len(completed.es) - firstRsIndex and checked if len(unprocessedElements) exceeded it. This allows us to selectively reschedule only the new SDF residual restrictions while completely preserving the first commit for normal splits (preventing any double-counting of livePending). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
