tkaymak opened a new issue, #40786: URL: https://github.com/apache/beam/issues/40786
Part of #36841. When one unbounded source PCollection feeds two branches of a single streaming query, for example Flatten of a PCollection with itself, Spark plans one MicroBatchScanExec per branch and does not deduplicate streaming relations, so each micro batch opens two readers for the same split. The sibling scan fix that follows #40766 gives the second scan a detached reader restored from the start mark, never cached and writing no marks. The two scans still agree only when the source yields the same records up to the same cut. With the default configuration (maxBatchDurationMillis 500 ms, no record quota) a batch cut by the deadline can end at different records per sibling, so the detached branch can lose or duplicate the tail of that batch while the next batch continues from the cached reader's mark. Bounded inputs via Create that drain within the deadline are exact. Possible fixes are recording the cached scan's cut (record count or end mark) so siblings replay to it, or a per batch buffer keyed by split and epoch shared by all scans of the key. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
