Eliaaazzz commented on PR #39823: URL: https://github.com/apache/beam/pull/39823#issuecomment-5520084881
Good question, I dug into this. I checked and could not find a ValidatesRunner test that was failing before and turns green with this fix, and I think there is a structural reason. The recent runs of both Spark postcommits are green on master, so I focused on the excluded tests. I lifted the UsesTestStreamWithMultipleStages exclusion locally and ran TestStreamTest on this branch. testMultiStage still fails at pipeline translation with "EVENT_TIME not yet supported in streaming mode" from StatefulStreamingParDoEvaluator, so its blocker is stateful event time timer support and it never reaches the watermark code. The other five tests selected by that run pass, and the full enabled validatesRunnerStreaming suite is green on this branch locally, 284 tests, no failures, one skipped. The drop needs at least two source ids, one updating in a batch while another already registered source with an unfinished watermark stays idle. A TestStream reports its watermark only on watermark events, and the only VR test I found with two TestStreams, testMultipleStreams, advances both to infinity together, so it does not exercise this case. It passes before and after this change. That is why the direct regression coverage here is the new GlobalWatermarkHolderTest case. Two existing CreateStream scripts relied on the drop erasing their stalled source, with retention they now advance their watermarks to infinity explicitly like the other scripts. If a VR level regression test would be useful I can add one with two TestStreams, one advancing while the other holds its watermark after registering. I would verify it fails on master before pushing it. I will resolve the conflict from the #39825 merge and trigger both Spark postcommits on this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
