je-ik commented on PR #31391: URL: https://github.com/apache/beam/pull/31391#issuecomment-2130976269
> * This sounds similar to [[runners-flink] Fix watermark emission for empty splits (#29816) #30969](https://github.com/apache/beam/pull/30969), what is the difference here ? The fix in #30969 was related, but different. Source can be empty _temporarily_ or _finally_. The fact, that the source is empty for ever is signaled by watermark going to infinity. Then the split can be closed (and this results in watermark move, because closed split does not hold watermark anymore). This PR fixes the other case - when the source is not emitting any data, but _does not_ move watermark to infinity, but rather uses some idle source policy. Before this PR no watermark was emitted downstream _until at least one element was emitted from the source_. This is fixed now. > > * I also observed similar issue on JmsIO on Dataflow runner ("watermark does not increase when there is no incoming data for a while") and the fix [[DRAFT] Attempt fix Jms watermark #30337](https://github.com/apache/beam/pull/30337) didn't work. I am wondering if [[Bug]: FlinkRunner does not emit watermark with empty source #31390](https://github.com/apache/beam/issues/31390) is generic at SDK level and a fix could posed in general ? All these fixes relate to Flink only. These issues were introduced by source refactoring in FlinkRunner, so nothing that can be extended to a general case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
