[ 
https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890855#comment-15890855
 ] 

Eugene Kirpichov commented on BEAM-849:
---------------------------------------

I disagree that "unbounded pipelines" can't finish successfully.

- Dataflow runner supports draining of pipelines, which leads to successful 
termination.
- It is possible to run a pipeline like Create.of(1, 2, 3) + ParDo(do nothing) 
using a streaming runner, and it should terminate rather than hang.
- One might ask "why run such a pipeline with a streaming runner", but it makes 
a lot more sense if the ParDo is splittable. E.g. Create.of(filename) + 
ParDo(tail file) + ParDo(process records) could use the low-latency 
capabilities of a streaming runner, but successfully terminate when the file is 
somehow "finalized". As a more mundane example - tests in 
https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
 should pass in streaming runners as well as batch runners.
- "Unbounded pipeline" is in general not a Beam concept - we should have a 
batch/streaming-agnostic meaning of "finished" in "waitUntilFinished". I 
propose the one that Dataflow runner uses for deciding when drain is completed: 
"all watermarks have progressed to infinity".

> Redesign PipelineResult API
> ---------------------------
>
>                 Key: BEAM-849
>                 URL: https://issues.apache.org/jira/browse/BEAM-849
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-core
>            Reporter: Pei He
>
> Current state: 
> Jira https://issues.apache.org/jira/browse/BEAM-443 addresses 
> waitUntilFinish() and cancel(). 
> However, there are additional work around PipelineResult: 
> need clearly defined contract and verification across all runners 
> need to revisit how to handle metrics/aggregators 
> need to be able to get logs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Reply via email to