[jira] [Comment Edited] (BEAM-849) Redesign PipelineResult API

2017-03-01 Thread Amit Sela (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890885#comment-15890885
 ] 

Amit Sela edited comment on BEAM-849 at 3/1/17 7:35 PM:


I disagree with stating that "Create.of(filename) + ParDo(tail file) + 
ParDo(process records)" in streaming leverages "low-latency", that's very 
runner specific, and generally inaccurate - how is sending a "filename" to 
worker/s + data locality slower than streaming the file to processors ? 


was (Author: amitsela):
I disagree with stating that "Create.of(filename) + ParDo(tail file) + 
ParDo(process records)" in streaming leverages "low-latency", that's very 
runner specific, and generally inaccurate - how is sending a "filename" to 
worker/s + data locality faster than streaming the file to processors ? 

> Redesign PipelineResult API
> ---
>
> Key: BEAM-849
> URL: https://issues.apache.org/jira/browse/BEAM-849
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Pei He
>
> Current state: 
> Jira https://issues.apache.org/jira/browse/BEAM-443 addresses 
> waitUntilFinish() and cancel(). 
> However, there are additional work around PipelineResult: 
> need clearly defined contract and verification across all runners 
> need to revisit how to handle metrics/aggregators 
> need to be able to get logs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Comment Edited] (BEAM-849) Redesign PipelineResult API

2017-01-08 Thread Stas Levin (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782396#comment-15782396
 ] 

Stas Levin edited comment on BEAM-849 at 1/8/17 12:21 PM:
--

In addition, the following points have been brought up in a recent [dev list 
thread|https://lists.apache.org/thread.html/79d78e8e114ac97e804f2c01570419404983d6fe40540471698c07c6@%3Cdev.beam.apache.org%3E]:

1.  The fact both {{waitUntilFinish()}} and {{cancel()}} return a {{State}} 
which may or may not be consistent with the {{State}} returned by 
(independently) calling {{getSate()}} is somewhat confusing.  Perhaps it would 
make sense to have {{waitUntilFinish()}} return {{boolean}} to indicate 
timeouts while {{cancel()}} can be {{void}}. This way the only way to get the 
{{State}} would be via {{getSate}} which could help things keep consistent, i.e.

{code:java}
State getState();
boolean waitUntilFinish(...);
void cancel() throws IOException;
{code}

2.  At the moment {{waitUntilFinish()}} returns {{null}} to indicate a timeout, 
which is a bit counter-intuitive for users in light of {{State}} being an enum 
(so they might not expect {{null}} to be ever returned).


was (Author: staslev):
In addition, the following points have been brought up in a recent [dev list 
thread|https://lists.apache.org/thread.html/79d78e8e114ac97e804f2c01570419404983d6fe40540471698c07c6@%3Cdev.beam.apache.org%3E]:

1.  The fact both {{waitUntilFinish()}} and {{cancel()}} return a {{State}} 
which may or may not be consistent with the {{State}} returned by 
(independently) calling {{getSate()}} is somewhat confusing.  Perhaps it would 
make sense to have {{waitUntilFinish()}} return {{boolean}} to indicate 
timeouts while {{cancel()}} can be {{void}}. This way the only way to get the 
{{State}} would be via {{getSate}} which could help things keep consistent.

{code:java}
void getState();
boolean waitUntilFinish(...);
State cancel() throws IOException;
{code}

2.  At the moment {{waitUntilFinish()}} returns {{null}} to indicate a timeout, 
which is a bit counter-intuitive for users in light of {{State}} being an enum 
(so they might not expect {{null}} to be ever returned).

> Redesign PipelineResult API
> ---
>
> Key: BEAM-849
> URL: https://issues.apache.org/jira/browse/BEAM-849
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Pei He
>
> Current state: 
> Jira https://issues.apache.org/jira/browse/BEAM-443 addresses 
> waitUntilFinish() and cancel(). 
> However, there are additional work around PipelineResult: 
> need clearly defined contract and verification across all runners 
> need to revisit how to handle metrics/aggregators 
> need to be able to get logs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)