[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981678#comment-15981678
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2417


> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958324#comment-15958324
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/2417


> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958325#comment-15958325
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

GitHub user pabloem reopened a pull request:

https://github.com/apache/beam/pull/2417

[BEAM-1148] Port PAssert away from Aggregators

Still running into trouble with the direct runner, but unit tests in other 
runners are passing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam play-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2417


commit a32d9ac4f2e8c3898ca9b7e3346be2d948bb4f9b
Author: Pablo 
Date:   2017-03-29T21:49:53Z

Playing with PAssert.java

commit c14080573be086c407537500f29a6e87d8420ffd
Author: Pablo Estrada 
Date:   2017-04-05T00:03:25Z

Removing extra line

commit 32da5ed589e1109131012dff05038e19f082fa3c
Author: Pablo Estrada 
Date:   2017-04-05T00:16:37Z

Small change

commit 2aad08648d97ead9e74ab99a92b4a873886efda5
Author: Pablo Estrada 
Date:   2017-04-05T00:29:08Z

Fix issue

commit a27e24352aed33fe88e05bfbc129ca48add30053
Author: Pablo Estrada 
Date:   2017-04-05T00:31:05Z

Fix issue

commit 091785089bdd974509e04988bbf11d7ed9d53ef5
Author: Pablo Estrada 
Date:   2017-04-05T07:56:10Z

Serializable Stack Traces

commit b6af5429e2f6dcca7dec5bc6429ddb0d1a496d72
Author: Pablo 
Date:   2017-04-05T18:31:30Z

Further fixing

commit d6cdee3bca90ad3d4e7064d891f0136c4567409e
Author: Pablo 
Date:   2017-04-05T20:01:27Z

Styling

commit 8ee6b7ddcafe4a60527c49d0667efb75a3629b03
Author: Pablo 
Date:   2017-04-06T01:28:57Z

Addressing comments




> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958229#comment-15958229
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/2417


> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958230#comment-15958230
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

GitHub user pabloem reopened a pull request:

https://github.com/apache/beam/pull/2417

[BEAM-1148] Port PAssert away from Aggregators

Still running into trouble with the direct runner, but unit tests in other 
runners are passing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam play-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2417


commit a32d9ac4f2e8c3898ca9b7e3346be2d948bb4f9b
Author: Pablo 
Date:   2017-03-29T21:49:53Z

Playing with PAssert.java

commit c14080573be086c407537500f29a6e87d8420ffd
Author: Pablo Estrada 
Date:   2017-04-05T00:03:25Z

Removing extra line

commit 32da5ed589e1109131012dff05038e19f082fa3c
Author: Pablo Estrada 
Date:   2017-04-05T00:16:37Z

Small change

commit 2aad08648d97ead9e74ab99a92b4a873886efda5
Author: Pablo Estrada 
Date:   2017-04-05T00:29:08Z

Fix issue

commit a27e24352aed33fe88e05bfbc129ca48add30053
Author: Pablo Estrada 
Date:   2017-04-05T00:31:05Z

Fix issue

commit 091785089bdd974509e04988bbf11d7ed9d53ef5
Author: Pablo Estrada 
Date:   2017-04-05T07:56:10Z

Serializable Stack Traces

commit b6af5429e2f6dcca7dec5bc6429ddb0d1a496d72
Author: Pablo 
Date:   2017-04-05T18:31:30Z

Further fixing

commit d6cdee3bca90ad3d4e7064d891f0136c4567409e
Author: Pablo 
Date:   2017-04-05T20:01:27Z

Styling

commit 8ee6b7ddcafe4a60527c49d0667efb75a3629b03
Author: Pablo 
Date:   2017-04-06T01:28:57Z

Addressing comments




> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956043#comment-15956043
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/2417


> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954429#comment-15954429
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2417

[BEAM-1148] Port PAssert away from Aggregators

Still running into trouble with the direct runner, but unit tests in other 
runners are passing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam play-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2417


commit a32d9ac4f2e8c3898ca9b7e3346be2d948bb4f9b
Author: Pablo 
Date:   2017-03-29T21:49:53Z

Playing with PAssert.java




> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939312#comment-15939312
 ] 

Eugene Kirpichov commented on BEAM-1148:


One option is to let PAssert output the failures (or success) to a file in the 
job's temp directory.

Another option: in case of success the pipeline should simply terminate 
successfully (even if it's being run using "streaming" mode, by a runner that 
has a concept of modes - according to the termination semantics); so it only 
remains to make sure that in case of failure the whole pipeline fails rather 
than hangs. There should be an API to fail a pipeline, or to watch failures 
within it. The treatment of failures, e.g. by Dataflow batch vs. streaming 
runners, is inconsistent. That's probably topic for a separate JIRA.

> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937237#comment-15937237
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2291

[BEAM-1148] Port PAssert away from Aggregators

This PR contains the following changes:
* Changing `PAssert.java` code to use Metrics instead of Aggregators
* Changing `TestSparkRunner.java` code to use Metrics. The change here also 
required adding up different Metrics counters because Metrics are per-step; 
while Aggregators in Spark seem to be adding accross steps.

Additional bug fixes/code improvements:
* Fixing a bug in `MetricFiltering`
* Changing the Spark runner to use `MetricFiltering` instead of its own 
filtering implementation - if that's okay.

* I ran `mvn clean verify` successfully.

Are there other places where I should be porting Aggregators checks to 
Metrics?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam fix-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2291.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2291


commit c6fe4986a720e5e9235be889f3a48844c0c85f0f
Author: Pablo 
Date:   2017-03-22T21:25:31Z

Tentative fixing PAssert for Spark runner tests.

commit 12342525888a6249122ab9327b71aa6cd75eba90
Author: Pablo 
Date:   2017-03-22T22:00:23Z

Fixing lint issue




> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)