[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958324#comment-15958324
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/2417


> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958325#comment-15958325
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

GitHub user pabloem reopened a pull request:

https://github.com/apache/beam/pull/2417

[BEAM-1148] Port PAssert away from Aggregators

Still running into trouble with the direct runner, but unit tests in other 
runners are passing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam play-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2417


commit a32d9ac4f2e8c3898ca9b7e3346be2d948bb4f9b
Author: Pablo 
Date:   2017-03-29T21:49:53Z

Playing with PAssert.java

commit c14080573be086c407537500f29a6e87d8420ffd
Author: Pablo Estrada 
Date:   2017-04-05T00:03:25Z

Removing extra line

commit 32da5ed589e1109131012dff05038e19f082fa3c
Author: Pablo Estrada 
Date:   2017-04-05T00:16:37Z

Small change

commit 2aad08648d97ead9e74ab99a92b4a873886efda5
Author: Pablo Estrada 
Date:   2017-04-05T00:29:08Z

Fix issue

commit a27e24352aed33fe88e05bfbc129ca48add30053
Author: Pablo Estrada 
Date:   2017-04-05T00:31:05Z

Fix issue

commit 091785089bdd974509e04988bbf11d7ed9d53ef5
Author: Pablo Estrada 
Date:   2017-04-05T07:56:10Z

Serializable Stack Traces

commit b6af5429e2f6dcca7dec5bc6429ddb0d1a496d72
Author: Pablo 
Date:   2017-04-05T18:31:30Z

Further fixing

commit d6cdee3bca90ad3d4e7064d891f0136c4567409e
Author: Pablo 
Date:   2017-04-05T20:01:27Z

Styling

commit 8ee6b7ddcafe4a60527c49d0667efb75a3629b03
Author: Pablo 
Date:   2017-04-06T01:28:57Z

Addressing comments




> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2417: [BEAM-1148] Port PAssert away from Aggregators

[pabloem]

GitHub user pabloem reopened a pull request:

https://github.com/apache/beam/pull/2417

[BEAM-1148] Port PAssert away from Aggregators

Still running into trouble with the direct runner, but unit tests in other 
runners are passing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam play-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2417


commit a32d9ac4f2e8c3898ca9b7e3346be2d948bb4f9b
Author: Pablo 
Date:   2017-03-29T21:49:53Z

Playing with PAssert.java

commit c14080573be086c407537500f29a6e87d8420ffd
Author: Pablo Estrada 
Date:   2017-04-05T00:03:25Z

Removing extra line

commit 32da5ed589e1109131012dff05038e19f082fa3c
Author: Pablo Estrada 
Date:   2017-04-05T00:16:37Z

Small change

commit 2aad08648d97ead9e74ab99a92b4a873886efda5
Author: Pablo Estrada 
Date:   2017-04-05T00:29:08Z

Fix issue

commit a27e24352aed33fe88e05bfbc129ca48add30053
Author: Pablo Estrada 
Date:   2017-04-05T00:31:05Z

Fix issue

commit 091785089bdd974509e04988bbf11d7ed9d53ef5
Author: Pablo Estrada 
Date:   2017-04-05T07:56:10Z

Serializable Stack Traces

commit b6af5429e2f6dcca7dec5bc6429ddb0d1a496d72
Author: Pablo 
Date:   2017-04-05T18:31:30Z

Further fixing

commit d6cdee3bca90ad3d4e7064d891f0136c4567409e
Author: Pablo 
Date:   2017-04-05T20:01:27Z

Styling

commit 8ee6b7ddcafe4a60527c49d0667efb75a3629b03
Author: Pablo 
Date:   2017-04-06T01:28:57Z

Addressing comments




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2417: [BEAM-1148] Port PAssert away from Aggregators

[pabloem]

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/2417


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-501) Update website skin

[Frances Perry (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958315#comment-15958315
 ] 

Frances Perry commented on BEAM-501:


Checked with JB that he's not actively working on this right now. Reassigning 
to Jeremy, who has some great thoughts ;-)

> Update website skin
> ---
>
> Key: BEAM-501
> URL: https://issues.apache.org/jira/browse/BEAM-501
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: Jeremy Weinstein
>
> Update the main landing page and website skin as discussed here
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-501) Update website skin

[Frances Perry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frances Perry reassigned BEAM-501:
--

Assignee: Jeremy Weinstein  (was: Jean-Baptiste Onofré)

> Update website skin
> ---
>
> Key: BEAM-501
> URL: https://issues.apache.org/jira/browse/BEAM-501
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: Jeremy Weinstein
>
> Update the main landing page and website skin as discussed here
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1851) Sample.fixedSizedGlobally documentation should include single worker memory constraint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958310#comment-15958310
 ] 

ASF GitHub Bot commented on BEAM-1851:
--

GitHub user mdshalda opened a pull request:

https://github.com/apache/beam/pull/2451

[BEAM-1851] Update documentation for fixedSizeGlobally…

to be clear about memory constraints, like the documentation for "any".




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mdshalda/beam mshalda/beam-1851

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2451.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2451


commit 15048e1cfeccaa5d036269c6b39b1b3941275b75
Author: mdshalda 
Date:   2017-04-06T05:23:00Z

BEAM-1851: Update documentation for for fixedSizeGlobally to be clear about 
memory constraints.




> Sample.fixedSizedGlobally documentation should include single worker memory 
> constraint
> --
>
> Key: BEAM-1851
> URL: https://issues.apache.org/jira/browse/BEAM-1851
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.6.0
>Reporter: Mark Shalda
>Assignee: Mark Shalda
>Priority: Trivial
>  Labels: doc
>
> Sample.any mentions in the documentation:
> " All of the elements of the output {@code PCollection} should fit into main 
> memory of a single worker machine.  This operation does not run in parallel."
> This should also be mentioned for fixedSizedGlobally.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2451: [BEAM-1851] Update documentation for fixedSizeGloba...

[mdshalda]

GitHub user mdshalda opened a pull request:

https://github.com/apache/beam/pull/2451

[BEAM-1851] Update documentation for fixedSizeGlobally…

to be clear about memory constraints, like the documentation for "any".




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mdshalda/beam mshalda/beam-1851

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2451.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2451


commit 15048e1cfeccaa5d036269c6b39b1b3941275b75
Author: mdshalda 
Date:   2017-04-06T05:23:00Z

BEAM-1851: Update documentation for for fixedSizeGlobally to be clear about 
memory constraints.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #2756

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2417: [BEAM-1148] Port PAssert away from Aggregators

[pabloem]

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/2417


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958229#comment-15958229
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/2417


> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958230#comment-15958230
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

GitHub user pabloem reopened a pull request:

https://github.com/apache/beam/pull/2417

[BEAM-1148] Port PAssert away from Aggregators

Still running into trouble with the direct runner, but unit tests in other 
runners are passing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam play-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2417


commit a32d9ac4f2e8c3898ca9b7e3346be2d948bb4f9b
Author: Pablo 
Date:   2017-03-29T21:49:53Z

Playing with PAssert.java

commit c14080573be086c407537500f29a6e87d8420ffd
Author: Pablo Estrada 
Date:   2017-04-05T00:03:25Z

Removing extra line

commit 32da5ed589e1109131012dff05038e19f082fa3c
Author: Pablo Estrada 
Date:   2017-04-05T00:16:37Z

Small change

commit 2aad08648d97ead9e74ab99a92b4a873886efda5
Author: Pablo Estrada 
Date:   2017-04-05T00:29:08Z

Fix issue

commit a27e24352aed33fe88e05bfbc129ca48add30053
Author: Pablo Estrada 
Date:   2017-04-05T00:31:05Z

Fix issue

commit 091785089bdd974509e04988bbf11d7ed9d53ef5
Author: Pablo Estrada 
Date:   2017-04-05T07:56:10Z

Serializable Stack Traces

commit b6af5429e2f6dcca7dec5bc6429ddb0d1a496d72
Author: Pablo 
Date:   2017-04-05T18:31:30Z

Further fixing

commit d6cdee3bca90ad3d4e7064d891f0136c4567409e
Author: Pablo 
Date:   2017-04-05T20:01:27Z

Styling

commit 8ee6b7ddcafe4a60527c49d0667efb75a3629b03
Author: Pablo 
Date:   2017-04-06T01:28:57Z

Addressing comments




> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #997

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #1532

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #2216

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2449: Fix Compile Break in PTransformMatchersTest

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2449


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Fix Compile Break in PTransformMatchersTest

[tgroh]

Fix Compile Break in PTransformMatchersTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f6e91470
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f6e91470
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f6e91470

Branch: refs/heads/master
Commit: f6e91470c5e0c805b87dce0101f9e898ccaac116
Parents: 2078895
Author: Thomas Groh 
Authored: Wed Apr 5 18:27:46 2017 -0700
Committer: Thomas Groh 
Committed: Wed Apr 5 19:26:21 2017 -0700

--
 .../runners/core/construction/PTransformMatchersTest.java | 10 ++
 1 file changed, 10 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f6e91470/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
--
diff --git 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
index 1d12194..d053f62 100644
--- 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
+++ 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
@@ -38,6 +38,8 @@ import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.Materialization;
+import org.apache.beam.sdk.transforms.Materializations;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.Sum;
@@ -350,6 +352,14 @@ public class PTransformMatchersTest implements 
Serializable {
 ViewFn, Iterable> viewFn =
 new ViewFn, Iterable>() {
   @Override
+  public Materialization> 
getMaterialization() {
+@SuppressWarnings({"rawtypes", "unchecked"})
+Materialization> materialization =
+(Materialization) Materializations.iterable();
+return materialization;
+  }
+
+  @Override
   public Iterable apply(Iterable contents) {
 return Collections.emptyList();
   }

[1/2] beam git commit: This closes #2449

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 20788953f -> 5fd5828d1


This closes #2449


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5fd5828d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5fd5828d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5fd5828d

Branch: refs/heads/master
Commit: 5fd5828d13044028a5b2068c4d1922eb7b27e5a1
Parents: 2078895 f6e9147
Author: Thomas Groh 
Authored: Wed Apr 5 19:26:21 2017 -0700
Committer: Thomas Groh 
Committed: Wed Apr 5 19:26:21 2017 -0700

--
 .../runners/core/construction/PTransformMatchersTest.java | 10 ++
 1 file changed, 10 insertions(+)
--

[jira] [Commented] (BEAM-1887) Switch ParDo execution to use new DoFn in Apex runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958177#comment-15958177
 ] 

ASF GitHub Bot commented on BEAM-1887:
--

GitHub user tweise opened a pull request:

https://github.com/apache/beam/pull/2450

[BEAM-1887] Switch Apex ParDo to new DoFn.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @kennknowles @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tweise/beam BEAM-1887_Apex-NewDoFn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2450.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2450


commit 629887cb124134cdae06087cb5159efa834df0b9
Author: Thomas Weise 
Date:   2017-04-04T07:33:35Z

BEAM-1887 Switch Apex ParDo to new DoFn.




> Switch ParDo execution to use new DoFn in Apex runner
> -
>
> Key: BEAM-1887
> URL: https://issues.apache.org/jira/browse/BEAM-1887
> Project: Beam
>  Issue Type: Task
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>
> The Apex operator is still using OldDoFn and that needs to change to support 
> splittable fn, state and timers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2450: [BEAM-1887] Switch Apex ParDo to new DoFn.

[tweise]

GitHub user tweise opened a pull request:

https://github.com/apache/beam/pull/2450

[BEAM-1887] Switch Apex ParDo to new DoFn.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @kennknowles @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tweise/beam BEAM-1887_Apex-NewDoFn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2450.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2450


commit 629887cb124134cdae06087cb5159efa834df0b9
Author: Thomas Weise 
Date:   2017-04-04T07:33:35Z

BEAM-1887 Switch Apex ParDo to new DoFn.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1891) Java @Autovalue: currently doesn't have a good coder

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-1891:
-

Assignee: (was: Davor Bonaci)

> Java @Autovalue: currently doesn't have a good coder
> 
>
> Key: BEAM-1891
> URL: https://issues.apache.org/jira/browse/BEAM-1891
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>  Labels: newbie, starter
>
> In Java, @AutoValue classes are something that we would like developers to be 
> able to use in PCollections in Beam.
> However, there doesn't appear to be a good existing Beam Coder for Autovalue 
> generated classes:
> * Avrocoder doesn't work
> * Serializable Coder works, but has other problems (larger/less efficient)
> This is discussed fully at 
> https://lists.apache.org/thread.html/29617096819824d5c12247a246d316b763d9e583a21fff2f1c430077@%3Cdev.beam.apache.org%3E
>  
> We may need to implement an AutoValueCoder.
> cc [~pabloem] [~bchambers]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1891) Java @Autovalue: currently doesn't have a good coder

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1891:
--
Priority: Minor  (was: Major)

> Java @Autovalue: currently doesn't have a good coder
> 
>
> Key: BEAM-1891
> URL: https://issues.apache.org/jira/browse/BEAM-1891
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Priority: Minor
>  Labels: newbie, starter
>
> In Java, @AutoValue classes are something that we would like developers to be 
> able to use in PCollections in Beam.
> However, there doesn't appear to be a good existing Beam Coder for Autovalue 
> generated classes:
> * Avrocoder doesn't work
> * Serializable Coder works, but has other problems (larger/less efficient)
> This is discussed fully at 
> https://lists.apache.org/thread.html/29617096819824d5c12247a246d316b763d9e583a21fff2f1c430077@%3Cdev.beam.apache.org%3E
>  
> We may need to implement an AutoValueCoder.
> cc [~pabloem] [~bchambers]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1891) Java @Autovalue: currently doesn't have a good coder

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1891:
--
Labels: newbie starter  (was: starter)

> Java @Autovalue: currently doesn't have a good coder
> 
>
> Key: BEAM-1891
> URL: https://issues.apache.org/jira/browse/BEAM-1891
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Priority: Minor
>  Labels: newbie, starter
>
> In Java, @AutoValue classes are something that we would like developers to be 
> able to use in PCollections in Beam.
> However, there doesn't appear to be a good existing Beam Coder for Autovalue 
> generated classes:
> * Avrocoder doesn't work
> * Serializable Coder works, but has other problems (larger/less efficient)
> This is discussed fully at 
> https://lists.apache.org/thread.html/29617096819824d5c12247a246d316b763d9e583a21fff2f1c430077@%3Cdev.beam.apache.org%3E
>  
> We may need to implement an AutoValueCoder.
> cc [~pabloem] [~bchambers]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #2755

[Apache Jenkins Server]

See 


Changes:

[altay] Replace Any type with a KV type for inputs of a GroupByKey step

[tgroh] Use WinodwMappingFn where possible

[tgroh] Add Initial Materializations

[tgroh] Remove unused import in ViewFn

[dhalperi] DataflowPipelineJob: handle concurrent cancel and finish

--
[...truncated 1.20 MB...]
2017-04-06T01:44:26.900 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-core ---
2017-04-06T01:44:27.044 [INFO] 
2017-04-06T01:44:27.044 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-core ---
2017-04-06T01:44:27.200 [INFO] Building jar: 

2017-04-06T01:44:27.364 [INFO] 
2017-04-06T01:44:27.365 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-core ---
2017-04-06T01:44:27.469 [INFO] 
2017-04-06T01:44:27.469 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-core ---
2017-04-06T01:44:27.595 [INFO] Building jar: 

2017-04-06T01:44:27.727 [INFO] 
2017-04-06T01:44:27.728 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-core ---
2017-04-06T01:44:27.731 [INFO] Excluding 
org.apache.beam:beam-sdks-common-runner-api:jar:0.7.0-SNAPSHOT from the shaded 
jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 
from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from 
the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 
from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding com.google.code.gson:gson:jar:2.7 from 
the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
com.google.auth:google-auth-library-credentials:jar:0.6.0 from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
com.google.auth:google-auth-library-oauth2-http:jar:0.6.0 from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.netty:netty-common:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0 from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-04-06T01:44:27.731 [INFO] Excluding 
com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #1531

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add Initial Materializations

[tgroh] Remove unused import in ViewFn

[dhalperi] DataflowPipelineJob: handle concurrent cancel and finish

--
[...truncated 665.49 KB...]
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.auth:google-auth-library-oauth2-http:jar:0.6.0 from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding io.netty:netty-common:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0 from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0 from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0 from the 
shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 from the shaded 
jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.http-client:google-http-client:jar:1.22.0 from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
commons-logging:commons-logging:jar:1.1.1 from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding commons-codec:commons-codec:jar:1.3 
from the shaded jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.http-client:google-http-client-jackson:jar:1.22.0 from the shaded 
jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.http-client:google-http-client-jackson2:jar:1.22.0 from the shaded 
jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.http-client:google-http-client-protobuf:jar:1.22.0 from the shaded 
jar.
2017-04-06T01:37:20.823 [INFO] Excluding 
com.google.oauth-client:google-oauth-client:jar:1.22.0 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.google.cloud.bigdataoss:gcsio:jar:1.4.5 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.google.api-client:google-api-client-java6:jar:1.22.0 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.google.api-client:google-api-client-jackson2:jar:1.22.0 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 from the shaded 
jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Including com.google.guava:guava:jar:20.0 in the 
shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.7.2 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.7.2 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.7.2 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding org.apache.avro:avro:jar:1.8.1 from 
the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-04-06T01:37:20.824 [INFO] Excluding 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #996

[Apache Jenkins Server]

See 


Changes:

[dhalperi] DataflowPipelineJob: handle concurrent cancel and finish

--
[...truncated 679.85 KB...]
Running org.apache.beam.sdk.util.NumberedShardedFileTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.038 sec - in 
org.apache.beam.sdk.util.NumberedShardedFileTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.PTupleTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.PTupleTest
Running org.apache.beam.sdk.util.UnownedOutputStreamTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedOutputStreamTest
Running org.apache.beam.sdk.util.gcsfs.GcsPathTest
Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.gcsfs.GcsPathTest
Running org.apache.beam.sdk.util.CombineFnUtilTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.CombineFnUtilTest
Running org.apache.beam.sdk.util.ReshuffleTriggerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ReshuffleTriggerTest
Running org.apache.beam.sdk.util.BucketingFunctionTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.BucketingFunctionTest
Running org.apache.beam.sdk.util.NameUtilsTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.NameUtilsTest
Running org.apache.beam.sdk.util.UserCodeExceptionTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.UserCodeExceptionTest
Running org.apache.beam.sdk.util.SerializerTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.SerializerTest
Running org.apache.beam.sdk.util.MovingFunctionTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.MovingFunctionTest
Running org.apache.beam.sdk.util.AttemptBoundedExponentialBackOffTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.AttemptBoundedExponentialBackOffTest
Running org.apache.beam.sdk.util.IOChannelUtilsTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.IOChannelUtilsTest

Results :

Tests run: 2673, Failures: 0, Errors: 0, Skipped: 3

[JENKINS] Recording test results
2017-04-06T01:30:36.918 [INFO] 
2017-04-06T01:30:36.918 [INFO] --- jacoco-maven-plugin:0.7.8:report (report) @ 
beam-sdks-java-core ---
2017-04-06T01:30:37.007 [INFO] Loading execution data file 

2017-04-06T01:30:37.319 [INFO] Analyzed bundle 'Apache Beam :: SDKs :: Java :: 
Core' with 1194 classes
2017-04-06T01:30:38.875 [INFO] 
2017-04-06T01:30:38.875 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-core ---
2017-04-06T01:30:38.975 [INFO] 
2017-04-06T01:30:38.976 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-core ---
2017-04-06T01:30:39.127 [INFO] Building jar: 

2017-04-06T01:30:39.284 [INFO] 
2017-04-06T01:30:39.284 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-core ---
2017-04-06T01:30:39.385 [INFO] 
2017-04-06T01:30:39.386 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-core ---
2017-04-06T01:30:39.523 [INFO] Building jar: 

2017-04-06T01:30:39.653 [INFO] 
2017-04-06T01:30:39.654 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-core ---
2017-04-06T01:30:39.657 [INFO] Excluding 
org.apache.beam:beam-sdks-common-runner-api:jar:0.7.0-SNAPSHOT from the shaded 
jar.
2017-04-06T01:30:39.657 [INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:30:39.657 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:30:39.657 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-04-06T01:30:39.657 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2017-04-06T01:30:39.657 [INFO] Excluding 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #2215

[Apache Jenkins Server]

See 


Changes:

[dhalperi] DataflowPipelineJob: handle concurrent cancel and finish

--
[...truncated 682.39 KB...]
2017-04-06T01:31:11.277 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-core ---
2017-04-06T01:31:11.378 [INFO] 
2017-04-06T01:31:11.378 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-core ---
2017-04-06T01:31:11.512 [INFO] Building jar: 

2017-04-06T01:31:11.632 [INFO] 
2017-04-06T01:31:11.632 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-core ---
2017-04-06T01:31:11.635 [INFO] Excluding 
org.apache.beam:beam-sdks-common-runner-api:jar:0.7.0-SNAPSHOT from the shaded 
jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 
from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from 
the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 
from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding com.google.code.gson:gson:jar:2.7 from 
the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.auth:google-auth-library-credentials:jar:0.6.0 from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.auth:google-auth-library-oauth2-http:jar:0.6.0 from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.netty:netty-common:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0 from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0 from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0 from the 
shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 from the shaded 
jar.
2017-04-06T01:31:11.635 [INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
2017-04-06T01:31:11.635 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3192

[Apache Jenkins Server]

See 


Changes:

[dhalperi] DataflowPipelineJob: handle concurrent cancel and finish

--
[...truncated 761.22 KB...]
2017-04-06T01:29:08.540 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-runners-parent ---
2017-04-06T01:29:08.545 [INFO] Replacing original artifact with shaded artifact.
2017-04-06T01:29:08.633 [INFO] 
2017-04-06T01:29:08.633 [INFO] --- maven-javadoc-plugin:2.10.4:jar 
(attach-javadocs) @ beam-runners-parent ---
2017-04-06T01:29:08.637 [INFO] Not executing Javadoc as the project is not a 
Java classpath-capable package
2017-04-06T01:29:08.728 [INFO] 
2017-04-06T01:29:08.728 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-runners-parent ---
2017-04-06T01:29:08.816 [INFO] 
2017-04-06T01:29:08.816 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-runners-parent ---
2017-04-06T01:29:08.904 [INFO] 
2017-04-06T01:29:08.904 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-runners-parent ---
2017-04-06T01:29:08.905 [INFO] Skipping pom project
2017-04-06T01:29:08.992 [INFO] 
2017-04-06T01:29:08.992 [INFO] --- apache-rat-plugin:0.12:check (default) @ 
beam-runners-parent ---
2017-04-06T01:29:08.997 [INFO] Enabled default license matchers.
2017-04-06T01:29:08.997 [INFO] Will parse SCM ignores for exclusions...
2017-04-06T01:29:08.997 [INFO] Finished adding exclusions from SCM ignore files.
2017-04-06T01:29:08.997 [INFO] 61 implicit excludes (use -debug for more 
details).
2017-04-06T01:29:08.997 [INFO] Exclude: **/target/**/*
2017-04-06T01:29:08.997 [INFO] Exclude: **/dependency-reduced-pom.xml
2017-04-06T01:29:08.997 [INFO] Exclude: **/hs_err_pid*.log
2017-04-06T01:29:08.997 [INFO] Exclude: .github/**/*
2017-04-06T01:29:08.997 [INFO] Exclude: **/*.iml
2017-04-06T01:29:08.997 [INFO] Exclude: **/.idea/**/*
2017-04-06T01:29:08.997 [INFO] Exclude: **/*.egg-info/**/*
2017-04-06T01:29:08.997 [INFO] Exclude: **/package-list
2017-04-06T01:29:08.997 [INFO] Exclude: **/user.avsc
2017-04-06T01:29:08.997 [INFO] Exclude: **/test/resources/**/*.txt
2017-04-06T01:29:08.997 [INFO] Exclude: **/test/**/.placeholder
2017-04-06T01:29:08.997 [INFO] Exclude: .repository/**/*
2017-04-06T01:29:08.997 [INFO] Exclude: **/nose-*.egg/**/*
2017-04-06T01:29:08.997 [INFO] Exclude: **/.eggs/**/*
2017-04-06T01:29:08.997 [INFO] Exclude: **/.tox/**/*
2017-04-06T01:29:08.997 [INFO] Exclude: **/.checkstyle
2017-04-06T01:29:08.997 [INFO] Exclude: **/.classpath
2017-04-06T01:29:08.997 [INFO] Exclude: **/.factorypath
2017-04-06T01:29:08.997 [INFO] Exclude: **/.project
2017-04-06T01:29:08.997 [INFO] Exclude: **/.settings/**/*
2017-04-06T01:29:09.012 [INFO] 601 resources included (use -debug for more 
details)
2017-04-06T01:29:09.236 [INFO] Rat check: Summary over all files. Unapproved: 
0, unknown: 0, generated: 0, approved: 601 licenses.
2017-04-06T01:29:09.325 [INFO] 
2017-04-06T01:29:09.325 [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ beam-runners-parent ---
2017-04-06T01:29:09.326 [INFO] Installing 

 to 

[JENKINS] Archiving disabled
2017-04-06T01:29:09.899 [INFO]  
   
2017-04-06T01:29:09.899 [INFO] 

2017-04-06T01:29:09.899 [INFO] Building Apache Beam :: Runners :: Core Java 
Construction 0.7.0-SNAPSHOT
2017-04-06T01:29:09.899 [INFO] 

2017-04-06T01:29:09.914 [INFO] 
2017-04-06T01:29:09.914 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-core-construction-java ---
2017-04-06T01:29:09.916 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/], excludes = [])
2017-04-06T01:29:09.966 [INFO] 
2017-04-06T01:29:09.967 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-core-construction-java ---
2017-04-06T01:29:10.378 [INFO] 
2017-04-06T01:29:10.378 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-core-construction-java ---
2017-04-06T01:29:11.039 [INFO] 
2017-04-06T01:29:11.039 [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ beam-runners-core-construction-java ---
2017-04-06T01:29:11.040 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-04-06T01:29:11.040 [INFO] skip non existing resourceDirectory 

[GitHub] beam pull request #2449: Fix Compile Break in PTransformMatchersTest

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2449

Fix Compile Break in PTransformMatchersTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam fix_view_testing_break

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2449.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2449


commit bdf93f23e6eec2b3b384428223d80d09a5245738
Author: Thomas Groh 
Date:   2017-04-06T01:27:46Z

Fix Compile Break in PTransformMatchersTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #995

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove unused import in ViewFn

--
[...truncated 676.45 KB...]
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.UploadIdResponseInterceptorTest
Running org.apache.beam.sdk.util.NumberedShardedFileTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.03 sec - in 
org.apache.beam.sdk.util.NumberedShardedFileTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.PTupleTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PTupleTest
Running org.apache.beam.sdk.util.UnownedOutputStreamTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.UnownedOutputStreamTest
Running org.apache.beam.sdk.util.gcsfs.GcsPathTest
Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.gcsfs.GcsPathTest
Running org.apache.beam.sdk.util.CombineFnUtilTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.CombineFnUtilTest
Running org.apache.beam.sdk.util.ReshuffleTriggerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReshuffleTriggerTest
Running org.apache.beam.sdk.util.BucketingFunctionTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.BucketingFunctionTest
Running org.apache.beam.sdk.util.NameUtilsTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.NameUtilsTest
Running org.apache.beam.sdk.util.UserCodeExceptionTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.UserCodeExceptionTest
Running org.apache.beam.sdk.util.SerializerTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.SerializerTest
Running org.apache.beam.sdk.util.MovingFunctionTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.MovingFunctionTest
Running org.apache.beam.sdk.util.AttemptBoundedExponentialBackOffTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.AttemptBoundedExponentialBackOffTest
Running org.apache.beam.sdk.util.IOChannelUtilsTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.IOChannelUtilsTest

Results :

Tests run: 2673, Failures: 0, Errors: 0, Skipped: 3

[JENKINS] Recording test results
2017-04-06T01:24:49.372 [INFO] 
2017-04-06T01:24:49.373 [INFO] --- jacoco-maven-plugin:0.7.8:report (report) @ 
beam-sdks-java-core ---
2017-04-06T01:24:49.445 [INFO] Loading execution data file 

2017-04-06T01:24:49.765 [INFO] Analyzed bundle 'Apache Beam :: SDKs :: Java :: 
Core' with 1194 classes
2017-04-06T01:24:51.211 [INFO] 
2017-04-06T01:24:51.211 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-core ---
2017-04-06T01:24:51.350 [INFO] 
2017-04-06T01:24:51.351 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-core ---
2017-04-06T01:24:51.498 [INFO] Building jar: 

2017-04-06T01:24:51.651 [INFO] 
2017-04-06T01:24:51.651 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-core ---
2017-04-06T01:24:51.751 [INFO] 
2017-04-06T01:24:51.751 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-core ---
2017-04-06T01:24:51.884 [INFO] Building jar: 

2017-04-06T01:24:52.009 [INFO] 
2017-04-06T01:24:52.009 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-core ---
2017-04-06T01:24:52.013 [INFO] Excluding 
org.apache.beam:beam-sdks-common-runner-api:jar:0.7.0-SNAPSHOT from the shaded 
jar.
2017-04-06T01:24:52.013 [INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:24:52.013 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:24:52.013 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-04-06T01:24:52.013 [INFO] 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #2214

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove unused import in ViewFn

--
[...truncated 680.45 KB...]
2017-04-06T01:24:18.584 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-core ---
2017-04-06T01:24:18.685 [INFO] 
2017-04-06T01:24:18.685 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-core ---
2017-04-06T01:24:18.810 [INFO] Building jar: 

2017-04-06T01:24:18.931 [INFO] 
2017-04-06T01:24:18.931 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-core ---
2017-04-06T01:24:18.934 [INFO] Excluding 
org.apache.beam:beam-sdks-common-runner-api:jar:0.7.0-SNAPSHOT from the shaded 
jar.
2017-04-06T01:24:18.934 [INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:24:18.934 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:24:18.934 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-04-06T01:24:18.934 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the 
shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 
from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from 
the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 
from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding com.google.code.gson:gson:jar:2.7 from 
the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.auth:google-auth-library-credentials:jar:0.6.0 from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.auth:google-auth-library-oauth2-http:jar:0.6.0 from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.netty:netty-common:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final 
from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0 from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0 from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0 from the 
shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 from the shaded 
jar.
2017-04-06T01:24:18.935 [INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
2017-04-06T01:24:18.935 [INFO] Excluding 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3191

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove unused import in ViewFn

--
[...truncated 761.42 KB...]
2017-04-06T01:24:18.812 [INFO] 
2017-04-06T01:24:18.812 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-runners-parent ---
2017-04-06T01:24:18.815 [INFO] Replacing original artifact with shaded artifact.
2017-04-06T01:24:18.865 [INFO] 
2017-04-06T01:24:18.865 [INFO] --- maven-javadoc-plugin:2.10.4:jar 
(attach-javadocs) @ beam-runners-parent ---
2017-04-06T01:24:18.868 [INFO] Not executing Javadoc as the project is not a 
Java classpath-capable package
2017-04-06T01:24:18.921 [INFO] 
2017-04-06T01:24:18.921 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-runners-parent ---
2017-04-06T01:24:18.972 [INFO] 
2017-04-06T01:24:18.972 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-runners-parent ---
2017-04-06T01:24:19.024 [INFO] 
2017-04-06T01:24:19.024 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-runners-parent ---
2017-04-06T01:24:19.025 [INFO] Skipping pom project
2017-04-06T01:24:19.075 [INFO] 
2017-04-06T01:24:19.075 [INFO] --- apache-rat-plugin:0.12:check (default) @ 
beam-runners-parent ---
2017-04-06T01:24:19.080 [INFO] Enabled default license matchers.
2017-04-06T01:24:19.080 [INFO] Will parse SCM ignores for exclusions...
2017-04-06T01:24:19.080 [INFO] Finished adding exclusions from SCM ignore files.
2017-04-06T01:24:19.080 [INFO] 61 implicit excludes (use -debug for more 
details).
2017-04-06T01:24:19.080 [INFO] Exclude: **/target/**/*
2017-04-06T01:24:19.080 [INFO] Exclude: **/dependency-reduced-pom.xml
2017-04-06T01:24:19.080 [INFO] Exclude: **/hs_err_pid*.log
2017-04-06T01:24:19.080 [INFO] Exclude: .github/**/*
2017-04-06T01:24:19.080 [INFO] Exclude: **/*.iml
2017-04-06T01:24:19.080 [INFO] Exclude: **/.idea/**/*
2017-04-06T01:24:19.080 [INFO] Exclude: **/*.egg-info/**/*
2017-04-06T01:24:19.080 [INFO] Exclude: **/package-list
2017-04-06T01:24:19.080 [INFO] Exclude: **/user.avsc
2017-04-06T01:24:19.080 [INFO] Exclude: **/test/resources/**/*.txt
2017-04-06T01:24:19.080 [INFO] Exclude: **/test/**/.placeholder
2017-04-06T01:24:19.080 [INFO] Exclude: .repository/**/*
2017-04-06T01:24:19.080 [INFO] Exclude: **/nose-*.egg/**/*
2017-04-06T01:24:19.080 [INFO] Exclude: **/.eggs/**/*
2017-04-06T01:24:19.081 [INFO] Exclude: **/.tox/**/*
2017-04-06T01:24:19.081 [INFO] Exclude: **/.checkstyle
2017-04-06T01:24:19.081 [INFO] Exclude: **/.classpath
2017-04-06T01:24:19.081 [INFO] Exclude: **/.factorypath
2017-04-06T01:24:19.081 [INFO] Exclude: **/.project
2017-04-06T01:24:19.081 [INFO] Exclude: **/.settings/**/*
2017-04-06T01:24:19.094 [INFO] 601 resources included (use -debug for more 
details)
2017-04-06T01:24:19.321 [INFO] Rat check: Summary over all files. Unapproved: 
0, unknown: 0, generated: 0, approved: 601 licenses.
2017-04-06T01:24:19.371 [INFO] 
2017-04-06T01:24:19.371 [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ beam-runners-parent ---
2017-04-06T01:24:19.373 [INFO] Installing 

 to 

[JENKINS] Archiving disabled
2017-04-06T01:24:19.985 [INFO]  
   
2017-04-06T01:24:19.986 [INFO] 

2017-04-06T01:24:19.986 [INFO] Building Apache Beam :: Runners :: Core Java 
Construction 0.7.0-SNAPSHOT
2017-04-06T01:24:19.986 [INFO] 

2017-04-06T01:24:20.001 [INFO] 
2017-04-06T01:24:20.001 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-core-construction-java ---
2017-04-06T01:24:20.002 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/], excludes = [])
2017-04-06T01:24:20.052 [INFO] 
2017-04-06T01:24:20.052 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-core-construction-java ---
2017-04-06T01:24:20.506 [INFO] 
2017-04-06T01:24:20.506 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-core-construction-java ---
2017-04-06T01:24:20.953 [INFO] 
2017-04-06T01:24:20.953 [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ beam-runners-core-construction-java ---
2017-04-06T01:24:20.954 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-04-06T01:24:20.954 [INFO] skip non existing resourceDirectory 

[jira] [Resolved] (BEAM-1880) DataflowPipelineJob.cancel() should behave gracefully if called concurrent with job termination

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1880.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> DataflowPipelineJob.cancel() should behave gracefully if called concurrent 
> with job termination
> ---
>
> Key: BEAM-1880
> URL: https://issues.apache.org/jira/browse/BEAM-1880
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> In Dataflow runner, we use job state to avoid canceling finished jobs, or to 
> notice when we tried to cancel a finished job. However, the job state is not 
> authoritative about whether the job is done, and cancel will throw an error 
> if they happen to be near each other in time. This causes flaky tests and bad 
> developer experience.
> Future changes to Dataflow service will make this much more likely to happen.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1880) DataflowPipelineJob.cancel() should behave gracefully if called concurrent with job termination

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958142#comment-15958142
 ] 

ASF GitHub Bot commented on BEAM-1880:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2428


> DataflowPipelineJob.cancel() should behave gracefully if called concurrent 
> with job termination
> ---
>
> Key: BEAM-1880
> URL: https://issues.apache.org/jira/browse/BEAM-1880
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> In Dataflow runner, we use job state to avoid canceling finished jobs, or to 
> notice when we tried to cancel a finished job. However, the job state is not 
> authoritative about whether the job is done, and cancel will throw an error 
> if they happen to be near each other in time. This causes flaky tests and bad 
> developer experience.
> Future changes to Dataflow service will make this much more likely to happen.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2428: [BEAM-1880] DataflowPipelineJob: handle concurrent ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2428


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2428

[dhalperi]

This closes #2428


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/20788953
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/20788953
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/20788953

Branch: refs/heads/master
Commit: 20788953f9cc51dc20aec44423dbdbdd53cc8741
Parents: 512ad13 09580a7
Author: Dan Halperin 
Authored: Wed Apr 5 18:20:36 2017 -0700
Committer: Dan Halperin 
Committed: Wed Apr 5 18:20:36 2017 -0700

--
 .../runners/dataflow/DataflowPipelineJob.java   | 14 +++-
 .../dataflow/DataflowPipelineJobTest.java   | 37 +++-
 2 files changed, 41 insertions(+), 10 deletions(-)
--

[1/2] beam git commit: DataflowPipelineJob: handle concurrent cancel and finish

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 512ad1355 -> 20788953f


DataflowPipelineJob: handle concurrent cancel and finish

This makes job.cancel() not throw an exception if cancel() is called while job 
is finished.
(Note that state.isTerminal() is not guaranteed to be up to date.)


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/09580a73
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/09580a73
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/09580a73

Branch: refs/heads/master
Commit: 09580a731e7b67564315a56e43658300011512eb
Parents: 512ad13
Author: Dan Halperin 
Authored: Tue Apr 4 15:33:21 2017 -0700
Committer: Dan Halperin 
Committed: Wed Apr 5 18:20:29 2017 -0700

--
 .../runners/dataflow/DataflowPipelineJob.java   | 14 +++-
 .../dataflow/DataflowPipelineJobTest.java   | 37 +++-
 2 files changed, 41 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/09580a73/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
index 5ad6f9f..7cb0f0e 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
@@ -369,7 +369,19 @@ public class DataflowPipelineJob implements PipelineResult 
{
 } catch (IOException e) {
   State state = getState();
   if (state.isTerminal()) {
-LOG.warn("Job is already terminated. State is {}", state);
+LOG.warn("Cancel failed because job is already terminated. State 
is {}", state);
+return state;
+  } else if (e.getMessage().contains("has terminated")) {
+// This handles the case where the getState() call above returns 
RUNNING but the cancel
+// was rejected because the job is in fact done. Hopefully, 
someday we can delete this
+// code if there is better consistency between the State and 
whether Cancel succeeds.
+//
+// Example message:
+//Workflow modification failed. Causes: (7603adc9e9bff51e): 
Cannot perform
+//operation 'cancel' on Job: 
2017-04-01_22_50_59-9269855660514862348. Job has
+//terminated in state SUCCESS: Workflow job: 
2017-04-01_22_50_59-9269855660514862348
+//succeeded.
+LOG.warn("Cancel failed because job is already terminated.", e);
 return state;
   } else {
 String errorMsg = String.format(

http://git-wip-us.apache.org/repos/asf/beam/blob/09580a73/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
index 108badd..e3d2e4e 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
@@ -618,7 +618,6 @@ public class DataflowPipelineJobTest {
 
 thrown.expect(IllegalArgumentException.class);
 thrown.expectMessage("not used in this pipeline");
-
 job.getAggregatorValues(aggregator);
   }
 
@@ -656,7 +655,6 @@ public class DataflowPipelineJobTest {
 thrown.expectCause(is(cause));
 thrown.expectMessage(aggregator.toString());
 thrown.expectMessage("when retrieving Aggregator values for");
-
 job.getAggregatorValues(aggregator);
   }
 
@@ -750,7 +748,7 @@ public class DataflowPipelineJobTest {
 Dataflow.Projects.Locations.Jobs.Update.class);
 when(mockJobs.update(eq(PROJECT_ID), eq(REGION_ID), eq(JOB_ID), 
any(Job.class)))
 .thenReturn(update);
-when(update.execute()).thenThrow(new IOException());
+when(update.execute()).thenThrow(new IOException("Some random 
IOException"));
 
 DataflowPipelineJob job = new DataflowPipelineJob(JOB_ID, options, null);
 
@@ -758,13 +756,34 @@ public class DataflowPipelineJobTest {
 thrown.expectMessage("Failed to cancel job in state RUNNING, "
 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #2213

[Apache Jenkins Server]

See 


--
[...truncated 617.17 KB...]
2017-04-06T01:17:44.044 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/woodstox/stax2-api/3.1.4/stax2-api-3.1.4.jar
 (159 KB at 299.9 KB/sec)
2017-04-06T01:17:44.044 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.2.1/snappy-java-1.1.2.1.jar
 (689 KB at 1306.1 KB/sec)
2017-04-06T01:17:44.044 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/woodstox/woodstox-core-asl/4.4.1/woodstox-core-asl-4.4.1.jar
2017-04-06T01:17:44.045 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auto/service/auto-service/1.0-rc2/auto-service-1.0-rc2.jar
2017-04-06T01:17:44.050 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar
 (469 KB at 879.2 KB/sec)
2017-04-06T01:17:44.050 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auto/auto-common/0.3/auto-common-0.3.jar
2017-04-06T01:17:44.061 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/joda-time/joda-time/2.4/joda-time-2.4.jar 
(573 KB at 1052.3 KB/sec)
2017-04-06T01:17:44.061 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auto/value/auto-value/1.3/auto-value-1.3.jar
2017-04-06T01:17:44.073 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auto/service/auto-service/1.0-rc2/auto-service-1.0-rc2.jar
 (11 KB at 18.2 KB/sec)
2017-04-06T01:17:44.073 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.7.2/jackson-dataformat-yaml-2.7.2.jar
2017-04-06T01:17:44.077 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auto/auto-common/0.3/auto-common-0.3.jar
 (39 KB at 68.0 KB/sec)
2017-04-06T01:17:44.078 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.15/snakeyaml-1.15.jar
2017-04-06T01:17:44.106 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/woodstox/woodstox-core-asl/4.4.1/woodstox-core-asl-4.4.1.jar
 (475 KB at 805.8 KB/sec)
2017-04-06T01:17:44.106 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-all/1.3/hamcrest-all-1.3.jar
2017-04-06T01:17:44.110 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.jar
 (2706 KB at 4555.2 KB/sec)
2017-04-06T01:17:44.110 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar
2017-04-06T01:17:44.122 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.7.2/jackson-dataformat-yaml-2.7.2.jar
 (314 KB at 518.1 KB/sec)
2017-04-06T01:17:44.122 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/mockito/mockito-all/1.9.5/mockito-all-1.9.5.jar
2017-04-06T01:17:44.129 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.15/snakeyaml-1.15.jar 
(263 KB at 429.7 KB/sec)
2017-04-06T01:17:44.130 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/dataflow/google-cloud-dataflow-java-proto-library-all/0.5.160304/google-cloud-dataflow-java-proto-library-all-0.5.160304.jar
2017-04-06T01:17:44.140 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar
 (8 KB at 12.2 KB/sec)
2017-04-06T01:17:44.140 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar
2017-04-06T01:17:44.157 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-all/1.3/hamcrest-all-1.3.jar
 (300 KB at 467.8 KB/sec)
2017-04-06T01:17:44.157 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar
2017-04-06T01:17:44.159 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/dataflow/google-cloud-dataflow-java-proto-library-all/0.5.160304/google-cloud-dataflow-java-proto-library-all-0.5.160304.jar
 (53 KB at 82.0 KB/sec)
2017-04-06T01:17:44.159 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/4.0/asm-4.0.jar
2017-04-06T01:17:44.189 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar
 (65 KB at 95.3 KB/sec)
2017-04-06T01:17:44.190 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/4.0/asm-4.0.jar (45 KB at 
66.8 KB/sec)
2017-04-06T01:17:44.190 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/objenesis/objenesis/1.2/objenesis-1.2.jar
2017-04-06T01:17:44.190 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar
2017-04-06T01:17:44.190 [INFO] Downloaded: 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #994

[Apache Jenkins Server]

See 


--
[...truncated 606.42 KB...]
2017-04-06T01:17:12.199 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client/1.22.0/google-oauth-client-1.22.0.jar
2017-04-06T01:17:12.206 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/http-client/google-http-client/1.22.0/google-http-client-1.22.0.jar
 (349 KB at 887.2 KB/sec)
2017-04-06T01:17:12.207 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigdataoss/gcsio/1.4.5/gcsio-1.4.5.jar
2017-04-06T01:17:12.209 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/http-client/google-http-client-jackson2/1.22.0/google-http-client-jackson2-1.22.0.jar
 (7 KB at 16.6 KB/sec)
2017-04-06T01:17:12.209 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-java6/1.22.0/google-api-client-java6-1.22.0.jar
2017-04-06T01:17:12.218 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/http-client/google-http-client-protobuf/1.22.0/google-http-client-protobuf-1.22.0.jar
 (6 KB at 12.4 KB/sec)
2017-04-06T01:17:12.218 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-jackson2/1.22.0/google-api-client-jackson2-1.22.0.jar
2017-04-06T01:17:12.220 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.0.1/httpclient-4.0.1.jar
 (285 KB at 698.3 KB/sec)
2017-04-06T01:17:12.220 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client-java6/1.22.0/google-oauth-client-java6-1.22.0.jar
2017-04-06T01:17:12.235 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client/1.22.0/google-oauth-client-1.22.0.jar
 (60 KB at 142.0 KB/sec)
2017-04-06T01:17:12.235 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigdataoss/util/1.4.5/util-1.4.5.jar
2017-04-06T01:17:12.237 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-java6/1.22.0/google-api-client-java6-1.22.0.jar
 (3 KB at 5.4 KB/sec)
2017-04-06T01:17:12.237 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-testlib/20.0/guava-testlib-20.0.jar
2017-04-06T01:17:12.248 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client-java6/1.22.0/google-oauth-client-java6-1.22.0.jar
 (11 KB at 24.4 KB/sec)
2017-04-06T01:17:12.248 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.2/jackson-core-2.7.2.jar
2017-04-06T01:17:12.251 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-jackson2/1.22.0/google-api-client-jackson2-1.22.0.jar
 (3 KB at 6.0 KB/sec)
2017-04-06T01:17:12.251 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.7.2/jackson-annotations-2.7.2.jar
2017-04-06T01:17:12.252 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigdataoss/gcsio/1.4.5/gcsio-1.4.5.jar
 (152 KB at 345.1 KB/sec)
2017-04-06T01:17:12.252 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.7.2/jackson-databind-2.7.2.jar
2017-04-06T01:17:12.289 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.7.2/jackson-annotations-2.7.2.jar
 (50 KB at 104.7 KB/sec)
2017-04-06T01:17:12.289 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.14/slf4j-api-1.7.14.jar
2017-04-06T01:17:12.296 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigdataoss/util/1.4.5/util-1.4.5.jar
 (59 KB at 120.6 KB/sec)
2017-04-06T01:17:12.296 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.jar
2017-04-06T01:17:12.310 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.2/jackson-core-2.7.2.jar
 (247 KB at 497.3 KB/sec)
2017-04-06T01:17:12.310 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro/1.8.1/avro-1.8.1.jar
2017-04-06T01:17:12.327 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.14/slf4j-api-1.7.14.jar
 (33 KB at 63.9 KB/sec)
2017-04-06T01:17:12.327 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar
2017-04-06T01:17:12.384 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-testlib/20.0/guava-testlib-20.0.jar
 (749 KB at 1312.8 KB/sec)
2017-04-06T01:17:12.384 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar

[GitHub] beam pull request #2448: Remove unused import in ViewFn

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2448


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Remove unused import in ViewFn

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 9704f2257 -> 512ad1355


Remove unused import in ViewFn


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1e4cd265
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1e4cd265
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1e4cd265

Branch: refs/heads/master
Commit: 1e4cd265e976e9d7ec9b8af372352468529f3224
Parents: 9704f22
Author: Thomas Groh 
Authored: Wed Apr 5 18:13:33 2017 -0700
Committer: Thomas Groh 
Committed: Wed Apr 5 18:13:33 2017 -0700

--
 .../core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java   | 1 -
 1 file changed, 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1e4cd265/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java
index ff53d1c..cdfcb88 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java
@@ -18,7 +18,6 @@
 package org.apache.beam.sdk.transforms;
 
 import java.io.Serializable;
-import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;
 

[2/2] beam git commit: This closes #2448

[tgroh]

This closes #2448


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/512ad135
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/512ad135
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/512ad135

Branch: refs/heads/master
Commit: 512ad1355519ef41c659386d7a500bbece556c96
Parents: 9704f22 1e4cd26
Author: Thomas Groh 
Authored: Wed Apr 5 18:15:29 2017 -0700
Committer: Thomas Groh 
Committed: Wed Apr 5 18:15:29 2017 -0700

--
 .../core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java   | 1 -
 1 file changed, 1 deletion(-)
--

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3189

[Apache Jenkins Server]

See 


Changes:

[tgroh] Use WinodwMappingFn where possible

[tgroh] Add Initial Materializations

--
[...truncated 664.02 KB...]
2017-04-06T01:13:15.403 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.1.2/doxia-decoration-model-1.1.2.jar
2017-04-06T01:13:15.408 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.1.2/doxia-core-1.1.2.jar
 (155 KB at 2493.4 KB/sec)
2017-04-06T01:13:15.409 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.1.2/doxia-module-xhtml-1.1.2.jar
2017-04-06T01:13:15.410 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.1.2/doxia-site-renderer-1.1.2.jar
 (50 KB at 782.0 KB/sec)
2017-04-06T01:13:15.410 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.1.2/doxia-module-fml-1.1.2.jar
2017-04-06T01:13:15.411 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar
 (298 KB at 4582.3 KB/sec)
2017-04-06T01:13:15.411 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2/commons-collections-3.2.jar
2017-04-06T01:13:15.435 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.1.2/doxia-decoration-model-1.1.2.jar
 (52 KB at 587.1 KB/sec)
2017-04-06T01:13:15.435 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.agent/0.7.8/org.jacoco.agent-0.7.8-runtime.jar
2017-04-06T01:13:15.436 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.1.2/doxia-module-xhtml-1.1.2.jar
 (15 KB at 166.5 KB/sec)
2017-04-06T01:13:15.436 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.core/0.7.8/org.jacoco.core-0.7.8.jar
2017-04-06T01:13:15.443 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.1.2/doxia-module-fml-1.1.2.jar
 (37 KB at 384.4 KB/sec)
2017-04-06T01:13:15.443 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-debug-all/5.1/asm-debug-all-5.1.jar
2017-04-06T01:13:15.480 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.core/0.7.8/org.jacoco.core-0.7.8.jar
 (122 KB at 920.4 KB/sec)
2017-04-06T01:13:15.480 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.8/org.jacoco.report-0.7.8.jar
2017-04-06T01:13:15.501 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.agent/0.7.8/org.jacoco.agent-0.7.8-runtime.jar
 (252 KB at 1642.5 KB/sec)
2017-04-06T01:13:15.502 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.8.1/xercesImpl-2.8.1.jar
 (1185 KB at 7593.2 KB/sec)
2017-04-06T01:13:15.502 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2/commons-collections-3.2.jar
 (558 KB at 3622.5 KB/sec)
2017-04-06T01:13:15.518 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-debug-all/5.1/asm-debug-all-5.1.jar
 (379 KB at 2227.4 KB/sec)
2017-04-06T01:13:15.518 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.8/org.jacoco.report-0.7.8.jar
 (125 KB at 731.8 KB/sec)
2017-04-06T01:13:15.619 [INFO] argLine set to 
-javaagent:
2017-04-06T01:13:15.708 [INFO] 
2017-04-06T01:13:15.709 [INFO] --- avro-maven-plugin:1.8.1:schema (schemas) @ 
beam-sdks-java-core ---
2017-04-06T01:13:15.712 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-compiler/1.8.1/avro-compiler-1.8.1.pom
2017-04-06T01:13:15.739 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-compiler/1.8.1/avro-compiler-1.8.1.pom
 (5 KB at 178.4 KB/sec)
2017-04-06T01:13:15.742 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.pom
2017-04-06T01:13:15.768 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.pom
 (3 KB at 101.0 KB/sec)
2017-04-06T01:13:15.769 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.7/slf4j-parent-1.7.7.pom
2017-04-06T01:13:15.796 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.7/slf4j-parent-1.7.7.pom
 (12 KB at 430.7 KB/sec)
2017-04-06T01:13:15.799 [INFO] 

[GitHub] beam pull request #2448: Remove unused import in ViewFn

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2448

Remove unused import in ViewFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
I got overeager to merge

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam fix_viewfn_style

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2448.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2448


commit 1e4cd265e976e9d7ec9b8af372352468529f3224
Author: Thomas Groh 
Date:   2017-04-06T01:13:33Z

Remove unused import in ViewFn




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #2212

[Apache Jenkins Server]

See 


Changes:

[tgroh] Use WinodwMappingFn where possible

[tgroh] Add Initial Materializations

--
[...truncated 595.20 KB...]
2017-04-06T01:12:47.370 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/woodstox/stax2-api/3.1.4/stax2-api-3.1.4.jar
2017-04-06T01:12:47.382 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.2.1/snappy-java-1.1.2.1.jar
 (689 KB at 1407.5 KB/sec)
2017-04-06T01:12:47.382 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/woodstox/woodstox-core-asl/4.4.1/woodstox-core-asl-4.4.1.jar
2017-04-06T01:12:47.384 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar
 (469 KB at 954.4 KB/sec)
2017-04-06T01:12:47.384 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auto/service/auto-service/1.0-rc2/auto-service-1.0-rc2.jar
2017-04-06T01:12:47.413 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/woodstox/stax2-api/3.1.4/stax2-api-3.1.4.jar
 (159 KB at 304.0 KB/sec)
2017-04-06T01:12:47.413 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auto/auto-common/0.3/auto-common-0.3.jar
2017-04-06T01:12:47.415 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auto/service/auto-service/1.0-rc2/auto-service-1.0-rc2.jar
 (11 KB at 19.4 KB/sec)
2017-04-06T01:12:47.415 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auto/value/auto-value/1.3/auto-value-1.3.jar
2017-04-06T01:12:47.428 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/joda-time/joda-time/2.4/joda-time-2.4.jar 
(573 KB at 1070.0 KB/sec)
2017-04-06T01:12:47.428 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.7.2/jackson-dataformat-yaml-2.7.2.jar
2017-04-06T01:12:47.443 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auto/auto-common/0.3/auto-common-0.3.jar
 (39 KB at 69.2 KB/sec)
2017-04-06T01:12:47.443 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.15/snakeyaml-1.15.jar
2017-04-06T01:12:47.444 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.jar
 (2706 KB at 4910.7 KB/sec)
2017-04-06T01:12:47.444 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-all/1.3/hamcrest-all-1.3.jar
2017-04-06T01:12:47.448 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/woodstox/woodstox-core-asl/4.4.1/woodstox-core-asl-4.4.1.jar
 (475 KB at 855.2 KB/sec)
2017-04-06T01:12:47.448 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar
2017-04-06T01:12:47.475 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar
 (8 KB at 13.1 KB/sec)
2017-04-06T01:12:47.475 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/mockito/mockito-all/1.9.5/mockito-all-1.9.5.jar
2017-04-06T01:12:47.479 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.7.2/jackson-dataformat-yaml-2.7.2.jar
 (314 KB at 534.9 KB/sec)
2017-04-06T01:12:47.479 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/dataflow/google-cloud-dataflow-java-proto-library-all/0.5.160304/google-cloud-dataflow-java-proto-library-all-0.5.160304.jar
2017-04-06T01:12:47.488 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.15/snakeyaml-1.15.jar 
(263 KB at 442.0 KB/sec)
2017-04-06T01:12:47.488 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar
2017-04-06T01:12:47.490 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-all/1.3/hamcrest-all-1.3.jar
 (300 KB at 501.5 KB/sec)
2017-04-06T01:12:47.490 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar
2017-04-06T01:12:47.511 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/dataflow/google-cloud-dataflow-java-proto-library-all/0.5.160304/google-cloud-dataflow-java-proto-library-all-0.5.160304.jar
 (53 KB at 85.2 KB/sec)
2017-04-06T01:12:47.511 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/4.0/asm-4.0.jar
2017-04-06T01:12:47.527 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar
 (65 KB at 101.1 KB/sec)
2017-04-06T01:12:47.527 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar
2017-04-06T01:12:47.539 [INFO] Downloaded: 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #993

[Apache Jenkins Server]

See 


Changes:

[tgroh] Use WinodwMappingFn where possible

[tgroh] Add Initial Materializations

--
[...truncated 588.55 KB...]
2017-04-06T01:12:16.132 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/http-client/google-http-client-jackson/1.22.0/google-http-client-jackson-1.22.0.jar
 (6 KB at 18.3 KB/sec)
2017-04-06T01:12:16.132 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client/1.22.0/google-oauth-client-1.22.0.jar
2017-04-06T01:12:16.150 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/http-client/google-http-client-jackson2/1.22.0/google-http-client-jackson2-1.22.0.jar
 (7 KB at 19.3 KB/sec)
2017-04-06T01:12:16.150 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigdataoss/gcsio/1.4.5/gcsio-1.4.5.jar
2017-04-06T01:12:16.151 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/http-client/google-http-client-protobuf/1.22.0/google-http-client-protobuf-1.22.0.jar
 (6 KB at 14.8 KB/sec)
2017-04-06T01:12:16.151 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-java6/1.22.0/google-api-client-java6-1.22.0.jar
2017-04-06T01:12:16.163 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client/1.22.0/google-oauth-client-1.22.0.jar
 (60 KB at 169.7 KB/sec)
2017-04-06T01:12:16.163 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-jackson2/1.22.0/google-api-client-jackson2-1.22.0.jar
2017-04-06T01:12:16.164 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.0.1/httpclient-4.0.1.jar
 (285 KB at 802.9 KB/sec)
2017-04-06T01:12:16.164 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client-java6/1.22.0/google-oauth-client-java6-1.22.0.jar
2017-04-06T01:12:16.165 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/http-client/google-http-client/1.22.0/google-http-client-1.22.0.jar
 (349 KB at 982.1 KB/sec)
2017-04-06T01:12:16.165 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigdataoss/util/1.4.5/util-1.4.5.jar
2017-04-06T01:12:16.177 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-java6/1.22.0/google-api-client-java6-1.22.0.jar
 (3 KB at 6.2 KB/sec)
2017-04-06T01:12:16.177 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-testlib/20.0/guava-testlib-20.0.jar
2017-04-06T01:12:16.189 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigdataoss/gcsio/1.4.5/gcsio-1.4.5.jar
 (152 KB at 399.8 KB/sec)
2017-04-06T01:12:16.189 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-jackson2/1.22.0/google-api-client-jackson2-1.22.0.jar
 (3 KB at 6.9 KB/sec)
2017-04-06T01:12:16.190 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.2/jackson-core-2.7.2.jar
2017-04-06T01:12:16.190 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.7.2/jackson-annotations-2.7.2.jar
2017-04-06T01:12:16.192 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client-java6/1.22.0/google-oauth-client-java6-1.22.0.jar
 (11 KB at 27.8 KB/sec)
2017-04-06T01:12:16.192 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.7.2/jackson-databind-2.7.2.jar
2017-04-06T01:12:16.196 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/cloud/bigdataoss/util/1.4.5/util-1.4.5.jar
 (59 KB at 151.0 KB/sec)
2017-04-06T01:12:16.196 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.14/slf4j-api-1.7.14.jar
2017-04-06T01:12:16.223 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.7.2/jackson-annotations-2.7.2.jar
 (50 KB at 120.7 KB/sec)
2017-04-06T01:12:16.223 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.jar
2017-04-06T01:12:16.229 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.14/slf4j-api-1.7.14.jar
 (33 KB at 78.4 KB/sec)
2017-04-06T01:12:16.229 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro/1.8.1/avro-1.8.1.jar
2017-04-06T01:12:16.241 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.2/jackson-core-2.7.2.jar
 (247 KB at 573.6 KB/sec)
2017-04-06T01:12:16.241 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar
2017-04-06T01:12:16.315 

[jira] [Commented] (BEAM-1198) ViewFn: explicitly decouple runner materialization of side inputs from SDK-specific mapping

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958134#comment-15958134
 ] 

ASF GitHub Bot commented on BEAM-1198:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2440


> ViewFn: explicitly decouple runner materialization of side inputs from 
> SDK-specific mapping
> ---
>
> Key: BEAM-1198
> URL: https://issues.apache.org/jira/browse/BEAM-1198
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-fn-api, beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: Not applicable, First stable release
>
>
> For side inputs, the field {{PCollectionView.fromIterableInternal}} implies 
> an "iterable" materialization of the contents of a PCollection, which is 
> adapted to the desired user-facing type within a UDF (today the 
> PCollectionView "is" the UDF)
> In practice, runners get adequate performance by special casing just a couple 
> of types of PCollectionView in a rather cumbersome manner.
> The design to improve this is https://s.apache.org/beam-side-inputs-1-pager 
> and this makes the de facto standard approach the actual model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: Add Initial Materializations

[tgroh]

Add Initial Materializations


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9b5d66ee
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9b5d66ee
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9b5d66ee

Branch: refs/heads/master
Commit: 9b5d66eec14501769086a12347960bdf52d62921
Parents: 447fb2c
Author: Thomas Groh 
Authored: Mon Apr 3 16:25:08 2017 -0700
Committer: Thomas Groh 
Committed: Wed Apr 5 18:09:27 2017 -0700

--
 .../beam/sdk/transforms/Materialization.java| 36 +
 .../beam/sdk/transforms/Materializations.java   | 53 
 .../org/apache/beam/sdk/transforms/ViewFn.java  |  6 +++
 .../apache/beam/sdk/util/PCollectionViews.java  | 25 +
 .../sdk/testing/PCollectionViewTesting.java | 18 +++
 5 files changed, 138 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9b5d66ee/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Materialization.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Materialization.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Materialization.java
new file mode 100644
index 000..7cd6256
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Materialization.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.transforms;
+
+import org.apache.beam.sdk.runners.PipelineRunner;
+
+/**
+ * How a view should be physically materialized by a {@link PipelineRunner}.
+ *
+ * A {@link PipelineRunner} will support some set of materializations, and 
will reject
+ * {@link ViewFn ViewFns} that require materializations it does not support. 
See
+ * {@link Materializations} for known implementations.
+ */
+public interface Materialization {
+  /**
+   * Gets the URN describing this {@link Materialization}. This is a stable, 
SDK-independent URN
+   * understood by a {@link PipelineRunner}.
+   */
+  String getUrn();
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/9b5d66ee/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Materializations.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Materializations.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Materializations.java
new file mode 100644
index 000..35925fa
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Materializations.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.transforms;
+
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/**
+ * Utility methods for constructing known {@link Materialization 
materializations} for a
+ * {@link ViewFn}.
+ */
+public class Materializations {
+  /**
+   * The URN for a {@link Materialization} where the primitive view type is an 
iterable of fully
+   * specified windowed values.
+   */
+  @Experimental(Kind.CORE_RUNNERS_ONLY)
+  

[GitHub] beam pull request #2440: [BEAM-1198] Add Initial Materializations

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2440


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #2440

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 447fb2c15 -> 9704f2257


This closes #2440


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9704f225
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9704f225
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9704f225

Branch: refs/heads/master
Commit: 9704f225732e41a4b0c880abb923832e7ee6c9e4
Parents: 447fb2c 9b5d66e
Author: Thomas Groh 
Authored: Wed Apr 5 18:09:27 2017 -0700
Committer: Thomas Groh 
Committed: Wed Apr 5 18:09:27 2017 -0700

--
 .../beam/sdk/transforms/Materialization.java| 36 +
 .../beam/sdk/transforms/Materializations.java   | 53 
 .../org/apache/beam/sdk/transforms/ViewFn.java  |  6 +++
 .../apache/beam/sdk/util/PCollectionViews.java  | 25 +
 .../sdk/testing/PCollectionViewTesting.java | 18 +++
 5 files changed, 138 insertions(+)
--

[jira] [Commented] (BEAM-846) Decouple side input window mapping from WindowFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958132#comment-15958132
 ] 

ASF GitHub Bot commented on BEAM-846:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2436


> Decouple side input window mapping from WindowFn
> 
>
> Key: BEAM-846
> URL: https://issues.apache.org/jira/browse/BEAM-846
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Robert Bradshaw
>Assignee: Thomas Groh
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Currently the main WindowFn provides as getSideInputWindow method. Instead, 
> this mapping should be specified per-side-input (thought the default mapping 
> would remain the same). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2436: [BEAM-846] Use getWindowMappingFn where possible

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2436


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Use WinodwMappingFn where possible

[tgroh]

Use WinodwMappingFn where possible

Migrates callers away from the user of WindowingStrategyInternal,
permitting future changes to have a configurable WindowMappingFn.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9e8cf0c5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9e8cf0c5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9e8cf0c5

Branch: refs/heads/master
Commit: 9e8cf0c5ea7f47a9d7ec05272d56508962c86918
Parents: cc5f78d
Author: Thomas Groh 
Authored: Wed Apr 5 11:53:03 2017 -0700
Committer: Thomas Groh 
Committed: Wed Apr 5 18:08:03 2017 -0700

--
 ...tputAndTimeBoundedSplittableProcessElementInvoker.java |  3 +--
 .../beam/runners/core/PushbackSideInputDoFnRunner.java|  2 +-
 .../apache/beam/runners/core/ReduceFnContextFactory.java  |  3 +--
 .../org/apache/beam/runners/core/SimpleDoFnRunner.java|  2 +-
 .../org/apache/beam/runners/core/SimpleOldDoFnRunner.java |  2 +-
 .../org/apache/beam/runners/core/ReduceFnRunnerTest.java  | 10 --
 .../java/org/apache/beam/sdk/transforms/DoFnTester.java   |  3 +--
 .../org/apache/beam/sdk/util/CombineContextFactory.java   |  2 +-
 8 files changed, 11 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9e8cf0c5/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
index 5aa7605..357094c 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
@@ -226,8 +226,7 @@ public class 
OutputAndTimeBoundedSplittableProcessElementInvoker<
 public  T sideInput(PCollectionView view) {
   return sideInputReader.get(
   view,
-  view.getWindowingStrategyInternal()
-  .getWindowFn()
+  view.getWindowMappingFn()
   
.getSideInputWindow(Iterables.getOnlyElement(element.getWindows(;
 }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/9e8cf0c5/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
index 2962832..4ad20b5 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
@@ -98,7 +98,7 @@ public class PushbackSideInputDoFnRunner 
implements DoFnRunner<
 }
 for (PCollectionView view : views) {
   BoundedWindow sideInputWindow =
-  
view.getWindowingStrategyInternal().getWindowFn().getSideInputWindow(mainInputWindow);
+  view.getWindowMappingFn().getSideInputWindow(mainInputWindow);
   if (!sideInputReader.isReady(view, sideInputWindow)) {
 return false;
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/9e8cf0c5/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
index 66a6ef8..8493474 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
@@ -514,8 +514,7 @@ class ReduceFnContextFactory {
 public  T sideInput(PCollectionView view) {
   return sideInputReader.get(
   view,
-  view.getWindowingStrategyInternal()
-  .getWindowFn()
+  view.getWindowMappingFn()
   .getSideInputWindow(mainInputWindow));
 }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/9e8cf0c5/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 

[1/2] beam git commit: This closes #2436

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master cc5f78dd0 -> 447fb2c15


This closes #2436


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/447fb2c1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/447fb2c1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/447fb2c1

Branch: refs/heads/master
Commit: 447fb2c15df2bca758c0ad262571c5b9ece46231
Parents: cc5f78d 9e8cf0c
Author: Thomas Groh 
Authored: Wed Apr 5 18:08:03 2017 -0700
Committer: Thomas Groh 
Committed: Wed Apr 5 18:08:03 2017 -0700

--
 ...tputAndTimeBoundedSplittableProcessElementInvoker.java |  3 +--
 .../beam/runners/core/PushbackSideInputDoFnRunner.java|  2 +-
 .../apache/beam/runners/core/ReduceFnContextFactory.java  |  3 +--
 .../org/apache/beam/runners/core/SimpleDoFnRunner.java|  2 +-
 .../org/apache/beam/runners/core/SimpleOldDoFnRunner.java |  2 +-
 .../org/apache/beam/runners/core/ReduceFnRunnerTest.java  | 10 --
 .../java/org/apache/beam/sdk/transforms/DoFnTester.java   |  3 +--
 .../org/apache/beam/sdk/util/CombineContextFactory.java   |  2 +-
 8 files changed, 11 insertions(+), 16 deletions(-)
--

[jira] [Created] (BEAM-1898) Need a SerializableThrowable for PAssert to capture the point of an assertion

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-1898:
---

 Summary: Need a SerializableThrowable for PAssert to capture the 
point of an assertion
 Key: BEAM-1898
 URL: https://issues.apache.org/jira/browse/BEAM-1898
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Pablo Estrada
Assignee: Davor Bonaci


For the regular Error class in Java, its stack trace is a transient attribute, 
so it's not serialized by any framework. We need a class that allows us to 
serialize these AssertionErrors to have them in PCollections.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1897) Remove Sink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958112#comment-15958112
 ] 

ASF GitHub Bot commented on BEAM-1897:
--

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/2447

[BEAM-1897] Remove Sink in favor of FileBasedSink

The Sink base class has turned out to be mostly useful for file sinks. 
Remove Sink, and replace it with FileBasedSink.

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam streaming_gcs_output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2447.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2447


commit 5cd1630bc938f64003ab6b6b59b16e95d11d078a
Author: Reuven Lax 
Date:   2017-04-05T19:13:44Z

Start the process of getting rid of Sink.

commit d148b0e1a02413d55f73e0a42a4f8be6bf059e80
Author: Reuven Lax 
Date:   2017-04-05T22:06:14Z

Remove Sink class, and rename Write to WriteFiles.

commit 0af0e60c416718839be157ee88626453cefc5d81
Author: Reuven Lax 
Date:   2017-04-06T00:22:15Z

Get rid of Sink and initialize. We keep standin versions of the old Write 
and Sink transforms around as a stopgap solution for HDFSFileSink.

commit c2f374342234c71fc132e7eee1dabd8ef55a9794
Author: Reuven Lax 
Date:   2017-04-06T00:36:43Z

Roll back IntelliJ's spurious renaming of all "Write" strings and comments 
to "WriteFiles"

commit 5a37b614699df730040992f300e98fc0910e2f29
Author: Reuven Lax 
Date:   2017-04-06T00:39:18Z

Fix a few more spurious renames.




> Remove Sink
> ---
>
> Key: BEAM-1897
> URL: https://issues.apache.org/jira/browse/BEAM-1897
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Davor Bonaci
>
> Sink is really only used for file sinks. We should remove the abstract base 
> class, and just keep around FileBasedSink for file sinks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2447: [BEAM-1897] Remove Sink in favor of FileBasedSink

[reuvenlax]

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/2447

[BEAM-1897] Remove Sink in favor of FileBasedSink

The Sink base class has turned out to be mostly useful for file sinks. 
Remove Sink, and replace it with FileBasedSink.

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam streaming_gcs_output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2447.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2447


commit 5cd1630bc938f64003ab6b6b59b16e95d11d078a
Author: Reuven Lax 
Date:   2017-04-05T19:13:44Z

Start the process of getting rid of Sink.

commit d148b0e1a02413d55f73e0a42a4f8be6bf059e80
Author: Reuven Lax 
Date:   2017-04-05T22:06:14Z

Remove Sink class, and rename Write to WriteFiles.

commit 0af0e60c416718839be157ee88626453cefc5d81
Author: Reuven Lax 
Date:   2017-04-06T00:22:15Z

Get rid of Sink and initialize. We keep standin versions of the old Write 
and Sink transforms around as a stopgap solution for HDFSFileSink.

commit c2f374342234c71fc132e7eee1dabd8ef55a9794
Author: Reuven Lax 
Date:   2017-04-06T00:36:43Z

Roll back IntelliJ's spurious renaming of all "Write" strings and comments 
to "WriteFiles"

commit 5a37b614699df730040992f300e98fc0910e2f29
Author: Reuven Lax 
Date:   2017-04-06T00:39:18Z

Fix a few more spurious renames.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1897) Remove Sink

[Reuven Lax (JIRA)]

Reuven Lax created BEAM-1897:


 Summary: Remove Sink
 Key: BEAM-1897
 URL: https://issues.apache.org/jira/browse/BEAM-1897
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Reuven Lax
Assignee: Davor Bonaci


Sink is really only used for file sinks. We should remove the abstract base 
class, and just keep around FileBasedSink for file sinks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1865) Input Coder of GroupByKey should be a KV Coder in the Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958088#comment-15958088
 ] 

ASF GitHub Bot commented on BEAM-1865:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2409


> Input Coder of GroupByKey should be a KV Coder in the Python SDK
> 
>
> Key: BEAM-1865
> URL: https://issues.apache.org/jira/browse/BEAM-1865
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>
> `Any` type is consistent with `KV` in python. The coder for `Any` type is a 
> fallback coder or a `FastPrimitivesCoder`, but for a `GroupByKey` operation 
> this needs to be a `TupleCoder` to ensure that the generated pipeline 
> representation is runnable on a runner in a different language (in the Fn API 
> world)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2409: [BEAM-1865]: Replace Any type with a KV type for in...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2409


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2409

[altay]

This closes #2409


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cc5f78dd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cc5f78dd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cc5f78dd

Branch: refs/heads/master
Commit: cc5f78dd01395c24e1164d88a3d5b5b39256b4ff
Parents: 697b19f 7fad739
Author: Ahmet Altay 
Authored: Wed Apr 5 17:21:49 2017 -0700
Committer: Ahmet Altay 
Committed: Wed Apr 5 17:21:49 2017 -0700

--
 sdks/python/apache_beam/runners/runner.py | 29 ++
 1 file changed, 29 insertions(+)
--

[1/2] beam git commit: Replace Any type with a KV type for inputs of a GroupByKey step

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 697b19fe5 -> cc5f78dd0


Replace Any type with a KV type for inputs of a GroupByKey step


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7fad7391
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7fad7391
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7fad7391

Branch: refs/heads/master
Commit: 7fad7391548b6399066c0d9e79949707d7a5914e
Parents: 697b19f
Author: Vikas Kedigehalli 
Authored: Sun Apr 2 20:46:55 2017 -0700
Committer: Ahmet Altay 
Committed: Wed Apr 5 17:21:14 2017 -0700

--
 sdks/python/apache_beam/runners/runner.py | 29 ++
 1 file changed, 29 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7fad7391/sdks/python/apache_beam/runners/runner.py
--
diff --git a/sdks/python/apache_beam/runners/runner.py 
b/sdks/python/apache_beam/runners/runner.py
index b203c8b..528b03f 100644
--- a/sdks/python/apache_beam/runners/runner.py
+++ b/sdks/python/apache_beam/runners/runner.py
@@ -119,6 +119,35 @@ class PipelineRunner(object):
   logging.error('Error while visiting %s', transform_node.full_label)
   raise
 
+class GroupByKeyInputVisitor(PipelineVisitor):
+  """A visitor that replaces `Any` element type for input `PCollection` of
+  a `GroupByKey` with a `KV` type.
+
+  TODO(BEAM-115): Once Python SDk is compatible with the new Runner API,
+  we could directly replace the coder instead of mutating the element type.
+  """
+  def visit_transform(self, transform_node):
+# Imported here to avoid circular dependencies.
+# pylint: disable=wrong-import-order, wrong-import-position
+from apache_beam import GroupByKey
+from apache_beam import typehints
+if isinstance(transform_node.transform, GroupByKey):
+  pcoll = transform_node.inputs[0]
+  input_type = pcoll.element_type
+  if not isinstance(input_type, typehints.TupleHint.TupleConstraint):
+if isinstance(input_type, typehints.AnyTypeConstraint):
+  # `Any` type needs to be replaced with a KV[Any, Any] to
+  # force a KV coder as the main output coder for the pcollection
+  # preceding a GroupByKey.
+  pcoll.element_type = typehints.KV[typehints.Any, typehints.Any]
+else:
+  # TODO: Handle other valid types,
+  # e.g. Union[KV[str, int], KV[str, float]]
+  raise ValueError(
+  "Input to GroupByKey must be of Tuple or Any type. "
+  "Found %s for %s" % (input_type, pcoll))
+
+pipeline.visit(GroupByKeyInputVisitor())
 pipeline.visit(RunVisitor(self))
 
   def clear(self, pipeline, node=None):

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1527

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1895) Create tranform in python sdk should be a custom source

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958075#comment-15958075
 ] 

ASF GitHub Bot commented on BEAM-1895:
--

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2446

[BEAM-1895] Python Create as a custom source

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam py_custom_create

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2446.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2446


commit 43b8c6ded42054734c3f6d207ffd14b74a048285
Author: Vikas Kedigehalli 
Date:   2017-04-03T17:01:45Z

Create as custom source




> Create tranform in python sdk should be a custom source
> ---
>
> Key: BEAM-1895
> URL: https://issues.apache.org/jira/browse/BEAM-1895
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Vikas Kedigehalli
>Assignee: Ahmet Altay
>
> This allows Create transform to be runner independent. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2446: [BEAM-1895] Python Create as a custom source

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2446

[BEAM-1895] Python Create as a custom source

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam py_custom_create

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2446.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2446


commit 43b8c6ded42054734c3f6d207ffd14b74a048285
Author: Vikas Kedigehalli 
Date:   2017-04-03T17:01:45Z

Create as custom source




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1895) Create tranform in python sdk should be a custom source

[Vikas Kedigehalli (JIRA)]

Vikas Kedigehalli created BEAM-1895:
---

 Summary: Create tranform in python sdk should be a custom source
 Key: BEAM-1895
 URL: https://issues.apache.org/jira/browse/BEAM-1895
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Vikas Kedigehalli
Assignee: Ahmet Altay


This allows Create transform to be runner independent. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #274

[Apache Jenkins Server]

See 


Changes:

[tgroh] Exclude UsesTestStream Tests for more builds

[kirpichov] Clarifies BigQueryIO javadoc

[tgroh] Suppress TestStream tests in Dataflow

[dhalperi] [BEAM-1881] Exclude protobuf-lite and replace with protobuf-java

[altay] Do not send on_success_matcher to remote executors

[tgroh] Remove single-matcher replacement API

[altay] Revert "Revert "Add ValueProvider class for FileBasedSource I/O

[dhalperi] Fixed a comment in AfterWatermark

[dhalperi] [BEAM-1269] Update bigtable library dependency

--
[...truncated 252.52 KB...]
error: unable to resolve reference refs/remotes/origin/pr/2347/merge: No such 
file or directory
 ! 2207b92...61649b9 refs/pull/2347/merge -> origin/pr/2347/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2348/merge: No such 
file or directory
 ! ec3e51d...fdbe070 refs/pull/2348/merge -> origin/pr/2348/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2350/merge: No such 
file or directory
 ! 822cf4d...61acff5 refs/pull/2350/merge -> origin/pr/2350/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2351/merge: No such 
file or directory
 ! 662028f...c9f9e2f refs/pull/2351/merge -> origin/pr/2351/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2352/merge: No such 
file or directory
 ! 6773c57...9a2631b refs/pull/2352/merge -> origin/pr/2352/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2353/merge: No such 
file or directory
 ! a67201d...ffb2932 refs/pull/2353/merge -> origin/pr/2353/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2358/merge: No such 
file or directory
 ! a9bcdcb...c251df6 refs/pull/2358/merge -> origin/pr/2358/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2362/merge: No such 
file or directory
 ! d55e900...b4e6a7f refs/pull/2362/merge -> origin/pr/2362/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2363/merge: No such 
file or directory
 ! 50c5048...41c3a3f refs/pull/2363/merge -> origin/pr/2363/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2364/merge: No such 
file or directory
 ! 75c86de...c768e09 refs/pull/2364/merge -> origin/pr/2364/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2367/merge: No such 
file or directory
 ! 0a97f02...6ce0b46 refs/pull/2367/merge -> origin/pr/2367/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2369/merge: No such 
file or directory
 ! a10c8c8...03bdfc1 refs/pull/2369/merge -> origin/pr/2369/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2375/merge: No such 
file or directory
 ! 0883a61...aa0dfb8 refs/pull/2375/merge -> origin/pr/2375/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2377/head: No such 
file or directory
 ! 973a8c9...2a7d9eb refs/pull/2377/head -> origin/pr/2377/head  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2377/merge: No such 
file or directory
 ! 4481291...46ac43b refs/pull/2377/merge -> origin/pr/2377/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2379/merge: No such 
file or directory
 ! bd373f4...50feb29 refs/pull/2379/merge -> origin/pr/2379/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2382/merge: No such 
file or directory
 ! 9feec00...a302f42 refs/pull/2382/merge -> origin/pr/2382/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2383/merge: No such 
file or directory
 ! dcc5e4d...7fd9d5b refs/pull/2383/merge -> origin/pr/2383/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2385/merge: No such 
file or directory
 ! 7ffba0c...fd20276 refs/pull/2385/merge -> origin/pr/2385/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2389/merge: No such 
file or directory
 ! 112e28d...64711d2 refs/pull/2389/merge -> origin/pr/2389/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2390/merge: No such 
file or directory
 ! 194e4d0...dd408be refs/pull/2390/merge -> origin/pr/2390/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2392/merge: No such 
file or directory
 ! 0a5b68b...7c97f22 refs/pull/2392/merge -> origin/pr/2392/merge  (unable to 
update local ref)
error: unable to resolve reference 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #2209

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1894) Race conditions in python direct runner eager mode

[Vikas Kedigehalli (JIRA)]

Vikas Kedigehalli created BEAM-1894:
---

 Summary: Race conditions in python direct runner eager mode
 Key: BEAM-1894
 URL: https://issues.apache.org/jira/browse/BEAM-1894
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Vikas Kedigehalli
Assignee: Ahmet Altay
 Fix For: First stable release


test_eager_pipeline 
(https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline_test.py#L283)
 fails with the following error:
ERROR: test_eager_pipeline (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline_test.py",
 line 285, in test_eager_pipeline
self.assertEqual([1, 4, 9], p | Create([1, 2, 3]) | Map(lambda x: x*x))
  File 
"/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/transforms/ptransform.py",
 line 387, in __ror__
p.run().wait_until_finish()
  File 
"/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline.py",
 line 160, in run
self.to_runner_api(), self.runner, self.options).run(False)
  File 
"/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline.py",
 line 169, in run
return self.runner.run(self)
  File 
"/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
 line 99, in run
result.wait_until_finish()
  File 
"/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
 line 166, in wait_until_finish
self._executor.await_completion()
  File 
"/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 336, in await_completion
self._executor.await_completion()
  File 
"/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 308, in __call__
uncommitted_bundle.get_elements_iterable())
  File 
"/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 176, in append_to_cache
self._cache.append(applied_ptransform, tag, elements)
  File 
"/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
 line 138, in append
self._cache[(applied_ptransform, tag)].extend(elements)
TypeError: 'NoneType' object has no attribute '__getitem__'


This is triggered when Create is changed to a custom source. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1418) MapElements and FlatMapElements should comply with PTransform style guide

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-1418:
-

Assignee: Eugene Kirpichov

> MapElements and FlatMapElements should comply with PTransform style guide
> -
>
> Key: BEAM-1418
> URL: https://issues.apache.org/jira/browse/BEAM-1418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>
> Type structure of these classes can be improved by slightly changing the API:
> FlatMapElements.into(TypeDescriptor).via(as usual).
> Likewise for MapElements. This allows getting rid of the awkward 
> MissingOutputTypeDescriptor intermediate class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1269) BigtableIO should make more efficient use of connections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958039#comment-15958039
 ] 

ASF GitHub Bot commented on BEAM-1269:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2377


> BigtableIO should make more efficient use of connections
> 
>
> Key: BEAM-1269
> URL: https://issues.apache.org/jira/browse/BEAM-1269
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Daniel Halperin
>  Labels: newbie, starter
>
> RIght now, {{BigtableIO}} opens up a new Bigtable session for every DoFn, in 
> the {{@Setup}} function. However, sessions can support multiple connections, 
> so perhaps this code should be modified to open up a smaller session pool and 
> then allocation connections in {{@StartBundle}}.
> This would likely make more efficient use of resources, especially for highly 
> multithreaded workers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2377: BEAM-1269: Update bigtable library dependency and a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2377


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1269] Update bigtable library dependency

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 74f799509 -> 697b19fe5


[BEAM-1269] Update bigtable library dependency

Enable cached data pools for efficiency and remove pegging data channel count 
to 1.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/badac8dc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/badac8dc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/badac8dc

Branch: refs/heads/master
Commit: badac8dc156d8c85d23802dd2092e9a8a4d1f6c1
Parents: 74f7995
Author: mdshalda 
Authored: Thu Mar 30 14:06:03 2017 -0700
Committer: Dan Halperin 
Committed: Wed Apr 5 16:42:43 2017 -0700

--
 pom.xml |  6 ++---
 sdks/java/io/google-cloud-platform/pom.xml  | 24 ++--
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java|  6 ++---
 3 files changed, 8 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/badac8dc/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 4ecfdd7..3f53b1f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -119,7 +119,7 @@
 1.4.5
 
0.5.160304
 20.0
-1.0.1
+1.2.0
 
0.1.0
 1.3
 2.7.2
@@ -127,9 +127,9 @@
 2.4
 4.12
 1.9.5
-4.1.6.Final
+4.1.8.Final
 1.5.0.Final
-3.1.0
+3.2.0
 v1-rev10-1.22.0
 1.7.14
 1.6.2

http://git-wip-us.apache.org/repos/asf/beam/blob/badac8dc/sdks/java/io/google-cloud-platform/pom.xml
--
diff --git a/sdks/java/io/google-cloud-platform/pom.xml 
b/sdks/java/io/google-cloud-platform/pom.xml
index f784e27..a5b2e56 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -32,7 +32,7 @@
   jar
 
   
-0.9.5.1
+0.9.6
   
 
   
@@ -81,26 +81,6 @@
   
 
   
-
-  
-org.apache.maven.plugins
-maven-dependency-plugin
-
-  
-analyze-only
-
-  
-
-
com.google.api.grpc:grpc-google-common-protos
-  
-
-  
-
-  
 
   
 
@@ -223,7 +203,7 @@
 
   io.netty
   netty-tcnative-boringssl-static
-  1.1.33.Fork18
+  1.1.33.Fork26
   runtime
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/badac8dc/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 2cdd11d..89c67a4 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -212,9 +212,8 @@ public class BigtableIO {
   // TODO: is there a better way to clone a Builder? Want it to be immune 
from user changes.
   BigtableOptions options = optionsBuilder.build();
 
-  // Set data channel count to one because there is only 1 scanner in this 
session
   BigtableOptions.Builder clonedBuilder = options.toBuilder()
-  .setDataChannelCount(1);
+  .setUseCachedDataPool(true);
   BigtableOptions optionsWithAgent =
   clonedBuilder.setUserAgent(getBeamSdkPartOfUserAgent()).build();
 
@@ -449,7 +448,8 @@ public class BigtableIO {
   .setBulkOptions(
   options.getBulkOptions().toBuilder()
   .setUseBulkApi(true)
-  .build());
+  .build())
+  .setUseCachedDataPool(true);
   BigtableOptions optionsWithAgent =
   clonedBuilder.setUserAgent(getBeamSdkPartOfUserAgent()).build();
   return new Write(optionsWithAgent, tableId, bigtableService);

[2/2] beam git commit: This closes #2377

[dhalperi]

This closes #2377


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/697b19fe
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/697b19fe
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/697b19fe

Branch: refs/heads/master
Commit: 697b19fe5768cdc85aeaa54c425cb3a61a6167f4
Parents: 74f7995 badac8d
Author: Dan Halperin 
Authored: Wed Apr 5 16:43:00 2017 -0700
Committer: Dan Halperin 
Committed: Wed Apr 5 16:43:00 2017 -0700

--
 pom.xml |  6 ++---
 sdks/java/io/google-cloud-platform/pom.xml  | 24 ++--
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java|  6 ++---
 3 files changed, 8 insertions(+), 28 deletions(-)
--

[jira] [Commented] (BEAM-1166) Source.getDefaultOutputCoder() documentation should mention CannotProvideCoderException

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958031#comment-15958031
 ] 

Stephen Sisk commented on BEAM-1166:


will try to get to this next week

> Source.getDefaultOutputCoder() documentation should mention 
> CannotProvideCoderException
> ---
>
> Key: BEAM-1166
> URL: https://issues.apache.org/jira/browse/BEAM-1166
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> Knowing that you can throw CannotProviderCoderException is an important part 
> of implementing getDefaultOutputCoder
> The documentation for PTransform's getDefaultOutputCoder mentions this class, 
> and we should do this for the Source class as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2444: updated documentation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2444


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2445: [BEAM-1892] File size estimation thresholding and p...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2445

[BEAM-1892] File size estimation thresholding and process reporting

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @chamikaramj PTAL


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1892-file-size-estimation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2445.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2445


commit 507d8a4b35b35de79729f813500197ae920ce9a8
Author: Sourabh Bajaj 
Date:   2017-04-05T23:36:18Z

[BEAM-1892] File size estimation thresholding and process reporting




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Fixed a comment in AfterWatermark

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 3500d694a -> 74f799509


Fixed a comment in AfterWatermark


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/174b631a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/174b631a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/174b631a

Branch: refs/heads/master
Commit: 174b631ac1b180505c4881795309ea6d2bc592f2
Parents: 3500d69
Author: Anil Muppalla 
Authored: Wed Apr 5 19:01:10 2017 -0400
Committer: Dan Halperin 
Committed: Wed Apr 5 16:37:57 2017 -0700

--
 .../org/apache/beam/sdk/transforms/windowing/AfterWatermark.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/174b631a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
index 37b73a6..7cd2601 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
@@ -55,7 +55,7 @@ import org.joda.time.Instant;
  *
  * Additionaly firings before or after the watermark can be requested by 
calling
  * {@code AfterWatermark.pastEndOfWindow.withEarlyFirings(OnceTrigger)} or
- * {@code AfterWatermark.pastEndOfWindow.withEarlyFirings(OnceTrigger)}.
+ * {@code AfterWatermark.pastEndOfWindow.withLateFirings(OnceTrigger)}.
  */
 @Experimental(Experimental.Kind.TRIGGER)
 public class AfterWatermark {

[jira] [Commented] (BEAM-1892) Log process during size estimation in filebasedsource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958030#comment-15958030
 ] 

ASF GitHub Bot commented on BEAM-1892:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2445

[BEAM-1892] File size estimation thresholding and process reporting

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @chamikaramj PTAL


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1892-file-size-estimation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2445.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2445


commit 507d8a4b35b35de79729f813500197ae920ce9a8
Author: Sourabh Bajaj 
Date:   2017-04-05T23:36:18Z

[BEAM-1892] File size estimation thresholding and process reporting




> Log process during size estimation in filebasedsource
> -
>
> Key: BEAM-1892
> URL: https://issues.apache.org/jira/browse/BEAM-1892
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>
> http://stackoverflow.com/questions/43095445/how-to-iterate-all-files-in-google-cloud-storage-to-be-used-as-dataflow-input
> The user mentioned that there was no output and a huge delay in submitting 
> the pipeline. The file size estimation process can be slow for really large 
> datasets and this reports no process to the end user right now. We should be 
> logging process and thresholding the pre submission size estimation as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2444

[dhalperi]

This closes #2444


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/74f79950
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/74f79950
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/74f79950

Branch: refs/heads/master
Commit: 74f7995097bee22a0701e27f31c7f0449a01e37c
Parents: 3500d69 174b631
Author: Dan Halperin 
Authored: Wed Apr 5 16:38:05 2017 -0700
Committer: Dan Halperin 
Committed: Wed Apr 5 16:38:05 2017 -0700

--
 .../org/apache/beam/sdk/transforms/windowing/AfterWatermark.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Created] (BEAM-1893) Add IO module for Couchbase

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-1893:


 Summary: Add IO module for Couchbase
 Key: BEAM-1893
 URL: https://issues.apache.org/jira/browse/BEAM-1893
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-extensions
Reporter: Xu Mingmin
Assignee: Xu Mingmin


Create a {{CouchbaseIO}} for Couchbase database.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1892) Log process during size estimation in filebasedsource

[Sourabh Bajaj (JIRA)]

Sourabh Bajaj created BEAM-1892:
---

 Summary: Log process during size estimation in filebasedsource
 Key: BEAM-1892
 URL: https://issues.apache.org/jira/browse/BEAM-1892
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Sourabh Bajaj
Assignee: Sourabh Bajaj


http://stackoverflow.com/questions/43095445/how-to-iterate-all-files-in-google-cloud-storage-to-be-used-as-dataflow-input

The user mentioned that there was no output and a huge delay in submitting the 
pipeline. The file size estimation process can be slow for really large 
datasets and this reports no process to the end user right now. We should be 
logging process and thresholding the pre submission size estimation as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1678) Create MemcachedIO

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958018#comment-15958018
 ] 

Xu Mingmin commented on BEAM-1678:
--

Hi [~iemejia], I thought {{Memcached}} is more likely a cache system without 
persistent. Can it be used as an IO for Couchbase?

> Create MemcachedIO
> --
>
> Key: BEAM-1678
> URL: https://issues.apache.org/jira/browse/BEAM-1678
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2443

[altay]

This closes #2443


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3500d694
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3500d694
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3500d694

Branch: refs/heads/master
Commit: 3500d694a555356888a8618d7c7635286c1b5ea8
Parents: 4b69d24 28a0ea8
Author: Ahmet Altay 
Authored: Wed Apr 5 16:21:55 2017 -0700
Committer: Ahmet Altay 
Committed: Wed Apr 5 16:21:55 2017 -0700

--
 sdks/python/apache_beam/examples/wordcount.py   |  32 ++--
 .../apache_beam/internal/gcp/json_value.py  |   6 +
 sdks/python/apache_beam/io/filebasedsource.py   |  54 --
 .../apache_beam/io/filebasedsource_test.py  |  24 +++
 sdks/python/apache_beam/io/fileio.py|  56 +--
 sdks/python/apache_beam/io/fileio_test.py   |  45 +++--
 .../runners/dataflow/internal/apiclient.py  |   1 +
 .../apache_beam/runners/direct/direct_runner.py |   9 +
 sdks/python/apache_beam/transforms/display.py   |   1 +
 .../apache_beam/transforms/display_test.py  |  36 
 .../apache_beam/utils/pipeline_options.py   |  92 ++-
 .../apache_beam/utils/pipeline_options_test.py  |  52 +-
 sdks/python/apache_beam/utils/value_provider.py | 110 +
 .../apache_beam/utils/value_provider_test.py| 165 +++
 14 files changed, 618 insertions(+), 65 deletions(-)
--

[GitHub] beam pull request #2443: [BEAM-802] Add ValueProvider class for FileBasedSou...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2443


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Revert "Revert "Add ValueProvider class for FileBasedSource I/O Transforms""

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 4b69d24af -> 3500d694a


Revert "Revert "Add ValueProvider class for FileBasedSource I/O Transforms""

This reverts commit 3eef246f761f92c626541e9008f8624b43bdcc09.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/28a0ea86
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/28a0ea86
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/28a0ea86

Branch: refs/heads/master
Commit: 28a0ea86721871a97f68b573019e818217266356
Parents: 4b69d24
Author: Maria Garcia Herrero 
Authored: Wed Apr 5 14:42:13 2017 -0700
Committer: Ahmet Altay 
Committed: Wed Apr 5 16:21:50 2017 -0700

--
 sdks/python/apache_beam/examples/wordcount.py   |  32 ++--
 .../apache_beam/internal/gcp/json_value.py  |   6 +
 sdks/python/apache_beam/io/filebasedsource.py   |  54 --
 .../apache_beam/io/filebasedsource_test.py  |  24 +++
 sdks/python/apache_beam/io/fileio.py|  56 +--
 sdks/python/apache_beam/io/fileio_test.py   |  45 +++--
 .../runners/dataflow/internal/apiclient.py  |   1 +
 .../apache_beam/runners/direct/direct_runner.py |   9 +
 sdks/python/apache_beam/transforms/display.py   |   1 +
 .../apache_beam/transforms/display_test.py  |  36 
 .../apache_beam/utils/pipeline_options.py   |  92 ++-
 .../apache_beam/utils/pipeline_options_test.py  |  52 +-
 sdks/python/apache_beam/utils/value_provider.py | 110 +
 .../apache_beam/utils/value_provider_test.py| 165 +++
 14 files changed, 618 insertions(+), 65 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/28a0ea86/sdks/python/apache_beam/examples/wordcount.py
--
diff --git a/sdks/python/apache_beam/examples/wordcount.py 
b/sdks/python/apache_beam/examples/wordcount.py
index 50c0328..27b9dcb 100644
--- a/sdks/python/apache_beam/examples/wordcount.py
+++ b/sdks/python/apache_beam/examples/wordcount.py
@@ -19,7 +19,6 @@
 
 from __future__ import absolute_import
 
-import argparse
 import logging
 import re
 
@@ -67,24 +66,29 @@ class WordExtractingDoFn(beam.DoFn):
 
 def run(argv=None):
   """Main entry point; defines and runs the wordcount pipeline."""
-  parser = argparse.ArgumentParser()
-  parser.add_argument('--input',
-  dest='input',
-  default='gs://dataflow-samples/shakespeare/kinglear.txt',
-  help='Input file to process.')
-  parser.add_argument('--output',
-  dest='output',
-  required=True,
-  help='Output file to write results to.')
-  known_args, pipeline_args = parser.parse_known_args(argv)
+  class WordcountOptions(PipelineOptions):
+@classmethod
+def _add_argparse_args(cls, parser):
+  parser.add_value_provider_argument(
+  '--input',
+  dest='input',
+  default='gs://dataflow-samples/shakespeare/kinglear.txt',
+  help='Input file to process.')
+  parser.add_value_provider_argument(
+  '--output',
+  dest='output',
+  required=True,
+  help='Output file to write results to.')
+  pipeline_options = PipelineOptions(argv)
+  wordcount_options = pipeline_options.view_as(WordcountOptions)
+
   # We use the save_main_session option because one or more DoFn's in this
   # workflow rely on global context (e.g., a module imported at module level).
-  pipeline_options = PipelineOptions(pipeline_args)
   pipeline_options.view_as(SetupOptions).save_main_session = True
   p = beam.Pipeline(options=pipeline_options)
 
   # Read the text file[pattern] into a PCollection.
-  lines = p | 'read' >> ReadFromText(known_args.input)
+  lines = p | 'read' >> ReadFromText(wordcount_options.input)
 
   # Count the occurrences of each word.
   counts = (lines
@@ -99,7 +103,7 @@ def run(argv=None):
 
   # Write the output using a "Write" transform that has side effects.
   # pylint: disable=expression-not-assigned
-  output | 'write' >> WriteToText(known_args.output)
+  output | 'write' >> WriteToText(wordcount_options.output)
 
   # Actually run the pipeline (all operations above are deferred).
   result = p.run()

http://git-wip-us.apache.org/repos/asf/beam/blob/28a0ea86/sdks/python/apache_beam/internal/gcp/json_value.py
--
diff --git a/sdks/python/apache_beam/internal/gcp/json_value.py 
b/sdks/python/apache_beam/internal/gcp/json_value.py
index c8b5393..4099c1a 100644
--- a/sdks/python/apache_beam/internal/gcp/json_value.py
+++ b/sdks/python/apache_beam/internal/gcp/json_value.py
@@ -25,6 +25,8 @@ except ImportError:
   extra_types = None
 # pylint: 

[jira] [Commented] (BEAM-802) Support Dynamic PipelineOptions for python

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957999#comment-15957999
 ] 

ASF GitHub Bot commented on BEAM-802:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2443


> Support Dynamic PipelineOptions for python
> --
>
> Key: BEAM-802
> URL: https://issues.apache.org/jira/browse/BEAM-802
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
>   Original Estimate: 1,680h
>  Remaining Estimate: 1,680h
>
> Goal:  Enable users to run pipelines from templates filled via CL (pipeline 
> options)
> Background: Currently, the Runner creates the JSON pipeline description which 
> can be sent to the worker as is, since everything is already defined there 
> (with links to gs:// for input and binaries). With the parametrized approach, 
> those descriptions are empty and filled by the user or defaulted, so the 
> pipeline needs to be stored somewhere first until the values become available.
> Tasks:
> 1- Create template-style pipeline description (TemplateRunner)
> The graph description is now a template (some parts are not filled) that 
> needs to be saved.
> 2- Define values to inject to the template (ValueProviders API)
> The placeholders can be filled with default values (static) or with dynamic 
> key/value pairs provided at runtime (dynamic)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-301) Add a Beam SQL DSL

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957983#comment-15957983
 ] 

Xu Mingmin commented on BEAM-301:
-

[~julianhyde] It's great that {{EMIT}}  is supported soon. Will look into it 
for more details. QQ, from parser/grammar, is {{EMIT}} limited to Streaming or 
workable for both? We would like to remove {{STREAM}} in Beam SQL as BEAM model 
is unified for both streaming/batch data.

> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Xu Mingmin
>
> The SQL DSL helps developers to build a Beam pipeline from SQL statement in 
> String directly. 
> In Phase I, it starts to support INSERT/SELECT queries with FILTERs, one 
> example SQL as below:
> {code}
> INSERT INTO `SUB_USEREVENT` (`SITEID`, `PAGEID`, `PAGENAME`, `EVENTTIMESTAMP`)
> (SELECT STREAM `USEREVENT`.`SITEID`, `USEREVENT`.`PAGEID`, 
> `USEREVENT`.`PAGENAME`, `USEREVENT`.`EVENTTIMESTAMP`
> FROM `USEREVENT` AS `USEREVENT`
> WHERE `USEREVENT`.`SITEID` > 10)
> {code}
> A design doc is available at 
> https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_0a1Bz5BsCROMzCU/edit?usp=sharing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-644) Primitive to shift the watermark while assigning timestamps

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957959#comment-15957959
 ] 

Eugene Kirpichov commented on BEAM-644:
---

I don't think there's an SDF-specific issue here: it applies just as well to a 
non-splittable DoFn that would take a filename as input, and potentially 
produce elements whose timestamps are behind the timestamp of the input element 
(filename).

Speaking of updates - no, I don't think there've been any updates. Perhaps in 
practice this can be solved by feeding the SDF elements with a timestamp of 
"infinite past", and from then on, relying on the SDF's own output watermark 
reporting.

> Primitive to shift the watermark while assigning timestamps
> ---
>
> Key: BEAM-644
> URL: https://issues.apache.org/jira/browse/BEAM-644
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, beam-model-runner-api
>Reporter: Kenneth Knowles
>
> There is a general need, especially important in the presence of 
> SplittableDoFn, to be able to assign new timestamps to elements without 
> making them late or droppable.
>  - DoFn.withAllowedTimestampSkew is inadequate, because it simply allows one 
> to produce late data, but does not allow one to shift the watermark so the 
> new data is on-time.
>  - For a SplittableDoFn, one may receive an element such as the name of a log 
> file that contains elements for the day preceding the log file. The timestamp 
> on the filename must currently be the beginning of the log. If such elements 
> are constantly flowing, it may be OK, but since we don't know that element is 
> coming, in that absence of data, the watermark may advance. We need a way to 
> keep it far enough back even in the absence of data holding it back.
> One idea is a new primitive ShiftWatermark / AdjustTimestamps with the 
> following pieces:
>  - A constant duration (positive or negative) D by which to shift the 
> watermark.
>  - A function from TimestampedElement to new timestamp that is >= t + D
> So, for example, AdjustTimestamps(<-60 minutes>, f) would allow f to make 
> timestamps up to 60 minutes earlier.
> With this primitive added, outputWithTimestamp and withAllowedTimestampSkew 
> could be removed, simplifying DoFn.
> Alternatively, all of this functionality could be bolted on to DoFn.
> This ticket is not a proposal, but a record of the issue and ideas that were 
> mentioned.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #2752

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-301) Add a Beam SQL DSL

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957945#comment-15957945
 ] 

Julian Hyde edited comment on BEAM-301 at 4/5/17 10:47 PM:
---

A couple of recent developments in Calcite's streaming SQL:
* A proposed {{EMIT}} clause to match Beam's "triggering" (the term "trigger" 
has some unfortunate connotations in the SQL world). Appears on slide 15 of [my 
"Streaming SQL" talk at FlinkForward 
2016|https://www.slideshare.net/julianhyde/streaming-sql-at-flinkforward-berlin-20160912].
* A {{MATCH_RECOGNIZE}} clause, inspired by Oracle, good for non-streaming and 
streaming queries, but clearly good for "Complex Event Processing" (CEP) kinds 
of applications. Currently we have parser and validator support but no 
implementation; see CALCITE-1570.


was (Author: julianhyde):
A couple of recent developments in Calcite's streaming SQL:
* A proposed {{EMIT}} clause to match Beam's "triggering" (the term "trigger" 
has some unfortunate connotations in the SQL world). Appears on slide 15 of 
[https://www.slideshare.net/julianhyde/streaming-sql-at-flinkforward-berlin-20160912|my
 "Streaming SQL" talk at FlinkForward].
* A {{MATCH_RECOGNIZE}} clause, inspired by Oracle, good for non-streaming and 
streaming queries, but clearly good for "Complex Event Processing" (CEP) kinds 
of applications. Currently we have parser and validator support but no 
implementation; see CALCITE-1570.

> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Xu Mingmin
>
> The SQL DSL helps developers to build a Beam pipeline from SQL statement in 
> String directly. 
> In Phase I, it starts to support INSERT/SELECT queries with FILTERs, one 
> example SQL as below:
> {code}
> INSERT INTO `SUB_USEREVENT` (`SITEID`, `PAGEID`, `PAGENAME`, `EVENTTIMESTAMP`)
> (SELECT STREAM `USEREVENT`.`SITEID`, `USEREVENT`.`PAGEID`, 
> `USEREVENT`.`PAGENAME`, `USEREVENT`.`EVENTTIMESTAMP`
> FROM `USEREVENT` AS `USEREVENT`
> WHERE `USEREVENT`.`SITEID` > 10)
> {code}
> A design doc is available at 
> https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_0a1Bz5BsCROMzCU/edit?usp=sharing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-301) Add a Beam SQL DSL

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957945#comment-15957945
 ] 

Julian Hyde commented on BEAM-301:
--

A couple of recent developments in Calcite's streaming SQL:
* A proposed {{EMIT}} clause to match Beam's "triggering" (the term "trigger" 
has some unfortunate connotations in the SQL world). Appears on slide 15 of 
[https://www.slideshare.net/julianhyde/streaming-sql-at-flinkforward-berlin-20160912|my
 "Streaming SQL" talk at FlinkForward].
* A {{MATCH_RECOGNIZE}} clause, inspired by Oracle, good for non-streaming and 
streaming queries, but clearly good for "Complex Event Processing" (CEP) kinds 
of applications. Currently we have parser and validator support but no 
implementation; see CALCITE-1570.

> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Xu Mingmin
>
> The SQL DSL helps developers to build a Beam pipeline from SQL statement in 
> String directly. 
> In Phase I, it starts to support INSERT/SELECT queries with FILTERs, one 
> example SQL as below:
> {code}
> INSERT INTO `SUB_USEREVENT` (`SITEID`, `PAGEID`, `PAGENAME`, `EVENTTIMESTAMP`)
> (SELECT STREAM `USEREVENT`.`SITEID`, `USEREVENT`.`PAGEID`, 
> `USEREVENT`.`PAGENAME`, `USEREVENT`.`EVENTTIMESTAMP`
> FROM `USEREVENT` AS `USEREVENT`
> WHERE `USEREVENT`.`SITEID` > 10)
> {code}
> A design doc is available at 
> https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_0a1Bz5BsCROMzCU/edit?usp=sharing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

[Mairbek Khadikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957937#comment-15957937
 ] 

Mairbek Khadikov commented on BEAM-1542:


Hi Guy, I work on Cloud Spanner at Google. Let us know if you need any help 
with the Sink. I'm going to leave few comments in the PR as well.

If you are hitting the following exception
Exception in thread "main" 
com.google.common.util.concurrent.UncheckedExecutionException: 
org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: 
Could not initialize class com.google.cloud.spanner.spi.SpannerErrorInterceptor
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
at com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
at 
org.apache.beam.runners.direct.DoFnLifecycleManager.get(DoFnLifecycleManager.java:61)
at 
org.apache.beam.runners.direct.ParDoEvaluatorFactory.createEvaluator(ParDoEvaluatorFactory.java:124)
at 
org.apache.beam.runners.direct.ParDoEvaluatorFactory.forApplication(ParDoEvaluatorFactory.java:77)
at 
org.apache.beam.runners.direct.TransformEvaluatorRegistry.forApplication(TransformEvaluatorRegistry.java:96)
at 
org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:100)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


It should be resolved with https://github.com/apache/beam/pull/2377

> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Guy Molinari
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3179

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-1572) Add per-stage matching of scope in metrics for the DirectRunner

[Pablo Estrada (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Estrada resolved BEAM-1572.
-
   Resolution: Fixed
Fix Version/s: First stable release

> Add per-stage matching of scope in metrics for the DirectRunner
> ---
>
> Key: BEAM-1572
> URL: https://issues.apache.org/jira/browse/BEAM-1572
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
> Fix For: First stable release
>
>
> e.g. Metrics with scope "Top/Outer/Inner" should be matched by queries with 
> "Top/Outer" scope.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1381) Implement DataflowMetrics.query method

[Pablo Estrada (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Estrada resolved BEAM-1381.
-
   Resolution: Fixed
Fix Version/s: First stable release

> Implement DataflowMetrics.query method
> --
>
> Key: BEAM-1381
> URL: https://issues.apache.org/jira/browse/BEAM-1381
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
> Fix For: First stable release
>
>
> Once metrics can bequeried from the dataflow service, we must implement the 
> logic for this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1579) Runners should verify that PT overrides converged

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957913#comment-15957913
 ] 

ASF GitHub Bot commented on BEAM-1579:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2424


> Runners should verify that PT overrides converged
> -
>
> Key: BEAM-1579
> URL: https://issues.apache.org/jira/browse/BEAM-1579
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-dataflow, runner-direct
>Reporter: Eugene Kirpichov
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> PT overrides are applied in order, see 
> https://issues.apache.org/jira/browse/BEAM-1578 .
> To make sure that the order is correct and avoid confusing errors, after 
> applying the overrides in order, we should verify that the pipeline has 
> converged, i.e. the overrides no longer match.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1579) Runners should verify that PT overrides converged

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1579.
---
   Resolution: Fixed
Fix Version/s: First stable release

> Runners should verify that PT overrides converged
> -
>
> Key: BEAM-1579
> URL: https://issues.apache.org/jira/browse/BEAM-1579
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-dataflow, runner-direct
>Reporter: Eugene Kirpichov
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> PT overrides are applied in order, see 
> https://issues.apache.org/jira/browse/BEAM-1578 .
> To make sure that the order is correct and avoid confusing errors, after 
> applying the overrides in order, we should verify that the pipeline has 
> converged, i.e. the overrides no longer match.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: Remove single-matcher replacement API

[tgroh]

Remove single-matcher replacement API

This makes it more difficult for runner authors to use the discouraged
API that doesn't validate ordering of replacements.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/753bc9cc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/753bc9cc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/753bc9cc

Branch: refs/heads/master
Commit: 753bc9cc2bcfd130e1523eabf97d3de791c4cfd2
Parents: 9fb4fc3
Author: Thomas Groh 
Authored: Tue Apr 4 10:20:49 2017 -0700
Committer: Thomas Groh 
Committed: Wed Apr 5 15:26:56 2017 -0700

--
 .../org/apache/beam/runners/spark/TestSparkRunner.java| 10 ++
 .../core/src/main/java/org/apache/beam/sdk/Pipeline.java  |  2 +-
 2 files changed, 7 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/753bc9cc/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
index be9ff2e..988a82b 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
@@ -27,6 +27,7 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.util.concurrent.Uninterruptibles;
 import java.io.File;
 import java.io.IOException;
+import java.util.Collections;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.TimeUnit;
@@ -217,10 +218,11 @@ public final class TestSparkRunner extends 
PipelineRunner {
 
   @VisibleForTesting
   void adaptBoundedReads(Pipeline pipeline) {
-pipeline.replace(
-PTransformOverride.of(
-
PTransformMatchers.classEqualTo(BoundedReadFromUnboundedSource.class),
-new AdaptedBoundedAsUnbounded.Factory()));
+pipeline.replaceAll(
+Collections.singletonList(
+PTransformOverride.of(
+
PTransformMatchers.classEqualTo(BoundedReadFromUnboundedSource.class),
+new AdaptedBoundedAsUnbounded.Factory(;
   }
 
   private static class AdaptedBoundedAsUnbounded extends PTransform {

http://git-wip-us.apache.org/repos/asf/beam/blob/753bc9cc/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
index fa8277f..11d781d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
@@ -228,7 +228,7 @@ public class Pipeline {
 });
   }
 
-  public void replace(final PTransformOverride override) {
+  private void replace(final PTransformOverride override) {
 final Collection matches = new ArrayList<>();
 transforms.visit(
 new PipelineVisitor.Defaults() {

[GitHub] beam pull request #2424: [BEAM-1579] Remove single-matcher replacement API

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2424


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---