[jira] [Assigned] (BEAM-2148) BeamKafkaCSVTable: Use a more robust library to parse CSV

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-2148:


Assignee: (was: Xu Mingmin)

> BeamKafkaCSVTable: Use a more robust library to parse CSV
> -
>
> Key: BEAM-2148
> URL: https://issues.apache.org/jira/browse/BEAM-2148
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: James Xu
>Priority: Minor
>
> use something like apache commons csv 
> (https://commons.apache.org/proper/commons-csv/) to parse the message.
> And the CSVRecordEncoder & CSVRecordDecoder can be refactored out to be used 
> by not only KafkaTable but also TextTable or any other table that are encoded 
> using CSV format.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2084) Distribution metrics should be queriable in the Dataflow Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995857#comment-15995857
 ] 

ASF GitHub Bot commented on BEAM-2084:
--

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2874

[BEAM-2084] Metrics are queriable from Python SDK

Unstructured metrics are no longer queriable. Distributions are now queried 
in  the examples.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam distros-qr-py

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2874.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2874


commit 532b594d62b4bd18e39498d4cbe06e06eb136225
Author: Pablo 
Date:   2017-05-03T22:50:15Z

Metrics are queriable from Python SDK




> Distribution metrics should be queriable in the Dataflow Runner
> ---
>
> Key: BEAM-2084
> URL: https://issues.apache.org/jira/browse/BEAM-2084
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2874: [BEAM-2084] Metrics are queriable from Python SDK

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2874

[BEAM-2084] Metrics are queriable from Python SDK

Unstructured metrics are no longer queriable. Distributions are now queried 
in  the examples.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam distros-qr-py

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2874.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2874


commit 532b594d62b4bd18e39498d4cbe06e06eb136225
Author: Pablo 
Date:   2017-05-03T22:50:15Z

Metrics are queriable from Python SDK




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3621

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #3028

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-828) Remove PipelineOptions from construction time in BigQueryIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995834#comment-15995834
 ] 

ASF GitHub Bot commented on BEAM-828:
-

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2873

[BEAM-828] BigQueryIO: Remove tempLocation usage at pipeline construction 
time

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
No semantic changes, just moves tempLocation access to runtime. 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam bq_templocation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2873.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2873


commit 78ed5740ab97cf6daa99fd62c385ef1c7bc10f35
Author: Vikas Kedigehalli 
Date:   2017-05-03T22:42:50Z

BigQueryIO: Remove tempLocation usage at pipeline construction time




> Remove PipelineOptions from construction time in BigQueryIO
> ---
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Vikas Kedigehalli
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2873: [BEAM-828] BigQueryIO: Remove tempLocation usage at...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2873

[BEAM-828] BigQueryIO: Remove tempLocation usage at pipeline construction 
time

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
No semantic changes, just moves tempLocation access to runtime. 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam bq_templocation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2873.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2873


commit 78ed5740ab97cf6daa99fd62c385ef1c7bc10f35
Author: Vikas Kedigehalli 
Date:   2017-05-03T22:42:50Z

BigQueryIO: Remove tempLocation usage at pipeline construction time




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1920) Add Spark 2.x support in Spark runner

[Chaoran Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995822#comment-15995822
 ] 

Chaoran Yu commented on BEAM-1920:
--

The current Spark 1.6.3 runner is compiled with Scala 2.10. As 2.11 is more 
widely used now, are we going to use 2.11 versions of Spark components?

> Add Spark 2.x support in Spark runner
> -
>
> Key: BEAM-1920
> URL: https://issues.apache.org/jira/browse/BEAM-1920
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> I have a branch working with both Spark 1 and Spark 2 backend.
> I'm preparing a pull request about that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-302) Add Scio Scala DSL to Beam

[Chaoran Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995814#comment-15995814
 ] 

Chaoran Yu commented on BEAM-302:
-

Thanks Neville for the information! I tried Spark 1.6.3 but it had Scala 
incompatibilities with Scio. The version of Spark 1.6.3 included in Beam Spark 
runner is compiled using Scala 2.10, but Scio is compiled using Scala 2.11. I 
had to change a few other dependencies to 2.10-compiled version such as 
https://mvnrepository.com/artifact/me.lyh/protobuf-generic_2.10, but still got 
errors. My team and myself will be thinking about contributing to get Spark 
runner to fully work with scio when we can spend more time on the project.

> Add Scio Scala DSL to Beam
> --
>
> Key: BEAM-302
> URL: https://issues.apache.org/jira/browse/BEAM-302
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3619

[Apache Jenkins Server]

See 

[GitHub] beam-site pull request #221: Migrate content from Create Your Pipeline to th...

[hadarhg]

Github user hadarhg closed the pull request at:

https://github.com/apache/beam-site/pull/221


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-828) Remove PipelineOptions from construction time in BigQueryIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995794#comment-15995794
 ] 

ASF GitHub Bot commented on BEAM-828:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2846


> Remove PipelineOptions from construction time in BigQueryIO
> ---
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Vikas Kedigehalli
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #229: Transfer some content from Create Your Pipeline...

[hadarhg]

GitHub user hadarhg opened a pull request:

https://github.com/apache/beam-site/pull/229

Transfer some content from Create Your Pipeline to Programming Guide



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hadarhg/incubator-beam-site cyp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/229.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #229


commit 60dcc42f8d5bbb3a002ca66ceaa4dd2b2bdb526a
Author: Hadar Hod 
Date:   2017-05-03T22:08:40Z

Transfer some content from Create Your Pipeline to the Programming Guide.

commit 3484eb530650e1ffd9c770c90c095c00d0ed1faf
Author: Hadar Hod 
Date:   2017-05-03T22:14:49Z

Remove Running Your Pipeline from TOC in Programming Guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2846: [BEAM-828] BigQueryIO: Remove PipelineOptions.getJo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2846


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Remove job name usages from BigQueryIO at pipeline construction time

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 57f449c4c -> 17f0843eb


Remove job name usages from BigQueryIO at pipeline construction time


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0ddf8d49
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0ddf8d49
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0ddf8d49

Branch: refs/heads/master
Commit: 0ddf8d49d94288e693494ac0685b0c6df78dcd3b
Parents: 57f449c
Author: Vikas Kedigehalli 
Authored: Tue May 2 13:55:32 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 3 15:15:02 2017 -0700

--
 .../sdk/io/gcp/bigquery/BigQueryHelpers.java| 61 -
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 32 +++--
 .../io/gcp/bigquery/BigQueryQuerySource.java| 40 +---
 .../sdk/io/gcp/bigquery/BigQuerySourceBase.java | 24 +++
 .../io/gcp/bigquery/BigQueryTableSource.java| 15 ++---
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 69 +++-
 .../sdk/io/gcp/bigquery/FakeJobService.java |  5 +-
 7 files changed, 94 insertions(+), 152 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0ddf8d49/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
index e04361c..3850cbd 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
@@ -256,15 +256,6 @@ public class BigQueryHelpers {
 }
   }
 
-  @VisibleForTesting
-  static class BeamJobUuidToBigQueryJobUuid
-  implements SerializableFunction {
-@Override
-public String apply(String from) {
-  return "beam_job_" + from;
-}
-  }
-
   static class TableSchemaToJsonSchema
   implements SerializableFunction {
 @Override
@@ -297,14 +288,6 @@ public class BigQueryHelpers {
 }
   }
 
-  static class TableRefToProjectId
-  implements SerializableFunction {
-@Override
-public String apply(TableReference from) {
-  return from.getProjectId();
-}
-  }
-
   @VisibleForTesting
   static class TableSpecToTableRef
   implements SerializableFunction {
@@ -314,39 +297,21 @@ public class BigQueryHelpers {
 }
   }
 
-  @VisibleForTesting
-  static class CreatePerBeamJobUuid
-  implements SerializableFunction {
-private final String stepUuid;
-
-CreatePerBeamJobUuid(String stepUuid) {
-  this.stepUuid = stepUuid;
-}
-
-@Override
-public String apply(String jobUuid) {
-  return stepUuid + "_" + jobUuid.replaceAll("-", "");
-}
+  static String createJobIdToken(String jobName, String stepUuid) {
+return String.format("beam_job_%s_%s", stepUuid, jobName.replaceAll("-", 
""));
   }
 
-  @VisibleForTesting
-  static class CreateJsonTableRefFromUuid
-  implements SerializableFunction {
-private final String executingProject;
-
-CreateJsonTableRefFromUuid(String executingProject) {
-  this.executingProject = executingProject;
-}
+  static String getExtractJobId(String jobIdToken) {
+return String.format("%s-extract", jobIdToken);
+  }
 
-@Override
-public TableReference apply(String jobUuid) {
-  String queryTempDatasetId = "temp_dataset_" + jobUuid;
-  String queryTempTableId = "temp_table_" + jobUuid;
-  TableReference queryTempTableRef = new TableReference()
-  .setProjectId(executingProject)
-  .setDatasetId(queryTempDatasetId)
-  .setTableId(queryTempTableId);
-  return queryTempTableRef;
-}
+  static TableReference createTempTableReference(String projectId, String 
jobUuid) {
+String queryTempDatasetId = "temp_dataset_" + jobUuid;
+String queryTempTableId = "temp_table_" + jobUuid;
+TableReference queryTempTableRef = new TableReference()
+.setProjectId(projectId)
+.setDatasetId(queryTempDatasetId)
+.setTableId(queryTempTableId);
+return queryTempTableRef;
   }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/0ddf8d49/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 

[2/2] beam git commit: This closes #2846: BigQueryIO: Remove PipelineOptions.getJobName usages at pipeline construction time

[kenn]

This closes #2846: BigQueryIO: Remove PipelineOptions.getJobName usages at 
pipeline construction time

  Remove job name usages from BigQueryIO at pipeline construction time


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/17f0843e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/17f0843e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/17f0843e

Branch: refs/heads/master
Commit: 17f0843eba34d5b9adbba523477464d0b0651a3d
Parents: 57f449c 0ddf8d4
Author: Kenneth Knowles 
Authored: Wed May 3 15:15:13 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 3 15:15:13 2017 -0700

--
 .../sdk/io/gcp/bigquery/BigQueryHelpers.java| 61 -
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 32 +++--
 .../io/gcp/bigquery/BigQueryQuerySource.java| 40 +---
 .../sdk/io/gcp/bigquery/BigQuerySourceBase.java | 24 +++
 .../io/gcp/bigquery/BigQueryTableSource.java| 15 ++---
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 69 +++-
 .../sdk/io/gcp/bigquery/FakeJobService.java |  5 +-
 7 files changed, 94 insertions(+), 152 deletions(-)
--

[jira] [Commented] (BEAM-1340) Remove or make private public bits of the SDK that shouldn't be public

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995786#comment-15995786
 ] 

ASF GitHub Bot commented on BEAM-1340:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2872

[BEAM-1340,BEAM-1345] Tighten the Dataflow runner's API surface

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DataflowRunner-surface

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2872.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2872


commit 5a8b617d966427adfd2c6e4e05b1a63ffdb7
Author: Kenneth Knowles 
Date:   2017-05-03T21:30:55Z

Tighten the Dataflow runner's API surface




> Remove or make private public bits of the SDK that shouldn't be public
> --
>
> Key: BEAM-1340
> URL: https://issues.apache.org/jira/browse/BEAM-1340
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-extensions
>Reporter: Kenneth Knowles
>Priority: Blocker
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> This JIRA is for the many small changes that do not merit their own JIRA 
> towards getting the SDK's API surface right. For example, removal of 
> `DoFn.InputProvider` and `DoFn.OutputReceiver`.
> While the above is not quite backwards incompatible, succeeding at this task 
> surely will be.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2872: [BEAM-1340,BEAM-1345] Tighten the Dataflow runner's...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2872

[BEAM-1340,BEAM-1345] Tighten the Dataflow runner's API surface

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DataflowRunner-surface

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2872.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2872


commit 5a8b617d966427adfd2c6e4e05b1a63ffdb7
Author: Kenneth Knowles 
Date:   2017-05-03T21:30:55Z

Tighten the Dataflow runner's API surface




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-2078) add BeamSQL feature branch in site

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin resolved BEAM-2078.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> add BeamSQL feature branch in site
> --
>
> Key: BEAM-2078
> URL: https://issues.apache.org/jira/browse/BEAM-2078
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql, website
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
> Fix For: Not applicable
>
>
> Add {{dsl_sql}} feature branch to page 
> 'https://beam.apache.org/contribute/work-in-progress/', to track the status.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2871: Split Coder's encode/decode methods into two method...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/2871

Split Coder's encode/decode methods into two methods depending on con…

…text.

This allows the outer context to be marked deprecated.  A follow-up PR will
remove the old method once all consumers have been updated.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam nested-coders

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2871.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2871


commit f9be62f224000f0777375a15548c02e96344f83f
Author: Robert Bradshaw 
Date:   2017-05-03T21:56:37Z

Split Coder's encode/decode methods into two methods depending on context.

This allows the outer context to be marked deprecated.  A follow-up PR will
remove the old method once all consumers have been updated.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #226: [BEAM-2078] add BeamSQL feature branch in site

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/226


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam-site git commit: This closes #226

[davor]

This closes #226


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/7b3e24f3
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/7b3e24f3
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/7b3e24f3

Branch: refs/heads/asf-site
Commit: 7b3e24f3b014dea09521889a61f4e51357b898d5
Parents: d7f4684 b4f5243
Author: Davor Bonaci 
Authored: Wed May 3 14:54:31 2017 -0700
Committer: Davor Bonaci 
Committed: Wed May 3 14:54:31 2017 -0700

--
 content/contribute/work-in-progress/index.html | 6 ++
 src/contribute/work-in-progress.md | 1 +
 2 files changed, 7 insertions(+)
--

[1/3] beam-site git commit: add SQL_DSL to page 'work-in-progress'

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site d7f468491 -> 7b3e24f3b


add SQL_DSL to page 'work-in-progress'


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/7f914af7
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/7f914af7
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/7f914af7

Branch: refs/heads/asf-site
Commit: 7f914af744307d69e88ec4d51408ec14b077238c
Parents: d7f4684
Author: mingmxu 
Authored: Sat Apr 29 19:11:45 2017 -0700
Committer: Davor Bonaci 
Committed: Wed May 3 14:52:12 2017 -0700

--
 src/contribute/work-in-progress.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/7f914af7/src/contribute/work-in-progress.md
--
diff --git a/src/contribute/work-in-progress.md 
b/src/contribute/work-in-progress.md
index 224dddb..979e458 100644
--- a/src/contribute/work-in-progress.md
+++ b/src/contribute/work-in-progress.md
@@ -26,5 +26,6 @@ Current branches include:
 |  |  |  |  |
 | Apache Gearpump Runner | 
[gearpump-runner](https://github.com/apache/beam/tree/gearpump-runner) | 
[runner-gearpump](https://issues.apache.org/jira/browse/BEAM/component/12330829)
 | [runner homepage]({{ site.baseurl }}/documentation/runners/gearpump/) |
 | Apache Spark 2.0 Runner | 
[runners-spark2](https://github.com/apache/beam/tree/runners-spark2) | - | 
[thread](https://lists.apache.org/thread.html/e38ac4e4914a6cb1b865b1f32a6ca06c2be28ea4aa0f6b18393de66f@%3Cdev.beam.apache.org%3E)
 |
+| Beam SQL DSL | [DSL_SQL](https://github.com/apache/beam/tree/DSL_SQL) | 
[dsl-sql](https://issues.apache.org/jira/browse/BEAM/component/12332480) | 
[BEAM-301](https://issues.apache.org/jira/browse/BEAM-301) |
 {:.table}
 

[jira] [Commented] (BEAM-2078) add BeamSQL feature branch in site

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995754#comment-15995754
 ] 

ASF GitHub Bot commented on BEAM-2078:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/226


> add BeamSQL feature branch in site
> --
>
> Key: BEAM-2078
> URL: https://issues.apache.org/jira/browse/BEAM-2078
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql, website
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> Add {{dsl_sql}} feature branch to page 
> 'https://beam.apache.org/contribute/work-in-progress/', to track the status.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/b4f5243c
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/b4f5243c
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/b4f5243c

Branch: refs/heads/asf-site
Commit: b4f5243c2329b6c1f130f20eee68a440f05ae722
Parents: 7f914af
Author: Davor Bonaci 
Authored: Wed May 3 14:54:30 2017 -0700
Committer: Davor Bonaci 
Committed: Wed May 3 14:54:30 2017 -0700

--
 content/contribute/work-in-progress/index.html | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/b4f5243c/content/contribute/work-in-progress/index.html
--
diff --git a/content/contribute/work-in-progress/index.html 
b/content/contribute/work-in-progress/index.html
index 07abdf1..215f9db 100644
--- a/content/contribute/work-in-progress/index.html
+++ b/content/contribute/work-in-progress/index.html
@@ -192,6 +192,12 @@
   -
   https://lists.apache.org/thread.html/e38ac4e4914a6cb1b865b1f32a6ca06c2be28ea4aa0f6b18393de66f@%3Cdev.beam.apache.org%3E;>thread
 
+
+  Beam SQL DSL
+  https://github.com/apache/beam/tree/DSL_SQL;>DSL_SQL
+  https://issues.apache.org/jira/browse/BEAM/component/12332480;>dsl-sql
+  https://issues.apache.org/jira/browse/BEAM-301;>BEAM-301
+
   
 
 

[GitHub] beam pull request #2870: DataflowRunner: integration test GCP-IO

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2870

DataflowRunner: integration test GCP-IO

Triggered under `-DskipITs=false -Pdataflow-runner`

R: @davorbonaci 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam dataflow-its

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2870.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2870


commit cc340318b4c665d42c83842f9d935da51edc485e
Author: Dan Halperin 
Date:   2017-05-03T18:52:02Z

DataflowRunner: integration test GCP-IO

Triggered under `-DskipITs=false -Pdataflow-runner`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1642) Combine transformation evaluation fails on direct runner with Avro as a fallback coder

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995736#comment-15995736
 ] 

ASF GitHub Bot commented on BEAM-1642:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2771


> Combine transformation evaluation fails on direct runner with Avro as a 
> fallback coder
> --
>
> Key: BEAM-1642
> URL: https://issues.apache.org/jira/browse/BEAM-1642
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 0.4.0, 0.5.0, 0.6.0
>Reporter: Sathya Hariesh
>Assignee: Robert Bradshaw
>Priority: Trivial
> Fix For: First stable release
>
>
> This does not happen in 0.3.0. Would probably need to investigate the root 
> cause.
> Steps to reproduce:
> public class WordCount {
> public static void main(String[] args) {
> DirectOptions directPipelineOptions = 
> PipelineOptionsFactory.as(DirectOptions.class);
> directPipelineOptions.setRunner(DirectRunner.class);
> Pipeline pipeline = Pipeline.create(directPipelineOptions);
> 
> pipeline.getCoderRegistry().setFallbackCoderProvider(AvroCoder.PROVIDER);
> pipeline.apply(Create.of(Arrays.asList(100, 
> 200))).apply(Count.globally());
> pipeline.run().waitUntilFinish();
> }
> }
> Stack trace:
> Exception in thread "main" java.lang.ClassCastException: java.lang.Object 
> cannot be cast to java.lang.Iterable
>   at 
> org.apache.beam.runners.direct.ViewEvaluatorFactory$1.processElement(ViewEvaluatorFactory.java:81)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/3] beam git commit: Closes #2771

[robertwb]

Closes #2771


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/57f449c4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/57f449c4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/57f449c4

Branch: refs/heads/master
Commit: 57f449c4c41ad96cd30336dd55875efbdf21c4e6
Parents: 9b6b906 3e6b02a
Author: Robert Bradshaw 
Authored: Wed May 3 14:36:04 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed May 3 14:36:04 2017 -0700

--
 .../beam/runners/direct/DirectRunnerTest.java   | 10 ++
 .../org/apache/beam/sdk/values/TypedPValue.java | 20 ++--
 .../beam/sdk/transforms/CombineFnsTest.java |  2 +-
 3 files changed, 21 insertions(+), 11 deletions(-)
--

[GitHub] beam pull request #2771: [BEAM-1642] Try PTransform-based coder inference be...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2771


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam git commit: Try PTransform-based coder inference before using fallback coder.

[robertwb]

Try PTransform-based coder inference before using fallback coder.

This is particularly important for fallback coders that claim
to provide a coder for Object (or equivalently an unconstrained
type parameter).  See BEAM-1642.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/289d2dec
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/289d2dec
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/289d2dec

Branch: refs/heads/master
Commit: 289d2decba57ff990f44567d1a16c189cfbd1cc8
Parents: 9b6b906
Author: Robert Bradshaw 
Authored: Fri Apr 28 15:53:42 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed May 3 14:36:03 2017 -0700

--
 .../beam/runners/direct/DirectRunnerTest.java   | 10 ++
 .../org/apache/beam/sdk/values/TypedPValue.java | 20 ++--
 2 files changed, 20 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/289d2dec/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
index 6f9adc4..83881fc 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
@@ -578,4 +578,14 @@ public class DirectRunnerTest implements Serializable {
   return underlying.getDefaultOutputCoder();
 }
   }
+
+  @Test
+  public void fallbackCoderProviderAllowsInference() {
+// See https://issues.apache.org/jira/browse/BEAM-1642
+Pipeline p = getPipeline();
+p.getCoderRegistry().setFallbackCoderProvider(
+org.apache.beam.sdk.coders.AvroCoder.PROVIDER);
+p.apply(Create.of(Arrays.asList(100, 
200))).apply(Count.globally());
+p.run().waitUntilFinish();
+  }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/289d2dec/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypedPValue.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypedPValue.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypedPValue.java
index 54af747..f473776 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypedPValue.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypedPValue.java
@@ -140,7 +140,16 @@ public abstract class TypedPValue extends PValueBase 
implements PValue {
   return coderOrFailure;
 }
 
-// Second option for a coder: Look in the coder registry.
+// Second option for a coder: use the default Coder from the producing 
PTransform.
+CannotProvideCoderException inputCoderException;
+try {
+  return new CoderOrFailure<>(
+  ((PTransform) transform).getDefaultOutputCoder(input, this), null);
+} catch (CannotProvideCoderException exc) {
+  inputCoderException = exc;
+}
+
+// Third option for a coder: Look in the coder registry.
 TypeDescriptor token = getTypeDescriptor();
 CannotProvideCoderException inferFromTokenException = null;
 if (token != null) {
@@ -162,15 +171,6 @@ public abstract class TypedPValue extends PValueBase 
implements PValue {
   }
 }
 
-// Third option for a coder: use the default Coder from the producing 
PTransform.
-CannotProvideCoderException inputCoderException;
-try {
-  return new CoderOrFailure<>(
-  ((PTransform) transform).getDefaultOutputCoder(input, this), null);
-} catch (CannotProvideCoderException exc) {
-  inputCoderException = exc;
-}
-
 // Build up the error message and list of causes.
 StringBuilder messageBuilder = new StringBuilder()
 .append("Unable to return a default Coder for ").append(this)

[1/3] beam git commit: Fix coder inference ambiguity in CombineFnsTest.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 9b6b9060b -> 57f449c4c


Fix coder inference ambiguity in CombineFnsTest.

String.class was being encoded with both StringUtf8Coder.of() and
NullableCoder.of(UserStringCoder.of()) in the same transform,
and the wrong one was being chosen.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3e6b02a6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3e6b02a6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3e6b02a6

Branch: refs/heads/master
Commit: 3e6b02a6649a0bc93ee8086eea07033dbb58492e
Parents: 289d2de
Author: Robert Bradshaw 
Authored: Wed May 3 12:09:24 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed May 3 14:36:03 2017 -0700

--
 .../test/java/org/apache/beam/sdk/transforms/CombineFnsTest.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3e6b02a6/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineFnsTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineFnsTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineFnsTest.java
index dcb8fdc..9250dfa 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineFnsTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineFnsTest.java
@@ -231,7 +231,7 @@ public class  CombineFnsTest {
 KV.of("b", KV.of(13, UserString.of("13",
 Arrays.asList(0L, 4L, 7L, 10L, 16L))
 .withCoder(KvCoder.of(
-StringUtf8Coder.of(),
+NullableCoder.of(StringUtf8Coder.of()),
 KvCoder.of(
 BigEndianIntegerCoder.of(), 
NullableCoder.of(UserStringCoder.of());
 

[jira] [Resolved] (BEAM-1902) Datastore IO never retries on errors

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1902.
---
   Resolution: Fixed
Fix Version/s: First stable release

> Datastore IO never retries on errors
> 
>
> Key: BEAM-1902
> URL: https://issues.apache.org/jira/browse/BEAM-1902
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Vikas Kedigehalli
> Fix For: First stable release
>
>
> {{retry_on_rpc_error}} assumes that the {{exception.code}} are HTTP errors 
> and tries to retry on error codes {{>= 500}}. However these are rpc errors 
> and all are < 500
> Existing rpc error codes: 
> https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1902) Datastore IO never retries on errors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995729#comment-15995729
 ] 

ASF GitHub Bot commented on BEAM-1902:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2865


> Datastore IO never retries on errors
> 
>
> Key: BEAM-1902
> URL: https://issues.apache.org/jira/browse/BEAM-1902
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Vikas Kedigehalli
>
> {{retry_on_rpc_error}} assumes that the {{exception.code}} are HTTP errors 
> and tries to retry on error codes {{>= 500}}. However these are rpc errors 
> and all are < 500
> Existing rpc error codes: 
> https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2865: [BEAM-1902] Retry on correct error codes for datast...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2865


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2865

[altay]

This closes #2865


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9b6b9060
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9b6b9060
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9b6b9060

Branch: refs/heads/master
Commit: 9b6b9060b1478073587eb24a134be56d93d69a57
Parents: f55d002 d8e2522
Author: Ahmet Altay 
Authored: Wed May 3 14:33:01 2017 -0700
Committer: Ahmet Altay 
Committed: Wed May 3 14:33:01 2017 -0700

--
 .../apache_beam/io/gcp/datastore/v1/helper.py   | 16 -
 .../io/gcp/datastore/v1/helper_test.py  | 24 +++-
 2 files changed, 28 insertions(+), 12 deletions(-)
--

[1/2] beam git commit: Retry on correct error codes for datastoreio

[altay]

Repository: beam
Updated Branches:
  refs/heads/master f55d00253 -> 9b6b9060b


Retry on correct error codes for datastoreio


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d8e2522e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d8e2522e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d8e2522e

Branch: refs/heads/master
Commit: d8e2522eb04a2a0b5cb28415e55d467d8905d841
Parents: f55d002
Author: Vikas Kedigehalli 
Authored: Wed May 3 13:14:20 2017 -0700
Committer: Ahmet Altay 
Committed: Wed May 3 14:32:59 2017 -0700

--
 .../apache_beam/io/gcp/datastore/v1/helper.py   | 16 -
 .../io/gcp/datastore/v1/helper_test.py  | 24 +++-
 2 files changed, 28 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d8e2522e/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
--
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py 
b/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
index d544226..a61884f 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
@@ -24,13 +24,13 @@ try:
   from google.cloud.proto.datastore.v1 import datastore_pb2
   from google.cloud.proto.datastore.v1 import entity_pb2
   from google.cloud.proto.datastore.v1 import query_pb2
+  from google.rpc import code_pb2
   from googledatastore import PropertyFilter, CompositeFilter
   from googledatastore import helper as datastore_helper
   from googledatastore.connection import Datastore
   from googledatastore.connection import RPCError
-  QUERY_NOT_FINISHED = query_pb2.QueryResultBatch.NOT_FINISHED
 except ImportError:
-  QUERY_NOT_FINISHED = None
+  pass
 # pylint: enable=wrong-import-order, wrong-import-position
 
 from apache_beam.internal.gcp import auth
@@ -129,8 +129,12 @@ def make_partition(project, namespace):
 def retry_on_rpc_error(exception):
   """A retry filter for Cloud Datastore RPCErrors."""
   if isinstance(exception, RPCError):
-return exception.code >= 500
-  # TODO(vikasrk): Figure out what other errors should be retried.
+err_code = exception.code
+# TODO(BEAM-2156): put these codes in a global list and use that instead.
+return (err_code == code_pb2.DEADLINE_EXCEEDED or
+err_code == code_pb2.UNAVAILABLE or
+err_code == code_pb2.UNKNOWN or
+err_code == code_pb2.INTERNAL)
   return False
 
 
@@ -221,7 +225,6 @@ class QueryIterator(object):
 
   Entities are read in batches. Retries on failures.
   """
-  _NOT_FINISHED = QUERY_NOT_FINISHED
   # Maximum number of results to request per query.
   _BATCH_SIZE = 500
 
@@ -265,4 +268,5 @@ class QueryIterator(object):
   # read).
   more_results = ((self._limit > 0) and
   ((num_results == self._BATCH_SIZE) or
-   (resp.batch.more_results == self._NOT_FINISHED)))
+   (resp.batch.more_results ==
+query_pb2.QueryResultBatch.NOT_FINISHED)))

http://git-wip-us.apache.org/repos/asf/beam/blob/d8e2522e/sdks/python/apache_beam/io/gcp/datastore/v1/helper_test.py
--
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/helper_test.py 
b/sdks/python/apache_beam/io/gcp/datastore/v1/helper_test.py
index 582a5b3..5d4bb6f 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/helper_test.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/helper_test.py
@@ -33,6 +33,7 @@ try:
   from google.cloud.proto.datastore.v1 import entity_pb2
   from google.cloud.proto.datastore.v1 import query_pb2
   from google.cloud.proto.datastore.v1.entity_pb2 import Key
+  from google.rpc import code_pb2
   from googledatastore.connection import RPCError
   from googledatastore import helper as datastore_helper
 except ImportError:
@@ -49,19 +50,22 @@ class HelperTest(unittest.TestCase):
 self._query.kind.add().name = 'dummy_kind'
 patch_retry(self, helper)
 
-  def permanent_datastore_failure(self, req):
-raise RPCError("dummy", 500, "failed")
+  def permanent_retriable_datastore_failure(self, req):
+raise RPCError("dummy", code_pb2.UNAVAILABLE, "failed")
 
-  def transient_datastore_failure(self, req):
+  def transient_retriable_datastore_failure(self, req):
 if self._transient_fail_count:
   self._transient_fail_count -= 1
-  raise RPCError("dummy", 500, "failed")
+  raise RPCError("dummy", code_pb2.INTERNAL, "failed")
 else:
   return datastore_pb2.RunQueryResponse()
 
+  def non_retriable_datastore_failure(self, req):
+raise RPCError("dummy", code_pb2.UNAUTHENTICATED, "failed")
+
   

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2642

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #2102

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-1871] Move GCP specific serialization CloudObject and supporting

--
[...truncated 585.04 KB...]
{
  "kind": "ParallelDo", 
  "name": "s13", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert_that/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s15", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3027

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2076) DirectRunner: minimal transitive API surface

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995695#comment-15995695
 ] 

ASF GitHub Bot commented on BEAM-2076:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2863


> DirectRunner: minimal transitive API surface
> 
>
> Key: BEAM-2076
> URL: https://issues.apache.org/jira/browse/BEAM-2076
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Daniel Halperin
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> The {{DirectRunner}} is likely to accidentally be on many users' classpath 
> when they are running on other runners. As such, it should have a minimal 
> transitive API surface, shading things it needs directly and need not expose.
> My base assumption is that {{runners-core}} should be shaded. There may be 
> others tho -- this merits a bit of a look.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2863: [BEAM-2076] Reduce visibility of DirectRunner class...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2863


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam git commit: Reduce visibility of DirectRunner classes

[tgroh]

http://git-wip-us.apache.org/repos/asf/beam/blob/9edd8599/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
index fc63406..67f2fd5 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
@@ -40,8 +40,6 @@ import org.apache.beam.runners.core.StateNamespace;
 import org.apache.beam.runners.core.StateNamespaces;
 import org.apache.beam.runners.core.StateTag;
 import org.apache.beam.runners.core.StateTags;
-import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
-import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
 import org.apache.beam.runners.direct.ParDoMultiOverrideFactory.StatefulParDo;
 import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
 import org.apache.beam.sdk.coders.StringUtf8Coder;

http://git-wip-us.apache.org/repos/asf/beam/blob/9edd8599/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StepTransformResultTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StepTransformResultTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StepTransformResultTest.java
index 0d94b7a..45ae678 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StepTransformResultTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StepTransformResultTest.java
@@ -24,7 +24,6 @@ import static org.hamcrest.Matchers.hasItem;
 import static org.junit.Assert.assertThat;
 
 import org.apache.beam.runners.direct.CommittedResult.OutputType;
-import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.Create;

http://git-wip-us.apache.org/repos/asf/beam/blob/9edd8599/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
index b9c6e64..d1712e0 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
@@ -26,7 +26,6 @@ import static org.mockito.Mockito.when;
 import com.google.common.collect.Iterables;
 import java.util.Collection;
 import java.util.Collections;
-import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import 
org.apache.beam.runners.direct.TestStreamEvaluatorFactory.DirectTestStreamFactory.DirectTestStream;
 import org.apache.beam.runners.direct.TestStreamEvaluatorFactory.TestClock;
 import 
org.apache.beam.runners.direct.TestStreamEvaluatorFactory.TestStreamIndex;

http://git-wip-us.apache.org/repos/asf/beam/blob/9edd8599/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TransformExecutorTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TransformExecutorTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TransformExecutorTest.java
index e66ffcf..1f95327 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TransformExecutorTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TransformExecutorTest.java
@@ -36,7 +36,6 @@ import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
 import java.util.concurrent.atomic.AtomicBoolean;
 import org.apache.beam.runners.direct.CommittedResult.OutputType;
-import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.Create;

http://git-wip-us.apache.org/repos/asf/beam/blob/9edd8599/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
 

[3/3] beam git commit: Reduce visibility of DirectRunner classes

[tgroh]

Reduce visibility of DirectRunner classes

Move inner classes of the DirectRunner to reduce total API Surface.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9edd8599
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9edd8599
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9edd8599

Branch: refs/heads/master
Commit: 9edd8599c28228cd5d7d5df1084f7e63684964d8
Parents: b263cb7
Author: Thomas Groh 
Authored: Wed Apr 26 13:38:37 2017 -0700
Committer: Thomas Groh 
Committed: Wed May 3 14:12:05 2017 -0700

--
 .../direct/AbstractModelEnforcement.java|   1 -
 .../direct/BoundedReadEvaluatorFactory.java |   2 -
 .../beam/runners/direct/BundleFactory.java  |   3 +-
 .../org/apache/beam/runners/direct/Clock.java   |   2 +-
 .../runners/direct/CloningBundleFactory.java|   2 -
 .../beam/runners/direct/CommittedBundle.java|  82 +
 .../beam/runners/direct/CommittedResult.java|   1 -
 .../beam/runners/direct/CompletionCallback.java |   1 -
 .../CopyOnAccessInMemoryStateInternals.java |   2 +-
 .../apache/beam/runners/direct/DirectGraph.java |  12 +-
 .../beam/runners/direct/DirectMetrics.java  |   1 -
 .../beam/runners/direct/DirectRunner.java   | 122 ++-
 .../beam/runners/direct/EmptyInputProvider.java |   1 -
 .../runners/direct/EmptyTransformEvaluator.java |  50 
 .../beam/runners/direct/EvaluationContext.java  |   3 -
 .../runners/direct/ExecutorServiceFactory.java  |   2 +-
 .../direct/ExecutorServiceParallelExecutor.java |   1 -
 .../runners/direct/FlattenEvaluatorFactory.java |   2 -
 .../GroupAlsoByWindowEvaluatorFactory.java  |   2 -
 .../direct/GroupByKeyOnlyEvaluatorFactory.java  |   2 -
 .../ImmutabilityCheckingBundleFactory.java  |   2 -
 .../direct/ImmutabilityEnforcementFactory.java  |   1 -
 .../direct/ImmutableListBundleFactory.java  |   2 -
 .../beam/runners/direct/ModelEnforcement.java   |  10 +-
 .../runners/direct/ModelEnforcementFactory.java |   3 +-
 .../beam/runners/direct/NanosOffsetClock.java   |   2 +-
 .../runners/direct/PCollectionViewWriter.java   |  34 ++
 .../beam/runners/direct/ParDoEvaluator.java |   1 -
 .../runners/direct/ParDoEvaluatorFactory.java   |   1 -
 .../direct/PassthroughTransformEvaluator.java   |   1 -
 .../beam/runners/direct/PipelineExecutor.java   |   1 -
 .../beam/runners/direct/RootInputProvider.java  |   1 -
 .../runners/direct/RootProviderRegistry.java|   1 -
 ...littableProcessElementsEvaluatorFactory.java |   1 -
 .../direct/StatefulParDoEvaluatorFactory.java   |   1 -
 .../runners/direct/StepTransformResult.java |   3 +-
 .../direct/TestStreamEvaluatorFactory.java  |   2 -
 .../beam/runners/direct/TransformEvaluator.java |   3 +-
 .../direct/TransformEvaluatorFactory.java   |   6 +-
 .../direct/TransformEvaluatorRegistry.java  |   1 -
 .../beam/runners/direct/TransformExecutor.java  |   1 -
 .../beam/runners/direct/TransformResult.java|   3 +-
 .../direct/UnboundedReadEvaluatorFactory.java   |   2 -
 .../beam/runners/direct/UncommittedBundle.java  |  57 +
 .../runners/direct/ViewEvaluatorFactory.java|   3 +-
 .../beam/runners/direct/WatermarkManager.java   |   3 +-
 .../runners/direct/WindowEvaluatorFactory.java  |   2 -
 .../direct/BoundedReadEvaluatorFactoryTest.java |   2 -
 .../direct/CloningBundleFactoryTest.java|   2 -
 .../runners/direct/CommittedResultTest.java |  12 +-
 .../beam/runners/direct/DirectMetricsTest.java  |   1 -
 .../runners/direct/EvaluationContextTest.java   |   3 -
 .../direct/FlattenEvaluatorFactoryTest.java |   2 -
 .../direct/GroupByKeyEvaluatorFactoryTest.java  |   2 -
 .../GroupByKeyOnlyEvaluatorFactoryTest.java |   2 -
 .../ImmutabilityCheckingBundleFactoryTest.java  |   2 -
 .../ImmutabilityEnforcementFactoryTest.java |   1 -
 .../direct/ImmutableListBundleFactoryTest.java  |   2 -
 .../beam/runners/direct/ParDoEvaluatorTest.java |   1 -
 .../StatefulParDoEvaluatorFactoryTest.java  |   2 -
 .../runners/direct/StepTransformResultTest.java |   1 -
 .../direct/TestStreamEvaluatorFactoryTest.java  |   1 -
 .../runners/direct/TransformExecutorTest.java   |   1 -
 .../UnboundedReadEvaluatorFactoryTest.java  |   2 -
 .../direct/ViewEvaluatorFactoryTest.java|   2 -
 .../runners/direct/WatermarkManagerTest.java|   2 -
 .../direct/WindowEvaluatorFactoryTest.java  |   2 -
 67 files changed, 214 insertions(+), 273 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9edd8599/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AbstractModelEnforcement.java
--
diff --git 

[1/3] beam git commit: This closes #2863

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master b263cb776 -> f55d00253


This closes #2863


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f55d0025
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f55d0025
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f55d0025

Branch: refs/heads/master
Commit: f55d00253ef63adb96f96b0491b79d1024b905b6
Parents: b263cb7 9edd859
Author: Thomas Groh 
Authored: Wed May 3 14:12:05 2017 -0700
Committer: Thomas Groh 
Committed: Wed May 3 14:12:05 2017 -0700

--
 .../direct/AbstractModelEnforcement.java|   1 -
 .../direct/BoundedReadEvaluatorFactory.java |   2 -
 .../beam/runners/direct/BundleFactory.java  |   3 +-
 .../org/apache/beam/runners/direct/Clock.java   |   2 +-
 .../runners/direct/CloningBundleFactory.java|   2 -
 .../beam/runners/direct/CommittedBundle.java|  82 +
 .../beam/runners/direct/CommittedResult.java|   1 -
 .../beam/runners/direct/CompletionCallback.java |   1 -
 .../CopyOnAccessInMemoryStateInternals.java |   2 +-
 .../apache/beam/runners/direct/DirectGraph.java |  12 +-
 .../beam/runners/direct/DirectMetrics.java  |   1 -
 .../beam/runners/direct/DirectRunner.java   | 122 ++-
 .../beam/runners/direct/EmptyInputProvider.java |   1 -
 .../runners/direct/EmptyTransformEvaluator.java |  50 
 .../beam/runners/direct/EvaluationContext.java  |   3 -
 .../runners/direct/ExecutorServiceFactory.java  |   2 +-
 .../direct/ExecutorServiceParallelExecutor.java |   1 -
 .../runners/direct/FlattenEvaluatorFactory.java |   2 -
 .../GroupAlsoByWindowEvaluatorFactory.java  |   2 -
 .../direct/GroupByKeyOnlyEvaluatorFactory.java  |   2 -
 .../ImmutabilityCheckingBundleFactory.java  |   2 -
 .../direct/ImmutabilityEnforcementFactory.java  |   1 -
 .../direct/ImmutableListBundleFactory.java  |   2 -
 .../beam/runners/direct/ModelEnforcement.java   |  10 +-
 .../runners/direct/ModelEnforcementFactory.java |   3 +-
 .../beam/runners/direct/NanosOffsetClock.java   |   2 +-
 .../runners/direct/PCollectionViewWriter.java   |  34 ++
 .../beam/runners/direct/ParDoEvaluator.java |   1 -
 .../runners/direct/ParDoEvaluatorFactory.java   |   1 -
 .../direct/PassthroughTransformEvaluator.java   |   1 -
 .../beam/runners/direct/PipelineExecutor.java   |   1 -
 .../beam/runners/direct/RootInputProvider.java  |   1 -
 .../runners/direct/RootProviderRegistry.java|   1 -
 ...littableProcessElementsEvaluatorFactory.java |   1 -
 .../direct/StatefulParDoEvaluatorFactory.java   |   1 -
 .../runners/direct/StepTransformResult.java |   3 +-
 .../direct/TestStreamEvaluatorFactory.java  |   2 -
 .../beam/runners/direct/TransformEvaluator.java |   3 +-
 .../direct/TransformEvaluatorFactory.java   |   6 +-
 .../direct/TransformEvaluatorRegistry.java  |   1 -
 .../beam/runners/direct/TransformExecutor.java  |   1 -
 .../beam/runners/direct/TransformResult.java|   3 +-
 .../direct/UnboundedReadEvaluatorFactory.java   |   2 -
 .../beam/runners/direct/UncommittedBundle.java  |  57 +
 .../runners/direct/ViewEvaluatorFactory.java|   3 +-
 .../beam/runners/direct/WatermarkManager.java   |   3 +-
 .../runners/direct/WindowEvaluatorFactory.java  |   2 -
 .../direct/BoundedReadEvaluatorFactoryTest.java |   2 -
 .../direct/CloningBundleFactoryTest.java|   2 -
 .../runners/direct/CommittedResultTest.java |  12 +-
 .../beam/runners/direct/DirectMetricsTest.java  |   1 -
 .../runners/direct/EvaluationContextTest.java   |   3 -
 .../direct/FlattenEvaluatorFactoryTest.java |   2 -
 .../direct/GroupByKeyEvaluatorFactoryTest.java  |   2 -
 .../GroupByKeyOnlyEvaluatorFactoryTest.java |   2 -
 .../ImmutabilityCheckingBundleFactoryTest.java  |   2 -
 .../ImmutabilityEnforcementFactoryTest.java |   1 -
 .../direct/ImmutableListBundleFactoryTest.java  |   2 -
 .../beam/runners/direct/ParDoEvaluatorTest.java |   1 -
 .../StatefulParDoEvaluatorFactoryTest.java  |   2 -
 .../runners/direct/StepTransformResultTest.java |   1 -
 .../direct/TestStreamEvaluatorFactoryTest.java  |   1 -
 .../runners/direct/TransformExecutorTest.java   |   1 -
 .../UnboundedReadEvaluatorFactoryTest.java  |   2 -
 .../direct/ViewEvaluatorFactoryTest.java|   2 -
 .../runners/direct/WatermarkManagerTest.java|   2 -
 .../direct/WindowEvaluatorFactoryTest.java  |   2 -
 67 files changed, 214 insertions(+), 273 deletions(-)
--

[jira] [Commented] (BEAM-2136) AvroCoderTest.testTwoClassLoaders fails on beam_PostCommit_Java_ValidatesRunner_Dataflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995689#comment-15995689
 ] 

ASF GitHub Bot commented on BEAM-2136:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2861


> AvroCoderTest.testTwoClassLoaders fails on 
> beam_PostCommit_Java_ValidatesRunner_Dataflow
> 
>
> Key: BEAM-2136
> URL: https://issues.apache.org/jira/browse/BEAM-2136
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Kenneth Knowles
>Priority: Minor
>
> Example failure:
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/3003/org.apache.beam$beam-sdks-java-core/testReport/org.apache.beam.sdk.coders/AvroCoderTest/testTwoClassLoaders/
> java.lang.NullPointerException
>   at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:165)
>   at 
> org.apache.beam.sdk.coders.AvroCoderTest$InterceptingUrlClassLoader.loadClass(AvroCoderTest.java:179)
>   at 
> org.apache.beam.sdk.coders.AvroCoderTest.testTwoClassLoaders(AvroCoderTest.java:199)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:307)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runners.Suite.runChild(Suite.java:128)
>   at org.junit.runners.Suite.runChild(Suite.java:27)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:202)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:158)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:81)
>   at 
> org.apache.maven.plugin.surefire.InPluginVMSurefireStarter.runSuitesInProcess(InPluginVMSurefireStarter.java:84)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1060)
>   at 
> 

[jira] [Commented] (BEAM-2152) Authentication fails if there is an unauthenticated gcloud tool even if application default credentials are available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995688#comment-15995688
 ] 

ASF GitHub Bot commented on BEAM-2152:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2869

[BEAM-2152] Remove gcloud auth as application default credentials manages it


Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-2152-remove-gcloud-auth

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2869.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2869


commit 6d0db5f5bf6e1256b2923e56091a028b1251ad37
Author: Sourabh Bajaj 
Date:   2017-05-03T20:50:46Z

[BEAM-2152] Remove gcloud auth as application default credentials does it




> Authentication fails if there is an unauthenticated gcloud tool even if 
> application default credentials are available
> -
>
> Key: BEAM-2152
> URL: https://issues.apache.org/jira/browse/BEAM-2152
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>
> In a machine that has valid application default credentials, if {{gcloud}} 
> tool is not installed authentication works. If {{gcloud}} tool is recently 
> installed but has not authenticated yet authentication fails with {{You do 
> not currently have an active account selected.}}
> Authentication code should fallback to default method in this case. (Or the 
> {{gcloud}} based authentication needs to be fully removed.
> cc: [~lcwik] 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2869: [BEAM-2152] Remove gcloud auth as application defau...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2869

[BEAM-2152] Remove gcloud auth as application default credentials manages it


Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-2152-remove-gcloud-auth

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2869.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2869


commit 6d0db5f5bf6e1256b2923e56091a028b1251ad37
Author: Sourabh Bajaj 
Date:   2017-05-03T20:50:46Z

[BEAM-2152] Remove gcloud auth as application default credentials does it




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-969) Add a gearpump runner web page under "learn/runners"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995678#comment-15995678
 ] 

ASF GitHub Bot commented on BEAM-969:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/223


> Add a gearpump runner web page under "learn/runners"
> 
>
> Key: BEAM-969
> URL: https://issues.apache.org/jira/browse/BEAM-969
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-gearpump, website
>Reporter: Manu Zhang
>Assignee: Huafeng Wang
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/4] beam-site git commit: This closes #223: Add a gearpump runner web page

[kenn]

This closes #223: Add a gearpump runner web page


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/94f50512
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/94f50512
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/94f50512

Branch: refs/heads/asf-site
Commit: 94f505129854c9ccf92d418d5b1f6919e86c014f
Parents: 3cafa86 c717c8b
Author: Kenneth Knowles 
Authored: Wed May 3 13:59:16 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 3 13:59:16 2017 -0700

--
 src/_data/capability-matrix.yml   | 124 ++---
 src/contribute/work-in-progress.md|   2 +-
 src/documentation/runners/gearpump.md | 141 +
 src/get-started/beam-overview.md  |   8 +-
 src/images/logos/runners/gearpump.png | Bin 0 -> 2643 bytes
 5 files changed, 261 insertions(+), 14 deletions(-)
--

[GitHub] beam-site pull request #223: [BEAM-969] Add a gearpump runner web page

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/223


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/4] beam-site git commit: Minor touch-ups to Gearpump listing on overview

[kenn]

Minor touch-ups to Gearpump listing on overview

 - Alphabetical order
 - Add a space


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/da7d6c0e
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/da7d6c0e
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/da7d6c0e

Branch: refs/heads/asf-site
Commit: da7d6c0e61282fb2261131614eb633e9cafee689
Parents: 94f5051
Author: Kenneth Knowles 
Authored: Wed May 3 13:59:48 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 3 13:59:48 2017 -0700

--
 src/get-started/beam-overview.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/da7d6c0e/src/get-started/beam-overview.md
--
diff --git a/src/get-started/beam-overview.md b/src/get-started/beam-overview.md
index 07b1ad9..491f901 100644
--- a/src/get-started/beam-overview.md
+++ b/src/get-started/beam-overview.md
@@ -34,10 +34,10 @@ Beam currently supports Runners that work with the 
following distributed process
  alt="Apache Apex">
 * Apache Flink 
+* Apache Gearpump (incubating) 
 * Apache Spark 
-* Apache Gearpump(incubating) 
 * Google Cloud Dataflow 
 
@@ -59,4 +59,4 @@ Get started using Beam for your data processing tasks.
 
 ## Contribute
 
-Beam is an [Apache Software Foundation](http://www.apache.org) project, 
available under the Apache v2 license. Beam is an open source community and 
contributions are greatly appreciated! If you'd like to contribute, please see 
the [Contribute]({{ site.baseurl }}/contribute/) section.
\ No newline at end of file
+Beam is an [Apache Software Foundation](http://www.apache.org) project, 
available under the Apache v2 license. Beam is an open source community and 
contributions are greatly appreciated! If you'd like to contribute, please see 
the [Contribute]({{ site.baseurl }}/contribute/) section.

[4/4] beam-site git commit: Regenerate website

[kenn]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/d7f46849
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/d7f46849
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/d7f46849

Branch: refs/heads/asf-site
Commit: d7f46849112abafd182eb107aca486e00df07087
Parents: da7d6c0
Author: Kenneth Knowles 
Authored: Wed May 3 14:03:01 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 3 14:03:01 2017 -0700

--
 content/contribute/work-in-progress/index.html  |   2 +-
 .../runners/capability-matrix/index.html| 286 +
 .../documentation/runners/gearpump/index.html   | 313 +++
 content/get-started/beam-overview/index.html|  12 +-
 content/images/logos/runners/gearpump.png   | Bin 0 -> 2643 bytes
 5 files changed, 611 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/d7f46849/content/contribute/work-in-progress/index.html
--
diff --git a/content/contribute/work-in-progress/index.html 
b/content/contribute/work-in-progress/index.html
index 90eff24..07abdf1 100644
--- a/content/contribute/work-in-progress/index.html
+++ b/content/contribute/work-in-progress/index.html
@@ -184,7 +184,7 @@
   Apache Gearpump Runner
   https://github.com/apache/beam/tree/gearpump-runner;>gearpump-runner
   https://issues.apache.org/jira/browse/BEAM/component/12330829;>runner-gearpump
-  https://github.com/apache/beam/blob/gearpump-runner/runners/gearpump/README.md;>README
+  runner homepage
 
 
   Apache Spark 2.0 Runner

http://git-wip-us.apache.org/repos/asf/beam-site/blob/d7f46849/content/documentation/runners/capability-matrix/index.html
--
diff --git a/content/documentation/runners/capability-matrix/index.html 
b/content/documentation/runners/capability-matrix/index.html
index 3faf1f2..24fa451 100644
--- a/content/documentation/runners/capability-matrix/index.html
+++ b/content/documentation/runners/capability-matrix/index.html
@@ -203,6 +203,8 @@
   
 Apache Apex
   
+Apache Gearpump
+  
   
   
   
@@ -233,6 +235,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -263,6 +270,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -293,6 +305,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -323,6 +340,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -353,6 +375,11 @@
 ~
 
 
+
+
+~
+
+
   
   
   
@@ -383,6 +410,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -413,6 +445,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -443,6 +480,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -473,6 +515,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -495,6 +542,8 @@
   
 Apache Apex
   
+Apache Gearpump
+  
   
   
   
@@ -525,6 +574,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -555,6 +609,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -585,6 +644,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -615,6 +679,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -645,6 +714,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -675,6 +749,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -705,6 +784,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -727,6 +811,8 @@
   
 Apache Apex
   
+Apache Gearpump
+  
   
   
   
@@ -757,6 +843,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -787,6 +878,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -817,6 +913,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -847,6 +948,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -877,6 +983,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -907,6 +1018,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -937,6 +1053,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -967,6 +1088,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -989,6 +1115,8 @@
   
 Apache Apex
   
+Apache Gearpump
+  
   
   
   
@@ -1019,6 +1147,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -1049,6 +1182,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -1079,6 +1217,11 @@
 
 
 
+
+
+
+
+
   
   
   
@@ -1115,6 +1258,8 @@
   
 Apache 
Apex
   
+Apache 
Gearpump
+  
   
   
   
@@ -1145,6 +1290,11 @@
 Yes: fully 
supportedSupported through Apex operator that wraps the 
function and processes data as single element bundles.
 
 
+
+
+Yes: fully 
supportedGearpump wraps the per-element transformation 
function into processor execution.
+
+
   
   
   
@@ -1175,6 +1325,11 @@
 Yes: 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3617

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995649#comment-15995649
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/2868

[BEAM-1871] Migrate ReleaseInfo away from Google API client GenericJson

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam thin_sdk_core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2868.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2868


commit 292f7e753170e4b032a7a456492875340890ded1
Author: Luke Cwik 
Date:   2017-05-03T20:48:07Z

[BEAM-1871] Migrate ReleaseInfo away from Google API client GenericJson




> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-32) Consider not emitting empty ON_TIME pane unless requested

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-32:

Fix Version/s: (was: First stable release)

> Consider not emitting empty ON_TIME pane unless requested
> -
>
> Key: BEAM-32
> URL: https://issues.apache.org/jira/browse/BEAM-32
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, runner-core
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: Triggers, Windowing, backward-incompatible
>
> Today, the ReduceFnRunner sets a timers and emits an empty ON_TIME pane as 
> long as the trigger allows it. This could be controlled in a manner analogous 
> to the empty final pane at window expiration (also owned by the 
> ReduceFnRunner).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2152) Authentication fails if there is an unauthenticated gcloud tool even if application default credentials are available

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj reassigned BEAM-2152:
---

 Assignee: Sourabh Bajaj
Fix Version/s: First stable release

> Authentication fails if there is an unauthenticated gcloud tool even if 
> application default credentials are available
> -
>
> Key: BEAM-2152
> URL: https://issues.apache.org/jira/browse/BEAM-2152
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>
> In a machine that has valid application default credentials, if {{gcloud}} 
> tool is not installed authentication works. If {{gcloud}} tool is recently 
> installed but has not authenticated yet authentication fails with {{You do 
> not currently have an active account selected.}}
> Authentication code should fallback to default method in this case. (Or the 
> {{gcloud}} based authentication needs to be fully removed.
> cc: [~lcwik] 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-32) Consider not emitting empty ON_TIME pane unless requested

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-32?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995636#comment-15995636
 ] 

Kenneth Knowles commented on BEAM-32:
-

Yea, this seems like a really problematic situation.

This only happens when there was some data (to make the window active) and it 
was all emitted in an early pane.

 - In accumulating mode, we will always re-emit the full result, which is fine. 
A user may at some point in their pipeline filter for just the on-time result 
and that will work.
 - In discarding mode, it will be like a punctuation that says "you've got all 
the data". In this case the user would have to filter out late data, rather 
than have a positive filter for on-time. That also seems fine.

The impact of this is that users cannot safely use identity-less {{CombineFn}} 
in discarding mode with early triggering. It is easy to adjoin an identity 
element to any type so it isn't quite as critical as I had feared.

Noting also BEAM-210 that lets users turn it off, but I'm a bit unsure about 
the best default.

> Consider not emitting empty ON_TIME pane unless requested
> -
>
> Key: BEAM-32
> URL: https://issues.apache.org/jira/browse/BEAM-32
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, runner-core
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: Triggers, Windowing, backward-incompatible
> Fix For: First stable release
>
>
> Today, the ReduceFnRunner sets a timers and emits an empty ON_TIME pane as 
> long as the trigger allows it. This could be controlled in a manner analogous 
> to the empty final pane at window expiration (also owned by the 
> ReduceFnRunner).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2867: Inline TypedPValue

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2867

Inline TypedPValue

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This has exactly one implementation, and this is not expected to change.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam inline_typed_pvalue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2867.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2867


commit 748ab60e096c686c067a132d672642565d335526
Author: Thomas Groh 
Date:   2017-05-03T20:42:49Z

Inline TypedPValue

This has exactly one implementation, and this is not expected to change.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2157) HadoopFileSystemModuleTest Failed in Some JDK Versions on Jenkins

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995617#comment-15995617
 ] 

Luke Cwik commented on BEAM-2157:
-

It might be something simple like "contains" -> "containsInAnyOrder" on the 
matcher.

> HadoopFileSystemModuleTest Failed in Some JDK Versions on Jenkins
> -
>
> Key: BEAM-2157
> URL: https://issues.apache.org/jira/browse/BEAM-2157
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Mark Liu
>
> I'm running Java Postcommit test in multiple JDK versions and 
> HadoopFileSystemModuleTest failed in the following JDK versions:
> JDK 1.7
> OpenJDK 7 (on Ubuntu only)
> OpenJDK 8 (on Ubuntu only)
> Error log:
> {code}
> estConfigurationSerializationDeserialization(org.apache.beam.sdk.io.hdfs.HadoopFileSystemModuleTest)
>   Time elapsed: 0.022 s  <<< FAILURE!
> java.lang.AssertionError: 
> Expected: iterable containing 

[jira] [Updated] (BEAM-2157) HadoopFileSystemModuleTest Failed in Some JDK Versions on Jenkins

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-2157:
---
Description: 
I'm running Java Postcommit test in multiple JDK versions and 
HadoopFileSystemModuleTest failed in the following JDK versions:

JDK 1.7
OpenJDK 7 (on Ubuntu only)
OpenJDK 8 (on Ubuntu only)

Error log:
{code}
estConfigurationSerializationDeserialization(org.apache.beam.sdk.io.hdfs.HadoopFileSystemModuleTest)
  Time elapsed: 0.022 s  <<< FAILURE!
java.lang.AssertionError: 

Expected: iterable containing 

[jira] [Commented] (BEAM-32) Consider not emitting empty ON_TIME pane unless requested

[Ben Chambers (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-32?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995613#comment-15995613
 ] 

Ben Chambers commented on BEAM-32:
--

http://stackoverflow.com/questions/43765921/combine-perkey-receives-empty-groups-when-a-repeatedly-trigger-is-used/43769182#43769182

Emmitting an empty creates a problem for a CombineFn. Specifically, it requires 
an identity. For some classes of combine Fn there is no natural identity. For 
example, "the smallest string combineFn" would have "the largest string" as its 
identity. But we shouldn't actually instantiate the largest string, so then the 
combineFn needs to have an artificial identity (such as "null") created, and 
handled.

> Consider not emitting empty ON_TIME pane unless requested
> -
>
> Key: BEAM-32
> URL: https://issues.apache.org/jira/browse/BEAM-32
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, runner-core
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: Triggers, Windowing, backward-incompatible
> Fix For: First stable release
>
>
> Today, the ReduceFnRunner sets a timers and emits an empty ON_TIME pane as 
> long as the trigger allows it. This could be controlled in a manner analogous 
> to the empty final pane at window expiration (also owned by the 
> ReduceFnRunner).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2156) Clean up import guards for GCP libraries in the python SDK

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995610#comment-15995610
 ] 

Ahmet Altay commented on BEAM-2156:
---

One idea is to have an internal api to check for some extra features and 
optionally import stuff based on the available features.

> Clean up import guards for GCP libraries in the python SDK
> --
>
> Key: BEAM-2156
> URL: https://issues.apache.org/jira/browse/BEAM-2156
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Vikas Kedigehalli
>Assignee: Ahmet Altay
>Priority: Minor
>
> For protecting against environments that do not have gcp libraries installed 
> we have import guards like 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py#L21
>  which is ugly. Need to come up with a better approach to handle such 
> scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2157) HadoopFileSystemModuleTest Failed in Some JDK Versions on Jenkins

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-2157:
---
Description: 
I'm running Java Postcommit test in multiple JDK versions and 
HadoopFileSystemModuleTest failed in the following JDK versions:

JDK 1.7
OpenJDK 7 (on Ubuntu only)
OpenJDK 8 (on Ubuntu only)

However, this test was passed in JDK 1.8 which is the one we are currently used 
in all pre/post-commit.

Jenkins build link is 
[here|https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_JDK_Versions_Test/6/]

Note: This Jenkins branch is still not published, but environment should be set 
up correctly. Build (total 4 JDK versions) can be triggered by commenting "Run 
Java JDK Versions Test" in PR.

  was:
I'm running Java Postcommit test in multiple JDK versions and 
HadoopFileSystemModuleTest failed in the following JDK versions:

JDK 1.7
OpenJDK 7 (on Ubuntu only)
OpenJDK 8 (on Ubuntu only)

However, this test was passed in JDK 1.8 which is the one we are currently used 
in all pre/post-commit.

Jenkins build link is 
[here|https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_JDK_Versions_Test/6/]

Note: This Jenkins branch is still not published, but environment should be set 
up correctly.


> HadoopFileSystemModuleTest Failed in Some JDK Versions on Jenkins
> -
>
> Key: BEAM-2157
> URL: https://issues.apache.org/jira/browse/BEAM-2157
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Mark Liu
>
> I'm running Java Postcommit test in multiple JDK versions and 
> HadoopFileSystemModuleTest failed in the following JDK versions:
> JDK 1.7
> OpenJDK 7 (on Ubuntu only)
> OpenJDK 8 (on Ubuntu only)
> However, this test was passed in JDK 1.8 which is the one we are currently 
> used in all pre/post-commit.
> Jenkins build link is 
> [here|https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_JDK_Versions_Test/6/]
> Note: This Jenkins branch is still not published, but environment should be 
> set up correctly. Build (total 4 JDK versions) can be triggered by commenting 
> "Run Java JDK Versions Test" in PR.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-2020) Move CloudObject to Dataflow runner

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-2020.
---
Resolution: Fixed

> Move CloudObject to Dataflow runner
> ---
>
> Key: BEAM-2020
> URL: https://issues.apache.org/jira/browse/BEAM-2020
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> This entails primarily eliminating Coder.asCloudObject() by adding the needed 
> accessors, and possibly a serialization registrar discipline, for coders in 
> the Runner API proto.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-32) Consider not emitting empty ON_TIME pane unless requested

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-32:

Fix Version/s: First stable release

> Consider not emitting empty ON_TIME pane unless requested
> -
>
> Key: BEAM-32
> URL: https://issues.apache.org/jira/browse/BEAM-32
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, runner-core
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: Triggers, Windowing, backward-incompatible
> Fix For: First stable release
>
>
> Today, the ReduceFnRunner sets a timers and emits an empty ON_TIME pane as 
> long as the trigger allows it. This could be controlled in a manner analogous 
> to the empty final pane at window expiration (also owned by the 
> ReduceFnRunner).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2157) HadoopFileSystemModuleTest Failed in Some JDK Versions on Jenkins

[Mark Liu (JIRA)]

Mark Liu created BEAM-2157:
--

 Summary: HadoopFileSystemModuleTest Failed in Some JDK Versions on 
Jenkins
 Key: BEAM-2157
 URL: https://issues.apache.org/jira/browse/BEAM-2157
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Mark Liu


I'm running Java Postcommit test in multiple JDK versions and 
HadoopFileSystemModuleTest failed in the following JDK versions:

JDK 1.7
OpenJDK 7 (on Ubuntu only)
OpenJDK 8 (on Ubuntu only)

However, this test was passed in JDK 1.8 which is the one we are currently used 
in all pre/post-commit.

Jenkins build link is 
[here|https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_JDK_Versions_Test/6/]

Note: This Jenkins branch is still not published, but environment should be set 
up correctly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2150) Support for recursive wildcards in GcsPath

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995591#comment-15995591
 ] 

ASF GitHub Bot commented on BEAM-2150:
--

GitHub user meunierd opened a pull request:

https://github.com/apache/beam/pull/2866

[BEAM-2150] Relax regex to support wildcard globbing for GCS

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

Something I've noticed is that Beam's usage of the GCS API doesn't leverage 
delimiters so we're actually always iterating over the full set of objects 
after the prefix which is why this PR is so tiny.

Ideally, we can actually specify the delimiter `/` when not using  
recursive wildcards (`**`) for some efficiency gains.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/meunierd/beam 
BEAM-2150-gcs-recursive-wildcards

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2866.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2866


commit 6d0d6257040a9725b22f4c94bdaa2de388fd2e65
Author: Devon Meunier 
Date:   2017-05-03T20:22:16Z

[BEAM-2150] Relax regex to support wildcard globbing for GCS




> Support for recursive wildcards in GcsPath
> --
>
> Key: BEAM-2150
> URL: https://issues.apache.org/jira/browse/BEAM-2150
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Devon Meunier
>Assignee: Devon Meunier
>Priority: Minor
>
> When working with heavily nested folder structures in Google Cloud Storage, 
> it's great to make use of recursive wildcards, which the current API 
> explicitly does not support.
> This code hasn't been touched in 2 years so it's likely that simply no one's 
> gotten around to it yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1902) Datastore IO never retries on errors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995581#comment-15995581
 ] 

ASF GitHub Bot commented on BEAM-1902:
--

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2865

[BEAM-1902] Retry on correct error codes for datastoreio

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam ds_retry

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2865.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2865


commit bae0fc78f4599b931fd46aa53e00d5356b91d3f5
Author: Vikas Kedigehalli 
Date:   2017-05-03T20:14:20Z

Retry on correct error codes for datastoreio




> Datastore IO never retries on errors
> 
>
> Key: BEAM-1902
> URL: https://issues.apache.org/jira/browse/BEAM-1902
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Vikas Kedigehalli
>
> {{retry_on_rpc_error}} assumes that the {{exception.code}} are HTTP errors 
> and tries to retry on error codes {{>= 500}}. However these are rpc errors 
> and all are < 500
> Existing rpc error codes: 
> https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2865: [BEAM-1902] Retry on correct error codes for datast...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2865

[BEAM-1902] Retry on correct error codes for datastoreio

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam ds_retry

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2865.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2865


commit bae0fc78f4599b931fd46aa53e00d5356b91d3f5
Author: Vikas Kedigehalli 
Date:   2017-05-03T20:14:20Z

Retry on correct error codes for datastoreio




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2150) Support for recursive wildcards in GcsPath

[Devon Meunier (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995545#comment-15995545
 ] 

Devon Meunier commented on BEAM-2150:
-

Poking through the code, I think all that needs to change is the filtering 
semantics because we're already not using delimiters and returning everything 
after the prefix.

> Support for recursive wildcards in GcsPath
> --
>
> Key: BEAM-2150
> URL: https://issues.apache.org/jira/browse/BEAM-2150
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Devon Meunier
>Assignee: Devon Meunier
>Priority: Minor
>
> When working with heavily nested folder structures in Google Cloud Storage, 
> it's great to make use of recursive wildcards, which the current API 
> explicitly does not support.
> This code hasn't been touched in 2 years so it's likely that simply no one's 
> gotten around to it yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2156) Clean up import guards for GCP libraries in the python SDK

[Vikas Kedigehalli (JIRA)]

Vikas Kedigehalli created BEAM-2156:
---

 Summary: Clean up import guards for GCP libraries in the python SDK
 Key: BEAM-2156
 URL: https://issues.apache.org/jira/browse/BEAM-2156
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Vikas Kedigehalli
Assignee: Ahmet Altay
Priority: Minor


For protecting against environments that do not have gcp libraries installed we 
have import guards like 
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py#L21
 which is ugly. Need to come up with a better approach to handle such scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995515#comment-15995515
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2862


> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2862: [BEAM-1871] Move GCP specific serialization CloudOb...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2862


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam git commit: [BEAM-1871] Move GCP specific serialization CloudObject and supporting translation code to Dataflow runner module

[lcwik]

[BEAM-1871] Move GCP specific serialization CloudObject and supporting 
translation code to Dataflow runner module


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a5627b1a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a5627b1a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a5627b1a

Branch: refs/heads/master
Commit: a5627b1a64696d7526bc5aeec5a0b51571fb5ef1
Parents: 320f9af
Author: Luke Cwik <lc...@google.com>
Authored: Wed May 3 10:03:45 2017 -0700
Committer: Luke Cwik <lc...@google.com>
Committed: Wed May 3 12:46:17 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../dataflow/DataflowPipelineTranslator.java|  18 +-
 .../beam/runners/dataflow/DataflowRunner.java   |   2 +-
 .../beam/runners/dataflow/ReadTranslator.java   |   8 +-
 .../dataflow/internal/CustomSources.java|   6 +-
 .../util/AvroCoderCloudObjectTranslator.java|   2 -
 .../beam/runners/dataflow/util/CloudObject.java |   1 -
 .../runners/dataflow/util/CloudObjectKinds.java |   2 -
 .../dataflow/util/CloudObjectTranslator.java|   2 -
 .../dataflow/util/CloudObjectTranslators.java   |  23 +-
 .../runners/dataflow/util/CloudObjects.java |   1 -
 .../CoderCloudObjectTranslatorRegistrar.java|   1 -
 .../runners/dataflow/util/PropertyNames.java| 112 ++
 .../SerializableCoderCloudObjectTranslator.java |   2 -
 .../beam/runners/dataflow/util/Serializer.java  | 262 -
 .../beam/runners/dataflow/util/Structs.java | 372 +++
 .../DataflowPipelineTranslatorTest.java |  10 +-
 .../runners/dataflow/util/CloudObjectsTest.java |   1 -
 .../beam/runners/dataflow/util/StructsTest.java | 206 ++
 .../apache/beam/sdk/coders/CollectionCoder.java |  13 -
 .../apache/beam/sdk/coders/IterableCoder.java   |  13 -
 .../org/apache/beam/sdk/coders/KvCoder.java |  13 -
 .../beam/sdk/coders/LengthPrefixCoder.java  |  13 -
 .../org/apache/beam/sdk/coders/ListCoder.java   |  13 -
 .../org/apache/beam/sdk/coders/SetCoder.java|  16 -
 .../apache/beam/sdk/util/CloudKnownType.java| 143 ---
 .../org/apache/beam/sdk/util/CloudObject.java   | 187 --
 .../org/apache/beam/sdk/util/CoderUtils.java| 117 --
 .../org/apache/beam/sdk/util/PropertyNames.java | 112 --
 .../org/apache/beam/sdk/util/Serializer.java| 147 
 .../java/org/apache/beam/sdk/util/Structs.java  | 371 --
 .../java/org/apache/beam/sdk/util/Values.java   |  88 -
 .../org/apache/beam/sdk/util/WindowedValue.java |  22 --
 .../org/apache/beam/sdk/values/TupleTag.java|  22 --
 .../apache/beam/sdk/util/SerializerTest.java| 162 
 .../org/apache/beam/sdk/util/StructsTest.java   | 206 --
 36 files changed, 729 insertions(+), 1962 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a5627b1a/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index bbad156..30ef84d 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20170502
+
beam-master-20170503
 
1
 
6
   

http://git-wip-us.apache.org/repos/asf/beam/blob/a5627b1a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 28a9c1c..05edd28 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -21,15 +21,15 @@ import static 
com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
 import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.base.Strings.isNullOrEmpty;
+import static org.apache.beam.runners.dataflow.util.Structs.addBoolean;
+import static org.apache.beam.runners.dataflow.util.Structs.addDictionary;
+import static org.apache.beam.runners.dataflow.util.Structs.addList;
+import static org.apache.beam.runners.dataflow.util.Structs.addLong;
+import static org.apache.beam.runners.dataflow.util.Structs.addObject;
+import static org.apache.beam.runners.dataflow.util.Structs.addString;
+i

[1/3] beam git commit: [BEAM-1871] Move GCP specific serialization CloudObject and supporting translation code to Dataflow runner module

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 320f9affb -> aafa1bba9


http://git-wip-us.apache.org/repos/asf/beam/blob/a5627b1a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CloudObject.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CloudObject.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CloudObject.java
deleted file mode 100644
index 061e56a..000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CloudObject.java
+++ /dev/null
@@ -1,187 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util;
-
-
-import static com.google.common.base.Preconditions.checkNotNull;
-
-import com.google.api.client.json.GenericJson;
-import com.google.api.client.util.Key;
-import java.util.Map;
-import javax.annotation.Nullable;
-
-/**
- * A representation of an arbitrary Java object to be instantiated by Dataflow
- * workers.
- *
- * Typically, an object to be written by the SDK to the Dataflow service 
will
- * implement a method (typically called {@code asCloudObject()}) that returns a
- * {@code CloudObject} to represent the object in the protocol.  Once the
- * {@code CloudObject} is constructed, the method should explicitly add
- * additional properties to be presented during deserialization, representing
- * child objects by building additional {@code CloudObject}s.
- *
- * @deprecated replaced by {@code 
org.apache.beam.runners.dataflow.CloudKnownType}
- */
-@Deprecated
-public final class CloudObject extends GenericJson {
-  /**
-   * Constructs a {@code CloudObject} by copying the supplied serialized object
-   * spec, which must represent an SDK object serialized for transport via the
-   * Dataflow API.
-   *
-   * The most common use of this method is during deserialization on the 
worker,
-   * where it's used as a binding type during instance construction.
-   *
-   * @param spec supplies the serialized form of the object as a nested map
-   * @throws RuntimeException if the supplied map does not represent an SDK 
object
-   */
-  public static CloudObject fromSpec(Map spec) {
-CloudObject result = new CloudObject();
-result.putAll(spec);
-if (result.className == null) {
-  throw new RuntimeException("Unable to create an SDK object from " + spec
-  + ": Object class not specified (missing \""
-  + PropertyNames.OBJECT_TYPE_NAME + "\" field)");
-}
-return result;
-  }
-
-  /**
-   * Constructs a {@code CloudObject} to be used for serializing an instance of
-   * the supplied class for transport via the Dataflow API.  The instance
-   * parameters to be serialized must be supplied explicitly after the
-   * {@code CloudObject} is created, by using {@link CloudObject#put}.
-   *
-   * @param cls the class to use when deserializing the object on the worker
-   */
-  public static CloudObject forClass(Class cls) {
-CloudObject result = new CloudObject();
-result.className = checkNotNull(cls).getName();
-return result;
-  }
-
-  /**
-   * Constructs a {@code CloudObject} to be used for serializing data to be
-   * deserialized using the supplied class name the supplied class name for
-   * transport via the Dataflow API.  The instance parameters to be serialized
-   * must be supplied explicitly after the {@code CloudObject} is created, by
-   * using {@link CloudObject#put}.
-   *
-   * @param className the class to use when deserializing the object on the 
worker
-   */
-  public static CloudObject forClassName(String className) {
-CloudObject result = new CloudObject();
-result.className = checkNotNull(className);
-return result;
-  }
-
-  /**
-   * Constructs a {@code CloudObject} representing the given value.
-   * @param value the scalar value to represent.
-   */
-  public static CloudObject forString(String value) {
-CloudObject result = forClassName(CloudKnownType.TEXT.getUri());
-result.put(PropertyNames.SCALAR_FIELD_NAME, value);
-return result;
-  }
-
-  /**
-   * Constructs a {@code CloudObject} representing the given value.
-   * @param value the scalar value to represent.
-   

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1919

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2842: [BEAM-2051] Mark all PCollectionView methods intern...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2842


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2051) Reduce scope of the PCollectionView interface

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995508#comment-15995508
 ] 

ASF GitHub Bot commented on BEAM-2051:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2842


> Reduce scope of the PCollectionView interface
> -
>
> Key: BEAM-2051
> URL: https://issues.apache.org/jira/browse/BEAM-2051
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> Users should only ever use a PCollectionView class as a token to access a 
> view. A Runner can cast down to a more expressive type if required.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: This closes #2842

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 0d19d3cb2 -> 320f9affb


This closes #2842


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/320f9aff
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/320f9aff
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/320f9aff

Branch: refs/heads/master
Commit: 320f9affb3c0e1f7199d7db60bcb4d25acc3d7f3
Parents: 0d19d3c cc4ca8c
Author: Thomas Groh 
Authored: Wed May 3 12:42:15 2017 -0700
Committer: Thomas Groh 
Committed: Wed May 3 12:42:15 2017 -0700

--
 .../apache/beam/sdk/values/PCollectionView.java | 29 ++--
 1 file changed, 26 insertions(+), 3 deletions(-)
--

[2/2] beam git commit: Mark all PCollectionView methods internal

[tgroh]

Mark all PCollectionView methods internal

Add a note that the methods should not be considered to be accessible.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cc4ca8c0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cc4ca8c0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cc4ca8c0

Branch: refs/heads/master
Commit: cc4ca8c0b75f64f0bb6f705af3e80db9310d6647
Parents: 0d19d3c
Author: Thomas Groh 
Authored: Tue May 2 10:31:21 2017 -0700
Committer: Thomas Groh 
Committed: Wed May 3 12:42:15 2017 -0700

--
 .../apache/beam/sdk/values/PCollectionView.java | 29 ++--
 1 file changed, 26 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cc4ca8c0/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
index d65912b..71efc09 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
@@ -21,6 +21,7 @@ import java.io.Serializable;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.annotations.Internal;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
@@ -44,54 +45,76 @@ import org.apache.beam.sdk.util.WindowingStrategy;
  * {@link View#asIterable()}, and {@link View#asMap()} for more detail on 
specific views
  * available in the SDK.
  *
+ * NOTE: View methods should not be considered to be user-accessible. They 
are implementation
+ * details of how a Runner obtains information required to construct a view, 
and are subject to
+ * change at any time.
+ *
  * @param  the type of the value(s) accessible via this {@link 
PCollectionView}
  */
 public interface PCollectionView extends PValue, Serializable {
   /**
-   * Gets the {@link PCollection} this {@link PCollectionView} was created 
from.
+   * For internal use only.
+   *
+   * Gets the {@link PCollection} this {@link PCollectionView} was created 
from.
*
* The {@link PCollection} may not be available in all contexts.
*/
   @Nullable
+  @Internal
   PCollection getPCollection();
 
   /**
+   * For internal use only.
+   *
* @deprecated this method will be removed entirely. The {@link PCollection} 
underlying a side
* input, is part of the side input's specification with a {@link ParDo} 
transform, which will
* obtain that information via a package-private channel.
*/
   @Deprecated
+  @Internal
   TupleTag> getTagInternal();
 
   /**
+   * For internal use only.
+   *
* @deprecated this method will be removed entirely. The {@link ViewFn} for 
a side input is an
* attribute of the side input's specification with a {@link ParDo} 
transform, which will
* obtain this specification via a package-private channel.
*/
   @Deprecated
+  @Internal
   ViewFn, T> getViewFn();
 
   /**
-   * Returns the {@link WindowMappingFn} used to map windows from a main input 
to the side input of
-   * this {@link PCollectionView}.
+   * For internal use only.
+   *
+   * Returns the {@link WindowMappingFn} used to map windows from a main 
input to the side input
+   * of this {@link PCollectionView}.
*/
   @Experimental(Kind.CORE_RUNNERS_ONLY)
+  @Internal
   WindowMappingFn getWindowMappingFn();
 
   /**
+   * For internal use only.
+   *
* @deprecated this method will be removed entirely. The {@link PCollection} 
underlying a side
* input, including its {@link WindowingStrategy}, is part of the side 
input's specification
* with a {@link ParDo} transform, which will obtain that information 
via a package-private
* channel.
*/
   @Deprecated
+  @Internal
   WindowingStrategy getWindowingStrategyInternal();
 
   /**
+   * For internal use only.
+   *
* @deprecated this method will be removed entirely. The {@link PCollection} 
underlying a side
* input, including its {@link Coder}, is part of the side input's 
specification with a {@link
* ParDo} transform, which will obtain that information via a 
package-private channel.
*/
   @Deprecated
+  @Internal
   Coder> getCoderInternal();
 }

[jira] [Commented] (BEAM-2150) Support for recursive wildcards in GcsPath

[Devon Meunier (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995490#comment-15995490
 ] 

Devon Meunier commented on BEAM-2150:
-

With GCS you're not actually walking a filesystem. With **, you simply remove 
the delimiter restriction and filter.

> Support for recursive wildcards in GcsPath
> --
>
> Key: BEAM-2150
> URL: https://issues.apache.org/jira/browse/BEAM-2150
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Devon Meunier
>Assignee: Devon Meunier
>Priority: Minor
>
> When working with heavily nested folder structures in Google Cloud Storage, 
> it's great to make use of recursive wildcards, which the current API 
> explicitly does not support.
> This code hasn't been touched in 2 years so it's likely that simply no one's 
> gotten around to it yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #2100

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2145) Do not publish javadoc for Java SDK's transforms.reflect directory

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995407#comment-15995407
 ] 

Daniel Halperin commented on BEAM-2145:
---

Given that Davor will not be actively working on it, it should be left open for 
someone from the community to pick it up. I believe that the fix for this is 
O(1 line) and probably simpler than filing the issue in the first place. We 
can't expect Davor to handle all these little taxes.

> Do not publish javadoc for Java SDK's transforms.reflect directory
> --
>
> Key: BEAM-2145
> URL: https://issues.apache.org/jira/browse/BEAM-2145
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, website
>Reporter: Kenneth Knowles
>Priority: Blocker
> Fix For: First stable release
>
>
> This is where the utilities for analyzing a {{DoFn}} live. They need to be in 
> the core SDK for {{DoFnTester}}, otherwise they would be in 
> runner-core-construction.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2145) Do not publish javadoc for Java SDK's transforms.reflect directory

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995367#comment-15995367
 ] 

Kenneth Knowles commented on BEAM-2145:
---

This should be owned by the next release manager if not before. As it is the 
first stable release I presume Davor will be aware and the best person to hand 
it to the release manager, no? It shouldn't be ownerless I think.

> Do not publish javadoc for Java SDK's transforms.reflect directory
> --
>
> Key: BEAM-2145
> URL: https://issues.apache.org/jira/browse/BEAM-2145
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, website
>Reporter: Kenneth Knowles
>Priority: Blocker
> Fix For: First stable release
>
>
> This is where the utilities for analyzing a {{DoFn}} live. They need to be in 
> the core SDK for {{DoFnTester}}, otherwise they would be in 
> runner-core-construction.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3025

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Dataflow #368

[Apache Jenkins Server]

See 


Changes:

[klk] Use LinkedHashMap for step contexts in BaseExecutionContext

[dhalperi] Remove IoChannelUtils from PackageUtil

[dhalperi] [BEAM-59] FileChecksumMatcherTest: switch from IOChannelUtils to java

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0d19d3cb210ef179e1831b971f7cf6d3d6bb1dde (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0d19d3cb210ef179e1831b971f7cf6d3d6bb1dde
 > git rev-list 3b61f6a9034b3dbb8b08f695b7b526656660c8a6 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson8822402256232841916.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson6143726776491349836.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson5896650652856289214.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Dataflow] $ /bin/bash -xe 
/tmp/hudson3659993746409279566.sh
+ python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.pkb_results --official=true 
--benchmarks=dpb_wordcount_benchmark 
--dpb_dataflow_staging_location=gs://temp-storage-for-perf-tests/staging 
--dpb_wordcount_input=dataflow-samples/shakespeare/kinglear.txt 
--config_override=dpb_wordcount_benchmark.dpb_service.service_type=dataflow
WARNING:root:File resource loader root perfkitbenchmarker/data/ycsb is not a 
directory.
2017-05-03 18:04:37,849 33130c4f MainThread INFO Verbose logging to: 
/tmp/perfkitbenchmarker/runs/33130c4f/pkb.log
2017-05-03 18:04:37,849 33130c4f MainThread INFO PerfKitBenchmarker 
version: v1.11.0-48-ge557505

[jira] [Commented] (BEAM-1199) Condense recordAsOutput, finishSpecifyingOutput from POutput

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995343#comment-15995343
 ] 

ASF GitHub Bot commented on BEAM-1199:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2864

[BEAM-1199] Remove POutput#recordAsOutput

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Add PValue#setDefaultName, for composite POutputs to use when they are
being finalized.

Expand POutput#finishSpecifyingOutput to take the name of the transform
that is being finished.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam condense_finish_output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2864.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2864


commit cc04e3356f9232049a00babccd76e9b9f1630f67
Author: Thomas Groh 
Date:   2017-05-03T00:53:57Z

Remove POutput#recordAsOutput

Add PValue#setDefaultName, for composite POutputs to use when they are
being finalized.

Expand POutput#finishSpecifyingOutput to take the name of the transform
that is being finished.




> Condense recordAsOutput, finishSpecifyingOutput from POutput
> 
>
> Key: BEAM-1199
> URL: https://issues.apache.org/jira/browse/BEAM-1199
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: First stable release
>
>
> {{recordAsOutput}} and {{finishSpecifyingOutput}} are both methods which are 
> called after an output has been attached to a PTransform application. They 
> can be combined to only have one method that does any after-production work 
> (such as the initial run of Coder inference)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2864: [BEAM-1199] Remove POutput#recordAsOutput

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2864

[BEAM-1199] Remove POutput#recordAsOutput

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Add PValue#setDefaultName, for composite POutputs to use when they are
being finalized.

Expand POutput#finishSpecifyingOutput to take the name of the transform
that is being finished.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam condense_finish_output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2864.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2864


commit cc04e3356f9232049a00babccd76e9b9f1630f67
Author: Thomas Groh 
Date:   2017-05-03T00:53:57Z

Remove POutput#recordAsOutput

Add PValue#setDefaultName, for composite POutputs to use when they are
being finalized.

Expand POutput#finishSpecifyingOutput to take the name of the transform
that is being finished.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Python_Verify #2099

[Apache Jenkins Server]

See 


Changes:

[klk] Use LinkedHashMap for step contexts in BaseExecutionContext

--
[...truncated 508.52 KB...]
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/pyparsing-2.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging 
appdirs pyparsing
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:132:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:132:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-35.0.2.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/pyparsing-2.2.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-35.0.2.zip
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging 
appdirs pyparsing
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting setuptools (from pyhamcrest->-r 

[GitHub] beam pull request #2863: Reduce visibility of DirectRunner classes

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2863

Reduce visibility of DirectRunner classes

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Move inner classes of the DirectRunner to reduce total API Surface.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam direct_runner_visibility

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2863.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2863






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-2113) Apex Runner is not able to submit any job to YARN

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2113:
-

Assignee: Kenneth Knowles  (was: Thomas Weise)

> Apex Runner is not able to submit any job to YARN
> -
>
> Key: BEAM-2113
> URL: https://issues.apache.org/jira/browse/BEAM-2113
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Affects Versions: First stable release
> Environment: CentOS7 + HDP-2.5.5
> OpenJDK 1.8.0_131-b11
>Reporter: Tibor Kiss
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> Trying to execute the WordCount example using Apex Runner:
> {code}
> $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount 
> -Dexec.args="--inputFile=/tmp/input/pom.xml --output=/tmp/output/ 
> --runner=ApexRunner --embeddedExecution=false" -Papex-runner
> ...
> ...
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ beam-examples-java ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/root/.m2/repository/org/slf4j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/root/.m2/repository/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]
> Apr 28, 2017 1:51:13 PM org.apache.beam.runners.apex.ApexYarnLauncher 
> launchApp
> INFO: Executing: hadoop org.apache.beam.runners.apex.ApexYarnLauncher 
> /tmp/beam-runner-apex1201917541012785769params with 
> {HADOOP_USER_CLASSPATH_FIRST=1, 
> HADOOP_CLASSPATH=/tmp/beam-runners-apex-5810119727972429445.jar:/root/.m2/repository/org/apache/beam/beam-sdks-java-core/0.7.0-SNAPSHOT/beam-sdks-java-core-0.7.0-SNAPSHOT.j
> ar:/root/.m2/repository/org/apache/beam/beam-sdks-common-runner-api/0.7.0-SNAPSHOT/beam-sdks-common-runner-api-0.7.0-SNAPSHOT.jar:/root/.m2/repository/com/google/http-client/google-http-client-jackson/1.22.0/google-http-client-jackson-1.22.0.jar:/root/.m2/repository/com/google/http-client/google-http-client-protobuf/1.22.0/g
> oogle-http-client-protobuf-1.22.0.jar:/root/.m2/repository/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar:/root/.m2/repository/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar:/root/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.8.8/jackson-core-2.8.8.jar:/root/.m2/repository/com/fasterxml/ja
> ckson/core/jackson-annotations/2.8.8/jackson-annotations-2.8.8.jar:/root/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.8.8/jackson-databind-2.8.8.jar:/root/.m2/repository/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.jar:/root/.m2/repository/org/xerial/snappy/snappy-java/1.1.4-M3/snappy-java-1.1.4-M3.jar:/ro
> ot/.m2/repository/org/apache/commons/commons-compress/1.9/commons-compress-1.9.jar:/root/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/root/.m2/repository/org/apache/beam/beam-sdks-java-extensions-gcp-core/0.7.0-SNAPSHOT/beam-sdks-java-extensions-gcp-core-0.7.0-SNAPSHOT.jar:/root/.m2/repository/c
> om/google/http-client/google-http-client-jackson2/1.22.0/google-http-client-jackson2-1.22.0.jar:/root/.m2/repository/com/google/cloud/bigdataoss/gcsio/1.4.5/gcsio-1.4.5.jar:/root/.m2/repository/com/google/apis/google-api-services-cloudresourcemanager/v1-rev6-1.22.0/google-api-services-cloudresourcemanager-v1-rev6-1.22.0.jar:
> /root/.m2/repository/com/google/apis/google-api-services-storage/v1-rev71-1.22.0/google-api-services-storage-v1-rev71-1.22.0.jar:/root/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.7.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-0.7.0-SNAPSHOT.jar:/root/.m2/repository/org/apache/beam/beam-sdks
> -java-extensions-protobuf/0.7.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-0.7.0-SNAPSHOT.jar:/root/.m2/repository/com/google/api/grpc/grpc-google-pubsub-v1/0.1.0/grpc-google-pubsub-v1-0.1.0.jar:/root/.m2/repository/com/google/api/grpc/grpc-google-iam-v1/0.1.0/grpc-google-iam-v1-0.1.0.jar:/root/.m2/repository/io/grpc/grpc-a
> uth/1.2.0/grpc-auth-1.2.0.jar:/root/.m2/repository/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.jar:/root/.m2/repository/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar:/root/.m2/repository/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.jar:/root/.m2/repository/com/google/instrumentation/ins
> 

[jira] [Commented] (BEAM-59) Switch from IOChannelFactory to FileSystems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995302#comment-15995302
 ] 

ASF GitHub Bot commented on BEAM-59:


Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2860


> Switch from IOChannelFactory to FileSystems
> ---
>
> Key: BEAM-59
> URL: https://issues.apache.org/jira/browse/BEAM-59
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: First stable release
>
>
> Right now, FileBasedSource and FileBasedSink communication is mediated by 
> IOChannelFactory. There are a number of issues:
> * Global configuration -- e.g., all 'gs://' URIs use the same credentials. 
> This should be per-source/per-sink/etc.
> * Supported APIs -- currently IOChannelFactory is in the "non-public API" 
> util package and subject to change. We need users to be able to add new 
> backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be 
> broken.
> * Per-backend features: e.g., creating buckets in GCS/s3, setting expiration 
> time, etc.
> Updates:
> Design docs posted on dev@ list:
> Part 1: IOChannelFactory Redesign: 
> https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit#
> Part 2: Configurable BeamFileSystem:
> https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2860: [BEAM-59] FileChecksumMatcherTest: switch from IOCh...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2860


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2860

[dhalperi]

This closes #2860


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0d19d3cb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0d19d3cb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0d19d3cb

Branch: refs/heads/master
Commit: 0d19d3cb210ef179e1831b971f7cf6d3d6bb1dde
Parents: f33039b fb9ebb6
Author: Dan Halperin 
Authored: Wed May 3 10:43:54 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 3 10:43:54 2017 -0700

--
 .../org/apache/beam/sdk/testing/FileChecksumMatcherTest.java  | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)
--

[jira] [Commented] (BEAM-2053) Add HDFS file provider to wordcount examples

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995299#comment-15995299
 ] 

Daniel Halperin commented on BEAM-2053:
---

This is a likely dupe of BEAM-59 and BEAM-2005.

> Add HDFS file provider to wordcount examples
> 
>
> Key: BEAM-2053
> URL: https://issues.apache.org/jira/browse/BEAM-2053
> Project: Beam
>  Issue Type: Task
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Daniel Halperin
>Priority: Minor
>
> Instructions for running the example on YARN refer to HDFS, but HDFS is 
> currently not supported in the example project:
> https://beam.apache.org/documentation/runners/apex/
> Using local files is sufficient in a sandbox, for multi-node cluster we need 
> a distributed FS.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: [BEAM-59] FileChecksumMatcherTest: switch from IOChannelUtils to java file API

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master f33039bfd -> 0d19d3cb2


[BEAM-59] FileChecksumMatcherTest: switch from IOChannelUtils to java file API

These are unit tests always running on local filesystem.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fb9ebb67
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fb9ebb67
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fb9ebb67

Branch: refs/heads/master
Commit: fb9ebb678b9c2d2bba1d00baedbfa0f5360927db
Parents: f33039b
Author: Dan Halperin 
Authored: Wed May 3 08:43:51 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 3 10:43:44 2017 -0700

--
 .../org/apache/beam/sdk/testing/FileChecksumMatcherTest.java  | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fb9ebb67/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/FileChecksumMatcherTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/FileChecksumMatcherTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/FileChecksumMatcherTest.java
index 5438479..4ee6750 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/FileChecksumMatcherTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/FileChecksumMatcherTest.java
@@ -27,7 +27,6 @@ import java.io.IOException;
 import java.nio.charset.StandardCharsets;
 import java.util.regex.Pattern;
 import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.util.IOChannelUtils;
 import org.junit.Rule;
 import org.junit.Test;
 import org.junit.rules.ExpectedException;
@@ -111,7 +110,7 @@ public class FileChecksumMatcherTest {
 FileChecksumMatcher matcher =
 new FileChecksumMatcher(
 "90552392c28396935fe4f123bd0b5c2d0f6260c8",
-IOChannelUtils.resolve(tmpFolder.getRoot().getPath(), "result-*"));
+tmpFolder.getRoot().toPath().resolve("result-*").toString());
 
 assertThat(pResult, matcher);
   }
@@ -123,7 +122,7 @@ public class FileChecksumMatcherTest {
 FileChecksumMatcher matcher =
 new FileChecksumMatcher(
 "da39a3ee5e6b4b0d3255bfef95601890afd80709",
-IOChannelUtils.resolve(tmpFolder.getRoot().getPath(), "*"));
+tmpFolder.getRoot().toPath().resolve("*").toString());
 
 assertThat(pResult, matcher);
   }
@@ -140,7 +139,7 @@ public class FileChecksumMatcherTest {
 Pattern.compile("(?x) result (?\\d+) - total 
(?\\d+)");
 FileChecksumMatcher matcher = new FileChecksumMatcher(
 "90552392c28396935fe4f123bd0b5c2d0f6260c8",
-IOChannelUtils.resolve(tmpFolder.getRoot().getPath(), "*"),
+tmpFolder.getRoot().toPath().resolve("*").toString(),
 customizedTemplate);
 
 assertThat(pResult, matcher);

[jira] [Assigned] (BEAM-2053) Add HDFS file provider to wordcount examples

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2053:
-

Assignee: Daniel Halperin

> Add HDFS file provider to wordcount examples
> 
>
> Key: BEAM-2053
> URL: https://issues.apache.org/jira/browse/BEAM-2053
> Project: Beam
>  Issue Type: Task
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Daniel Halperin
>Priority: Minor
>
> Instructions for running the example on YARN refer to HDFS, but HDFS is 
> currently not supported in the example project:
> https://beam.apache.org/documentation/runners/apex/
> Using local files is sufficient in a sandbox, for multi-node cluster we need 
> a distributed FS.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2053) Add HDFS file provider to wordcount examples

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995286#comment-15995286
 ] 

Kenneth Knowles commented on BEAM-2053:
---

I believe [~dhalp...@google.com] has changes done or currently underway that 
will clarify this.

> Add HDFS file provider to wordcount examples
> 
>
> Key: BEAM-2053
> URL: https://issues.apache.org/jira/browse/BEAM-2053
> Project: Beam
>  Issue Type: Task
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Daniel Halperin
>Priority: Minor
>
> Instructions for running the example on YARN refer to HDFS, but HDFS is 
> currently not supported in the example project:
> https://beam.apache.org/documentation/runners/apex/
> Using local files is sufficient in a sandbox, for multi-node cluster we need 
> a distributed FS.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3613

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2155) Investigate problem creating Map PCollection view on Dataflow

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2155:
-

 Summary: Investigate problem creating Map PCollection view on 
Dataflow
 Key: BEAM-2155
 URL: https://issues.apache.org/jira/browse/BEAM-2155
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Kenneth Knowles
Assignee: Daniel Halperin


A user reports a side input seemingly never becoming available: 
http://stackoverflow.com/questions/43755254/dataflow-map-side-input-issue

The code seems good at first glance, and it works in the direct runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)