[jira] [Resolved] (BEAM-1960) Hadoop InputFormat - Add Kubernetes large and small cluster Scripts for Cassandra and Elasticsearch tests

[Dipti Kulkarni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dipti Kulkarni resolved BEAM-1960.
--
   Resolution: Fixed
Fix Version/s: First stable release

> Hadoop InputFormat - Add Kubernetes large and small cluster Scripts for 
> Cassandra and Elasticsearch tests
> -
>
> Key: BEAM-1960
> URL: https://issues.apache.org/jira/browse/BEAM-1960
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Dipti Kulkarni
>Assignee: Dipti Kulkarni
> Fix For: First stable release
>
>
> Add Kubernetes cluster scripts for Elasticsearch and Cassandra for both large 
> and small clusters to run Elasticsearch and Cassandra ITs and Performance 
> tests.
> Create separate clusters for small - integration test and large- performance 
> test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-1960) Hadoop InputFormat - Add Kubernetes large and small cluster Scripts for Cassandra and Elasticsearch tests

[Dipti Kulkarni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dipti Kulkarni closed BEAM-1960.


Merged into the main code.

> Hadoop InputFormat - Add Kubernetes large and small cluster Scripts for 
> Cassandra and Elasticsearch tests
> -
>
> Key: BEAM-1960
> URL: https://issues.apache.org/jira/browse/BEAM-1960
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Dipti Kulkarni
>Assignee: Dipti Kulkarni
> Fix For: First stable release
>
>
> Add Kubernetes cluster scripts for Elasticsearch and Cassandra for both large 
> and small clusters to run Elasticsearch and Cassandra ITs and Performance 
> tests.
> Create separate clusters for small - integration test and large- performance 
> test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1336) A StateSpec that doesn't care about the key shouldn't be forced to declare it as type Object

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978046#comment-15978046
 ] 

ASF GitHub Bot commented on BEAM-1336:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2627

[BEAM-1336] Separate StateSpec and KeyedStateSpec

Before this change, almost all uses of state had a type variable that
existing only to support the esoteric use of a KeyedCombineFn in
a state cell. Now those cases use a more verbose form of KeyedStateSpec.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam StateSpec

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2627.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2627


commit c05bda1a5922e18ab29022403f7b9e00f4cea941
Author: Kenneth Knowles 
Date:   2017-04-21T03:46:37Z

Separate StateSpec and KeyedStateSpec

Before this change, almost all uses of state had a type variable that
existing only to support the esoteric use of a KeyedCombineFn in
a state cell. Now those cases use a more verbose form of KeyedStateSpec.




> A StateSpec that doesn't care about the key shouldn't be forced to declare it 
> as type Object
> 
>
> Key: BEAM-1336
> URL: https://issues.apache.org/jira/browse/BEAM-1336
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: starter
>
> In the state API as it exists today, if (as is almost often the case) you are 
> writing a {{StateSpec}} other than a {{KeyedCombiningState}} the 
> key type is irrelevant and the user just has to write {{Object}} there. This 
> was carried over from {{StateTag}} and is an artifact of the visitor pattern 
> there and the difficulty of getting all the types to line up.
> I think simplifying the visitor pattern to be more of just a syntax traversal 
> might alleviate the issue and allow us to drop this noise from the syntax.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1989) clean SyntaxWarning

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-1989:
--
Summary: clean SyntaxWarning  (was: SyntaxWarning in bigquery.py)

> clean SyntaxWarning
> ---
>
> Key: BEAM-1989
> URL: https://issues.apache.org/jira/browse/BEAM-1989
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>
> apache_beam/io/gcp/bigquery.py:326: SyntaxWarning: import * only allowed at 
> module level
>   def __init__(self, table=None, dataset=None, project=None, query=None,
> apache_beam/io/gcp/bigquery.py:431: SyntaxWarning: import * only allowed at 
> module level
>   def __init__(self, table, dataset=None, project=None, schema=None,
> cc: [~sb2nov][~chamikara]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2567: Fix GCP import check in bigquery.py

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2567


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Fix GCP import check in bigquery.py

[altay]

Repository: beam
Updated Branches:
  refs/heads/master bda16078e -> b3827955e


Fix GCP import check in bigquery.py


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3b6490d2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3b6490d2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3b6490d2

Branch: refs/heads/master
Commit: 3b6490d2f77c5ed8b7f181876660e6620162cd06
Parents: bda1607
Author: Charles Chen 
Authored: Tue Apr 18 12:27:56 2017 +0800
Committer: Ahmet Altay 
Committed: Thu Apr 20 20:57:31 2017 -0700

--
 sdks/python/apache_beam/io/gcp/bigquery.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3b6490d2/sdks/python/apache_beam/io/gcp/bigquery.py
--
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py 
b/sdks/python/apache_beam/io/gcp/bigquery.py
index 891f62a..0db965f 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -370,7 +370,7 @@ class BigQuerySource(dataflow_io.NativeSource):
 # Import here to avoid adding the dependency for local running scenarios.
 try:
   # pylint: disable=wrong-import-order, wrong-import-position
-  from apitools.base.py import *
+  from apitools.base import py  # pylint: disable=unused-variable
 except ImportError:
   raise ImportError(
   'Google Cloud IO not available, '
@@ -480,7 +480,7 @@ class BigQuerySink(dataflow_io.NativeSink):
 # Import here to avoid adding the dependency for local running scenarios.
 try:
   # pylint: disable=wrong-import-order, wrong-import-position
-  from apitools.base.py import *
+  from apitools.base import py  # pylint: disable=unused-variable
 except ImportError:
   raise ImportError(
   'Google Cloud IO not available, '

[jira] [Assigned] (BEAM-885) Move PipelineOptions from Pipeline.create() to Pipeline.run()

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-885:


Assignee: Kenneth Knowles  (was: Thomas Groh)

> Move PipelineOptions from Pipeline.create() to Pipeline.run()
> -
>
> Key: BEAM-885
> URL: https://issues.apache.org/jira/browse/BEAM-885
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The specification of a Pipeline should be independent of its PipelineOptions. 
> This delays specification of the options, including choices like Pipeline 
> Runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1022) WindowNamespace and WindowAndTriggerNamespace should not use Java object equality when comparing windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977991#comment-15977991
 ] 

ASF GitHub Bot commented on BEAM-1022:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2626

[BEAM-1022] Use Structural Value when comparing Window(AndTrigger) 
Namespaces

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This makes sure we don't use a potentially default equals method and
instead compare based on the Coder, which is how Beam defines equality.

Improve GlobalWindow and IntervalWindow structuralValue methods.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam structural_values_state_namespaces

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2626.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2626


commit 4915efe8be7b9a2a4cbfe83449e23132e658ac41
Author: Thomas Groh 
Date:   2017-04-21T03:00:07Z

Coder.structuralValue(T) should never throw

In the worst case, encoding to a byte array should never fail due to IO.

commit 74455ddb8a9c43230efb5058255cb1e5756000ae
Author: Thomas Groh 
Date:   2017-04-21T03:00:55Z

Use Structural Value when comparing Window(AndTrigger) Namespaces

This makes sure we don't use a potentially default equals method and
instead compare based on the Coder, which is how Beam defines equality.

Improve GlobalWindow and IntervalWindow structuralValue methods.




> WindowNamespace and WindowAndTriggerNamespace should not use Java object 
> equality when comparing windows
> 
>
> Key: BEAM-1022
> URL: https://issues.apache.org/jira/browse/BEAM-1022
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2626: [BEAM-1022] Use Structural Value when comparing Win...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2626

[BEAM-1022] Use Structural Value when comparing Window(AndTrigger) 
Namespaces

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This makes sure we don't use a potentially default equals method and
instead compare based on the Coder, which is how Beam defines equality.

Improve GlobalWindow and IntervalWindow structuralValue methods.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam structural_values_state_namespaces

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2626.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2626


commit 4915efe8be7b9a2a4cbfe83449e23132e658ac41
Author: Thomas Groh 
Date:   2017-04-21T03:00:07Z

Coder.structuralValue(T) should never throw

In the worst case, encoding to a byte array should never fail due to IO.

commit 74455ddb8a9c43230efb5058255cb1e5756000ae
Author: Thomas Groh 
Date:   2017-04-21T03:00:55Z

Use Structural Value when comparing Window(AndTrigger) Namespaces

This makes sure we don't use a potentially default equals method and
instead compare based on the Coder, which is how Beam defines equality.

Improve GlobalWindow and IntervalWindow structuralValue methods.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2625: Coder.structuralValue(T) should never throw

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2625

Coder.structuralValue(T) should never throw

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
In the worst case, encoding to a byte array should never fail due to IO.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam 
coder_structural_values_never_throws

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2625.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2625


commit 4915efe8be7b9a2a4cbfe83449e23132e658ac41
Author: Thomas Groh 
Date:   2017-04-21T03:00:07Z

Coder.structuralValue(T) should never throw

In the worst case, encoding to a byte array should never fail due to IO.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1022) WindowNamespace and WindowAndTriggerNamespace should not use Java object equality when comparing windows

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977986#comment-15977986
 ] 

Kenneth Knowles commented on BEAM-1022:
---

Ah, they should use Coder.structuralValue. Then it can be fast as well as 
correct.

> WindowNamespace and WindowAndTriggerNamespace should not use Java object 
> equality when comparing windows
> 
>
> Key: BEAM-1022
> URL: https://issues.apache.org/jira/browse/BEAM-1022
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1022) WindowNamespace and WindowAndTriggerNamespace should not use Java object equality when comparing windows

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1022:
--
Summary: WindowNamespace and WindowAndTriggerNamespace should not use Java 
object equality when comparing windows  (was: StateNamespace should not use 
Java object equality when comparing windows)

> WindowNamespace and WindowAndTriggerNamespace should not use Java object 
> equality when comparing windows
> 
>
> Key: BEAM-1022
> URL: https://issues.apache.org/jira/browse/BEAM-1022
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1022) WindowNamespace and WindowAndTriggerNamespace should not use Java object equality when comparing windows

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977985#comment-15977985
 ] 

Thomas Groh commented on BEAM-1022:
---

WindowNamespace and WIndowAndTriggerNamespace both also compare windows 
directly in their {{equals}} methods

> WindowNamespace and WindowAndTriggerNamespace should not use Java object 
> equality when comparing windows
> 
>
> Key: BEAM-1022
> URL: https://issues.apache.org/jira/browse/BEAM-1022
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1022) StateNamespace should not use Java object equality when comparing windows

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977973#comment-15977973
 ] 

Thomas Groh commented on BEAM-1022:
---

Specifically, 
https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/StateNamespaces.java#L211
 uses the window for the Cache Key, but there is no requirement that window 
defines a meaningful {{equals}} or {{hashCode}}

> StateNamespace should not use Java object equality when comparing windows
> -
>
> Key: BEAM-1022
> URL: https://issues.apache.org/jira/browse/BEAM-1022
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-2040) Occasional build failures caused by AutoValue

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-2040.
-
   Resolution: Fixed
Fix Version/s: First stable release

> Occasional build failures caused by AutoValue
> -
>
> Key: BEAM-2040
> URL: https://issues.apache.org/jira/browse/BEAM-2040
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: First stable release
>
>
> The following flaky compile failures appear to be fixed in AutoValue 1.4
> {code}
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile 
> (default-compile) on project beam-sdks-java-extensions-gcp-core: Fatal error 
> compiling: java.lang.AssertionError: java.io.IOException: Stream closed -> 
> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile 
> (default-compile) on project beam-sdks-java-extensions-gcp-core: Fatal error 
> compiling
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
>   at 
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:181)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Fatal error 
> compiling
>   at 
> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:954)
>   at 
> org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:137)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>   ... 11 more
> Caused by: org.codehaus.plexus.compiler.CompilerException: 
> java.lang.AssertionError: java.io.IOException: Stream closed
>   at 
> org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:173)
>   at 
> org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile(JavacCompiler.java:174)
>   at 
> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:943)
>   ... 14 more
> Caused by: java.lang.RuntimeException: java.lang.AssertionError: 
> java.io.IOException: Stream closed
>   at com.sun.tools.javac.main.Main.compile(Main.java:553)
>   at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
>   at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
>   at 
> org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:126)
>   ... 16 more
> Caused by: java.lang.AssertionError: java.io.IOException: Stream closed
>   at 
> com.google.auto.value.processor.TemplateVars.parsedTemplateForResource(TemplateVars.java:111)
>   at 
> com.google.auto.value.processor.AutoValueTemplateVars.(AutoValueTemplateVars.java:184)
>   at 
> com.google.auto.value.processor.AutoValueProcessor.processType(AutoValueProcessor.java:441)
>   at 
> com.google.auto.value.processor.AutoValueProcessor.process(AutoValueProcessor.java:150)
>   at 
> com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:794)
>   at 
> com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:705)
>   at 
> com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1800(JavacProcessingEnvironment.java:91)
>   at 
> com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1037)
>   at 
> 

[jira] [Commented] (BEAM-2040) Occasional build failures caused by AutoValue

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977970#comment-15977970
 ] 

ASF GitHub Bot commented on BEAM-2040:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2624


> Occasional build failures caused by AutoValue
> -
>
> Key: BEAM-2040
> URL: https://issues.apache.org/jira/browse/BEAM-2040
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> The following flaky compile failures appear to be fixed in AutoValue 1.4
> {code}
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile 
> (default-compile) on project beam-sdks-java-extensions-gcp-core: Fatal error 
> compiling: java.lang.AssertionError: java.io.IOException: Stream closed -> 
> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile 
> (default-compile) on project beam-sdks-java-extensions-gcp-core: Fatal error 
> compiling
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
>   at 
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:181)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Fatal error 
> compiling
>   at 
> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:954)
>   at 
> org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:137)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>   ... 11 more
> Caused by: org.codehaus.plexus.compiler.CompilerException: 
> java.lang.AssertionError: java.io.IOException: Stream closed
>   at 
> org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:173)
>   at 
> org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile(JavacCompiler.java:174)
>   at 
> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:943)
>   ... 14 more
> Caused by: java.lang.RuntimeException: java.lang.AssertionError: 
> java.io.IOException: Stream closed
>   at com.sun.tools.javac.main.Main.compile(Main.java:553)
>   at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
>   at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
>   at 
> org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:126)
>   ... 16 more
> Caused by: java.lang.AssertionError: java.io.IOException: Stream closed
>   at 
> com.google.auto.value.processor.TemplateVars.parsedTemplateForResource(TemplateVars.java:111)
>   at 
> com.google.auto.value.processor.AutoValueTemplateVars.(AutoValueTemplateVars.java:184)
>   at 
> com.google.auto.value.processor.AutoValueProcessor.processType(AutoValueProcessor.java:441)
>   at 
> com.google.auto.value.processor.AutoValueProcessor.process(AutoValueProcessor.java:150)
>   at 
> com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:794)
>   at 
> com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:705)
>   at 
> com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1800(JavacProcessingEnvironment.java:91)
>   at 
> com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1037)
>   at 
> 

[jira] [Updated] (BEAM-1022) StateNamespace should not use Java object equality when comparing windows

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1022:
--
Summary: StateNamespace should not use Java object equality when comparing 
windows  (was: TableNamespace should not use Java object equality when 
comparing windows)

> StateNamespace should not use Java object equality when comparing windows
> -
>
> Key: BEAM-1022
> URL: https://issues.apache.org/jira/browse/BEAM-1022
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2624: [BEAM-2040] Upgrade autovalue

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2624


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Upgrade autovalue

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 28665490f -> bda16078e


Upgrade autovalue

I believe we are occasionally being hit by these issues fixed in
release.

> We guard against spurious exceptions due to a JDK bug in reading resources 
> from jars. (#365)
>
> We don't propagate an exception if a corrupt jar is found in extension 
> loading.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7e64a8d2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7e64a8d2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7e64a8d2

Branch: refs/heads/master
Commit: 7e64a8d2592dfdba8451654777b246bd473c5330
Parents: 2866549
Author: Dan Halperin 
Authored: Thu Apr 20 19:00:52 2017 -0700
Committer: Dan Halperin 
Committed: Thu Apr 20 19:00:57 2017 -0700

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7e64a8d2/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 638c97f..d7199ef 100644
--- a/pom.xml
+++ b/pom.xml
@@ -114,7 +114,7 @@
 1.4.0
 1.3.0
 1.0-rc2
-1.3
+1.4.1
 0.6.1
 1.22.0
 1.4.5

[2/2] beam git commit: [BEAM-2040] Upgrade autovalue

[lcwik]

[BEAM-2040] Upgrade autovalue

This closes #2624


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bda16078
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bda16078
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bda16078

Branch: refs/heads/master
Commit: bda16078edb48ca5ade4aa5d5a7745a3ab39
Parents: 2866549 7e64a8d
Author: Luke Cwik 
Authored: Thu Apr 20 19:38:40 2017 -0700
Committer: Luke Cwik 
Committed: Thu Apr 20 19:38:40 2017 -0700

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Updated] (BEAM-1316) DoFn#startBundle should not be able to output

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1316:
--
Fix Version/s: First stable release

> DoFn#startBundle should not be able to output
> -
>
> Key: BEAM-1316
> URL: https://issues.apache.org/jira/browse/BEAM-1316
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
> Fix For: First stable release
>
>
> While within startBundle and finishBundle, the window in which elements are 
> output is not generally defined. Elements must always be output from within a 
> windowed context, or the {{WindowFn}} used by the {{PCollection}} may not 
> operate appropriately.
> startBundle and finishBundle are suitable for operational duties, similarly 
> to {{setup}} and {{teardown}}, but within the scope of some collection of 
> input elements. This includes actions such as clearing field state within a 
> DoFn and ensuring all live RPCs complete successfully before committing 
> inputs.
> Sometimes it might be reasonable to output from {{@FinishBundle}} but it is 
> hard to imagine a situation where output from {{@StartBundle}} is useful in a 
> way that doesn't seriously abuse things.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1247) Session state should not be lost when discardingFiredPanes

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977956#comment-15977956
 ] 

Kenneth Knowles commented on BEAM-1247:
---

Grabbed it because it is a blocker, but upon inspection I think it is already 
fixed.

> Session state should not be lost when discardingFiredPanes
> --
>
> Key: BEAM-1247
> URL: https://issues.apache.org/jira/browse/BEAM-1247
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Today when {{discardingFiredPanes}} the entirety of state is cleared, 
> including the state of evolving sessions. This means that with multiple 
> triggerings a single session shows up as multiple. This also stymies 
> downstream stateful computations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2040) Occasional build failures caused by AutoValue

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977949#comment-15977949
 ] 

ASF GitHub Bot commented on BEAM-2040:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2624

[BEAM-2040] Upgrade autovalue

I believe we are occasionally being hit by these issues fixed in
release.

> We guard against spurious exceptions due to a JDK bug in reading 
resources from jars. (#365)
>
> We don't propagate an exception if a corrupt jar is found in extension 
loading.

R: @lukecwik @tgroh @klk @jbonofre  or anyone else ;)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam autovalue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2624


commit 7e64a8d2592dfdba8451654777b246bd473c5330
Author: Dan Halperin 
Date:   2017-04-21T02:00:52Z

Upgrade autovalue

I believe we are occasionally being hit by these issues fixed in
release.

> We guard against spurious exceptions due to a JDK bug in reading 
resources from jars. (#365)
>
> We don't propagate an exception if a corrupt jar is found in extension 
loading.




> Occasional build failures caused by AutoValue
> -
>
> Key: BEAM-2040
> URL: https://issues.apache.org/jira/browse/BEAM-2040
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> The following flaky compile failures appear to be fixed in AutoValue 1.4
> {code}
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile 
> (default-compile) on project beam-sdks-java-extensions-gcp-core: Fatal error 
> compiling: java.lang.AssertionError: java.io.IOException: Stream closed -> 
> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile 
> (default-compile) on project beam-sdks-java-extensions-gcp-core: Fatal error 
> compiling
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
>   at 
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:181)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Fatal error 
> compiling
>   at 
> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:954)
>   at 
> org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:137)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>   ... 11 more
> Caused by: org.codehaus.plexus.compiler.CompilerException: 
> java.lang.AssertionError: java.io.IOException: Stream closed
>   at 
> org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:173)
>   at 
> org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile(JavacCompiler.java:174)
>   at 
> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:943)
>   ... 14 more
> Caused by: java.lang.RuntimeException: java.lang.AssertionError: 
> java.io.IOException: Stream closed
>   at com.sun.tools.javac.main.Main.compile(Main.java:553)
>   at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
>   at 

[GitHub] beam pull request #2624: [BEAM-2040] Upgrade autovalue

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2624

[BEAM-2040] Upgrade autovalue

I believe we are occasionally being hit by these issues fixed in
release.

> We guard against spurious exceptions due to a JDK bug in reading 
resources from jars. (#365)
>
> We don't propagate an exception if a corrupt jar is found in extension 
loading.

R: @lukecwik @tgroh @klk @jbonofre  or anyone else ;)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam autovalue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2624


commit 7e64a8d2592dfdba8451654777b246bd473c5330
Author: Dan Halperin 
Date:   2017-04-21T02:00:52Z

Upgrade autovalue

I believe we are occasionally being hit by these issues fixed in
release.

> We guard against spurious exceptions due to a JDK bug in reading 
resources from jars. (#365)
>
> We don't propagate an exception if a corrupt jar is found in extension 
loading.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2040) Occasional build failures caused by AutoValue

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-2040:
-

 Summary: Occasional build failures caused by AutoValue
 Key: BEAM-2040
 URL: https://issues.apache.org/jira/browse/BEAM-2040
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Daniel Halperin
Assignee: Daniel Halperin


The following flaky compile failures appear to be fixed in AutoValue 1.4

{code}
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) 
on project beam-sdks-java-extensions-gcp-core: Fatal error compiling: 
java.lang.AssertionError: java.io.IOException: Stream closed -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) 
on project beam-sdks-java-extensions-gcp-core: Fatal error compiling
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
at 
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:181)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.maven.plugin.MojoExecutionException: Fatal error compiling
at 
org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:954)
at 
org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:137)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 11 more
Caused by: org.codehaus.plexus.compiler.CompilerException: 
java.lang.AssertionError: java.io.IOException: Stream closed
at 
org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:173)
at 
org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile(JavacCompiler.java:174)
at 
org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:943)
... 14 more
Caused by: java.lang.RuntimeException: java.lang.AssertionError: 
java.io.IOException: Stream closed
at com.sun.tools.javac.main.Main.compile(Main.java:553)
at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
at 
org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess(JavaxToolsCompiler.java:126)
... 16 more
Caused by: java.lang.AssertionError: java.io.IOException: Stream closed
at 
com.google.auto.value.processor.TemplateVars.parsedTemplateForResource(TemplateVars.java:111)
at 
com.google.auto.value.processor.AutoValueTemplateVars.(AutoValueTemplateVars.java:184)
at 
com.google.auto.value.processor.AutoValueProcessor.processType(AutoValueProcessor.java:441)
at 
com.google.auto.value.processor.AutoValueProcessor.process(AutoValueProcessor.java:150)
at 
com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:794)
at 
com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:705)
at 
com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1800(JavacProcessingEnvironment.java:91)
at 
com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1037)
at 
com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1178)
at 
com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1171)
at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:857)
at com.sun.tools.javac.main.Main.compile(Main.java:523)
... 19 more
Caused by: java.io.IOException: Stream closed
at 

[GitHub] beam pull request #2623: [BEAM-1871] Inline and remove POutputValueBase

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2623

[BEAM-1871] Inline and remove POutputValueBase

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This removes a fairly extraneous base class that dates from before Beam and 
Runner API design revisions. Better to have fewer and simpler places to enter 
our APIs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam clean-values

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2623.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2623


commit b9d89b510a34aa5fa44db00cbfa7b5ff188ccc18
Author: Kenneth Knowles 
Date:   2017-04-21T01:47:57Z

Inline and remove POutputValueBase




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (BEAM-2012) Rebuild the Dataflow worker and remove the BoundedSource.splitIntoBundles deprecated method

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-2012.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Rebuild the Dataflow worker and remove the BoundedSource.splitIntoBundles 
> deprecated method
> ---
>
> Key: BEAM-2012
> URL: https://issues.apache.org/jira/browse/BEAM-2012
> Project: Beam
>  Issue Type: Task
>  Components: runner-dataflow
>Reporter: Etienne Chauchot
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> Linked to https://issues.apache.org/jira/browse/BEAM-1272
> Just a reminder for building/cleaning as suggested by [~dhalp...@google.com]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2012) Rebuild the Dataflow worker and remove the BoundedSource.splitIntoBundles deprecated method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977913#comment-15977913
 ] 

ASF GitHub Bot commented on BEAM-2012:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2621


> Rebuild the Dataflow worker and remove the BoundedSource.splitIntoBundles 
> deprecated method
> ---
>
> Key: BEAM-2012
> URL: https://issues.apache.org/jira/browse/BEAM-2012
> Project: Beam
>  Issue Type: Task
>  Components: runner-dataflow
>Reporter: Etienne Chauchot
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> Linked to https://issues.apache.org/jira/browse/BEAM-1272
> Just a reminder for building/cleaning as suggested by [~dhalp...@google.com]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2621

[dhalperi]

This closes #2621


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/28665490
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/28665490
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/28665490

Branch: refs/heads/master
Commit: 28665490f6ad0cad091f4f936a8f113617fd3f27
Parents: e4eae7b abe9e94
Author: Dan Halperin 
Authored: Thu Apr 20 18:29:22 2017 -0700
Committer: Dan Halperin 
Committed: Thu Apr 20 18:29:22 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 .../src/main/java/org/apache/beam/sdk/io/BoundedSource.java | 9 -
 2 files changed, 1 insertion(+), 10 deletions(-)
--

[jira] [Commented] (BEAM-1867) Element counts missing on Cloud Dataflow when PCollection has anything other than hardcoded name pattern

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977907#comment-15977907
 ] 

ASF GitHub Bot commented on BEAM-1867:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2618


> Element counts missing on Cloud Dataflow when PCollection has anything other 
> than hardcoded name pattern
> 
>
> Key: BEAM-1867
> URL: https://issues.apache.org/jira/browse/BEAM-1867
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: First stable release
>
>
> In 0.6.0 and 0.7.0-SNAPSHOT (and possibly all past versions, these are just 
> those where it is confirmed) element count and byte metrics are not reported 
> correctly when the output PCollection for a primitive transform is not 
> {{transformname + ".out" + index}}.
> In 0.7.0-SNAPSHOT, the DataflowRunner uses pipeline surgery to replace the 
> composite {{ParDoSingle}} (that contains a {{ParDoMulti}}) with a 
> Dataflow-specific non-composite {{ParDoSingle}}. So metrics are reported for 
> names like {{"ParDoSingle(MyDoFn).out"}} when they should be reported for 
> {{"ParDoSingle/ParDoMulti(MyDoFn).out"}}. So all single-output ParDo 
> transforms lack these metrics on their outputs.
> In 0.6.0 the same problem occurs if the user ever uses 
> {{PCollection.setName}} to give their collection a meaningful name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2618: [BEAM-1867] Use step-derived PCollection names in D...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2618


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam git commit: This closes #2618: Use step-derived PCollection names in Dataflow

[kenn]

This closes #2618: Use step-derived PCollection names in Dataflow

  Derive Dataflow output names from steps, not PCollection names
  Make crashing errors in Structs unchecked exceptions


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e4eae7bc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e4eae7bc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e4eae7bc

Branch: refs/heads/master
Commit: e4eae7bce4da75744379711d8f45d00b2ba07ad3
Parents: 524165a c9ed8f9
Author: Kenneth Knowles 
Authored: Thu Apr 20 18:19:18 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Apr 20 18:19:18 2017 -0700

--
 .../dataflow/DataflowPipelineTranslator.java|  7 +-
 .../DataflowPipelineTranslatorTest.java | 94 
 .../java/org/apache/beam/sdk/util/Structs.java  | 74 +++
 3 files changed, 131 insertions(+), 44 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e4eae7bc/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
--

[1/3] beam git commit: Make crashing errors in Structs unchecked exceptions

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 524165ac9 -> e4eae7bce


Make crashing errors in Structs unchecked exceptions


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4c0bdd6c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4c0bdd6c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4c0bdd6c

Branch: refs/heads/master
Commit: 4c0bdd6c002b83c67daedd5e01ee2ad0dd47c233
Parents: b47fd52
Author: Kenneth Knowles 
Authored: Thu Apr 20 14:32:29 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Apr 20 14:32:29 2017 -0700

--
 .../java/org/apache/beam/sdk/util/Structs.java  | 74 
 1 file changed, 31 insertions(+), 43 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4c0bdd6c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Structs.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Structs.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Structs.java
index d50b74a..a4be054 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Structs.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Structs.java
@@ -26,23 +26,22 @@ import java.util.Map;
 import javax.annotation.Nullable;
 
 /**
- * A collection of static methods for manipulating datastructure 
representations
- * transferred via the Dataflow API.
+ * A collection of static methods for manipulating datastructure 
representations transferred via the
+ * Dataflow API.
  */
 public final class Structs {
-  private Structs() {}  // Non-instantiable
+  private Structs() {} // Non-instantiable
 
-  public static String getString(Map map, String name) throws 
Exception {
+  public static String getString(Map map, String name) {
 return getValue(map, name, String.class, "a string");
   }
 
   public static String getString(
-  Map map, String name, @Nullable String defaultValue)
-  throws Exception {
+  Map map, String name, @Nullable String defaultValue) {
 return getValue(map, name, String.class, "a string", defaultValue);
   }
 
-  public static byte[] getBytes(Map map, String name) throws 
Exception {
+  public static byte[] getBytes(Map map, String name) {
 @Nullable byte[] result = getBytes(map, name, null);
 if (result == null) {
   throw new ParameterNotFoundException(name, map);
@@ -51,8 +50,8 @@ public final class Structs {
   }
 
   @Nullable
-  public static byte[] getBytes(Map map, String name, 
@Nullable byte[] defaultValue)
-  throws Exception {
+  public static byte[] getBytes(
+  Map map, String name, @Nullable byte[] defaultValue) {
 @Nullable String jsonString = getString(map, name, null);
 if (jsonString == null) {
   return defaultValue;
@@ -63,41 +62,38 @@ public final class Structs {
 return StringUtils.jsonStringToByteArray(jsonString);
   }
 
-  public static Boolean getBoolean(Map map, String name) 
throws Exception {
+  public static Boolean getBoolean(Map map, String name) {
 return getValue(map, name, Boolean.class, "a boolean");
   }
 
   @Nullable
   public static Boolean getBoolean(
-  Map map, String name, @Nullable Boolean defaultValue)
-  throws Exception {
+  Map map, String name, @Nullable Boolean defaultValue) {
 return getValue(map, name, Boolean.class, "a boolean", defaultValue);
   }
 
-  public static Long getLong(Map map, String name) throws 
Exception {
+  public static Long getLong(Map map, String name) {
 return getValue(map, name, Long.class, "a long");
   }
 
   @Nullable
-  public static Long getLong(Map map, String name, @Nullable 
Long defaultValue)
-  throws Exception {
+  public static Long getLong(Map map, String name, @Nullable 
Long defaultValue) {
 return getValue(map, name, Long.class, "a long", defaultValue);
   }
 
-  public static Integer getInt(Map map, String name) throws 
Exception {
+  public static Integer getInt(Map map, String name) {
 return getValue(map, name, Integer.class, "an int");
   }
 
   @Nullable
-  public static Integer getInt(Map map, String name, @Nullable 
Integer defaultValue)
-  throws Exception {
+  public static Integer getInt(
+  Map map, String name, @Nullable Integer defaultValue) {
 return getValue(map, name, Integer.class, "an int", defaultValue);
   }
 
   @Nullable
   public static List getStrings(
-  Map map, 

[2/3] beam git commit: Derive Dataflow output names from steps, not PCollection names

[kenn]

Derive Dataflow output names from steps, not PCollection names

Long ago, PCollection names were assigned after transform replacements took
place, because this happened interleaved with pipeline construction. Now,
runner-independent graphs are constructed with named PCollections and when
replacements occur, the names are preserved. This exposed a bug in Dataflow
whereby the names of steps and the names of PCollections are tightly coupled.

This change uses the mandatory derived names during translation, shielding
users from the bug.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c9ed8f9a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c9ed8f9a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c9ed8f9a

Branch: refs/heads/master
Commit: c9ed8f9a69d2b3f17e782f4bd0da9bd4305f2320
Parents: 4c0bdd6
Author: Kenneth Knowles 
Authored: Thu Apr 20 15:32:51 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Apr 20 15:32:51 2017 -0700

--
 .../dataflow/DataflowPipelineTranslator.java|  7 +-
 .../DataflowPipelineTranslatorTest.java | 94 
 2 files changed, 100 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c9ed8f9a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index abeca4d..0c0a2ef 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -656,7 +656,12 @@ public class DataflowPipelineTranslator {
 
   Map outputInfo = new HashMap<>();
   addString(outputInfo, PropertyNames.OUTPUT_NAME, Long.toString(id));
-  addString(outputInfo, PropertyNames.USER_NAME, value.getName());
+
+  String stepName = getString(properties, PropertyNames.USER_NAME);
+  String generatedName = String.format(
+  "%s.out%d", stepName, outputInfoList.size());
+
+  addString(outputInfo, PropertyNames.USER_NAME, generatedName);
   if (value instanceof PCollection
   && 
translator.runner.doesPCollectionRequireIndexedFormat((PCollection) value)) {
 addBoolean(outputInfo, PropertyNames.USE_INDEXED_FORMAT, true);

http://git-wip-us.apache.org/repos/asf/beam/blob/c9ed8f9a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
index 5016d88..9396169 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
@@ -772,6 +772,100 @@ public class DataflowPipelineTranslatorTest implements 
Serializable {
   }
 
   /**
+   * Test that in translation the name for a collection (in this case just a 
Create output) is
+   * overriden to be what the Dataflow service expects.
+   */
+  @Test
+  public void testNamesOverridden() throws Exception {
+DataflowPipelineOptions options = buildPipelineOptions();
+DataflowRunner runner = DataflowRunner.fromOptions(options);
+options.setStreaming(false);
+DataflowPipelineTranslator translator = 
DataflowPipelineTranslator.fromOptions(options);
+
+Pipeline pipeline = Pipeline.create(options);
+
+pipeline.apply("Jazzy", Create.of(3)).setName("foobizzle");
+
+runner.replaceTransforms(pipeline);
+
+Job job = translator.translate(pipeline,
+runner,
+Collections.emptyList()).getJob();
+
+// The Create step
+Step step = job.getSteps().get(0);
+
+// This is the name that is "set by the user" that the Dataflow translator 
must override
+String userSpecifiedName =
+Structs.getString(
+Structs.getListOfMaps(
+step.getProperties(),
+PropertyNames.OUTPUT_INFO,
+null).get(0),
+PropertyNames.USER_NAME);
+
+// This is the calculated name that must actually be used
+

[jira] [Commented] (BEAM-1917) add 'equal' function to Filter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977867#comment-15977867
 ] 

ASF GitHub Bot commented on BEAM-1917:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/2622

[BEAM-1917] add 'equal' function to Filter

a minor change to add `equal` in 
[Filter.java](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-1917

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2622.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2622


commit 6fd1d57b187915d92e07e051da0550c63fb2be66
Author: mingmxu 
Date:   2017-04-20T21:47:10Z

Add `equal` method to Filter transform

to trigger travis-ci




> add 'equal' function to Filter
> --
>
> Key: BEAM-1917
> URL: https://issues.apache.org/jira/browse/BEAM-1917
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
>
> add an {{equal}} function to avoid {{PredicateT predicate}} for the basic 
> comparison, logic similar as existing {{greaterThan}} ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2622: [BEAM-1917] add 'equal' function to Filter

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/2622

[BEAM-1917] add 'equal' function to Filter

a minor change to add `equal` in 
[Filter.java](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-1917

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2622.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2622


commit 6fd1d57b187915d92e07e051da0550c63fb2be66
Author: mingmxu 
Date:   2017-04-20T21:47:10Z

Add `equal` method to Filter transform

to trigger travis-ci




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2012) Rebuild the Dataflow worker and remove the BoundedSource.splitIntoBundles deprecated method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977857#comment-15977857
 ] 

ASF GitHub Bot commented on BEAM-2012:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2621

[BEAM-2012] Remove BoundedSource.splitIntoBundles for real

R: @tgroh 
CC: @echauchot

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam upgrade-worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2621.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2621


commit 069457a6aead1f2f884563814922bb21c671a8b2
Author: Dan Halperin 
Date:   2017-04-21T00:27:12Z

[BEAM-2012] Remove BoundedSource.splitIntoBundles for real




> Rebuild the Dataflow worker and remove the BoundedSource.splitIntoBundles 
> deprecated method
> ---
>
> Key: BEAM-2012
> URL: https://issues.apache.org/jira/browse/BEAM-2012
> Project: Beam
>  Issue Type: Task
>  Components: runner-dataflow
>Reporter: Etienne Chauchot
>Assignee: Daniel Halperin
>
> Linked to https://issues.apache.org/jira/browse/BEAM-1272
> Just a reminder for building/cleaning as suggested by [~dhalp...@google.com]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2621: [BEAM-2012] Remove BoundedSource.splitIntoBundles f...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2621

[BEAM-2012] Remove BoundedSource.splitIntoBundles for real

R: @tgroh 
CC: @echauchot

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam upgrade-worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2621.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2621


commit 069457a6aead1f2f884563814922bb21c671a8b2
Author: Dan Halperin 
Date:   2017-04-21T00:27:12Z

[BEAM-2012] Remove BoundedSource.splitIntoBundles for real




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1964) Upgrade pylint to 1.7.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977847#comment-15977847
 ] 

ASF GitHub Bot commented on BEAM-1964:
--

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/2619


> Upgrade pylint to 1.7.0
> ---
>
> Key: BEAM-1964
> URL: https://issues.apache.org/jira/browse/BEAM-1964
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Aviem Zur
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>
> Pre-commit tests seem to all be failing on pylint
> For example: 
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/9493/consoleFull



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2619: [BEAM-1964] Remove inner import star

[sb2nov]

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/2619


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PerformanceTests_JDBC #139

[Apache Jenkins Server]

See 


Changes:

[dhalperi] Add javadoc to the Metric interfaces

[dhalperi] Fix javadoc typo

[klk] Remove Sink in favor of FileBasedSink

[lcwik] [BEAM-1786, BEAM-1871] Add the ability to register coder factories for

[lcwik] [BEAM-1871] Move over GcsPath to gcp-core

[dhalperi] [BEAM-59] FileBasedSource: convert to FileSystem

[kirpichov] [BEAM-1992] Move Count.PerElement javadoc to a public place

[kirpichov] Fix erroneous use of .expand() in KafkaIO

[altay] Remove _unused_options_id

--
[...truncated 712.77 KB...]
at 
com.google.cloud.dataflow.worker.runners.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:261)
at 
com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:55)
at 
com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:43)
at 
com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:78)
at 
com.google.cloud.dataflow.worker.runners.worker.MapTaskExecutorFactory.create(MapTaskExecutorFactory.java:152)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.doWork(DataflowWorker.java:271)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:243)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:127)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:94)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
(bd46f0e301a1ccd): java.lang.RuntimeException: 
org.apache.beam.sdk.util.UserCodeException: org.postgresql.util.PSQLException: 
The connection attempt failed.
at 
com.google.cloud.dataflow.worker.runners.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:289)
at 
com.google.cloud.dataflow.worker.runners.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:261)
at 
com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:55)
at 
com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:43)
at 
com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:78)
at 
com.google.cloud.dataflow.worker.runners.worker.MapTaskExecutorFactory.create(MapTaskExecutorFactory.java:152)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.doWork(DataflowWorker.java:271)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:243)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:127)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
at 
com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:94)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.beam.sdk.util.UserCodeException: 
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at 
org.apache.beam.sdk.io.jdbc.JdbcIO$Read$ReadFn$auxiliary$5eJuSjwp.invokeSetup(Unknown
 Source)
at 
com.google.cloud.dataflow.worker.runners.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:66)
at 
com.google.cloud.dataflow.worker.runners.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:48)
at 
com.google.cloud.dataflow.worker.runners.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:99)
at 
com.google.cloud.dataflow.worker.runners.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:70)
at 
com.google.cloud.dataflow.worker.runners.worker.MapTaskExecutorFactory.createParDoOperation(MapTaskExecutorFactory.java:363)
at 

[jira] [Updated] (BEAM-1415) PubsubIO should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-1415:
---
Summary: PubsubIO should comply with PTransform style guide  (was: PubsubIO 
should comply with PTransfrom style guide)

> PubsubIO should comply with PTransform style guide
> --
>
> Key: BEAM-1415
> URL: https://issues.apache.org/jira/browse/BEAM-1415
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>  Labels: backward-incompatible, easy, starter
> Fix For: First stable release
>
>
> Suggested changes:
> - Rename builder methods such as .subscription(), .topic() etc. to 
> .withSubscription, .withTopic()
> - Replace use of Coder from the API (.withCoder()) with a SerializableFunction
> - Rename .withAttributes() to something else, because it sounds like this is 
> a function that sets attributes.
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #333

[Apache Jenkins Server]

See 


Changes:

[dhalperi] Add javadoc to the Metric interfaces

[dhalperi] Fix javadoc typo

[klk] Remove Sink in favor of FileBasedSink

[lcwik] [BEAM-1786, BEAM-1871] Add the ability to register coder factories for

[lcwik] [BEAM-1871] Move over GcsPath to gcp-core

[dhalperi] [BEAM-59] FileBasedSource: convert to FileSystem

[kirpichov] [BEAM-1992] Move Count.PerElement javadoc to a public place

[kirpichov] Fix erroneous use of .expand() in KafkaIO

[altay] Remove _unused_options_id

--
[...truncated 268.19 KB...]
 * [new ref] refs/pull/2551/head -> origin/pr/2551/head
 * [new ref] refs/pull/2551/merge -> origin/pr/2551/merge
 * [new ref] refs/pull/2552/head -> origin/pr/2552/head
 * [new ref] refs/pull/2552/merge -> origin/pr/2552/merge
 * [new ref] refs/pull/2553/head -> origin/pr/2553/head
 * [new ref] refs/pull/2553/merge -> origin/pr/2553/merge
 * [new ref] refs/pull/2554/head -> origin/pr/2554/head
 * [new ref] refs/pull/2554/merge -> origin/pr/2554/merge
 * [new ref] refs/pull/2555/head -> origin/pr/2555/head
 * [new ref] refs/pull/2555/merge -> origin/pr/2555/merge
 * [new ref] refs/pull/2556/head -> origin/pr/2556/head
 * [new ref] refs/pull/2556/merge -> origin/pr/2556/merge
 * [new ref] refs/pull/2557/head -> origin/pr/2557/head
 * [new ref] refs/pull/2557/merge -> origin/pr/2557/merge
 * [new ref] refs/pull/2558/head -> origin/pr/2558/head
 * [new ref] refs/pull/2558/merge -> origin/pr/2558/merge
 * [new ref] refs/pull/2559/head -> origin/pr/2559/head
 * [new ref] refs/pull/2559/merge -> origin/pr/2559/merge
 * [new ref] refs/pull/2560/head -> origin/pr/2560/head
 * [new ref] refs/pull/2560/merge -> origin/pr/2560/merge
 * [new ref] refs/pull/2561/head -> origin/pr/2561/head
 * [new ref] refs/pull/2561/merge -> origin/pr/2561/merge
 * [new ref] refs/pull/2562/head -> origin/pr/2562/head
 * [new ref] refs/pull/2562/merge -> origin/pr/2562/merge
 * [new ref] refs/pull/2563/head -> origin/pr/2563/head
 * [new ref] refs/pull/2564/head -> origin/pr/2564/head
 * [new ref] refs/pull/2564/merge -> origin/pr/2564/merge
 * [new ref] refs/pull/2565/head -> origin/pr/2565/head
 * [new ref] refs/pull/2565/merge -> origin/pr/2565/merge
 * [new ref] refs/pull/2566/head -> origin/pr/2566/head
 * [new ref] refs/pull/2566/merge -> origin/pr/2566/merge
 * [new ref] refs/pull/2567/head -> origin/pr/2567/head
 * [new ref] refs/pull/2567/merge -> origin/pr/2567/merge
 * [new ref] refs/pull/2568/head -> origin/pr/2568/head
 * [new ref] refs/pull/2569/head -> origin/pr/2569/head
 * [new ref] refs/pull/2569/merge -> origin/pr/2569/merge
 * [new ref] refs/pull/2570/head -> origin/pr/2570/head
 * [new ref] refs/pull/2570/merge -> origin/pr/2570/merge
 * [new ref] refs/pull/2571/head -> origin/pr/2571/head
 * [new ref] refs/pull/2571/merge -> origin/pr/2571/merge
 * [new ref] refs/pull/2572/head -> origin/pr/2572/head
 * [new ref] refs/pull/2572/merge -> origin/pr/2572/merge
 * [new ref] refs/pull/2573/head -> origin/pr/2573/head
 * [new ref] refs/pull/2573/merge -> origin/pr/2573/merge
 * [new ref] refs/pull/2574/head -> origin/pr/2574/head
 * [new ref] refs/pull/2575/head -> origin/pr/2575/head
 * [new ref] refs/pull/2575/merge -> origin/pr/2575/merge
 * [new ref] refs/pull/2576/head -> origin/pr/2576/head
 * [new ref] refs/pull/2577/head -> origin/pr/2577/head
 * [new ref] refs/pull/2577/merge -> origin/pr/2577/merge
 * [new ref] refs/pull/2578/head -> origin/pr/2578/head
 * [new ref] refs/pull/2578/merge -> origin/pr/2578/merge
 * [new ref] refs/pull/2579/head -> origin/pr/2579/head
 * [new ref] refs/pull/2579/merge -> origin/pr/2579/merge
 * [new ref] refs/pull/2580/head -> origin/pr/2580/head
 * [new ref] refs/pull/2581/head -> origin/pr/2581/head
 * [new ref] refs/pull/2581/merge -> origin/pr/2581/merge
 * [new ref] refs/pull/2582/head -> origin/pr/2582/head
 * [new ref] refs/pull/2582/merge -> origin/pr/2582/merge
 * [new ref] refs/pull/2583/head -> origin/pr/2583/head
 * [new ref] refs/pull/2583/merge -> origin/pr/2583/merge
 * [new ref] refs/pull/2584/head -> origin/pr/2584/head
 * [new ref] refs/pull/2585/head -> origin/pr/2585/head
 * [new ref] refs/pull/2585/merge -> origin/pr/2585/merge
 * [new ref] refs/pull/2586/head -> origin/pr/2586/head
 * [new ref] refs/pull/2586/merge -> origin/pr/2586/merge
 * [new ref] refs/pull/2587/head -> 

[jira] [Commented] (BEAM-1999) Change signature of set_runtime_options in value_provider.py

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977821#comment-15977821
 ] 

ASF GitHub Bot commented on BEAM-1999:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2608


> Change signature of set_runtime_options in value_provider.py
> 
>
> Key: BEAM-1999
> URL: https://issues.apache.org/jira/browse/BEAM-1999
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: María GH
>
> Change signature of this function to remove _unused_options_id, and add 
> kwargs to make it flexible for future changes



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1415) PubsubIO should comply with PTransfrom style guide

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reassigned BEAM-1415:
--

Assignee: Eugene Kirpichov

> PubsubIO should comply with PTransfrom style guide
> --
>
> Key: BEAM-1415
> URL: https://issues.apache.org/jira/browse/BEAM-1415
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>  Labels: backward-incompatible, easy, starter
> Fix For: First stable release
>
>
> Suggested changes:
> - Rename builder methods such as .subscription(), .topic() etc. to 
> .withSubscription, .withTopic()
> - Replace use of Coder from the API (.withCoder()) with a SerializableFunction
> - Rename .withAttributes() to something else, because it sounds like this is 
> a function that sets attributes.
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2608

[altay]

This closes #2608


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/524165ac
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/524165ac
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/524165ac

Branch: refs/heads/master
Commit: 524165ac95d01e6c4e4b97911e5e4cce78677a93
Parents: aa899e4 2ab6f39
Author: Ahmet Altay 
Authored: Thu Apr 20 16:57:03 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 20 16:57:03 2017 -0700

--
 .../apache_beam/runners/dataflow/internal/apiclient.py   | 1 -
 sdks/python/apache_beam/runners/direct/direct_runner.py  | 8 +++-
 sdks/python/apache_beam/utils/value_provider.py  | 2 +-
 sdks/python/apache_beam/utils/value_provider_test.py | 4 ++--
 4 files changed, 6 insertions(+), 9 deletions(-)
--

[jira] [Commented] (BEAM-1428) KinesisIO should comply with PTransform style guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977817#comment-15977817
 ] 

ASF GitHub Bot commented on BEAM-1428:
--

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/2620

[BEAM-1428] KinesisIO should comply with PTransform style guide

https://issues.apache.org/jira/browse/BEAM-1428

There's lots of other issues in KinesisIO, but this at least brings its API 
surface to a level where making it more style-compliant will not require 
backward-incompatible changes.

R: @jbonofre 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam kinesis-style

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2620.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2620


commit 65b5b0f09dd9a89871a2793e1d8da59f6f45d5a4
Author: Eugene Kirpichov 
Date:   2017-04-20T23:48:23Z

[BEAM-1428] KinesisIO should comply with PTransform style guide




> KinesisIO should comply with PTransform style guide
> ---
>
> Key: BEAM-1428
> URL: https://issues.apache.org/jira/browse/BEAM-1428
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>  Labels: backward-incompatible, starter
>
> Suggested changes:
> - KinesisIO.Read should be a PTransform itself
> - It should have builder methods .withBlah() for setting the parameters, 
> instead of the current somewhat strange combination of the from() factory 
> methods and the using() methods



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1964) Upgrade pylint to 1.7.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977793#comment-15977793
 ] 

ASF GitHub Bot commented on BEAM-1964:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2619

[BEAM-1964] Remove inner import star

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam 
BEAM-1964-remove-inner-import-start

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2619.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2619


commit caac238209746b472e54592ab48d8b12f0d05850
Author: Sourabh Bajaj 
Date:   2017-04-20T23:32:22Z

[BEAM-1964] Remove inner import star




> Upgrade pylint to 1.7.0
> ---
>
> Key: BEAM-1964
> URL: https://issues.apache.org/jira/browse/BEAM-1964
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Aviem Zur
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>
> Pre-commit tests seem to all be failing on pylint
> For example: 
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/9493/consoleFull



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1414) CountingInput should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reassigned BEAM-1414:
--

Assignee: Eugene Kirpichov

> CountingInput should comply with PTransform style guide
> ---
>
> Key: BEAM-1414
> URL: https://issues.apache.org/jira/browse/BEAM-1414
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>
> Suggested changes:
> - Rename the whole class and its inner transforms to sound more verb-like, 
> e.g.: GenerateRange.Bounded/Unbounded (as opposed to current 
> CountingInput.BoundedCountingInput)
> - Provide a more unified API between bounded and unbounded cases: 
> GenerateRange.from(100) should return a GenerateRange.Unbounded; 
> GenerateRange.from(100).to(200) should return a GenerateRange.Bounded. They 
> both should accept a timestampFn. The unbounded one _should not_ have a 
> withMaxNumRecords builder - that's redundant with specifying the range.
> - (optional) Use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2617

[jkff]

This closes #2617


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/aa899e4c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/aa899e4c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/aa899e4c

Branch: refs/heads/master
Commit: aa899e4ce4cb1d360bfdc53fab318f6bb14c42aa
Parents: 8be1dac 0259126
Author: Eugene Kirpichov 
Authored: Thu Apr 20 16:18:26 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Apr 20 16:18:26 2017 -0700

--
 .../src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[1/2] beam git commit: Fix erroneous use of .expand() in KafkaIO

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 8be1dacab -> aa899e4ce


Fix erroneous use of .expand() in KafkaIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/02591269
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/02591269
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/02591269

Branch: refs/heads/master
Commit: 0259126920058ffcd08a235fee68f7cfd3d6ffe4
Parents: 8be1dac
Author: Eugene Kirpichov 
Authored: Thu Apr 20 15:07:58 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Apr 20 16:18:14 2017 -0700

--
 .../src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/02591269/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index 47657bb..fbd96eb 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -561,8 +561,8 @@ public class KafkaIO {
 
 @Override
 public PCollection> expand(PBegin begin) {
-  return read
-  .expand(begin)
+  return begin
+  .apply(read)
   .apply("Remove Kafka Metadata",
   ParDo.of(new DoFn, KV>() {
 @ProcessElement

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3399

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1867) Element counts missing on Cloud Dataflow when PCollection has anything other than hardcoded name pattern

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977752#comment-15977752
 ] 

ASF GitHub Bot commented on BEAM-1867:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2618

[BEAM-1867] Use step-derived PCollection names in Dataflow

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @bjchambers 

This mitigates an issue in Dataflow. I also removed some checked exceptions 
that are never caught and probably never should be.

I have empirically checked that the element counts and byte sizes are 
restored by this change, and added unit tests to the translator. Integration 
tests TBD.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam Dataflow-PCollection-names

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2618.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2618


commit 4c0bdd6c002b83c67daedd5e01ee2ad0dd47c233
Author: Kenneth Knowles 
Date:   2017-04-20T21:32:29Z

Make crashing errors in Structs unchecked exceptions

commit c9ed8f9a69d2b3f17e782f4bd0da9bd4305f2320
Author: Kenneth Knowles 
Date:   2017-04-20T22:32:51Z

Derive Dataflow output names from steps, not PCollection names

Long ago, PCollection names were assigned after transform replacements took
place, because this happened interleaved with pipeline construction. Now,
runner-independent graphs are constructed with named PCollections and when
replacements occur, the names are preserved. This exposed a bug in Dataflow
whereby the names of steps and the names of PCollections are tightly 
coupled.

This change uses the mandatory derived names during translation, shielding
users from the bug.




> Element counts missing on Cloud Dataflow when PCollection has anything other 
> than hardcoded name pattern
> 
>
> Key: BEAM-1867
> URL: https://issues.apache.org/jira/browse/BEAM-1867
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: First stable release
>
>
> In 0.6.0 and 0.7.0-SNAPSHOT (and possibly all past versions, these are just 
> those where it is confirmed) element count and byte metrics are not reported 
> correctly when the output PCollection for a primitive transform is not 
> {{transformname + ".out" + index}}.
> In 0.7.0-SNAPSHOT, the DataflowRunner uses pipeline surgery to replace the 
> composite {{ParDoSingle}} (that contains a {{ParDoMulti}}) with a 
> Dataflow-specific non-composite {{ParDoSingle}}. So metrics are reported for 
> names like {{"ParDoSingle(MyDoFn).out"}} when they should be reported for 
> {{"ParDoSingle/ParDoMulti(MyDoFn).out"}}. So all single-output ParDo 
> transforms lack these metrics on their outputs.
> In 0.6.0 the same problem occurs if the user ever uses 
> {{PCollection.setName}} to give their collection a meaningful name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2618: [BEAM-1867] Use step-derived PCollection names in D...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2618

[BEAM-1867] Use step-derived PCollection names in Dataflow

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @bjchambers 

This mitigates an issue in Dataflow. I also removed some checked exceptions 
that are never caught and probably never should be.

I have empirically checked that the element counts and byte sizes are 
restored by this change, and added unit tests to the translator. Integration 
tests TBD.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam Dataflow-PCollection-names

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2618.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2618


commit 4c0bdd6c002b83c67daedd5e01ee2ad0dd47c233
Author: Kenneth Knowles 
Date:   2017-04-20T21:32:29Z

Make crashing errors in Structs unchecked exceptions

commit c9ed8f9a69d2b3f17e782f4bd0da9bd4305f2320
Author: Kenneth Knowles 
Date:   2017-04-20T22:32:51Z

Derive Dataflow output names from steps, not PCollection names

Long ago, PCollection names were assigned after transform replacements took
place, because this happened interleaved with pipeline construction. Now,
runner-independent graphs are constructed with named PCollections and when
replacements occur, the names are preserved. This exposed a bug in Dataflow
whereby the names of steps and the names of PCollections are tightly 
coupled.

This change uses the mandatory derived names during translation, shielding
users from the bug.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3398

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2577: [BEAM-1992] Move Count.PerElement javadoc to a publ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2577


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1992) Count.perElement javadoc refers to Count.PerElement, but Count.PerElement is private

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977686#comment-15977686
 ] 

ASF GitHub Bot commented on BEAM-1992:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2577


> Count.perElement javadoc refers to Count.PerElement, but Count.PerElement is 
> private
> 
>
> Key: BEAM-1992
> URL: https://issues.apache.org/jira/browse/BEAM-1992
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Wesley Tanaka
>Assignee: Wesley Tanaka
>Priority: Minor
>
> https://beam.apache.org/documentation/sdks/javadoc/0.6.0/org/apache/beam/sdk/transforms/Count.html#perElement--
> says:
> "See Count.PerElement for more details."
> But Count.PerElement isn't in 
> https://beam.apache.org/documentation/sdks/javadoc/0.6.0/allclasses-frame.html
>  -- probably since 
> https://github.com/apache/beam/commit/9957c895b1b1c3e491f288d17e70445c9864742a#diff-c225a00920a8d3505eb15163cb8c573bR108



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3397

[Apache Jenkins Server]

See 

[2/2] beam git commit: This closes #2577

[jkff]

This closes #2577


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8be1daca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8be1daca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8be1daca

Branch: refs/heads/master
Commit: 8be1dacab8bfcf471575039339ba67a9e2d80b8b
Parents: e449188 eb56f99
Author: Eugene Kirpichov 
Authored: Thu Apr 20 15:12:05 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Apr 20 15:12:05 2017 -0700

--
 .../org/apache/beam/sdk/transforms/Count.java   | 26 ++--
 1 file changed, 13 insertions(+), 13 deletions(-)
--

[1/2] beam git commit: [BEAM-1992] Move Count.PerElement javadoc to a public place

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master e44918881 -> 8be1dacab


[BEAM-1992] Move Count.PerElement javadoc to a public place


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/eb56f993
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/eb56f993
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/eb56f993

Branch: refs/heads/master
Commit: eb56f9934726d51c07f1703caaa01cb80b27cc56
Parents: e449188
Author: wtanaka.com 
Authored: Mon Apr 17 21:57:10 2017 -1000
Committer: Eugene Kirpichov 
Committed: Thu Apr 20 15:12:01 2017 -0700

--
 .../org/apache/beam/sdk/transforms/Count.java   | 26 ++--
 1 file changed, 13 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/eb56f993/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Count.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Count.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Count.java
index fd91430..abda1c6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Count.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Count.java
@@ -71,24 +71,17 @@ public class Count {
   }
 
   /**
-   * Returns a {@link PerElement Count.PerElement} {@link PTransform} that 
counts the number of
-   * occurrences of each element in its input {@link PCollection}.
+   * Returns a {@link PTransform} that counts the number of occurrences of 
each element
+   * in its input {@link PCollection}.
*
-   * See {@link PerElement Count.PerElement} for more details.
-   */
-  public static  PTransform>> 
perElement() {
-return new PerElement<>();
-  }
-
-  /**
-   * {@code Count.PerElement} takes a {@code PCollection} and returns a
+   * The returned {@code PTransform} takes a {@code PCollection} and 
returns a
* {@code PCollection>} representing a map from each distinct 
element of the input
* {@code PCollection} to the number of times that element occurs in the 
input. Each key in the
* output {@code PCollection} is unique.
*
-   * This transform compares two values of type {@code T} by first encoding 
each element using
-   * the input {@code PCollection}'s {@code Coder}, then comparing the encoded 
bytes. Because of
-   * this, the input coder must be deterministic.
+   * The returned transform compares two values of type {@code T} by first 
encoding each
+   * element using the input {@code PCollection}'s {@code Coder}, then 
comparing the encoded
+   * bytes. Because of this, the input coder must be deterministic.
* (See {@link org.apache.beam.sdk.coders.Coder#verifyDeterministic()} for 
more detail).
* Performing the comparison in this manner admits efficient parallel 
evaluation.
*
@@ -101,6 +94,13 @@ public class Count {
* PCollection> wordCounts =
* words.apply(Count.perElement());
* } 
+   */
+  public static  PTransform>> 
perElement() {
+return new PerElement<>();
+  }
+
+  /**
+   * Private implementation of {@link #perElement()}.
*
* @param  the type of the elements of the input {@code PCollection}, and 
the type of the keys
* of the output {@code PCollection}

[jira] [Commented] (BEAM-59) IOChannelFactory rethinking/redesign

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977685#comment-15977685
 ] 

ASF GitHub Bot commented on BEAM-59:


Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2563


> IOChannelFactory rethinking/redesign
> 
>
> Key: BEAM-59
> URL: https://issues.apache.org/jira/browse/BEAM-59
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Pei He
> Fix For: First stable release
>
>
> Right now, FileBasedSource and FileBasedSink communication is mediated by 
> IOChannelFactory. There are a number of issues:
> * Global configuration -- e.g., all 'gs://' URIs use the same credentials. 
> This should be per-source/per-sink/etc.
> * Supported APIs -- currently IOChannelFactory is in the "non-public API" 
> util package and subject to change. We need users to be able to add new 
> backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be 
> broken.
> * Per-backend features: e.g., creating buckets in GCS/s3, setting expiration 
> time, etc.
> Updates:
> Design docs posted on dev@ list:
> Part 1: IOChannelFactory Redesign: 
> https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit#
> Part 2: Configurable BeamFileSystem:
> https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/3] beam git commit: [BEAM-59] FileBasedSource: convert to FileSystem

[dhalperi]

[BEAM-59] FileBasedSource: convert to FileSystem

* Make ResourceId serializable
* Update all implementations


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/49b8b230
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/49b8b230
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/49b8b230

Branch: refs/heads/master
Commit: 49b8b2302f4cbc67690617dee788356de2337d65
Parents: 4f8b1cc
Author: Dan Halperin 
Authored: Thu Apr 20 15:08:34 2017 -0700
Committer: Dan Halperin 
Committed: Thu Apr 20 15:08:34 2017 -0700

--
 .../UnboundedReadFromBoundedSource.java |   3 +-
 .../UnboundedReadFromBoundedSourceTest.java |  14 +-
 .../translation/wrappers/SourceInputFormat.java |   3 +-
 .../beam/runners/spark/io/MicrobatchSource.java |   4 +-
 .../java/org/apache/beam/sdk/io/AvroSource.java |  43 +--
 .../apache/beam/sdk/io/BlockBasedSource.java|  10 +-
 .../apache/beam/sdk/io/CompressedSource.java|  26 +-
 .../org/apache/beam/sdk/io/FileBasedSource.java | 355 +++
 .../org/apache/beam/sdk/io/FileSystems.java |  32 +-
 .../org/apache/beam/sdk/io/LocalResourceId.java |  29 +-
 .../java/org/apache/beam/sdk/io/TFRecordIO.java |  17 +-
 .../java/org/apache/beam/sdk/io/TextIO.java |  21 +-
 .../java/org/apache/beam/sdk/io/XmlSource.java  |  12 +-
 .../org/apache/beam/sdk/io/fs/MatchResult.java  |   3 +-
 .../org/apache/beam/sdk/io/fs/ResourceId.java   |   3 +-
 .../org/apache/beam/sdk/util/AvroUtils.java |  24 +-
 .../org/apache/beam/sdk/io/AvroSourceTest.java  |  15 +-
 .../beam/sdk/io/CompressedSourceTest.java   |  12 +-
 .../apache/beam/sdk/io/FileBasedSourceTest.java | 192 +++---
 .../org/apache/beam/sdk/util/AvroUtilsTest.java |   9 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java  |   5 +-
 .../org/apache/beam/sdk/util/gcsfs/GcsPath.java |   5 +-
 .../sdk/io/gcp/bigtable/BigtableIOTest.java |   6 +-
 .../inputformat/HadoopInputFormatIOTest.java|   2 +-
 24 files changed, 349 insertions(+), 496 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/49b8b230/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
index f67af8a..6b99522 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
@@ -61,8 +61,7 @@ import org.slf4j.LoggerFactory;
 /**
  * {@link PTransform} that converts a {@link BoundedSource} as an {@link 
UnboundedSource}.
  *
- * {@link BoundedSource} is read directly without calling
- * {@link BoundedSource#split},
+ * {@link BoundedSource} is read directly without calling {@link 
BoundedSource#split},
  * and element timestamps are propagated. While any elements remain, the 
watermark is the beginning
  * of time {@link BoundedWindow#TIMESTAMP_MIN_VALUE}, and after all elements 
have been produced
  * the watermark goes to the end of time {@link 
BoundedWindow#TIMESTAMP_MAX_VALUE}.

http://git-wip-us.apache.org/repos/asf/beam/blob/49b8b230/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
--
diff --git 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
index c905cf5..0e48a9d 100644
--- 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
+++ 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
@@ -44,8 +44,10 @@ import org.apache.beam.sdk.io.CountingSource;
 import org.apache.beam.sdk.io.FileBasedSource;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
 import org.apache.beam.sdk.testing.CoderProperties;
 

[GitHub] beam pull request #2563: [BEAM-59] FileBasedSource: convert to FileSystem fr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2563


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam git commit: This closes #2563

[dhalperi]

This closes #2563


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e4491888
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e4491888
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e4491888

Branch: refs/heads/master
Commit: e44918881794ef80c83a6a119ec52861bece28a9
Parents: 4f8b1cc 49b8b23
Author: Dan Halperin 
Authored: Thu Apr 20 15:10:14 2017 -0700
Committer: Dan Halperin 
Committed: Thu Apr 20 15:10:14 2017 -0700

--
 .../UnboundedReadFromBoundedSource.java |   3 +-
 .../UnboundedReadFromBoundedSourceTest.java |  14 +-
 .../translation/wrappers/SourceInputFormat.java |   3 +-
 .../beam/runners/spark/io/MicrobatchSource.java |   4 +-
 .../java/org/apache/beam/sdk/io/AvroSource.java |  43 +--
 .../apache/beam/sdk/io/BlockBasedSource.java|  10 +-
 .../apache/beam/sdk/io/CompressedSource.java|  26 +-
 .../org/apache/beam/sdk/io/FileBasedSource.java | 355 +++
 .../org/apache/beam/sdk/io/FileSystems.java |  32 +-
 .../org/apache/beam/sdk/io/LocalResourceId.java |  29 +-
 .../java/org/apache/beam/sdk/io/TFRecordIO.java |  17 +-
 .../java/org/apache/beam/sdk/io/TextIO.java |  21 +-
 .../java/org/apache/beam/sdk/io/XmlSource.java  |  12 +-
 .../org/apache/beam/sdk/io/fs/MatchResult.java  |   3 +-
 .../org/apache/beam/sdk/io/fs/ResourceId.java   |   3 +-
 .../org/apache/beam/sdk/util/AvroUtils.java |  24 +-
 .../org/apache/beam/sdk/io/AvroSourceTest.java  |  15 +-
 .../beam/sdk/io/CompressedSourceTest.java   |  12 +-
 .../apache/beam/sdk/io/FileBasedSourceTest.java | 192 +++---
 .../org/apache/beam/sdk/util/AvroUtilsTest.java |   9 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java  |   5 +-
 .../org/apache/beam/sdk/util/gcsfs/GcsPath.java |   5 +-
 .../sdk/io/gcp/bigtable/BigtableIOTest.java |   6 +-
 .../inputformat/HadoopInputFormatIOTest.java|   2 +-
 24 files changed, 349 insertions(+), 496 deletions(-)
--

[1/3] beam git commit: [BEAM-59] FileBasedSource: convert to FileSystem

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 4f8b1cc22 -> e44918881


http://git-wip-us.apache.org/repos/asf/beam/blob/49b8b230/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSourceTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSourceTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSourceTest.java
index 94a29da..c15e667 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSourceTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSourceTest.java
@@ -26,11 +26,10 @@ import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
 import static org.junit.Assert.fail;
-import static org.mockito.Mockito.when;
 
-import com.google.common.collect.ImmutableList;
 import java.io.ByteArrayOutputStream;
 import java.io.File;
+import java.io.FileNotFoundException;
 import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.nio.channels.ReadableByteChannel;
@@ -39,21 +38,23 @@ import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
 import java.util.NoSuchElementException;
 import java.util.Random;
+import javax.annotation.Nullable;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.io.FileBasedSource.FileBasedReader;
 import org.apache.beam.sdk.io.Source.Reader;
+import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
 import org.apache.beam.sdk.testing.NeedsRunner;
 import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.sdk.util.IOChannelFactory;
-import org.apache.beam.sdk.util.IOChannelUtils;
 import org.apache.beam.sdk.values.PCollection;
 import org.junit.Rule;
 import org.junit.Test;
@@ -62,7 +63,6 @@ import org.junit.rules.ExpectedException;
 import org.junit.rules.TemporaryFolder;
 import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
-import org.mockito.Mockito;
 
 /**
  * Tests code common to all file-based sources.
@@ -70,7 +70,7 @@ import org.mockito.Mockito;
 @RunWith(JUnit4.class)
 public class FileBasedSourceTest {
 
-  Random random = new Random(0L);
+  private Random random = new Random(0L);
 
   @Rule public final TestPipeline p = TestPipeline.create();
   @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
@@ -89,16 +89,16 @@ public class FileBasedSourceTest {
 final String splitHeader;
 
 public TestFileBasedSource(String fileOrPattern, long minBundleSize, 
String splitHeader) {
-  super(fileOrPattern, minBundleSize);
+  super(StaticValueProvider.of(fileOrPattern), minBundleSize);
   this.splitHeader = splitHeader;
 }
 
 public TestFileBasedSource(
-String fileOrPattern,
+Metadata fileOrPattern,
 long minBundleSize,
 long startOffset,
 long endOffset,
-String splitHeader) {
+@Nullable String splitHeader) {
   super(fileOrPattern, minBundleSize, startOffset, endOffset);
   this.splitHeader = splitHeader;
 }
@@ -113,7 +113,7 @@ public class FileBasedSourceTest {
 
 @Override
 protected FileBasedSource createForSubrangeOfFile(
-String fileName, long start, long end) {
+Metadata fileName, long start, long end) {
   return new TestFileBasedSource(fileName, getMinBundleSize(), start, end, 
splitHeader);
 }
 
@@ -397,30 +397,12 @@ public class FileBasedSourceTest {
   }
 
   @Test
-  public void testSplittingUsingFullThreadPool() throws Exception {
-int numFiles = FileBasedSource.THREAD_POOL_SIZE * 5;
-File file0 = null;
-for (int i = 0; i < numFiles; i++) {
-  List data = createStringDataset(3, 1000);
-  File file = createFileWithData("file" + i, data);
-  if (i == 0) {
-file0 = file;
-  }
-}
-
-TestFileBasedSource source =
-new TestFileBasedSource(file0.getParent() + "/" + "file*", 
Long.MAX_VALUE, null);
-List> splits = 
source.split(Long.MAX_VALUE, null);
-assertEquals(numFiles, splits.size());
-  }
-
-  @Test
   public void testSplittingFailsOnEmptyFileExpansion() throws Exception {
 PipelineOptions options = PipelineOptionsFactory.create();
 String missingFilePath = tempFolder.newFolder().getAbsolutePath() + 
"/missing.txt";
 TestFileBasedSource source = new TestFileBasedSource(missingFilePath, 
Long.MAX_VALUE, null);
-thrown.expect(IllegalArgumentException.class);
-thrown.expectMessage(String.format("Unable to 

[jira] [Commented] (BEAM-1897) Remove Sink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977617#comment-15977617
 ] 

ASF GitHub Bot commented on BEAM-1897:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2447


> Remove Sink
> ---
>
> Key: BEAM-1897
> URL: https://issues.apache.org/jira/browse/BEAM-1897
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Davor Bonaci
>
> Sink is really only used for file sinks. We should remove the abstract base 
> class, and just keep around FileBasedSink for file sinks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-2021) Fix Java's Coder class hierarchy

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977613#comment-15977613
 ] 

Luke Cwik edited comment on BEAM-2021 at 4/20/17 9:49 PM:
--

Won't core construction have proto shaded and then shaded proto versions won't 
align?

byte[]/bytebuffer was with the goal of allowing a user to shade their version 
of proto in libraries as well.


was (Author: lcwik):
Won't core construction have proto shaded and then shaded proto versions won't 
align.

byte[]/bytebuffer was with the goal of allowing a user to shade their version 
of proto in libraries as well.

> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/4] beam git commit: Remove Sink in favor of FileBasedSink

[kenn]

http://git-wip-us.apache.org/repos/asf/beam/blob/6a6a1a8c/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java
new file mode 100644
index 000..4f45cc9
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java
@@ -0,0 +1,555 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.ThreadLocalRandom;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.io.FileBasedSink.FileBasedWriteOperation;
+import org.apache.beam.sdk.io.FileBasedSink.FileBasedWriter;
+import org.apache.beam.sdk.io.FileBasedSink.FileResult;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollection.IsBounded;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.PDone;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link PTransform} that writes to a {@link FileBasedSink}. A write begins 
with a sequential
+ * global initialization of a sink, followed by a parallel write, and ends 
with a sequential
+ * finalization of the write. The output of a write is {@link PDone}.
+ *
+ * By default, every bundle in the input {@link PCollection} will be 
processed by a
+ * {@link org.apache.beam.sdk.io.FileBasedSink.FileBasedWriteOperation}, so 
the number of output
+ * will vary based on runner behavior, though at least 1 output will always be 
produced. The
+ * exact parallelism of the write stage can be controlled using {@link 
WriteFiles#withNumShards},
+ * typically used to control how many files are produced or to globally limit 
the number of
+ * workers connecting to an external service. However, this option can often 
hurt performance: it
+ * adds an additional {@link GroupByKey} to the pipeline.
+ *
+ * Example usage with runner-determined sharding:
+ *
+ * {@code p.apply(WriteFiles.to(new MySink(...)));}
+ *
+ * Example usage with a fixed number of shards:
+ *
+ * {@code p.apply(WriteFiles.to(new MySink(...)).withNumShards(3));}
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class WriteFiles extends PTransform {
+  private static final Logger LOG = LoggerFactory.getLogger(WriteFiles.class);
+
+  private static final int UNKNOWN_SHARDNUM = -1;
+  private static final int UNKNOWN_NUMSHARDS = -1;
+
+  private FileBasedSink sink;
+  private FileBasedWriteOperation writeOperation;
+  // This allows the number of shards to be dynamically computed based on the 
input
+  // PCollection.
+  @Nullable
+  private final PTransform 
computeNumShards;
+  // We don't use a side input for static sharding, as we want 

[1/4] beam git commit: Remove Sink in favor of FileBasedSink

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 33078d20c -> 4f8b1cc22


http://git-wip-us.apache.org/repos/asf/beam/blob/6a6a1a8c/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/Sink.java
--
diff --git 
a/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/Sink.java 
b/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/Sink.java
new file mode 100644
index 000..ee349a9
--- /dev/null
+++ b/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/Sink.java
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.hdfs;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.display.HasDisplayData;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+
+/**
+ * This class is deprecated, and only exists for HDFSFileSink.
+ */
+@Deprecated
+public abstract class Sink implements Serializable, HasDisplayData {
+  /**
+   * Ensures that the sink is valid and can be written to before the write 
operation begins. One
+   * should use {@link com.google.common.base.Preconditions} to implement this 
method.
+   */
+  public abstract void validate(PipelineOptions options);
+
+  /**
+   * Returns an instance of a {@link WriteOperation} that can write to this 
Sink.
+   */
+  public abstract WriteOperation createWriteOperation(PipelineOptions 
options);
+
+  /**
+   * {@inheritDoc}
+   *
+   * By default, does not register any display data. Implementors may 
override this method
+   * to provide their own display data.
+   */
+  @Override
+  public void populateDisplayData(DisplayData.Builder builder) {}
+
+  /**
+   * A {@link WriteOperation} defines the process of a parallel write of 
objects to a Sink.
+   *
+   * The {@code WriteOperation} defines how to perform initialization and 
finalization of a
+   * parallel write to a sink as well as how to create a {@link Sink.Writer} 
object that can write
+   * a bundle to the sink.
+   *
+   * Since operations in Beam may be run multiple times for redundancy or 
fault-tolerance,
+   * the initialization and finalization defined by a WriteOperation must 
be idempotent.
+   *
+   * {@code WriteOperation}s may be mutable; a {@code WriteOperation} is 
serialized after the
+   * call to {@code initialize} method and deserialized before calls to
+   * {@code createWriter} and {@code finalized}. However, it is not
+   * reserialized after {@code createWriter}, so {@code createWriter} should 
not mutate the
+   * state of the {@code WriteOperation}.
+   *
+   * See {@link Sink} for more detailed documentation about the process of 
writing to a Sink.
+   *
+   * @param  The type of objects to write
+   * @param  The result of a per-bundle write
+   */
+  public abstract static class WriteOperation implements 
Serializable {
+/**
+ * Performs initialization before writing to the sink. Called before 
writing begins.
+ */
+public abstract void initialize(PipelineOptions options) throws Exception;
+
+/**
+ * Indicates that the operation will be performing windowed writes.
+ */
+public abstract void setWindowedWrites(boolean windowedWrites);
+
+/**
+ * Given an Iterable of results from bundle writes, performs finalization 
after writing and
+ * closes the sink. Called after all bundle writes are complete.
+ *
+ * The results that are passed to finalize are those returned by 
bundles that completed
+ * successfully. Although bundles may have been run multiple times (for 
fault-tolerance), only
+ * one writer result will be passed to finalize for each bundle. An 
implementation of finalize
+ * should perform clean up of any failed and successfully retried bundles. 
 Note that these
+ * failed bundles will not have their writer result passed to finalize, so 
finalize should be
+ * capable of locating any temporary/partial output written by failed 
bundles.
+  

[3/4] beam git commit: Remove Sink in favor of FileBasedSink

[kenn]

Remove Sink in favor of FileBasedSink


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6a6a1a8c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6a6a1a8c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6a6a1a8c

Branch: refs/heads/master
Commit: 6a6a1a8c0d39965e540dbe74ddf73c839ca46889
Parents: 8319369
Author: Reuven Lax 
Authored: Wed Apr 5 12:13:44 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Apr 20 13:49:36 2017 -0700

--
 .../core/construction/PTransformMatchers.java   |   6 +-
 .../construction/PTransformMatchersTest.java|  16 +-
 .../direct/WriteWithShardingFactory.java|  15 +-
 .../direct/WriteWithShardingFactoryTest.java|  18 +-
 .../beam/runners/flink/WriteSinkITCase.java | 192 -
 .../beam/runners/dataflow/DataflowRunner.java   |  16 +-
 .../DataflowPipelineTranslatorTest.java |   2 +-
 .../java/org/apache/beam/sdk/io/AvroIO.java |   6 +-
 .../org/apache/beam/sdk/io/FileBasedSink.java   |  93 +--
 .../main/java/org/apache/beam/sdk/io/Sink.java  | 295 
 .../java/org/apache/beam/sdk/io/TFRecordIO.java |   6 +-
 .../java/org/apache/beam/sdk/io/TextIO.java |   8 +-
 .../main/java/org/apache/beam/sdk/io/Write.java | 603 
 .../java/org/apache/beam/sdk/io/WriteFiles.java | 555 +++
 .../main/java/org/apache/beam/sdk/io/XmlIO.java |  37 +-
 .../java/org/apache/beam/sdk/io/XmlSink.java|   4 +-
 .../java/org/apache/beam/sdk/values/PDone.java  |   3 +-
 .../apache/beam/sdk/io/FileBasedSinkTest.java   |  73 --
 .../java/org/apache/beam/sdk/io/SimpleSink.java |  95 +++
 .../org/apache/beam/sdk/io/WriteFilesTest.java  | 457 
 .../java/org/apache/beam/sdk/io/WriteTest.java  | 705 ---
 .../beam/sdk/runners/TransformTreeTest.java |   6 +-
 .../apache/beam/sdk/io/hdfs/HDFSFileSink.java   |   6 +-
 .../java/org/apache/beam/sdk/io/hdfs/Sink.java  | 195 +
 .../java/org/apache/beam/sdk/io/hdfs/Write.java | 582 +++
 .../beam/sdk/io/hdfs/HDFSFileSinkTest.java  |   1 -
 26 files changed, 2014 insertions(+), 1981 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6a6a1a8c/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
index 09946bc..b2bd7d9 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
@@ -22,7 +22,7 @@ import java.util.HashSet;
 import java.util.Set;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
-import org.apache.beam.sdk.io.Write;
+import org.apache.beam.sdk.io.WriteFiles;
 import org.apache.beam.sdk.runners.PTransformMatcher;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.DoFn;
@@ -268,8 +268,8 @@ public class PTransformMatchers {
 return new PTransformMatcher() {
   @Override
   public boolean matches(AppliedPTransform application) {
-if (application.getTransform() instanceof Write) {
-  Write write = (Write) application.getTransform();
+if (application.getTransform() instanceof WriteFiles) {
+  WriteFiles write = (WriteFiles) application.getTransform();
   return write.getSharding() == null && write.getNumShards() == null;
 }
 return false;

http://git-wip-us.apache.org/repos/asf/beam/blob/6a6a1a8c/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
--
diff --git 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
index 4084cdc..d9bc1e7 100644
--- 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
+++ 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
@@ -30,7 +30,7 @@ import java.util.Collections;
 import org.apache.beam.sdk.coders.VarIntCoder;
 import org.apache.beam.sdk.coders.VoidCoder;
 import 

[jira] [Commented] (BEAM-2021) Fix Java's Coder class hierarchy

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977613#comment-15977613
 ] 

Luke Cwik commented on BEAM-2021:
-

Won't core construction have proto shaded and then shaded proto versions won't 
align.

byte[]/bytebuffer was with the goal of allowing a user to shade their version 
of proto in libraries as well.

> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2021) Fix Java's Coder class hierarchy

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977592#comment-15977592
 ] 

Kenneth Knowles commented on BEAM-2021:
---

If it lives in core construction, it can just have {{Message getPayload}}.

> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1786) AutoService registration of coders, like we do with PipelineRunners

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-1786.
-
Resolution: Fixed

> AutoService registration of coders, like we do with PipelineRunners
> ---
>
> Key: BEAM-1786
> URL: https://issues.apache.org/jira/browse/BEAM-1786
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Today, registering coders for auxiliary data types for a library transform is 
> not very convenient. It the appears in an output/covariant position then it 
> might be possible to use {{getDefaultOutputCoder}} to solve things. But for 
> writes/contravariant positions this is not applicable and the library 
> transform must contort itself to avoid requiring the user to come up with a 
> coder for a type they don't own.
> Probably the best case today is an explicit call to 
> {{LibraryTransform.registerCoders(Pipeline)}} which is far too manual.
> This could likely be solved quite easily with {{@AutoService}} and a static 
> global coder registry, as we do with pipeline runners.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1786) AutoService registration of coders, like we do with PipelineRunners

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977572#comment-15977572
 ] 

Luke Cwik commented on BEAM-1786:
-

Users should look at 
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistrar.java


> AutoService registration of coders, like we do with PipelineRunners
> ---
>
> Key: BEAM-1786
> URL: https://issues.apache.org/jira/browse/BEAM-1786
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Today, registering coders for auxiliary data types for a library transform is 
> not very convenient. It the appears in an output/covariant position then it 
> might be possible to use {{getDefaultOutputCoder}} to solve things. But for 
> writes/contravariant positions this is not applicable and the library 
> transform must contort itself to avoid requiring the user to come up with a 
> coder for a type they don't own.
> Probably the best case today is an explicit call to 
> {{LibraryTransform.registerCoders(Pipeline)}} which is far too manual.
> This could likely be solved quite easily with {{@AutoService}} and a static 
> global coder registry, as we do with pipeline runners.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977566#comment-15977566
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2616


> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2616: [BEAM-1871] Move over GcsPath to gcp-core

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2616


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #1170

[Apache Jenkins Server]

See 


--
[...truncated 239.39 KB...]
 x [deleted] (none) -> origin/pr/942/head
 x [deleted] (none) -> origin/pr/942/merge
 x [deleted] (none) -> origin/pr/943/head
 x [deleted] (none) -> origin/pr/943/merge
 x [deleted] (none) -> origin/pr/944/head
 x [deleted] (none) -> origin/pr/945/head
 x [deleted] (none) -> origin/pr/945/merge
 x [deleted] (none) -> origin/pr/946/head
 x [deleted] (none) -> origin/pr/946/merge
 x [deleted] (none) -> origin/pr/947/head
 x [deleted] (none) -> origin/pr/947/merge
 x [deleted] (none) -> origin/pr/948/head
 x [deleted] (none) -> origin/pr/948/merge
 x [deleted] (none) -> origin/pr/949/head
 x [deleted] (none) -> origin/pr/949/merge
 x [deleted] (none) -> origin/pr/95/head
 x [deleted] (none) -> origin/pr/95/merge
 x [deleted] (none) -> origin/pr/950/head
 x [deleted] (none) -> origin/pr/951/head
 x [deleted] (none) -> origin/pr/951/merge
 x [deleted] (none) -> origin/pr/952/head
 x [deleted] (none) -> origin/pr/952/merge
 x [deleted] (none) -> origin/pr/953/head
 x [deleted] (none) -> origin/pr/954/head
 x [deleted] (none) -> origin/pr/954/merge
 x [deleted] (none) -> origin/pr/955/head
 x [deleted] (none) -> origin/pr/955/merge
 x [deleted] (none) -> origin/pr/956/head
 x [deleted] (none) -> origin/pr/957/head
 x [deleted] (none) -> origin/pr/958/head
 x [deleted] (none) -> origin/pr/959/head
 x [deleted] (none) -> origin/pr/959/merge
 x [deleted] (none) -> origin/pr/96/head
 x [deleted] (none) -> origin/pr/96/merge
 x [deleted] (none) -> origin/pr/960/head
 x [deleted] (none) -> origin/pr/960/merge
 x [deleted] (none) -> origin/pr/961/head
 x [deleted] (none) -> origin/pr/962/head
 x [deleted] (none) -> origin/pr/962/merge
 x [deleted] (none) -> origin/pr/963/head
 x [deleted] (none) -> origin/pr/963/merge
 x [deleted] (none) -> origin/pr/964/head
 x [deleted] (none) -> origin/pr/965/head
 x [deleted] (none) -> origin/pr/965/merge
 x [deleted] (none) -> origin/pr/966/head
 x [deleted] (none) -> origin/pr/967/head
 x [deleted] (none) -> origin/pr/967/merge
 x [deleted] (none) -> origin/pr/968/head
 x [deleted] (none) -> origin/pr/968/merge
 x [deleted] (none) -> origin/pr/969/head
 x [deleted] (none) -> origin/pr/969/merge
 x [deleted] (none) -> origin/pr/97/head
 x [deleted] (none) -> origin/pr/97/merge
 x [deleted] (none) -> origin/pr/970/head
 x [deleted] (none) -> origin/pr/970/merge
 x [deleted] (none) -> origin/pr/971/head
 x [deleted] (none) -> origin/pr/971/merge
 x [deleted] (none) -> origin/pr/972/head
 x [deleted] (none) -> origin/pr/973/head
 x [deleted] (none) -> origin/pr/974/head
 x [deleted] (none) -> origin/pr/974/merge
 x [deleted] (none) -> origin/pr/975/head
 x [deleted] (none) -> origin/pr/975/merge
 x [deleted] (none) -> origin/pr/976/head
 x [deleted] (none) -> origin/pr/976/merge
 x [deleted] (none) -> origin/pr/977/head
 x [deleted] (none) -> origin/pr/977/merge
 x [deleted] (none) -> origin/pr/978/head
 x [deleted] (none) -> origin/pr/978/merge
 x [deleted] (none) -> origin/pr/979/head
 x [deleted] (none) -> origin/pr/979/merge
 x [deleted] (none) -> origin/pr/98/head
 x [deleted] (none) -> origin/pr/980/head
 x [deleted] (none) -> origin/pr/980/merge
 x [deleted] (none) -> origin/pr/981/head
 x [deleted] (none) -> origin/pr/982/head
 x [deleted] (none) -> origin/pr/982/merge
 x [deleted] (none) -> origin/pr/983/head
 x [deleted] (none) -> origin/pr/983/merge
 x [deleted] (none) -> origin/pr/984/head
 x [deleted] (none) -> origin/pr/984/merge
 x [deleted] (none) -> origin/pr/985/head
 x [deleted] (none) -> origin/pr/985/merge
 x [deleted] (none) -> origin/pr/986/head
 x [deleted] (none) -> origin/pr/986/merge
 x [deleted] (none) -> origin/pr/987/head
 x [deleted] (none) -> origin/pr/988/head
 x [deleted] (none) -> origin/pr/988/merge
 x [deleted] (none) -> 

[jira] [Created] (BEAM-2039) number the chapters in page 'Beam Programming Guide'

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-2039:


 Summary: number the chapters in page 'Beam Programming Guide'
 Key: BEAM-2039
 URL: https://issues.apache.org/jira/browse/BEAM-2039
 Project: Beam
  Issue Type: Task
  Components: website
Reporter: Xu Mingmin
Assignee: Davor Bonaci
Priority: Minor


Now more and more content are added to page 'Beam Programming 
Guide'(https://beam.apache.org/documentation/programming-guide/). Not easy to 
read to me, a well numbered table content could give some help.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1786) AutoService registration of coders, like we do with PipelineRunners

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977558#comment-15977558
 ] 

ASF GitHub Bot commented on BEAM-1786:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2605


> AutoService registration of coders, like we do with PipelineRunners
> ---
>
> Key: BEAM-1786
> URL: https://issues.apache.org/jira/browse/BEAM-1786
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Today, registering coders for auxiliary data types for a library transform is 
> not very convenient. It the appears in an output/covariant position then it 
> might be possible to use {{getDefaultOutputCoder}} to solve things. But for 
> writes/contravariant positions this is not applicable and the library 
> transform must contort itself to avoid requiring the user to come up with a 
> coder for a type they don't own.
> Probably the best case today is an explicit call to 
> {{LibraryTransform.registerCoders(Pipeline)}} which is far too manual.
> This could likely be solved quite easily with {{@AutoService}} and a static 
> global coder registry, as we do with pipeline runners.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: [BEAM-1786, BEAM-1871] Add the ability to register coder factories for classes allowing TableRowJsonCoder to move to sdks/java/io/google-cloud-platform

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master b47fd52cb -> 36a12d023


[BEAM-1786, BEAM-1871] Add the ability to register coder factories for classes 
allowing TableRowJsonCoder to move to sdks/java/io/google-cloud-platform


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3c811f5e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3c811f5e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3c811f5e

Branch: refs/heads/master
Commit: 3c811f5eabc823c5895a7b3c62379370bff8b22c
Parents: b47fd52
Author: Luke Cwik 
Authored: Wed Apr 19 17:59:15 2017 -0700
Committer: Luke Cwik 
Committed: Thu Apr 20 14:19:03 2017 -0700

--
 .../org/apache/beam/runners/core/StateTags.java |   6 +-
 .../beam/runners/core/ReduceFnTester.java   |  12 +--
 .../apache/beam/runners/core/StateTagTest.java  |   3 +-
 .../spark/translation/SparkRuntimeContext.java  |   3 +-
 sdks/java/core/pom.xml  |   5 -
 .../main/java/org/apache/beam/sdk/Pipeline.java |   3 +-
 .../apache/beam/sdk/coders/CoderFactories.java  |  22 +++-
 .../apache/beam/sdk/coders/CoderRegistrar.java  |  45 
 .../apache/beam/sdk/coders/CoderRegistry.java   | 108 ++-
 .../apache/beam/sdk/coders/StringUtf8Coder.java |   2 +-
 .../beam/sdk/coders/TableRowJsonCoder.java  |  88 ---
 .../apache/beam/sdk/util/state/StateSpecs.java  |   6 +-
 .../org/apache/beam/SdkCoreApiSurfaceTest.java  |   1 -
 .../beam/sdk/coders/CoderRegistryTest.java  |  77 +++--
 .../beam/sdk/coders/DefaultCoderTest.java   |   8 +-
 .../beam/sdk/coders/TableRowJsonCoderTest.java  |  94 
 .../beam/sdk/transforms/LatestFnTest.java   |   2 +-
 .../org/apache/beam/sdk/transforms/SumTest.java |   2 +-
 .../transforms/reflect/DoFnInvokersTest.java|   4 +-
 sdks/java/io/google-cloud-platform/pom.xml  |   5 +
 .../io/gcp/bigquery/BigQueryCoderRegistrar.java |  39 +++
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|   1 -
 .../sdk/io/gcp/bigquery/BigQuerySourceBase.java |   1 -
 .../io/gcp/bigquery/StreamingWriteTables.java   |   1 -
 .../sdk/io/gcp/bigquery/TableRowInfoCoder.java  |   1 -
 .../sdk/io/gcp/bigquery/TableRowJsonCoder.java  |  91 
 .../sdk/io/gcp/bigquery/TableRowWriter.java |   1 -
 .../bigquery/BigQueryCoderRegistrarTest.java|  40 +++
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java |   1 -
 .../io/gcp/bigquery/FakeBigQueryServices.java   |   1 -
 .../sdk/io/gcp/bigquery/FakeJobService.java |   1 -
 .../io/gcp/bigquery/TableRowJsonCoderTest.java  |  95 
 32 files changed, 479 insertions(+), 290 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3c811f5e/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTags.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTags.java 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTags.java
index 77ae8f5..3a45569 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTags.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTags.java
@@ -47,11 +47,7 @@ import org.apache.beam.sdk.util.state.WatermarkHoldState;
 @Experimental(Kind.STATE)
 public class StateTags {
 
-  private static final CoderRegistry STANDARD_REGISTRY = new CoderRegistry();
-
-  static {
-STANDARD_REGISTRY.registerStandardCoders();
-  }
+  private static final CoderRegistry STANDARD_REGISTRY = 
CoderRegistry.createDefault();
 
   /** @deprecated for migration purposes only */
   @Deprecated

http://git-wip-us.apache.org/repos/asf/beam/blob/3c811f5e/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnTester.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnTester.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnTester.java
index 914550e..549fd8a 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnTester.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnTester.java
@@ -180,8 +180,7 @@ public class ReduceFnTester {
   Coder outputCoder)
   throws Exception {
 
-CoderRegistry registry = new CoderRegistry();
-registry.registerStandardCoders();
+CoderRegistry registry = CoderRegistry.createDefault();
 AppliedCombineFn fn =
 AppliedCombineFn.withInputCoder(
 combineFn, registry, 

[2/2] beam git commit: [BEAM-1786, BEAM-1871] Add the ability to register coder factories

[lcwik]

[BEAM-1786, BEAM-1871] Add the ability to register coder factories

This closes #2605


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/36a12d02
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/36a12d02
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/36a12d02

Branch: refs/heads/master
Commit: 36a12d02371b2fdc216a9b7f7bf1b3ff11c2c53b
Parents: b47fd52 3c811f5
Author: Luke Cwik 
Authored: Thu Apr 20 14:19:31 2017 -0700
Committer: Luke Cwik 
Committed: Thu Apr 20 14:19:31 2017 -0700

--
 .../org/apache/beam/runners/core/StateTags.java |   6 +-
 .../beam/runners/core/ReduceFnTester.java   |  12 +--
 .../apache/beam/runners/core/StateTagTest.java  |   3 +-
 .../spark/translation/SparkRuntimeContext.java  |   3 +-
 sdks/java/core/pom.xml  |   5 -
 .../main/java/org/apache/beam/sdk/Pipeline.java |   3 +-
 .../apache/beam/sdk/coders/CoderFactories.java  |  22 +++-
 .../apache/beam/sdk/coders/CoderRegistrar.java  |  45 
 .../apache/beam/sdk/coders/CoderRegistry.java   | 108 ++-
 .../apache/beam/sdk/coders/StringUtf8Coder.java |   2 +-
 .../beam/sdk/coders/TableRowJsonCoder.java  |  88 ---
 .../apache/beam/sdk/util/state/StateSpecs.java  |   6 +-
 .../org/apache/beam/SdkCoreApiSurfaceTest.java  |   1 -
 .../beam/sdk/coders/CoderRegistryTest.java  |  77 +++--
 .../beam/sdk/coders/DefaultCoderTest.java   |   8 +-
 .../beam/sdk/coders/TableRowJsonCoderTest.java  |  94 
 .../beam/sdk/transforms/LatestFnTest.java   |   2 +-
 .../org/apache/beam/sdk/transforms/SumTest.java |   2 +-
 .../transforms/reflect/DoFnInvokersTest.java|   4 +-
 sdks/java/io/google-cloud-platform/pom.xml  |   5 +
 .../io/gcp/bigquery/BigQueryCoderRegistrar.java |  39 +++
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|   1 -
 .../sdk/io/gcp/bigquery/BigQuerySourceBase.java |   1 -
 .../io/gcp/bigquery/StreamingWriteTables.java   |   1 -
 .../sdk/io/gcp/bigquery/TableRowInfoCoder.java  |   1 -
 .../sdk/io/gcp/bigquery/TableRowJsonCoder.java  |  91 
 .../sdk/io/gcp/bigquery/TableRowWriter.java |   1 -
 .../bigquery/BigQueryCoderRegistrarTest.java|  40 +++
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java |   1 -
 .../io/gcp/bigquery/FakeBigQueryServices.java   |   1 -
 .../sdk/io/gcp/bigquery/FakeJobService.java |   1 -
 .../io/gcp/bigquery/TableRowJsonCoderTest.java  |  95 
 32 files changed, 479 insertions(+), 290 deletions(-)
--

[jira] [Commented] (BEAM-2005) Add a Hadoop FileSystem implementation of Beam's FileSystem

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977537#comment-15977537
 ] 

Stephen Sisk commented on BEAM-2005:


There are a couple possible ways we could adapt to this: 
* Potentially we could give different connections different schemas, but that 
falls apart if someone wants to use URIs generated elsewhere
* Start passing in the FileSystem object as an option on the read transform 
(like hadoop does) - this also incidentally solves the problem of "how will 
people know if hdfs is in the set of modules loaded on your system" problem 
that was discussed above - they'll need to instantiate the instance themselves 
and they'll go through their normal discovery mechanism for doing so.

> Add a Hadoop FileSystem implementation of Beam's FileSystem
> ---
>
> Key: BEAM-2005
> URL: https://issues.apache.org/jira/browse/BEAM-2005
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Beam's FileSystem creates an abstraction for reading from files in many 
> different places. 
> We should add a Hadoop FileSystem implementation 
> (https://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/fs/FileSystem.html)
>  - that would enable us to read from any file system that implements 
> FileSystem (including HDFS, azure, s3, etc..)
> I'm investigating this now.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2021) Fix Java's Coder class hierarchy

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977506#comment-15977506
 ] 

Luke Cwik commented on BEAM-2021:
-

I believe CoderEncoder interface lives in core construction (or another 
module...) and that implementers in other modules just give us 
byte[]/bytebuffer which we can decode to be an any.proto.

This way they can depend on and shade proto version and always give us 
byte[]/bytebuffer which we can then decode to our version of an Any proto.

> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977494#comment-15977494
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/2616

[BEAM-1871] Move over GcsPath to gcp-core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Note that google-auth-library-credentials was specified twice within the 
same pom file.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam thin_sdk_core3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2616.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2616






> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2616: [BEAM-1871] Move over GcsPath to gcp-core

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/2616

[BEAM-1871] Move over GcsPath to gcp-core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Note that google-auth-library-credentials was specified twice within the 
same pom file.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam thin_sdk_core3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2616.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2616






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-2036) PTransform identifier(name) documentation in website.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-2036:

Description: 
PTransform identifier(name) documentation in the website.
Section 'Applying transforms' - 
https://beam.apache.org/documentation/programming-guide/#transforms
See parent issue BEAM-2035 for more information.

  was:
PTransform identifier(name) documentation in website.
Section 'Applying transforms' - 
https://beam.apache.org/documentation/programming-guide/#transforms
See parent issue BEAM-2035 for more information.


> PTransform identifier(name) documentation in website.
> -
>
> Key: BEAM-2036
> URL: https://issues.apache.org/jira/browse/BEAM-2036
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Aviem Zur
>Assignee: Davor Bonaci
>
> PTransform identifier(name) documentation in the website.
> Section 'Applying transforms' - 
> https://beam.apache.org/documentation/programming-guide/#transforms
> See parent issue BEAM-2035 for more information.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2038) PTransform identifier(name) documentation in Python SDK.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-2038:

Description: 
PTransform identifier(name) documentation in Python SDK.

Specifically address this in in documentation of application of a 
{{PTransform}} in the pipeline and in code pertaining to metrics.

See parent issue BEAM-2035 for more information.
Take a look at the Java SDK subtask as well: BEAM-2037

  was:
PTransform identifier(name) documentation in Python SDK.

Specifically address this in address this in documentation on application of a 
{{PTransform}} in the pipeline and in code pertaining to metrics.

See parent issue BEAM-2035 for more information.
Take a look at the Java SDK subtask as well: BEAM-2037


> PTransform identifier(name) documentation in Python SDK.
> 
>
> Key: BEAM-2038
> URL: https://issues.apache.org/jira/browse/BEAM-2038
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py
>Reporter: Aviem Zur
>Assignee: Ahmet Altay
>
> PTransform identifier(name) documentation in Python SDK.
> Specifically address this in in documentation of application of a 
> {{PTransform}} in the pipeline and in code pertaining to metrics.
> See parent issue BEAM-2035 for more information.
> Take a look at the Java SDK subtask as well: BEAM-2037



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2021) Fix Java's Coder class hierarchy

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977470#comment-15977470
 ] 

Kenneth Knowles commented on BEAM-2021:
---

Yea, that's pretty much it. We'll have an added layer of "CoderEncoders" 
registered as services.


For core things, somewhat easy:
 - In SDK core: KvCoder with just basic accessors getKeyCoder() etc.
 - In core construction:
   - KvCoderEncoder that knows the URN and to put the getKeyCoder() and 
getValueCoder() into the component coders
   - If there were some payload that needed to be added, it can be built 
directly as a proto, so that it is cross-SDK (and cross-runner if they need to 
know about it). What the payload might be depends on the URN, so you can have a 
HeapCoder with explicit component coders but also a java-serialized comparator.

For non-core stuff like IOs or library transforms, it should be similar. In the 
extension library, include:
 - The FooCoder with its natural interface
 - A registered service for its FooCoderEncoder

So what I mean by the tricky bit is the design decision between:
 - CoderEncoder interface lives in the SDK and does not have proto on its API 
surface (or we figure out a way for this to be safe)
 - CoderEncoder interface lives in core construction (or another module...) and 
IOs that want to have cross-language/grokkable/compact encodings take a 
dependency
 - Other option?



> Fix Java's Coder class hierarchy
> 
>
> Key: BEAM-2021
> URL: https://issues.apache.org/jira/browse/BEAM-2021
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Affects Versions: First stable release
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> This is thoroughly out of hand. In the runner API world, there are two paths:
> 1. URN plus component coders plus custom payload (in the form of component 
> coders alongside an SdkFunctionSpec)
> 2. Custom coder (a single URN) and payload is serialized Java. I think this 
> never has component coders.
> The other base classes have now been shown to be extraneous: they favor 
> saving ~3 lines of boilerplate for rarely written code at the cost of 
> readability. Instead they should just be dropped.
> The custom payload is an Any proto in the runner API. But tying the Coder 
> interface to proto would be unfortunate from a design perspective and cannot 
> be done anyhow due to dependency hell.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2038) PTransform identifier(name) documentation in Python SDK.

[Aviem Zur (JIRA)]

Aviem Zur created BEAM-2038:
---

 Summary: PTransform identifier(name) documentation in Python SDK.
 Key: BEAM-2038
 URL: https://issues.apache.org/jira/browse/BEAM-2038
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py
Reporter: Aviem Zur
Assignee: Ahmet Altay


PTransform identifier(name) documentation in Python SDK.

Specifically address this in address this in documentation on application of 
{{PTransform}}s in the pipeline and in code pertaining to metrics.

See parent issue BEAM-2035 for more information.
Take a look at the Java SDK subtask as well: BEAM-2037



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2037) PTransform identifier(name) documentation in Java SDK.

[Aviem Zur (JIRA)]

Aviem Zur created BEAM-2037:
---

 Summary: PTransform identifier(name) documentation in Java SDK.
 Key: BEAM-2037
 URL: https://issues.apache.org/jira/browse/BEAM-2037
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-core
Reporter: Aviem Zur
Assignee: Davor Bonaci


PTransform identifier(name) documentation in Java SDK.

Specifically address this in address this in Javadoc of {{Pipeline#apply}}, 
{{PBegin#apply}}, {{PCollection#apply}}, {{Metrics}}, {{MetricsResult}}, 
{{MetricsFilter}}.

Documentation around Metrics {{step}} (Which corresponds to the PTransform 
application identifier) in the classes mentioned above is a bit confusing 
currently.

See parent issue BEAM-2035 for more information.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2036) PTransform identifier(name) documentation in website.

[Aviem Zur (JIRA)]

Aviem Zur created BEAM-2036:
---

 Summary: PTransform identifier(name) documentation in website.
 Key: BEAM-2036
 URL: https://issues.apache.org/jira/browse/BEAM-2036
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Aviem Zur
Assignee: Davor Bonaci


PTransform identifier(name) documentation in website.
Section 'Applying transforms' - 
https://beam.apache.org/documentation/programming-guide/#transforms
See parent issue BEAM-2035 for more information.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1773) Consider allowing Source#validate() to throw exception

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957130#comment-15957130
 ] 

Ted Yu edited comment on BEAM-1773 at 4/20/17 8:22 PM:
---

[~kenn] [~j...@nanthrax.net]:
Please kindly provide your feedback.


was (Author: yuzhih...@gmail.com):
[~kenn] [~j...@nanthrax.net]:

Please kindly provide your feedback.

> Consider allowing Source#validate() to throw exception
> --
>
> Key: BEAM-1773
> URL: https://issues.apache.org/jira/browse/BEAM-1773
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: beam-1773.v1.patch, beam-1773.v2.patch
>
>
> In HDFSFileSource.java :
> {code}
>   @Override
>   public void validate() {
> ...
>   } catch (IOException | InterruptedException e) {
> throw new RuntimeException(e);
>   }
> {code}
> Source#validate() should be allowed to throw exception so that we don't 
> resort to using RuntimeException.
> Here was related thread on mailing list:
> http://search-hadoop.com/m/Beam/gfKHFOwE0uETxae?subj=Re+why+Source+validate+is+not+declared+to+throw+any+exception



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-897) Datastore ITs have invalid PipelineOptions

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-897.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Datastore ITs have invalid PipelineOptions
> --
>
> Key: BEAM-897
> URL: https://issues.apache.org/jira/browse/BEAM-897
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Vikas Kedigehalli
> Fix For: Not applicable
>
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> Datastore ITs currently do have multiple incompatible defaults, and this 
> should be rectified.
> cc [~pei...@gmail.com] [~lcwik]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-976) Update examples README.md to fix instructions to run pipelines

[Vikas Kedigehalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977371#comment-15977371
 ] 

Vikas Kedigehalli commented on BEAM-976:


[~davor] Is https://github.com/apache/beam/tree/master/examples/java a legit 
doc that users can rely on? If so, it seems outdated / needs fixing. 

> Update examples README.md to fix instructions to run pipelines
> --
>
> Key: BEAM-976
> URL: https://issues.apache.org/jira/browse/BEAM-976
> Project: Beam
>  Issue Type: Task
>  Components: examples-java
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>Priority: Minor
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1956) Flatten operation should respect input type hints.

[Vikas Kedigehalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Kedigehalli reassigned BEAM-1956:
---

Assignee: Ahmet Altay  (was: Vikas Kedigehalli)

> Flatten operation should respect input type hints.
> --
>
> Key: BEAM-1956
> URL: https://issues.apache.org/jira/browse/BEAM-1956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Vikas Kedigehalli
>Assignee: Ahmet Altay
> Fix For: First stable release
>
>
> Input type hints are currently not respected by the Flatten operation and 
> instead `Any` type is chosen as a fallback. This could lead to using a pickle 
> coder even if there was a custom coder type hint provided for input 
> PCollections. 
> Also, this could lead to undesirable results, particularly, when a Flatten 
> operation is followed by a GroupByKey operation which requires the key coder 
> to be deterministic. Even if the user provides deterministic coder type hints 
> to their PCollections, defaulting to Any would result in using the pickle 
> coder (non-deterministic). As a result of this, CoGroupByKey is broken in 
> such scenarios where input PCollection coder is deterministic for the type 
> while pickle coder is not.   



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3396

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1353) Beam should comply with PTransform style guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977330#comment-15977330
 ] 

ASF GitHub Bot commented on BEAM-1353:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2615

Use the Input Coder in Dataflow Pubsub Write

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
If no element coder is specified, and one is required, use the element
coder of the input PCollection as the encoding input. The coder is known
to have the appropriate type.

R: @reuvenlax 

This is likely going away as part of 
https://issues.apache.org/jira/browse/BEAM-1353;
we should require a `T -> bytes` conversion to be explicitly specified 
rather than using a Coder.
Right now the input coder is not always defined. This fixes that bug.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam pubsub_write_coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2615.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2615


commit 4dd597621a9fce3fc95d6a265d4c9eaf70648326
Author: Thomas Groh 
Date:   2017-04-20T19:29:24Z

Use the Input Coder in Dataflow Pubsub Write

If no element coder is specified, and one is required, use the element
coder of the input PCollection as the encoding input. The coder is known
to have the appropriate type.




> Beam should comply with PTransform style guide
> --
>
> Key: BEAM-1353
> URL: https://issues.apache.org/jira/browse/BEAM-1353
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core, sdk-java-extensions, sdk-java-gcp, sdk-py
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>  Labels: backward-incompatible, easy, starter
> Fix For: First stable release
>
>
> https://beam.apache.org/contribute/ptransform-style-guide/ is a collection of 
> best practices for developing PTransform's.
> Some of these practices shaped up relatively recently, and some Beam 
> transforms violate them, introducing inconsistency into Beam APIs and 
> tempting new library writers and users into mimicking code that's best not 
> mimicked.
> This is an umbrella bug for cleanups bringing individual Beam transforms in 
> accordance with the style guide.
> Please file 1 bug per transform that needs to be fixed (e.g. "Bring KafkaIO 
> in accordance with style guide"); in the bug, describe which aspects need to 
> be fixed (e.g. "should use AutoValue; should explicitly specify type 
> parameters; should not use Coders for serialization") and link it with the 
> current bug.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2615: Use the Input Coder in Dataflow Pubsub Write

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2615

Use the Input Coder in Dataflow Pubsub Write

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
If no element coder is specified, and one is required, use the element
coder of the input PCollection as the encoding input. The coder is known
to have the appropriate type.

R: @reuvenlax 

This is likely going away as part of 
https://issues.apache.org/jira/browse/BEAM-1353;
we should require a `T -> bytes` conversion to be explicitly specified 
rather than using a Coder.
Right now the input coder is not always defined. This fixes that bug.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam pubsub_write_coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2615.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2615


commit 4dd597621a9fce3fc95d6a265d4c9eaf70648326
Author: Thomas Groh 
Date:   2017-04-20T19:29:24Z

Use the Input Coder in Dataflow Pubsub Write

If no element coder is specified, and one is required, use the element
coder of the input PCollection as the encoding input. The coder is known
to have the appropriate type.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #1725

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #2427

[Apache Jenkins Server]

See