[GitHub] beam pull request #3223: [BEAM-2354] Add a ReadStringsFromPubSub/WriteString...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3223

[BEAM-2354] Add a ReadStringsFromPubSub/WriteStringsToPubSub PTransform

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
I hid existing PubSubSource/PubSubSink which is backwards incompatible but 
the PTransform didn't work anyways.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam pubsub

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3223.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3223


commit 6d52705a1406db3965024203abb9cc9d897cff95
Author: Luke Cwik 
Date:   2017-05-25T01:14:59Z

[BEAM-2354] Add a ReadStringsFromPubSub/WriteStringsToPubSub PTransform




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2354) Add streaming support to Python DataflowRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024145#comment-16024145
 ] 

ASF GitHub Bot commented on BEAM-2354:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3223

[BEAM-2354] Add a ReadStringsFromPubSub/WriteStringsToPubSub PTransform

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
I hid existing PubSubSource/PubSubSink which is backwards incompatible but 
the PTransform didn't work anyways.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam pubsub

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3223.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3223


commit 6d52705a1406db3965024203abb9cc9d897cff95
Author: Luke Cwik 
Date:   2017-05-25T01:14:59Z

[BEAM-2354] Add a ReadStringsFromPubSub/WriteStringsToPubSub PTransform




> Add streaming support to Python DataflowRunner
> --
>
> Key: BEAM-2354
> URL: https://issues.apache.org/jira/browse/BEAM-2354
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Luke Cwik
>Assignee: Ahmet Altay
>
> Add support for Python streaming pipelines to the DataflowRunner.
> This is a tracking bug for minor work items not worthy of a JIRA.
> Major work items should have their own sub-task.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2313

[Apache Jenkins Server]

See 


--
[...truncated 42.31 KB...]

Processing ./target/.tox/dist/apache-beam-2.1.0.dev.zip
Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.1.0.dev0)
Collecting crcmod<2.0,>=1.7 (from apache-beam==2.1.0.dev0)
Collecting dill==0.2.6 (from apache-beam==2.1.0.dev0)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
apache-beam==2.1.0.dev0)
Collecting httplib2<0.10,>=0.8 (from apache-beam==2.1.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.1.0.dev0)
  Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==2.1.0.dev0)
Requirement already satisfied: protobuf<=3.3.0,>=3.2.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
apache-beam==2.1.0.dev0)
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.1.0.dev0)
Requirement already satisfied: futures>=2.2.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.1.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.1.0.dev0)
Requirement already satisfied: six>=1.5.2 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.1.0.dev0)
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.1.0.dev0)
  Using cached pbr-3.0.1-py2.py3-none-any.whl
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.1.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.1.0.dev0)
  Using cached pyasn1-0.2.3-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.1.0.dev0)
  Using cached pyasn1_modules-0.0.8-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==2.1.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
protobuf<=3.3.0,>=3.2.0->apache-beam==2.1.0.dev0)
Requirement already satisfied: appdirs>=1.4.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
setuptools->protobuf<=3.3.0,>=3.2.0->apache-beam==2.1.0.dev0)
Requirement already satisfied: packaging>=16.8 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
setuptools->protobuf<=3.3.0,>=3.2.0->apache-beam==2.1.0.dev0)
Requirement already satisfied: pyparsing in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
packaging>=16.8->setuptools->protobuf<=3.3.0,>=3.2.0->apache-beam==2.1.0.dev0)
Building wheels for collected packages: apache-beam
  Running setup.py bdist_wheel for apache-beam: started
  Running setup.py bdist_wheel for apache-beam: finished with status 'error'
  Complete output from command 

 -u -c "import setuptools, 
tokenize;__file__='/tmp/pip-ouS2wU-build/setup.py';f=getattr(tokenize, 'open', 
open)(__file__);code=f.read().replace('\r\n', 
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d 
/tmp/tmpfaAlMApip-wheel- --python-tag cp27:
  
:334:
 UserWarning: Normalizing '2.1.0.dev' to '2.1.0.dev0'
normalized_version,
  running bdist_wheel
  running build
  running build_py
  Traceback (most recent call last):
File "", line 1, in 
File "/tmp/pip-ouS2wU-build/setup.py", line 189, in 
  'test': generate_protos_first(test),
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
  dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
  self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File 
"
 line 199, in run
  self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
  self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/tmp/pip-ouS2wU-build/setup.py", line 132, in run
  gen_protos.generate_proto_files()
File 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #2168

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3222: [BEAM-115] Unify Java and Python WindowingStragegy ...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3222

[BEAM-115] Unify Java and Python WindowingStragegy representations.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam runner-windows

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3222.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3222


commit 5fb2043e7f25871f4e9ad7de0e6c3380290b0c21
Author: Robert Bradshaw 
Date:   2017-05-25T00:23:31Z

Unify Java and Python WindowingStragegy representations.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023946#comment-16023946
 ] 

ASF GitHub Bot commented on BEAM-115:
-

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3222

[BEAM-115] Unify Java and Python WindowingStragegy representations.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam runner-windows

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3222.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3222


commit 5fb2043e7f25871f4e9ad7de0e6c3380290b0c21
Author: Robert Bradshaw 
Date:   2017-05-25T00:23:31Z

Unify Java and Python WindowingStragegy representations.




> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-2321) gRPC configuration failure using DataflowRunner and Bigtable

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-2321.
-
   Resolution: Not A Problem
Fix Version/s: Not applicable

> gRPC configuration failure using DataflowRunner and Bigtable
> 
>
> Key: BEAM-2321
> URL: https://issues.apache.org/jira/browse/BEAM-2321
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.0.0
>Reporter: Nigel Kilmer
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> I'm attempting to run a pipeline that uses the DataflowRunner and writes to 
> Bigtable (v0.9.6.2). This exception is thrown (looks like it's when the 
> BigtableSession is being created):
> java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly 
> configured.
>   at 
> io.grpc.netty.GrpcSslContexts.selectApplicationProtocolConfig(GrpcSslContexts.java:174)
>   at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:151)
>   at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:139)
>   at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:109)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.createSslContext(BigtableSession.java:124)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.access$000(BigtableSession.java:81)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession$2.run(BigtableSession.java:151)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I can run the same pipeline without issue using the DirectRunner instead, and 
> it was also working for me using the 0.7.0 snapshot of Beam last week. I've 
> already checked with the cloud-bigtable-client project; they said that it 
> should be working since I have a dependency on netty_tcnative configured. The 
> fact that the same pipeline works with the DirectRunner and not with the 
> DataflowRunner makes me think it's a DataflowRunner bug.
> My pipeline is pretty simple; it looks like this:
> Pipeline p = Pipeline.create(gcpOptions);
> p.apply(PubsubIO.readProtos(TestProto.class)
> .fromSubscription(pubsubSubscription))
>   .apply(ParDo.of(new BigtableMutationTransform()))
>   
> .apply(BigtableIO.write().withBigtableOptions(bigtableOptionsBuilder).withTableId("table_id"));
> p.run();
> Let me know if you need more context.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2926

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2321) gRPC configuration failure using DataflowRunner and Bigtable

[Nigel Kilmer (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023922#comment-16023922
 ] 

Nigel Kilmer commented on BEAM-2321:


Yeah, you're correct, this is not actually a bug. I recreated the same pipeline 
so I can build it with Maven, and it works fine.

Looks like I have these versions of those libraries available:
gRPC: 1.3.0, Beam wants 1.2.0
protobuf: 3.1.0, Beam wants 3.2.0
netty-tcnative: 1.1.33.Fork9, Beam wants 1.1.33.Fork26

Since I'm stuck with those versions, I tried building Beam against them, but I 
got the same Jetty configuration error.

Anyway, thanks for the suggestions. I think this can be closed.

> gRPC configuration failure using DataflowRunner and Bigtable
> 
>
> Key: BEAM-2321
> URL: https://issues.apache.org/jira/browse/BEAM-2321
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.0.0
>Reporter: Nigel Kilmer
>Assignee: Daniel Halperin
>
> I'm attempting to run a pipeline that uses the DataflowRunner and writes to 
> Bigtable (v0.9.6.2). This exception is thrown (looks like it's when the 
> BigtableSession is being created):
> java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly 
> configured.
>   at 
> io.grpc.netty.GrpcSslContexts.selectApplicationProtocolConfig(GrpcSslContexts.java:174)
>   at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:151)
>   at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:139)
>   at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:109)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.createSslContext(BigtableSession.java:124)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.access$000(BigtableSession.java:81)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession$2.run(BigtableSession.java:151)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I can run the same pipeline without issue using the DirectRunner instead, and 
> it was also working for me using the 0.7.0 snapshot of Beam last week. I've 
> already checked with the cloud-bigtable-client project; they said that it 
> should be working since I have a dependency on netty_tcnative configured. The 
> fact that the same pipeline works with the DirectRunner and not with the 
> DataflowRunner makes me think it's a DataflowRunner bug.
> My pipeline is pretty simple; it looks like this:
> Pipeline p = Pipeline.create(gcpOptions);
> p.apply(PubsubIO.readProtos(TestProto.class)
> .fromSubscription(pubsubSubscription))
>   .apply(ParDo.of(new BigtableMutationTransform()))
>   
> .apply(BigtableIO.write().withBigtableOptions(bigtableOptionsBuilder).withTableId("table_id"));
> p.run();
> Let me know if you need more context.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #50

[Apache Jenkins Server]

See 


Changes:

[klk] Rename PCollections to PCollectionTranslation

[klk] Rename PTransforms to PTransformTranslation

[klk] Rename ParDos to ParDoTranslation

[klk] Rename WindowIntoTranslator to WindowIntoTranslation

[klk] Rename ReadTranslator to ReadTranslation

[klk] Rename Coders to CoderTranslation

[klk] Rename WindowingStrategies to WindowingStrategyTranslation

[klk] Rename Triggers to TriggerTranslation

[klk] Do not ever shrink allowed lateness

[klk] Adds large key tests to GroupByKeyTest

[dhalperi] Initial implementation of SpannerIO.Write

[dhalperi] Minor style, compilation, javadoc fixups

[dhalperi] Fix spanner dependency management

[dhalperi] Delete SpannerCSVLoader

[dhalperi] Refine Spanner API tests

[dhalperi] SpannerIO.Write cleanup and style fixes

[robertwb] Automatically generate Python proto and grpc files.

[robertwb] Remove auto-generated proto and grpc files.

[robertwb] A couple of worker fixes.

[robertwb] Adding a snippet for metrics

--
[...truncated 355.70 KB...]
2017-05-25T00:03:02.116 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.pom
2017-05-25T00:03:02.125 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.pom
 (3 KB at 260.1 KB/sec)
2017-05-25T00:03:02.129 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
2017-05-25T00:03:02.143 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
 (2 KB at 122.7 KB/sec)
2017-05-25T00:03:02.147 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
2017-05-25T00:03:02.157 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
 (6 KB at 500.6 KB/sec)
2017-05-25T00:03:02.161 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
2017-05-25T00:03:02.184 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
 (5 KB at 182.0 KB/sec)
2017-05-25T00:03:02.188 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
2017-05-25T00:03:02.196 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
 (2 KB at 213.4 KB/sec)
2017-05-25T00:03:02.200 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
2017-05-25T00:03:02.213 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
 (2 KB at 112.5 KB/sec)
2017-05-25T00:03:02.217 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
2017-05-25T00:03:02.252 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
 (3 KB at 76.3 KB/sec)
2017-05-25T00:03:02.257 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
2017-05-25T00:03:02.264 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
 (5 KB at 592.8 KB/sec)
2017-05-25T00:03:02.270 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom
2017-05-25T00:03:02.279 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom 
(2 KB at 156.8 KB/sec)
2017-05-25T00:03:02.283 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
2017-05-25T00:03:02.292 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
 (4 KB at 388.5 KB/sec)
2017-05-25T00:03:02.297 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
2017-05-25T00:03:02.305 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
 (3 KB at 259.3 KB/sec)
2017-05-25T00:03:02.310 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
2017-05-25T00:03:02.320 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
 (3 KB at 202.9 KB/sec)
2017-05-25T00:03:02.367 [INFO] Downloading: 

[jira] [Resolved] (BEAM-2338) GCS filepattern wildcard broken in Python SDK

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-2338.
--
Resolution: Fixed

> GCS filepattern wildcard broken in Python SDK
> -
>
> Key: BEAM-2338
> URL: https://issues.apache.org/jira/browse/BEAM-2338
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 2.0.0
>Reporter: Vilhelm von Ehrenheim
>Assignee: Sourabh Bajaj
> Fix For: 2.1.0
>
>
> Validation of file patterns containing wildcard (`*`) in GCS directories does 
> not always work. 
> Some kinds of patterns generates an error from here during validation:
> https://github.com/apache/beam/blob/v2.0.0/sdks/python/apache_beam/io/filebasedsource.py#L168
> I've tried a few different FileSystems match commands which confuses be a bit.
> Full path works:
> {noformat}
> >>> FileSystems.match(['gs://gcp-public-data-landsat/LC08/PRE/044/034/LC80440342016259LGN00/LC80440342016259LGN00_B1.TIF'],
> >>>  limits=[1])[0].metadata_list
> [FileMetadata(gs://gcp-public-data-landsat/LC08/PRE/044/034/LC80440342016259LGN00/LC80440342016259LGN00_B1.TIF,
>  74721736)]
> {noformat}
> Glob star on directory does not
> {noformat}
> >>> FileSystems.match(['gs://gcp-public-data-landsat/LC08/PRE/044/034/*/LC80440342016259LGN00_B1.TIF'],
> >>>  limits=[1])[0].metadata_list
> []
> {noformat}
> If adding a star on the file level only searching for TIF files it works (all 
> tough we match a different file but that is fine)
> {noformat}
> >>> FileSystems.match(['gs://gcp-public-data-landsat/LC08/PRE/044/034/*/*.TIF'],
> >>>  limits=[1])[0].metadata_list
> [FileMetadata(gs://gcp-public-data-landsat/LC08/PRE/044/034/LC80440342013106LGN01/LC80440342013106LGN01_B1.TIF,
>  65862791)]
> {noformat}
> Ok, Here comes the even more strange case. 
> Looking for the same file we found with the patterns that but with a star on 
> the dir we find it!!
> {noformat}
> >>> FileSystems.match(['gs://gcp-public-data-landsat/LC08/PRE/044/034/*/LC80440342013106LGN01_B1.TIF'],
> >>>  limits=[1])[0].metadata_list
> [FileMetadata(gs://gcp-public-data-landsat/LC08/PRE/044/034/LC80440342013106LGN01/LC80440342013106LGN01_B1.TIF,
>  65862791)]
> {noformat}
> Also looking at the first case again we will match if the star is placed late 
> enough in the pattern to make the directory unique.
> {noformat}
> >>> FileSystems.match(['gs://gcp-public-data-landsat/LC08/PRE/044/034/LC80440342016259LGN*/LC80440342016259LGN00_B1.TIF'],
> >>>  limits=[1])[0].metadata_list
> [FileMetadata(gs://gcp-public-data-landsat/LC08/PRE/044/034/LC80440342016259LGN00/LC80440342016259LGN00_B1.TIF,
>  74721736)]
> {noformat}
> but not if further up in the name
> {noformat}
> >>> FileSystems.match(['gs://gcp-public-data-landsat/LC08/PRE/044/034/LC8044034201*/LC80440342016259LGN00_B1.TIF'],
> >>>  limits=[1])[0].metadata_list
> []
> {noformat}
> My guess is that some folders are dropped from the list of matched 
> directories or something which is a bit concerning. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2312

[Apache Jenkins Server]

See 


Changes:

[robertwb] Automatically generate Python proto and grpc files.

[robertwb] Remove auto-generated proto and grpc files.

[robertwb] A couple of worker fixes.

[robertwb] Adding a snippet for metrics

--
[...truncated 577.87 KB...]
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/SideInput-s16.output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"user_name": "write/Write/WriteImpl/FinalizeWrite/SideInput-s16"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s17", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_finalize_write"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {
  "SideInput-s15": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s15"
  }, 
  "SideInput-s16": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s16"
  }
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/Write/WriteImpl/FinalizeWrite.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s7"
}, 
"serialized_fn": "", 
"user_name": "write/Write/WriteImpl/FinalizeWrite/Do"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2017-05-24_16_33_25-3759598730270575457]
root: INFO: To access the Dataflow monitoring console, please navigate to 

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

[Mairbek Khadikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023881#comment-16023881
 ] 

Mairbek Khadikov commented on BEAM-1542:


Created https://github.com/apache/beam-site/pull/252 

> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Mairbek Khadikov
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #252: Added Cloud Spanner Connector

[mairbek]

GitHub user mairbek opened a pull request:

https://github.com/apache/beam-site/pull/252

Added Cloud Spanner Connector



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mairbek/beam-site patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/252.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #252


commit 161732cd2098f3ae94cf4021b1d1460d6d230de1
Author: Mairbek Khadikov 
Date:   2017-05-24T23:30:08Z

Added Cloud Spanner Connector




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023868#comment-16023868
 ] 

Stephen Sisk commented on BEAM-1542:


hi! Mind adding this to the built-in IO transforms page? It's a simple change 
in 
https://github.com/apache/beam-site/blob/asf-site/src/documentation/io/built-in.md

This will help users see that this IO exists/is being worked on.

thanks!

> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Mairbek Khadikov
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2361) Add TikaIO to the list of in-progress transforms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023864#comment-16023864
 ] 

ASF GitHub Bot commented on BEAM-2361:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/250


> Add TikaIO to the list of in-progress transforms
> 
>
> Key: BEAM-2361
> URL: https://issues.apache.org/jira/browse/BEAM-2361
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Sergey Beryozkin
>Assignee: Sergey Beryozkin
>Priority: Minor
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #250: [BEAM-2361] Add TikaIO to the list of in-progre...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/250


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam-site git commit: This closes #250

[altay]

This closes #250


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/f03fb601
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/f03fb601
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/f03fb601

Branch: refs/heads/asf-site
Commit: f03fb601dda47c7ede157663ca140dfd1a5cc74d
Parents: 0b8e34d 047a210
Author: Ahmet Altay 
Authored: Wed May 24 16:16:00 2017 -0700
Committer: Ahmet Altay 
Committed: Wed May 24 16:16:00 2017 -0700

--
 content/documentation/io/built-in/index.html|  4 
 .../documentation/programming-guide/index.html  | 22 ++--
 .../sdks/python-custom-io/index.html| 20 +-
 .../sdks/python-type-safety/index.html  |  1 -
 .../get-started/wordcount-example/index.html|  2 +-
 src/documentation/io/built-in.md|  4 
 6 files changed, 30 insertions(+), 23 deletions(-)
--

[2/3] beam-site git commit: Regenerate website

[altay]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/047a2107
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/047a2107
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/047a2107

Branch: refs/heads/asf-site
Commit: 047a2107bdfacb5c2d89241c874ce033391e579a
Parents: 1f78f64
Author: Ahmet Altay 
Authored: Wed May 24 16:16:00 2017 -0700
Committer: Ahmet Altay 
Committed: Wed May 24 16:16:00 2017 -0700

--
 content/documentation/io/built-in/index.html|  4 
 .../documentation/programming-guide/index.html  | 22 ++--
 .../sdks/python-custom-io/index.html| 20 +-
 .../sdks/python-type-safety/index.html  |  1 -
 .../get-started/wordcount-example/index.html|  2 +-
 5 files changed, 26 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/047a2107/content/documentation/io/built-in/index.html
--
diff --git a/content/documentation/io/built-in/index.html 
b/content/documentation/io/built-in/index.html
index 79e47f7..6b3de1b 100644
--- a/content/documentation/io/built-in/index.html
+++ b/content/documentation/io/built-in/index.html
@@ -255,6 +255,10 @@
 RestIOJava
 https://issues.apache.org/jira/browse/BEAM-1946;>BEAM-1946
   
+  
+TikaIOJava
+https://issues.apache.org/jira/browse/BEAM-2328;>BEAM-2328
+  
 
 
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/047a2107/content/documentation/programming-guide/index.html
--
diff --git a/content/documentation/programming-guide/index.html 
b/content/documentation/programming-guide/index.html
index 56b5481..7b27b37 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -402,14 +402,14 @@
 
 
 
-p = beam.Pipeline(options=pipeline_options)
+with beam.Pipeline(options=pipeline_options) as p:
 
-lines = (p
- | beam.Create([
- 'To be, or not to be: that is the question: 
',
- 'Whether \'tis nobler in the mind to suffer ',
- 'The slings and arrows of outrageous fortune, 
',
- 'Or to take arms against a sea of troubles, 
']))
+  lines = (p
+   | beam.Create([
+   'To be, or not to be: that is the question: 
',
+   'Whether \'tis nobler in the mind to suffer ',
+   'The slings and arrows of outrageous fortune, 
',
+   'Or to take arms against a sea of troubles, 
']))
 
 
 
@@ -1128,10 +1128,10 @@ guest, [[], [order4]]
 lower_bound=pvalue.AsSingleton(avg_word_len)))
 
 # Mix and match.
-small_but_nontrivial = words | beam.FlatMap(filter_using_length,
-lower_bound=2,
-upper_bound=pvalue.AsSingleton(
-avg_word_len))
+small_but_nontrivial = words | beam.FlatMap(
+filter_using_length,
+lower_bound=2,
+upper_bound=pvalue.AsSingleton(avg_word_len))
 
 
 # We can also pass side inputs to a ParDo transform, which 
will get passed to its process method.

http://git-wip-us.apache.org/repos/asf/beam-site/blob/047a2107/content/documentation/sdks/python-custom-io/index.html
--
diff --git a/content/documentation/sdks/python-custom-io/index.html 
b/content/documentation/sdks/python-custom-io/index.html
index d6eb9f2..2952ea4 100644
--- a/content/documentation/sdks/python-custom-io/index.html
+++ b/content/documentation/sdks/python-custom-io/index.html
@@ -368,8 +368,8 @@
 
 To read data from the source in your pipeline, use the Read transform:
 
-p = 
beam.Pipeline(options=PipelineOptions())
-numbers = p | 'ProduceNumbers'  beam.io.Read(CountingSource(count))
+with 
beam.Pipeline(options=PipelineOptions()) as p:
+  numbers = p | 'ProduceNumbers'  beam.io.Read(CountingSource(count))
 
 
 
@@ -512,11 +512,11 @@ numbers = p | 'ProduceNumbers'  
beam.io.Read(CountingSource(count))
 
 The following code demonstrates how to write to the sink using the Write transform.
 
-p = 
beam.Pipeline(options=PipelineOptions())
-kvs = p | 'CreateKVs'  beam.Create(KVs)
+with 
beam.Pipeline(options=PipelineOptions()) as p:
+  kvs = p | 'CreateKVs'  beam.Create(KVs)
 
-kvs | 'WriteToSimpleKV'  beam.io.Write(
-SimpleKVSink('http://url_to_simple_kv/', final_table_name))
+  kvs | 'WriteToSimpleKV'  beam.io.Write(
+  SimpleKVSink('http://url_to_simple_kv/', final_table_name))
 
 
 
@@ -569,10 +569,10 @@ numbers = p | 'ProduceNumbers'  
ReadFromCountingSource(count)
 
 Finally, write 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #2167

[Apache Jenkins Server]

See 


Changes:

[robertwb] A couple of worker fixes.

[robertwb] Adding a snippet for metrics

--
[...truncated 486.14 KB...]
2017-05-24T22:40:36.879 [INFO] Running org.apache.beam.sdk.transforms.CreateTest
2017-05-24T22:40:37.438 [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.557 s - in org.apache.beam.sdk.transforms.CreateTest
2017-05-24T22:40:37.438 [INFO] Running 
org.apache.beam.sdk.transforms.DistinctTest
2017-05-24T22:40:37.599 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.159 s - in org.apache.beam.sdk.transforms.DistinctTest
2017-05-24T22:40:37.599 [INFO] Running 
org.apache.beam.sdk.transforms.CombineTest
2017-05-24T22:40:39.936 [INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 2.334 s - in org.apache.beam.sdk.transforms.CombineTest
2017-05-24T22:40:39.936 [INFO] Running 
org.apache.beam.sdk.io.CompressedSourceTest
2017-05-24T22:41:00.082 [INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 20.144 s - in org.apache.beam.sdk.io.CompressedSourceTest
2017-05-24T22:41:00.083 [INFO] Running org.apache.beam.sdk.io.AvroIOTest
2017-05-24T22:41:01.051 [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.966 s - in org.apache.beam.sdk.io.AvroIOTest
2017-05-24T22:41:01.051 [INFO] Running org.apache.beam.sdk.io.TextIOTest
2017-05-24T22:41:06.113 [INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 5.059 s - in org.apache.beam.sdk.io.TextIOTest
2017-05-24T22:41:06.113 [INFO] Running org.apache.beam.sdk.io.TFRecordIOTest
2017-05-24T22:41:08.432 [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 2.318 s - in org.apache.beam.sdk.io.TFRecordIOTest
2017-05-24T22:41:08.432 [INFO] Running 
org.apache.beam.sdk.runners.TransformTreeTest
2017-05-24T22:41:08.471 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.037 s - in org.apache.beam.sdk.runners.TransformTreeTest
2017-05-24T22:41:08.471 [INFO] Running 
org.apache.beam.sdk.testing.GatherAllPanesTest
2017-05-24T22:41:10.804 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 2.33 s - in org.apache.beam.sdk.testing.GatherAllPanesTest
2017-05-24T22:41:10.804 [INFO] Running 
org.apache.beam.sdk.coders.SerializableCoderTest
2017-05-24T22:41:10.881 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.075 s - in org.apache.beam.sdk.coders.SerializableCoderTest
2017-05-24T22:41:10.881 [INFO] Running org.apache.beam.sdk.coders.AvroCoderTest
2017-05-24T22:41:10.954 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.071 s - in org.apache.beam.sdk.coders.AvroCoderTest
2017-05-24T22:41:10.954 [INFO] Running org.apache.beam.sdk.transforms.CountTest
2017-05-24T22:41:11.172 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.216 s - in org.apache.beam.sdk.transforms.CountTest
2017-05-24T22:41:11.172 [INFO] Running org.apache.beam.sdk.transforms.KeysTest
2017-05-24T22:41:11.263 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.089 s - in org.apache.beam.sdk.transforms.KeysTest
2017-05-24T22:41:11.263 [INFO] Running org.apache.beam.sdk.transforms.TopTest
2017-05-24T22:41:11.711 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.446 s - in org.apache.beam.sdk.transforms.TopTest
2017-05-24T22:41:11.711 [INFO] Running 
org.apache.beam.sdk.transforms.ApproximateUniqueTest$ApproximateUniqueWithDuplicatesTest
2017-05-24T22:41:12.325 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.612 s - in 
org.apache.beam.sdk.transforms.ApproximateUniqueTest$ApproximateUniqueWithDuplicatesTest
2017-05-24T22:41:12.325 [INFO] Running 
org.apache.beam.sdk.transforms.ApproximateUniqueTest$ApproximateUniqueVariationsTest
2017-05-24T22:41:14.224 [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 1.897 s - in 
org.apache.beam.sdk.transforms.ApproximateUniqueTest$ApproximateUniqueVariationsTest
2017-05-24T22:41:14.224 [INFO] Running 
org.apache.beam.sdk.transforms.ApproximateUniqueTest$ApproximateUniqueMiscTest
2017-05-24T22:41:14.639 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.413 s - in 
org.apache.beam.sdk.transforms.ApproximateUniqueTest$ApproximateUniqueMiscTest
2017-05-24T22:41:14.639 [INFO] Running org.apache.beam.sdk.transforms.ViewTest
2017-05-24T22:41:16.908 [INFO] Tests run: 37, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 2.267 s - in org.apache.beam.sdk.transforms.ViewTest
2017-05-24T22:41:16.908 [INFO] Running 
org.apache.beam.sdk.transforms.GroupIntoBatchesTest
2017-05-24T22:41:17.263 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.353 s - in 
org.apache.beam.sdk.transforms.GroupIntoBatchesTest
2017-05-24T22:41:17.263 [INFO] 

[jira] [Commented] (BEAM-2328) Introduce Apache Tika Input component

[Sergey Beryozkin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023826#comment-16023826
 ] 

Sergey Beryozkin commented on BEAM-2328:


Sorry for a bit of a noise, I spotted in the docs that the site updates should 
be assigned to a different category, hence I opened BEAM-2361 and made this one 
related to it, hopefully I've made it nearly right this time :-) cheers

> Introduce Apache Tika Input component
> -
>
> Key: BEAM-2328
> URL: https://issues.apache.org/jira/browse/BEAM-2328
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas, sdk-java-extensions
>Reporter: Sergey Beryozkin
>Assignee: Sergey Beryozkin
> Fix For: 2.1.0
>
>
> Apache Tika is a popular project that offers an extensive support for parsing 
> the variety of file formats. It is used in many projects including Lucene and 
> Elastic Search. 
> Supporting a Tika Input (Read) at the Beam level would be of major interest 
> to many users.
> PR is to follow



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #251: [BEAM-2360] Add Beam Runner Guide

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam-site/pull/251

[BEAM-2360] Add Beam Runner Guide

I did a quick port of the doc at https://s.apache.org/beam-runner-guide.

I wanted to open this early, because there are technical issues to address:

 - The language toggles are inappropriate, as I'd like to include Java and 
Python on the same page, so a runner can read about them both. Similarly, it is 
not really useful to have a bit label as to which language is being shown.
 - The toggles actually totally break the page if I try to use syntax 
highlighting on the protobufs (I've left it in the state it should probably end 
up in to demonstrate).

Beyond the draft doc, I've added a ton of links to API docs and code. There 
may still be comments on the GDoc and I'll incorporate them here.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam-site runner-guide

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/251.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #251


commit b0095ec87d692a8f6827ef1f21c2e830f182c359
Author: Kenneth Knowles 
Date:   2017-05-24T13:58:59Z

Add Beam Runner Guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Issue Comment Deleted] (BEAM-2328) Introduce Apache Tika Input component

[Sergey Beryozkin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Beryozkin updated BEAM-2328:
---
Comment: was deleted

(was: Hi, pull request #250 has been created. thanks)

> Introduce Apache Tika Input component
> -
>
> Key: BEAM-2328
> URL: https://issues.apache.org/jira/browse/BEAM-2328
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas, sdk-java-extensions
>Reporter: Sergey Beryozkin
>Assignee: Sergey Beryozkin
> Fix For: 2.1.0
>
>
> Apache Tika is a popular project that offers an extensive support for parsing 
> the variety of file formats. It is used in many projects including Lucene and 
> Elastic Search. 
> Supporting a Tika Input (Read) at the Beam level would be of major interest 
> to many users.
> PR is to follow



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2360) Add guide for writing a Beam runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023820#comment-16023820
 ] 

ASF GitHub Bot commented on BEAM-2360:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam-site/pull/251

[BEAM-2360] Add Beam Runner Guide

I did a quick port of the doc at https://s.apache.org/beam-runner-guide.

I wanted to open this early, because there are technical issues to address:

 - The language toggles are inappropriate, as I'd like to include Java and 
Python on the same page, so a runner can read about them both. Similarly, it is 
not really useful to have a bit label as to which language is being shown.
 - The toggles actually totally break the page if I try to use syntax 
highlighting on the protobufs (I've left it in the state it should probably end 
up in to demonstrate).

Beyond the draft doc, I've added a ton of links to API docs and code. There 
may still be comments on the GDoc and I'll incorporate them here.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam-site runner-guide

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/251.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #251


commit b0095ec87d692a8f6827ef1f21c2e830f182c359
Author: Kenneth Knowles 
Date:   2017-05-24T13:58:59Z

Add Beam Runner Guide




> Add guide for writing a Beam runner
> ---
>
> Key: BEAM-2360
> URL: https://issues.apache.org/jira/browse/BEAM-2360
> Project: Beam
>  Issue Type: Wish
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> We now have enough runners and enough interest in more that we could really 
> use a guide for authoring a Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2361) Add TikaIO to the list of in-progress transforms

[Sergey Beryozkin (JIRA)]

Sergey Beryozkin created BEAM-2361:
--

 Summary: Add TikaIO to the list of in-progress transforms
 Key: BEAM-2361
 URL: https://issues.apache.org/jira/browse/BEAM-2361
 Project: Beam
  Issue Type: Task
  Components: website
Reporter: Sergey Beryozkin
Assignee: Sergey Beryozkin
Priority: Minor
 Fix For: 2.1.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2360) Add guide for writing a Beam runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2360:
--
Issue Type: Wish  (was: Bug)

> Add guide for writing a Beam runner
> ---
>
> Key: BEAM-2360
> URL: https://issues.apache.org/jira/browse/BEAM-2360
> Project: Beam
>  Issue Type: Wish
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> We now have enough runners and enough interest in more that we could really 
> use a guide for authoring a Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2925

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2360) Add guide for writing a Beam runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2360:
-

 Summary: Add guide for writing a Beam runner
 Key: BEAM-2360
 URL: https://issues.apache.org/jira/browse/BEAM-2360
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


We now have enough runners and enough interest in more that we could really use 
a guide for authoring a Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3221: [BEAM-1542] SpannerIO sink updates

[mairbek]

GitHub user mairbek opened a pull request:

https://github.com/apache/beam/pull/3221

[BEAM-1542] SpannerIO sink updates

Unit and integration test.
Logical mutation size estimation.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mairbek/beam mergespanner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3221.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3221


commit 8db6693c34712c899a1655d5305f2602a9120e9b
Author: MOLIG004 
Date:   2017-05-04T16:21:23Z

Initial implementation of SpannerIO.Write

This closes #2166.

commit 491f047726ec2146a634a870efa4a43b9d4d643d
Author: Dan Halperin 
Date:   2017-05-04T17:42:07Z

Minor style, compilation, javadoc fixups

commit 7a2d09a0d2783f074348d9b8a1f5bbab7e0a2871
Author: Dan Halperin 
Date:   2017-05-04T17:48:24Z

SpannerCSVLoader: update for TextIO API Change

commit 77be2762825def0c9a62c30b02b9d0a8ff2a15f2
Author: Dan Halperin 
Date:   2017-05-04T17:48:54Z

Fix spanner dependency management

Also minor cleanup alphabetization in root pom.xml

commit 4049047a38bd5dfa6facee8f3c68086c81873e91
Author: Mairbek Khadikov 
Date:   2017-05-12T19:59:27Z

Introduced MutationSizeEstimator.

commit 4392de9be272c045a9854340f887e420f25f0f4a
Author: MOLIG004 
Date:   2017-05-04T16:21:23Z

Initial implementation of SpannerIO.Write

This closes #2166.

commit b092be8df5cd96012c4d67b3defbd3554707edfe
Author: Dan Halperin 
Date:   2017-05-04T17:42:07Z

Minor style, compilation, javadoc fixups

commit e16339a3e699d7bea30f3f3a703b4c7c3ead0f59
Author: Dan Halperin 
Date:   2017-05-04T17:48:54Z

Fix spanner dependency management

Also minor cleanup alphabetization in root pom.xml

commit 9798534793286aeb3deacdc0e30f63032b6077fe
Author: Dan Halperin 
Date:   2017-05-04T17:48:24Z

Delete SpannerCSVLoader

This is not appropriate for examples. SpannerIO should be well-javadoced
and integration tested.

commit f495a414d6d6ec1a3098fb751f499680d070f7c5
Author: Dan Halperin 
Date:   2017-05-15T17:16:18Z

Refine Spanner API tests

And remove outdated Bigtable comment

commit ce5d04cc354c3d1bd3938eff94aa3474adac1e1b
Author: Dan Halperin 
Date:   2017-05-15T17:42:57Z

SpannerIO.Write cleanup and style fixes

* Rename to Write to match the rest of the SDK.
* Convert to AutoValue, delete toString.
* Drop .writeTo(), instead use .write() as default constructor.
* Temporarily drop withBatchSize, as its existence is not clearly
  justified.

commit 81e71f62c8c0ba2eb1f5b7e8240650539785fa52
Author: Mairbek Khadikov 
Date:   2017-05-15T18:20:18Z

Merge branch 'takeover-2166' of https://github.com/dhalperi/beam into 
mergespanner

# Conflicts:
#   
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
#   
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java

commit 16e4b832628205b1f560e5f37fb105d17f2b56ec
Author: Mairbek Khadikov 
Date:   2017-05-15T18:38:44Z

Use a new batch size limit.

commit 5a2214a2623fa6d8d4a7b50852b59dc7789837af
Author: Dan Halperin 
Date:   2017-05-15T17:42:57Z

SpannerIO.Write cleanup and style fixes

* Rename to Write to match the rest of the SDK.
* Convert to AutoValue, delete toString.
* Drop .writeTo(), instead use .write() as default constructor.
* Temporarily drop withBatchSize, as its existence is not clearly
  justified.

commit 5fc07fe297f6cf5fdd00bb3385d6a23baaf67104
Author: Mairbek Khadikov 
Date:   2017-05-15T21:59:22Z

Added a SpannerIO unit test.

commit 0b7b84dfbe17cbc0747197f34c8512610d01d86e
Author: Mairbek Khadikov 
Date:   2017-05-15T23:25:33Z

Renaming and documenting

commit 2b5a10e7b64148db2bbcfadbef81dee7c0e10347
Author: Mairbek Khadikov 
Date:   2017-05-15T23:56:23Z

 

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023791#comment-16023791
 ] 

ASF GitHub Bot commented on BEAM-1542:
--

GitHub user mairbek opened a pull request:

https://github.com/apache/beam/pull/3221

[BEAM-1542] SpannerIO sink updates

Unit and integration test.
Logical mutation size estimation.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mairbek/beam mergespanner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3221.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3221


commit 8db6693c34712c899a1655d5305f2602a9120e9b
Author: MOLIG004 
Date:   2017-05-04T16:21:23Z

Initial implementation of SpannerIO.Write

This closes #2166.

commit 491f047726ec2146a634a870efa4a43b9d4d643d
Author: Dan Halperin 
Date:   2017-05-04T17:42:07Z

Minor style, compilation, javadoc fixups

commit 7a2d09a0d2783f074348d9b8a1f5bbab7e0a2871
Author: Dan Halperin 
Date:   2017-05-04T17:48:24Z

SpannerCSVLoader: update for TextIO API Change

commit 77be2762825def0c9a62c30b02b9d0a8ff2a15f2
Author: Dan Halperin 
Date:   2017-05-04T17:48:54Z

Fix spanner dependency management

Also minor cleanup alphabetization in root pom.xml

commit 4049047a38bd5dfa6facee8f3c68086c81873e91
Author: Mairbek Khadikov 
Date:   2017-05-12T19:59:27Z

Introduced MutationSizeEstimator.

commit 4392de9be272c045a9854340f887e420f25f0f4a
Author: MOLIG004 
Date:   2017-05-04T16:21:23Z

Initial implementation of SpannerIO.Write

This closes #2166.

commit b092be8df5cd96012c4d67b3defbd3554707edfe
Author: Dan Halperin 
Date:   2017-05-04T17:42:07Z

Minor style, compilation, javadoc fixups

commit e16339a3e699d7bea30f3f3a703b4c7c3ead0f59
Author: Dan Halperin 
Date:   2017-05-04T17:48:54Z

Fix spanner dependency management

Also minor cleanup alphabetization in root pom.xml

commit 9798534793286aeb3deacdc0e30f63032b6077fe
Author: Dan Halperin 
Date:   2017-05-04T17:48:24Z

Delete SpannerCSVLoader

This is not appropriate for examples. SpannerIO should be well-javadoced
and integration tested.

commit f495a414d6d6ec1a3098fb751f499680d070f7c5
Author: Dan Halperin 
Date:   2017-05-15T17:16:18Z

Refine Spanner API tests

And remove outdated Bigtable comment

commit ce5d04cc354c3d1bd3938eff94aa3474adac1e1b
Author: Dan Halperin 
Date:   2017-05-15T17:42:57Z

SpannerIO.Write cleanup and style fixes

* Rename to Write to match the rest of the SDK.
* Convert to AutoValue, delete toString.
* Drop .writeTo(), instead use .write() as default constructor.
* Temporarily drop withBatchSize, as its existence is not clearly
  justified.

commit 81e71f62c8c0ba2eb1f5b7e8240650539785fa52
Author: Mairbek Khadikov 
Date:   2017-05-15T18:20:18Z

Merge branch 'takeover-2166' of https://github.com/dhalperi/beam into 
mergespanner

# Conflicts:
#   
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
#   
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java

commit 16e4b832628205b1f560e5f37fb105d17f2b56ec
Author: Mairbek Khadikov 
Date:   2017-05-15T18:38:44Z

Use a new batch size limit.

commit 5a2214a2623fa6d8d4a7b50852b59dc7789837af
Author: Dan Halperin 
Date:   2017-05-15T17:42:57Z

SpannerIO.Write cleanup and style fixes

* Rename to Write to match the rest of the SDK.
* Convert to AutoValue, delete toString.
* Drop .writeTo(), instead use .write() as default constructor.
* Temporarily drop withBatchSize, as its existence is not clearly
  justified.

commit 5fc07fe297f6cf5fdd00bb3385d6a23baaf67104
Author: Mairbek Khadikov 
Date:   2017-05-15T21:59:22Z

Added a SpannerIO unit test.

commit 

[GitHub] beam pull request #3200: Adding a snippet for metrics

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3200


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Closes #3200

[robertwb]

Closes #3200


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c0d19f9f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c0d19f9f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c0d19f9f

Branch: refs/heads/master
Commit: c0d19f9f7cccb222819dba26be52d85262f3f18f
Parents: 4084f71 f32cacb
Author: Robert Bradshaw 
Authored: Wed May 24 15:14:48 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed May 24 15:14:48 2017 -0700

--
 .../examples/snippets/snippets_test.py  | 62 
 1 file changed, 62 insertions(+)
--

[1/2] beam git commit: Adding a snippet for metrics

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 4084f71a1 -> c0d19f9f7


Adding a snippet for metrics


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f32cacb7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f32cacb7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f32cacb7

Branch: refs/heads/master
Commit: f32cacb7313b8352ddc054cd1b88d7a0462550db
Parents: 4084f71
Author: Pablo 
Authored: Mon May 22 15:39:44 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed May 24 15:14:41 2017 -0700

--
 .../examples/snippets/snippets_test.py  | 62 
 1 file changed, 62 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f32cacb7/sdks/python/apache_beam/examples/snippets/snippets_test.py
--
diff --git a/sdks/python/apache_beam/examples/snippets/snippets_test.py 
b/sdks/python/apache_beam/examples/snippets/snippets_test.py
index 6654fef..e302465 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets_test.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets_test.py
@@ -32,6 +32,8 @@ from apache_beam import typehints
 from apache_beam.coders.coders import ToStringCoder
 from apache_beam.options.pipeline_options import PipelineOptions
 from apache_beam.examples.snippets import snippets
+from apache_beam.metrics import Metrics
+from apache_beam.metrics.metric import MetricsFilter
 from apache_beam.testing.util import assert_that
 from apache_beam.testing.util import equal_to
 from apache_beam.utils.windowed_value import WindowedValue
@@ -689,6 +691,66 @@ class SnippetsTest(unittest.TestCase):
 expect = ['a; a...@example.com; x4312', 'b; b...@example.com; x8452']
 self.assertEqual(expect, self.get_output(result_path))
 
+  def test_model_use_and_query_metrics(self):
+"""DebuggingWordCount example snippets."""
+
+import re
+
+p = TestPipeline()  # Use TestPipeline for testing.
+words = p | beam.Create(['albert', 'sam', 'mark', 'sarah',
+ 'swati', 'daniel', 'andrea'])
+
+# pylint: disable=unused-variable
+# [START metrics_usage_example]
+class FilterTextFn(beam.DoFn):
+  """A DoFn that filters for a specific key based on a regex."""
+
+  def __init__(self, pattern):
+self.pattern = pattern
+# A custom metric can track values in your pipeline as it runs. Create
+# custom metrics to count unmatched words, and know the distribution of
+# word lengths in the input PCollection.
+self.word_len_dist = Metrics.distribution(self.__class__,
+  'word_len_dist')
+self.unmatched_words = Metrics.counter(self.__class__,
+   'unmatched_words')
+
+  def process(self, element):
+word = element
+self.word_len_dist.update(len(word))
+if re.match(self.pattern, word):
+  yield element
+else:
+  self.unmatched_words.inc()
+
+filtered_words = (
+words | 'FilterText' >> beam.ParDo(FilterTextFn('s.*')))
+# [END metrics_usage_example]
+# pylint: enable=unused-variable
+
+# [START metrics_check_values_example]
+result = p.run()
+result.wait_until_finish()
+
+custom_distribution = result.metrics().query(
+MetricsFilter().with_name('word_len_dist'))['distributions']
+custom_counter = result.metrics().query(
+MetricsFilter().with_name('unmatched_words'))['counters']
+
+if custom_distribution:
+  logging.info('The average word length was %d',
+   custom_distribution[0].committed.mean)
+if custom_counter:
+  logging.info('There were %d words that did not match the filter.',
+   custom_counter[0].committed)
+# [END metrics_check_values_example]
+
+# There should be 4 words that did not match
+self.assertEqual(custom_counter[0].committed, 4)
+# The shortest word is 3 characters, the longest is 6
+self.assertEqual(custom_distribution[0].committed.min, 3)
+self.assertEqual(custom_distribution[0].committed.max, 6)
+
   def test_model_join_using_side_inputs(self):
 name_list = ['a', 'b']
 email_list = [['a', 'a...@example.com'], ['b', 'b...@example.com']]

[2/2] beam git commit: Closes #3170

[robertwb]

Closes #3170


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4084f71a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4084f71a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4084f71a

Branch: refs/heads/master
Commit: 4084f71a10ac16c94b1cd88f0b9f9b2a1ca756e9
Parents: 24fb4cd e9cb40d
Author: Robert Bradshaw 
Authored: Wed May 24 15:09:13 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed May 24 15:09:13 2017 -0700

--
 .../python/apache_beam/runners/worker/operation_specs.py | 11 ++-
 sdks/python/apache_beam/runners/worker/operations.py |  2 +-
 2 files changed, 7 insertions(+), 6 deletions(-)
--

[GitHub] beam pull request #3170: A couple of worker fixes.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3170


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: A couple of worker fixes.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 24fb4cda6 -> 4084f71a1


A couple of worker fixes.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e9cb40dc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e9cb40dc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e9cb40dc

Branch: refs/heads/master
Commit: e9cb40dc37683effb2ded1bfaab6dfc04ade698b
Parents: 24fb4cd
Author: Robert Bradshaw 
Authored: Wed May 17 14:25:08 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed May 24 15:09:12 2017 -0700

--
 .../python/apache_beam/runners/worker/operation_specs.py | 11 ++-
 sdks/python/apache_beam/runners/worker/operations.py |  2 +-
 2 files changed, 7 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e9cb40dc/sdks/python/apache_beam/runners/worker/operation_specs.py
--
diff --git a/sdks/python/apache_beam/runners/worker/operation_specs.py 
b/sdks/python/apache_beam/runners/worker/operation_specs.py
index c03d9a2..db5eb76 100644
--- a/sdks/python/apache_beam/runners/worker/operation_specs.py
+++ b/sdks/python/apache_beam/runners/worker/operation_specs.py
@@ -305,7 +305,8 @@ def get_coder_from_spec(coder_spec):
   assert coder_spec is not None
 
   # Ignore the wrappers in these encodings.
-  # TODO(silviuc): Make sure with all the renamings that names below are ok.
+  ignored_wrappers = (
+  'com.google.cloud.dataflow.sdk.util.TimerOrElement$TimerOrElementCoder')
   if coder_spec['@type'] in ignored_wrappers:
 assert len(coder_spec['component_encodings']) == 1
 coder_spec = coder_spec['component_encodings'][0]
@@ -328,20 +329,20 @@ def get_coder_from_spec(coder_spec):
 return coders.WindowedValueCoder(value_coder, window_coder=window_coder)
   elif coder_spec['@type'] == 'kind:interval_window':
 assert ('component_encodings' not in coder_spec
-or len(coder_spec['component_encodings'] == 0))
+or not coder_spec['component_encodings'])
 return coders.IntervalWindowCoder()
   elif coder_spec['@type'] == 'kind:global_window':
 assert ('component_encodings' not in coder_spec
 or not coder_spec['component_encodings'])
-return coders.GlobalWindowCoder()
+return coders.coders.GlobalWindowCoder()
   elif coder_spec['@type'] == 'kind:length_prefix':
 assert len(coder_spec['component_encodings']) == 1
-return coders.LengthPrefixCoder(
+return coders.coders.LengthPrefixCoder(
 get_coder_from_spec(coder_spec['component_encodings'][0]))
 
   # We pass coders in the form "$" to make the job
   # description JSON more readable.
-  return coders.deserialize_coder(coder_spec['@type'])
+  return coders.coders.deserialize_coder(coder_spec['@type'])
 
 
 class MapTask(object):

http://git-wip-us.apache.org/repos/asf/beam/blob/e9cb40dc/sdks/python/apache_beam/runners/worker/operations.py
--
diff --git a/sdks/python/apache_beam/runners/worker/operations.py 
b/sdks/python/apache_beam/runners/worker/operations.py
index 5dbe57e..a44561d 100644
--- a/sdks/python/apache_beam/runners/worker/operations.py
+++ b/sdks/python/apache_beam/runners/worker/operations.py
@@ -281,7 +281,7 @@ class DoOperation(Operation):
 
   # Backwards compatibility for pre BEAM-733 SDKs.
   if isinstance(view_options, tuple):
-if view_class == pvalue.SingletonPCollectionView:
+if view_class == pvalue.AsSingleton:
   has_default, default = view_options
   view_options = {'default': default} if has_default else {}
 else:

[3/6] beam git commit: Remove auto-generated proto and grpc files.

[robertwb]

http://git-wip-us.apache.org/repos/asf/beam/blob/c2e85daa/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py
--
diff --git a/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py 
b/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py
deleted file mode 100644
index 08d7dad..000
--- a/sdks/python/apache_beam/runners/api/beam_fn_api_pb2_grpc.py
+++ /dev/null
@@ -1,205 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
-import grpc
-from grpc.framework.common import cardinality
-from grpc.framework.interfaces.face import utilities as face_utilities
-
-import beam_fn_api_pb2 as beam__fn__api__pb2
-
-# This module is experimental. No backwards-compatibility guarantees.
-
-
-class BeamFnControlStub(object):
-  """
-  Control Plane API
-
-  Progress reporting and splitting still need further vetting. Also, this may 
change
-  with the addition of new types of instructions/responses related to metrics.
-
-  An API that describes the work that a SDK harness is meant to do.
-  Stable
-  """
-
-  def __init__(self, channel):
-"""Constructor.
-
-Args:
-  channel: A grpc.Channel.
-"""
-self.Control = channel.stream_stream(
-'/org.apache.beam.fn.v1.BeamFnControl/Control',
-
request_serializer=beam__fn__api__pb2.InstructionResponse.SerializeToString,
-response_deserializer=beam__fn__api__pb2.InstructionRequest.FromString,
-)
-
-
-class BeamFnControlServicer(object):
-  """
-  Control Plane API
-
-  Progress reporting and splitting still need further vetting. Also, this may 
change
-  with the addition of new types of instructions/responses related to metrics.
-
-  An API that describes the work that a SDK harness is meant to do.
-  Stable
-  """
-
-  def Control(self, request_iterator, context):
-"""Instructions sent by the runner to the SDK requesting different types
-of work.
-"""
-context.set_code(grpc.StatusCode.UNIMPLEMENTED)
-context.set_details('Method not implemented!')
-raise NotImplementedError('Method not implemented!')
-
-
-def add_BeamFnControlServicer_to_server(servicer, server):
-  rpc_method_handlers = {
-  'Control': grpc.stream_stream_rpc_method_handler(
-  servicer.Control,
-  
request_deserializer=beam__fn__api__pb2.InstructionResponse.FromString,
-  
response_serializer=beam__fn__api__pb2.InstructionRequest.SerializeToString,
-  ),
-  }
-  generic_handler = grpc.method_handlers_generic_handler(
-  'org.apache.beam.fn.v1.BeamFnControl', rpc_method_handlers)
-  server.add_generic_rpc_handlers((generic_handler,))
-
-
-class BeamFnDataStub(object):
-  """Stable
-  """
-
-  def __init__(self, channel):
-"""Constructor.
-
-Args:
-  channel: A grpc.Channel.
-"""
-self.Data = channel.stream_stream(
-'/org.apache.beam.fn.v1.BeamFnData/Data',
-request_serializer=beam__fn__api__pb2.Elements.SerializeToString,
-response_deserializer=beam__fn__api__pb2.Elements.FromString,
-)
-
-
-class BeamFnDataServicer(object):
-  """Stable
-  """
-
-  def Data(self, request_iterator, context):
-"""Used to send data between harnesses.
-"""
-context.set_code(grpc.StatusCode.UNIMPLEMENTED)
-context.set_details('Method not implemented!')
-raise NotImplementedError('Method not implemented!')
-
-
-def add_BeamFnDataServicer_to_server(servicer, server):
-  rpc_method_handlers = {
-  'Data': grpc.stream_stream_rpc_method_handler(
-  servicer.Data,
-  request_deserializer=beam__fn__api__pb2.Elements.FromString,
-  response_serializer=beam__fn__api__pb2.Elements.SerializeToString,
-  ),
-  }
-  generic_handler = grpc.method_handlers_generic_handler(
-  'org.apache.beam.fn.v1.BeamFnData', rpc_method_handlers)
-  server.add_generic_rpc_handlers((generic_handler,))
-
-
-class BeamFnStateStub(object):
-
-  def __init__(self, channel):
-"""Constructor.
-
-Args:
-  channel: A grpc.Channel.
-"""
-self.State = channel.stream_stream(
-'/org.apache.beam.fn.v1.BeamFnState/State',
-

[1/6] beam git commit: Closes #3214

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 539e229fe -> 24fb4cda6


Closes #3214


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/24fb4cda
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/24fb4cda
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/24fb4cda

Branch: refs/heads/master
Commit: 24fb4cda6bec985ba1fb4893584b134b140b108b
Parents: 539e229 c2e85da
Author: Robert Bradshaw 
Authored: Wed May 24 15:01:39 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed May 24 15:01:39 2017 -0700

--
 .gitignore  |1 +
 pom.xml |2 +-
 sdks/python/MANIFEST.in |1 +
 sdks/python/apache_beam/runners/api/__init__.py |   13 +-
 .../apache_beam/runners/api/beam_fn_api_pb2.py  | 2742 -
 .../runners/api/beam_fn_api_pb2_grpc.py |  205 --
 .../runners/api/beam_runner_api_pb2.py  | 2872 --
 sdks/python/gen_protos.py   |  116 +
 sdks/python/run_pylint.sh   |1 +
 sdks/python/setup.py|   38 +-
 sdks/python/tox.ini |3 +
 11 files changed, 157 insertions(+), 5837 deletions(-)
--

[4/6] beam git commit: Remove auto-generated proto and grpc files.

[robertwb]

http://git-wip-us.apache.org/repos/asf/beam/blob/c2e85daa/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py
--
diff --git a/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py 
b/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py
deleted file mode 100644
index cb0b72b..000
--- a/sdks/python/apache_beam/runners/api/beam_fn_api_pb2.py
+++ /dev/null
@@ -1,2742 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-# source: beam_fn_api.proto
-
-import sys
-_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
-from google.protobuf import descriptor as _descriptor
-from google.protobuf import message as _message
-from google.protobuf import reflection as _reflection
-from google.protobuf import symbol_database as _symbol_database
-from google.protobuf import descriptor_pb2
-# @@protoc_insertion_point(imports)
-
-_sym_db = _symbol_database.Default()
-
-
-from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2
-from google.protobuf import timestamp_pb2 as 
google_dot_protobuf_dot_timestamp__pb2
-
-# This module is experimental. No backwards-compatibility guarantees.
-
-
-DESCRIPTOR = _descriptor.FileDescriptor(
-  name='beam_fn_api.proto',
-  package='org.apache.beam.fn.v1',
-  syntax='proto3',
-  
serialized_pb=_b('\n\x11\x62\x65\x61m_fn_api.proto\x12\x15org.apache.beam.fn.v1\x1a\x19google/protobuf/any.proto\x1a\x1fgoogle/protobuf/timestamp.proto\"t\n\x06Target\x12%\n\x1dprimitive_transform_reference\x18\x01
 \x01(\t\x12\x0c\n\x04name\x18\x02 
\x01(\t\x1a\x35\n\x04List\x12-\n\x06target\x18\x01 
\x03(\x0b\x32\x1d.org.apache.beam.fn.v1.Target\"&\n\x0bPCollection\x12\x17\n\x0f\x63oder_reference\x18\x01
 \x01(\t\"\xcb\x04\n\x12PrimitiveTransform\x12\n\n\x02id\x18\x01 
\x01(\t\x12:\n\rfunction_spec\x18\x02 
\x01(\x0b\x32#.org.apache.beam.fn.v1.FunctionSpec\x12\x45\n\x06inputs\x18\x03 
\x03(\x0b\x32\x35.org.apache.beam.fn.v1.PrimitiveTransform.InputsEntry\x12G\n\x07outputs\x18\x04
 
\x03(\x0b\x32\x36.org.apache.beam.fn.v1.PrimitiveTransform.OutputsEntry\x12N\n\x0bside_inputs\x18\x05
 
\x03(\x0b\x32\x39.org.apache.beam.fn.v1.PrimitiveTransform.SideInputsEntry\x12\x11\n\tstep_name\x18\x06
 \x01(\t\x1aQ\n\x0bInputsEntry\x12\x0b\n\x03key\x18\x01 
\x01(\t\x12\x31\n\x05value\x18\x02 \x01(\x0b\x32\"
 
.org.apache.beam.fn.v1.Target.List:\x02\x38\x01\x1aR\n\x0cOutputsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\x31\n\x05value\x18\x02 
\x01(\x0b\x32\".org.apache.beam.fn.v1.PCollection:\x02\x38\x01\x1aS\n\x0fSideInputsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12/\n\x05value\x18\x02 \x01(\x0b\x32 
.org.apache.beam.fn.v1.SideInput:\x02\x38\x01\"j\n\x0c\x46unctionSpec\x12\n\n\x02id\x18\x01
 \x01(\t\x12\x0b\n\x03urn\x18\x02 
\x01(\t\x12\x1d\n\x15\x65nvironment_reference\x18\x03 
\x01(\t\x12\"\n\x04\x64\x61ta\x18\x04 
\x01(\x0b\x32\x14.google.protobuf.Any\"o\n\tSideInput\x12,\n\x05input\x18\x01 
\x01(\x0b\x32\x1d.org.apache.beam.fn.v1.Target\x12\x34\n\x07view_fn\x18\x02 
\x01(\x0b\x32#.org.apache.beam.fn.v1.FunctionSpec\"f\n\x05\x43oder\x12:\n\rfunction_spec\x18\x01
 
\x01(\x0b\x32#.org.apache.beam.fn.v1.FunctionSpec\x12!\n\x19\x63omponent_coder_reference\x18\x02
 \x03(\t\"]\n\x0eRemoteGrpcPort\x12K\n\x16\x61pi_service_descriptor\x18\x01 
\x01(\x0b\x32+.org.apache.beam.fn.v1.ApiServiceDescriptor\"\xe8\x0
 2\n\x12InstructionRequest\x12\x16\n\x0einstruction_id\x18\x01 
\x01(\t\x12;\n\x08register\x18\xe8\x07 
\x01(\x0b\x32&.org.apache.beam.fn.v1.RegisterRequestH\x00\x12\x46\n\x0eprocess_bundle\x18\xe9\x07
 
\x01(\x0b\x32+.org.apache.beam.fn.v1.ProcessBundleRequestH\x00\x12W\n\x17process_bundle_progress\x18\xea\x07
 
\x01(\x0b\x32\x33.org.apache.beam.fn.v1.ProcessBundleProgressRequestH\x00\x12Q\n\x14process_bundle_split\x18\xeb\x07
 
\x01(\x0b\x32\x30.org.apache.beam.fn.v1.ProcessBundleSplitRequestH\x00\x42\t\n\x07request\"\xfd\x02\n\x13InstructionResponse\x12\x16\n\x0einstruction_id\x18\x01
 \x01(\t\x12\r\n\x05\x65rror\x18\x02 \x01(\t\x12<\n\x08register\x18\xe8\x07 
\x01(\x0b\x32\'.org.apache.beam.fn.v1.RegisterResponseH\x00\x12G\n\x0eprocess_bundle\x18\xe9\x07
 

[2/6] beam git commit: Remove auto-generated proto and grpc files.

[robertwb]

http://git-wip-us.apache.org/repos/asf/beam/blob/c2e85daa/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
--
diff --git a/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py 
b/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
deleted file mode 100644
index e8793b6..000
--- a/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
+++ /dev/null
@@ -1,2872 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-# source: beam_runner_api.proto
-
-import sys
-_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
-from google.protobuf.internal import enum_type_wrapper
-from google.protobuf import descriptor as _descriptor
-from google.protobuf import message as _message
-from google.protobuf import reflection as _reflection
-from google.protobuf import symbol_database as _symbol_database
-from google.protobuf import descriptor_pb2
-# @@protoc_insertion_point(imports)
-
-_sym_db = _symbol_database.Default()
-
-
-from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2
-
-# This module is experimental. No backwards-compatibility guarantees.
-
-
-DESCRIPTOR = _descriptor.FileDescriptor(
-  name='beam_runner_api.proto',
-  package='org.apache.beam.runner_api.v1',
-  syntax='proto3',
-  
serialized_pb=_b('\n\x15\x62\x65\x61m_runner_api.proto\x12\x1dorg.apache.beam.runner_api.v1\x1a\x19google/protobuf/any.proto\"\x8d\x07\n\nComponents\x12M\n\ntransforms\x18\x01
 
\x03(\x0b\x32\x39.org.apache.beam.runner_api.v1.Components.TransformsEntry\x12Q\n\x0cpcollections\x18\x02
 
\x03(\x0b\x32;.org.apache.beam.runner_api.v1.Components.PcollectionsEntry\x12`\n\x14windowing_strategies\x18\x03
 
\x03(\x0b\x32\x42.org.apache.beam.runner_api.v1.Components.WindowingStrategiesEntry\x12\x45\n\x06\x63oders\x18\x04
 
\x03(\x0b\x32\x35.org.apache.beam.runner_api.v1.Components.CodersEntry\x12Q\n\x0c\x65nvironments\x18\x05
 
\x03(\x0b\x32;.org.apache.beam.runner_api.v1.Components.EnvironmentsEntry\x1a\\\n\x0fTransformsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\x38\n\x05value\x18\x02 
\x01(\x0b\x32).org.apache.beam.runner_api.v1.PTransform:\x02\x38\x01\x1a_\n\x11PcollectionsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\x39\n\x05value\x18\x02 
\x01(\x0b\x32*.org.apache.beam.runner_api.v1.PCollection:\x02\
 x38\x01\x1al\n\x18WindowingStrategiesEntry\x12\x0b\n\x03key\x18\x01 
\x01(\t\x12?\n\x05value\x18\x02 
\x01(\x0b\x32\x30.org.apache.beam.runner_api.v1.WindowingStrategy:\x02\x38\x01\x1aS\n\x0b\x43odersEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\x33\n\x05value\x18\x02 
\x01(\x0b\x32$.org.apache.beam.runner_api.v1.Coder:\x02\x38\x01\x1a_\n\x11\x45nvironmentsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\x39\n\x05value\x18\x02 
\x01(\x0b\x32*.org.apache.beam.runner_api.v1.Environment:\x02\x38\x01\"\xe2\x06\n\x15MessageWithComponents\x12=\n\ncomponents\x18\x01
 
\x01(\x0b\x32).org.apache.beam.runner_api.v1.Components\x12\x35\n\x05\x63oder\x18\x02
 
\x01(\x0b\x32$.org.apache.beam.runner_api.v1.CoderH\x00\x12H\n\x0f\x63ombine_payload\x18\x03
 
\x01(\x0b\x32-.org.apache.beam.runner_api.v1.CombinePayloadH\x00\x12K\n\x11sdk_function_spec\x18\x04
 
\x01(\x0b\x32..org.apache.beam.runner_api.v1.SdkFunctionSpecH\x00\x12\x45\n\x0epar_do_payload\x18\x06
 \x01(\x0b\x32+.org.apache.beam.runner_api.v1.ParDoPayloadH\x00
 \x12?\n\nptransform\x18\x07 
\x01(\x0b\x32).org.apache.beam.runner_api.v1.PTransformH\x00\x12\x41\n\x0bpcollection\x18\x08
 
\x01(\x0b\x32*.org.apache.beam.runner_api.v1.PCollectionH\x00\x12\x42\n\x0cread_payload\x18\t
 
\x01(\x0b\x32*.org.apache.beam.runner_api.v1.ReadPayloadH\x00\x12>\n\nside_input\x18\x0b
 
\x01(\x0b\x32(.org.apache.beam.runner_api.v1.SideInputH\x00\x12O\n\x13window_into_payload\x18\x0c
 
\x01(\x0b\x32\x30.org.apache.beam.runner_api.v1.WindowIntoPayloadH\x00\x12N\n\x12windowing_strategy\x18\r
 
\x01(\x0b\x32\x30.org.apache.beam.runner_api.v1.WindowingStrategyH\x00\x12\x44\n\rfunction_spec\x18\x0e
 
\x01(\x0b\x32+.org.apache.beam.runner_api.v1.FunctionSpecH\x00\x42\x06\n\x04root\"\xa7\x01\n\x08Pipeline\x12=\n\ncomponents\x18\x01
 

[GitHub] beam pull request #3214: Automatically generate proto files for Python

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3214


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[6/6] beam git commit: Automatically generate Python proto and grpc files.

[robertwb]

Automatically generate Python proto and grpc files.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bb2b7064
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bb2b7064
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bb2b7064

Branch: refs/heads/master
Commit: bb2b7064960c458a63c627d96e948f38ef11a898
Parents: 539e229
Author: Robert Bradshaw 
Authored: Tue May 23 12:28:59 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed May 24 15:01:39 2017 -0700

--
 .gitignore  |   1 +
 pom.xml |   2 +-
 sdks/python/MANIFEST.in |   1 +
 sdks/python/apache_beam/runners/api/__init__.py |  13 +--
 sdks/python/gen_protos.py   | 116 +++
 sdks/python/run_pylint.sh   |   1 +
 sdks/python/setup.py|  38 +-
 sdks/python/tox.ini |   3 +
 8 files changed, 157 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bb2b7064/.gitignore
--
diff --git a/.gitignore b/.gitignore
index 1ecb993..6e077bd 100644
--- a/.gitignore
+++ b/.gitignore
@@ -25,6 +25,7 @@ sdks/python/**/*.egg
 sdks/python/LICENSE
 sdks/python/NOTICE
 sdks/python/README.md
+sdks/python/runners/api/*pb2*.*
 
 # Ignore IntelliJ files.
 .idea/

http://git-wip-us.apache.org/repos/asf/beam/blob/bb2b7064/pom.xml
--
diff --git a/pom.xml b/pom.xml
index c3a6b73..4f833cd 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1301,7 +1301,7 @@
   **/.settings/**/*
 
   
-  **/sdks/python/apache_beam/runners/api/*.py
+  **/apache_beam/runners/api/*_pb2*.py
 
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/bb2b7064/sdks/python/MANIFEST.in
--
diff --git a/sdks/python/MANIFEST.in b/sdks/python/MANIFEST.in
index 41d80ef..c97e57a 100644
--- a/sdks/python/MANIFEST.in
+++ b/sdks/python/MANIFEST.in
@@ -15,6 +15,7 @@
 # limitations under the License.
 #
 
+include gen_protos.py
 include README.md
 include NOTICE
 include LICENSE

http://git-wip-us.apache.org/repos/asf/beam/blob/bb2b7064/sdks/python/apache_beam/runners/api/__init__.py
--
diff --git a/sdks/python/apache_beam/runners/api/__init__.py 
b/sdks/python/apache_beam/runners/api/__init__.py
index bf95208..2750859 100644
--- a/sdks/python/apache_beam/runners/api/__init__.py
+++ b/sdks/python/apache_beam/runners/api/__init__.py
@@ -17,16 +17,5 @@
 
 """For internal use only; no backwards-compatibility guarantees.
 
-Checked in to avoid protoc dependency for Python development.
-
-Regenerate files with::
-
-protoc -I../common/runner-api/src/main/proto/ \
---python_out=apache_beam/runners/api/ \
-../common/runner-api/src/main/proto/*.proto
-
-protoc -I../common/{fn,runner}-api/src/main/proto/ \
---python_out=apache_beam/runners/api/ \
---grpc_python_out=apache_beam/runners/api/ \
-../common/fn-api/src/main/proto/*.proto
+Automatically generated when running setup.py sdist or build[_py].
 """

http://git-wip-us.apache.org/repos/asf/beam/blob/bb2b7064/sdks/python/gen_protos.py
--
diff --git a/sdks/python/gen_protos.py b/sdks/python/gen_protos.py
new file mode 100644
index 000..edaaff4
--- /dev/null
+++ b/sdks/python/gen_protos.py
@@ -0,0 +1,116 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Generates Python proto modules and grpc stubs for Beam protos."""
+
+import glob
+import logging
+import multiprocessing
+import os
+import pkg_resources
+import subprocess
+import sys
+import warnings
+
+
+GRPC_TOOLS = 'grpcio-tools>=1.3.5'
+

[5/6] beam git commit: Remove auto-generated proto and grpc files.

[robertwb]

Remove auto-generated proto and grpc files.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c2e85daa
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c2e85daa
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c2e85daa

Branch: refs/heads/master
Commit: c2e85daa68b444a9daeb582e686400bdee405964
Parents: bb2b706
Author: Robert Bradshaw 
Authored: Tue May 23 12:30:30 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed May 24 15:01:39 2017 -0700

--
 .../apache_beam/runners/api/beam_fn_api_pb2.py  | 2742 -
 .../runners/api/beam_fn_api_pb2_grpc.py |  205 --
 .../runners/api/beam_runner_api_pb2.py  | 2872 --
 3 files changed, 5819 deletions(-)
--

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #3205

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #2923

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023709#comment-16023709
 ] 

Daniel Halperin commented on BEAM-1542:
---

Thanks [~guymolinari]! Your original sink has now been merged (modulo a few 
cleanups) and [~mkhadikov] from Cloud Spanner team will likely build on it.

> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Mairbek Khadikov
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1542) Need Source/Sink for Spanner

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-1542:
-

Assignee: Mairbek Khadikov  (was: Guy Molinari)

> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Mairbek Khadikov
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023694#comment-16023694
 ] 

ASF GitHub Bot commented on BEAM-1542:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3146


> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Guy Molinari
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023695#comment-16023695
 ] 

ASF GitHub Bot commented on BEAM-1542:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2166


> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Guy Molinari
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2166: BEAM-1542 : Added SpannerIO Data Sink for Cloud Spa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2166


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3146: [BEAM-1542] Takeover #2166 with initial support for...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3146


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[5/7] beam git commit: Fix spanner dependency management

[dhalperi]

Fix spanner dependency management

Also minor cleanup alphabetization in root pom.xml


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8871eadc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8871eadc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8871eadc

Branch: refs/heads/master
Commit: 8871eadcc8413dab906656806b54f3d73b24893d
Parents: e53dc91
Author: Dan Halperin 
Authored: Thu May 4 10:48:54 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 24 14:12:52 2017 -0700

--
 pom.xml| 22 +++---
 sdks/java/io/google-cloud-platform/pom.xml |  8 +---
 2 files changed, 24 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8871eadc/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 3a6289d..c3a6b73 100644
--- a/pom.xml
+++ b/pom.xml
@@ -115,11 +115,13 @@
 0.5.160222
 1.4.0
 1.3.0
+1.0.0-rc2
 1.0-rc2
 1.4.1
 0.6.1
 1.22.0
 1.4.5
+1.0.2
 
0.5.160304
 20.0
 1.2.0
@@ -132,18 +134,20 @@
 1.9.5
 4.1.8.Final
 1.1.33.Fork26
-1.5.0.Final
 3.2.0
 v1-rev10-1.22.0
 1.7.14
+0.16.0-beta
 1.6.2
+4.3.5.RELEASE
 3.1.4
 v1-rev71-1.22.0
 4.4.1
-0.16.0-beta
 4.3.5.RELEASE
-2.0
 1.1.4-M3
+
+1.5.0.Final
+2.0
 2.20
 2.20
 3.6.1
@@ -611,6 +615,12 @@
   
 
   
+com.google.api
+api-common
+${google-api-common.version}
+  
+
+  
 com.google.api-client
 google-api-client
 ${google-clients.version}
@@ -869,6 +879,12 @@
 
   
 com.google.cloud
+google-cloud-core
+${google-cloud-core.version}
+  
+
+  
+com.google.cloud
 google-cloud-spanner
 ${spanner.version}
   

http://git-wip-us.apache.org/repos/asf/beam/blob/8871eadc/sdks/java/io/google-cloud-platform/pom.xml
--
diff --git a/sdks/java/io/google-cloud-platform/pom.xml 
b/sdks/java/io/google-cloud-platform/pom.xml
index 2181895..9143ccf 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -89,7 +89,6 @@
 
   com.google.api.grpc
   grpc-google-common-protos
-  ${grpc-google-common-protos.version}
 
 
 
@@ -100,7 +99,6 @@
 
   com.google.api
   api-common
-  ${api-common.version}
 
 
 
@@ -163,8 +161,12 @@
 
 
   com.google.cloud
+  google-cloud-core
+
+
+
+  com.google.cloud
   google-cloud-spanner
-  ${spanner.version}
 
 
 

[6/7] beam git commit: Delete SpannerCSVLoader

[dhalperi]

Delete SpannerCSVLoader

This is not appropriate for examples. SpannerIO should be well-javadoced
and integration tested.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/216b972b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/216b972b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/216b972b

Branch: refs/heads/master
Commit: 216b972b6b09c0bb612d7f716cfec68ace9777ec
Parents: 8871ead
Author: Dan Halperin 
Authored: Thu May 4 10:48:24 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 24 14:12:52 2017 -0700

--
 .../beam/examples/spanner/SpannerCSVLoader.java | 143 ---
 1 file changed, 143 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/216b972b/examples/java/src/main/java/org/apache/beam/examples/spanner/SpannerCSVLoader.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/spanner/SpannerCSVLoader.java
 
b/examples/java/src/main/java/org/apache/beam/examples/spanner/SpannerCSVLoader.java
deleted file mode 100644
index eee581d..000
--- 
a/examples/java/src/main/java/org/apache/beam/examples/spanner/SpannerCSVLoader.java
+++ /dev/null
@@ -1,143 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.examples.spanner;
-
-import com.google.cloud.spanner.Database;
-import com.google.cloud.spanner.DatabaseAdminClient;
-import com.google.cloud.spanner.Mutation;
-import com.google.cloud.spanner.Operation;
-import com.google.cloud.spanner.Spanner;
-import com.google.cloud.spanner.SpannerException;
-import com.google.cloud.spanner.SpannerOptions;
-import com.google.spanner.admin.database.v1.CreateDatabaseMetadata;
-import java.util.Collections;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.io.gcp.spanner.SpannerIO;
-import org.apache.beam.sdk.options.Default;
-import org.apache.beam.sdk.options.Description;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.options.Validation;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.PCollection;
-
-
-
-/**
- * Generalized bulk loader for importing CSV files into Spanner.
- *
- */
-public class SpannerCSVLoader {
-
-/**
- * Command options specification.
- */
-private interface Options extends PipelineOptions {
-@Description("Create a sample database")
-@Default.Boolean(false)
-boolean isCreateDatabase();
-void setCreateDatabase(boolean createDatabase);
-
-@Description("File to read from ")
-@Validation.Required
-String getInput();
-void setInput(String value);
-
-@Description("Instance ID to write to in Spanner")
-@Validation.Required
-String getInstanceId();
-void setInstanceId(String value);
-
-@Description("Database ID to write to in Spanner")
-@Validation.Required
-String getDatabaseId();
-void setDatabaseId(String value);
-
-@Description("Table name")
-@Validation.Required
-String getTable();
-void setTable(String value);
-  }
-
-
-  /**
-   * Constructs and executes the processing pipeline based upon command 
options.
-   */
-  public static void main(String[] args) throws Exception {
-  Options options = 
PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);
-
-  Pipeline p = Pipeline.create(options);
-  PCollection lines = 
p.apply(TextIO.Read.from(options.getInput()));
-  PCollection mutations = lines
-  .apply(ParDo.of(new NaiveParseCsvFn(options.getTable(;
-  mutations
-  .apply(SpannerIO.writeTo(options.getInstanceId(), 
options.getDatabaseId()));
-  p.run().waitUntilFinish();
-  }
-
-  public static void createDatabase(Options options) {
-  Spanner client = 

[2/7] beam git commit: Refine Spanner API tests

[dhalperi]

Refine Spanner API tests

And remove outdated Bigtable comment


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/65e740ed
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/65e740ed
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/65e740ed

Branch: refs/heads/master
Commit: 65e740ed2f265b550e70af867535d7bba7306298
Parents: 216b972
Author: Dan Halperin 
Authored: Mon May 15 10:16:18 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 24 14:12:52 2017 -0700

--
 .../java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java   | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/65e740ed/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java
index 8950452..91caded 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java
@@ -63,7 +63,10 @@ public class GcpApiSurfaceTest {
 
Matchers.equalTo(com.google.cloud.bigtable.grpc.BigtableClusterName.class),
 
Matchers.equalTo(com.google.cloud.bigtable.grpc.BigtableInstanceName.class),
 
Matchers.equalTo(com.google.cloud.bigtable.grpc.BigtableTableName.class),
-// via Bigtable, PR above out to fix.
+Matchers.equalTo(com.google.cloud.ByteArray.class),
+Matchers.equalTo(com.google.cloud.Date.class),
+Matchers.equalTo(com.google.cloud.Timestamp.class),
+classesInPackage("com.google.cloud.spanner"),
 classesInPackage("com.google.datastore.v1"),
 classesInPackage("com.google.protobuf"),
 classesInPackage("com.google.type"),
@@ -75,9 +78,6 @@ public class GcpApiSurfaceTest {
 classesInPackage("javax"),
 classesInPackage("org.apache.beam"),
 classesInPackage("org.apache.commons.logging"),
-classesInPackage("com.google.cloud"),
-classesInPackage("com.google.cloud.spanner"),
-// via Bigtable
 classesInPackage("org.joda.time"));
 
 assertThat(apiSurface, containsOnlyClassesMatching(allowedClasses));

[3/7] beam git commit: Minor style, compilation, javadoc fixups

[dhalperi]

Minor style, compilation, javadoc fixups


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e53dc915
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e53dc915
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e53dc915

Branch: refs/heads/master
Commit: e53dc915d8da30aa49fb28a632a5d890723a73dc
Parents: 9cd6c3b
Author: Dan Halperin 
Authored: Thu May 4 10:42:07 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 24 14:12:52 2017 -0700

--
 .../beam/sdk/io/gcp/spanner/SpannerIO.java  | 45 +---
 1 file changed, 20 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e53dc915/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
index 172ed8f..c9c81a5 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
@@ -19,10 +19,6 @@ package org.apache.beam.sdk.io.gcp.spanner;
 
 import static com.google.common.base.Preconditions.checkNotNull;
 
-import com.google.api.client.util.BackOff;
-import com.google.api.client.util.BackOffUtils;
-import com.google.api.client.util.Sleeper;
-
 import com.google.cloud.spanner.AbortedException;
 import com.google.cloud.spanner.DatabaseClient;
 import com.google.cloud.spanner.DatabaseId;
@@ -35,23 +31,25 @@ import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
 import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.DisplayData.Builder;
+import org.apache.beam.sdk.util.BackOff;
+import org.apache.beam.sdk.util.BackOffUtils;
 import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.Sleeper;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PDone;
 import org.joda.time.Duration;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-
-
-
 /**
- *  https://cloud.google.com/spanner/;>Google Cloud Spanner 
connectors.
+ * {@link PTransform Transforms} for reading from and writing to
+ * https://cloud.google.com/spanner;>Google Cloud Spanner.
  *
  * Reading from Cloud Spanner
  * Status: Not implemented.
@@ -61,9 +59,8 @@ import org.slf4j.LoggerFactory;
  *
  * {@link SpannerIO#writeTo} batches together and concurrently writes a set 
of {@link Mutation}s.
  * To configure Cloud Spanner sink, you must apply {@link SpannerIO#writeTo} 
transform to
- * {@link PCollection} and specify instance and database identifiers.
- * For example, following code sketches out a pipeline that imports data from 
the CSV file to Cloud
- * Spanner.
+ * {@link PCollection} and specify instance and database identifiers. For 
example, following code
+ * sketches out a pipeline that imports data from the CSV file to Cloud 
Spanner.
  *
  * {@code
  *
@@ -77,14 +74,10 @@ import org.slf4j.LoggerFactory;
  * p.run();
  *
  * }
-
  */
 @Experimental(Experimental.Kind.SOURCE_SINK)
 public class SpannerIO {
 
-  private SpannerIO() {
-  }
-
   @VisibleForTesting
   static final int SPANNER_MUTATIONS_PER_COMMIT_LIMIT = 2;
 
@@ -97,10 +90,11 @@ public class SpannerIO {
   }
 
   /**
-   * A {@link PTransform} that writes {@link Mutation} objects to Cloud 
Spanner.
+   * A {@link PTransform} that writes {@link Mutation} objects to Google Cloud 
Spanner.
*
* @see SpannerIO
*/
+  @Experimental(Experimental.Kind.SOURCE_SINK)
   public static class Writer extends PTransform {
 
 private final String instanceId;
@@ -130,7 +124,7 @@ public class SpannerIO {
 }
 
 @Override
-public void validate(PCollection input) {
+public void validate(PipelineOptions options) {
   checkNotNull(instanceId, "instanceId");
   checkNotNull(databaseId, "databaseId");
 }
@@ -152,19 +146,17 @@ public class SpannerIO {
   .addIfNotNull(DisplayData.item("databaseId", databaseId)
   .withLabel("Output Database"));
 }
-
   }
 
-
   /**
-   * {@link DoFn} that writes {@link Mutation}s to Cloud Spanner. Mutations 
are written in
+   * {@link DoFn} that writes {@link Mutation}s to Google 

[7/7] beam git commit: This closes #3146

[dhalperi]

This closes #3146


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/539e229f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/539e229f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/539e229f

Branch: refs/heads/master
Commit: 539e229fece27e0e33d19240f21fbaad1b7063a1
Parents: 0637df1 e207022
Author: Dan Halperin 
Authored: Wed May 24 14:12:54 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 24 14:12:54 2017 -0700

--
 pom.xml |  28 +-
 sdks/java/io/google-cloud-platform/pom.xml  |  35 ++-
 .../beam/sdk/io/gcp/spanner/SpannerIO.java  | 275 +++
 .../beam/sdk/io/gcp/spanner/package-info.java   |  23 ++
 .../beam/sdk/io/gcp/GcpApiSurfaceTest.java  |   6 +-
 5 files changed, 353 insertions(+), 14 deletions(-)
--

[1/7] beam git commit: SpannerIO.Write cleanup and style fixes

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 0637df1bc -> 539e229fe


SpannerIO.Write cleanup and style fixes

* Rename to Write to match the rest of the SDK.
* Convert to AutoValue, delete toString.
* Drop .writeTo(), instead use .write() as default constructor.
* Temporarily drop withBatchSize, as its existence is not clearly
  justified.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e2070229
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e2070229
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e2070229

Branch: refs/heads/master
Commit: e2070229f82ac4b78c470051cb94b68b58b75f16
Parents: 65e740e
Author: Dan Halperin 
Authored: Mon May 15 10:42:57 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 24 14:12:52 2017 -0700

--
 .../beam/sdk/io/gcp/spanner/SpannerIO.java  | 116 ++-
 1 file changed, 62 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e2070229/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
index c9c81a5..ec11931 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
@@ -19,6 +19,7 @@ package org.apache.beam.sdk.io.gcp.spanner;
 
 import static com.google.common.base.Preconditions.checkNotNull;
 
+import com.google.auto.value.AutoValue;
 import com.google.cloud.spanner.AbortedException;
 import com.google.cloud.spanner.DatabaseClient;
 import com.google.cloud.spanner.DatabaseId;
@@ -26,12 +27,11 @@ import com.google.cloud.spanner.Mutation;
 import com.google.cloud.spanner.Spanner;
 import com.google.cloud.spanner.SpannerOptions;
 import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.MoreObjects;
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
+import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
@@ -48,31 +48,29 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
- * {@link PTransform Transforms} for reading from and writing to
+ * Experimental {@link PTransform Transforms} for reading from and writing to
  * https://cloud.google.com/spanner;>Google Cloud Spanner.
  *
  * Reading from Cloud Spanner
- * Status: Not implemented.
+ *
+ * This functionality is not yet implemented.
  *
  * Writing to Cloud Spanner
- * Status: Experimental.
  *
- * {@link SpannerIO#writeTo} batches together and concurrently writes a set 
of {@link Mutation}s.
- * To configure Cloud Spanner sink, you must apply {@link SpannerIO#writeTo} 
transform to
- * {@link PCollection} and specify instance and database identifiers. For 
example, following code
- * sketches out a pipeline that imports data from the CSV file to Cloud 
Spanner.
+ * The Cloud Spanner {@link SpannerIO.Write} transform writes to Cloud 
Spanner by executing a
+ * collection of input row {@link Mutation Mutations}. The mutations grouped 
into batches for
+ * efficiency.
  *
- * {@code
+ * To configure the write transform, create an instance using {@link 
#write()} and then specify
+ * the destination Cloud Spanner instance ({@link 
Write#withInstanceId(String)} and destination
+ * database ({@link Write#withDatabaseId(String)}). For example:
  *
- * Pipeline p = ...;
- * // Read the CSV file.
- * PCollection lines = p.apply("Read CSV file", 
TextIO.Read.from(options.getInput()));
- * // Parse the line and convert to mutation.
- * PCollection mutations = lines.apply("Parse CSV", parseFromCsv());
+ * {@code
+ * // Earlier in the pipeline, create a PCollection of Mutations to be written 
to Cloud Spanner.
+ * PCollection mutations = ...;
  * // Write mutations.
- * mutations.apply("Write", SpannerIO.writeTo(options.getInstanceId(), 
options.getDatabaseId()));
- * p.run();
- *
+ * mutations.apply(
+ * "Write", 
SpannerIO.write().withInstanceId("instance").withDatabaseId("database"));
  * }
  */
 @Experimental(Experimental.Kind.SOURCE_SINK)
@@ -81,12 +79,14 @@ public class SpannerIO {
   @VisibleForTesting
   static final int SPANNER_MUTATIONS_PER_COMMIT_LIMIT = 2;
 
-/**
- * Creates an instance of {@link Writer}. 

[4/7] beam git commit: Initial implementation of SpannerIO.Write

[dhalperi]

Initial implementation of SpannerIO.Write

This closes #2166.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9cd6c3bd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9cd6c3bd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9cd6c3bd

Branch: refs/heads/master
Commit: 9cd6c3bdefd50473f9fc6fac359213dcd6b4e4d4
Parents: 0637df1
Author: MOLIG004 
Authored: Thu May 4 09:21:23 2017 -0700
Committer: Dan Halperin 
Committed: Wed May 24 14:12:52 2017 -0700

--
 .../beam/examples/spanner/SpannerCSVLoader.java | 143 ++
 pom.xml |   8 +
 sdks/java/io/google-cloud-platform/pom.xml  |  33 ++-
 .../beam/sdk/io/gcp/spanner/SpannerIO.java  | 272 +++
 .../beam/sdk/io/gcp/spanner/package-info.java   |  23 ++
 .../beam/sdk/io/gcp/GcpApiSurfaceTest.java  |   2 +
 6 files changed, 471 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9cd6c3bd/examples/java/src/main/java/org/apache/beam/examples/spanner/SpannerCSVLoader.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/spanner/SpannerCSVLoader.java
 
b/examples/java/src/main/java/org/apache/beam/examples/spanner/SpannerCSVLoader.java
new file mode 100644
index 000..eee581d
--- /dev/null
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/spanner/SpannerCSVLoader.java
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples.spanner;
+
+import com.google.cloud.spanner.Database;
+import com.google.cloud.spanner.DatabaseAdminClient;
+import com.google.cloud.spanner.Mutation;
+import com.google.cloud.spanner.Operation;
+import com.google.cloud.spanner.Spanner;
+import com.google.cloud.spanner.SpannerException;
+import com.google.cloud.spanner.SpannerOptions;
+import com.google.spanner.admin.database.v1.CreateDatabaseMetadata;
+import java.util.Collections;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.io.gcp.spanner.SpannerIO;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.Validation;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+
+
+
+/**
+ * Generalized bulk loader for importing CSV files into Spanner.
+ *
+ */
+public class SpannerCSVLoader {
+
+/**
+ * Command options specification.
+ */
+private interface Options extends PipelineOptions {
+@Description("Create a sample database")
+@Default.Boolean(false)
+boolean isCreateDatabase();
+void setCreateDatabase(boolean createDatabase);
+
+@Description("File to read from ")
+@Validation.Required
+String getInput();
+void setInput(String value);
+
+@Description("Instance ID to write to in Spanner")
+@Validation.Required
+String getInstanceId();
+void setInstanceId(String value);
+
+@Description("Database ID to write to in Spanner")
+@Validation.Required
+String getDatabaseId();
+void setDatabaseId(String value);
+
+@Description("Table name")
+@Validation.Required
+String getTable();
+void setTable(String value);
+  }
+
+
+  /**
+   * Constructs and executes the processing pipeline based upon command 
options.
+   */
+  public static void main(String[] args) throws Exception {
+  Options options = 
PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);
+
+  Pipeline p = Pipeline.create(options);
+  PCollection lines = 
p.apply(TextIO.Read.from(options.getInput()));
+  PCollection mutations = lines
+  .apply(ParDo.of(new NaiveParseCsvFn(options.getTable(;
+  mutations
+   

Jenkins build is back to normal : beam_PostCommit_Python_Verify #2309

[Apache Jenkins Server]

See 

[GitHub] beam-site pull request #250: [BEAM-2328] Add TikaIO to the list of in-progre...

[sberyozkin]

GitHub user sberyozkin opened a pull request:

https://github.com/apache/beam-site/pull/250

[BEAM-2328] Add TikaIO to the list of in-progress transforms



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sberyozkin/beam-site patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/250.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #250






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2333) Rehydrate Pipeline from Runner API proto

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023604#comment-16023604
 ] 

ASF GitHub Bot commented on BEAM-2333:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3216


> Rehydrate Pipeline from Runner API proto
> 
>
> Key: BEAM-2333
> URL: https://issues.apache.org/jira/browse/BEAM-2333
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3216: [BEAM-2333] Runner API translation renames

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3216


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[9/9] beam git commit: This closes #3216: [BEAM-2333] Runner API translation renames

[kenn]

This closes #3216: [BEAM-2333] Runner API translation renames

  Rename Triggers to TriggerTranslation
  Rename WindowingStrategies to WindowingStrategyTranslation
  Rename Coders to CoderTranslation
  Rename ReadTranslator to ReadTranslation
  Rename WindowIntoTranslator to WindowIntoTranslation
  Rename ParDos to ParDoTranslation
  Rename PTransforms to PTransformTranslation
  Rename PCollections to PCollectionTranslation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0637df1b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0637df1b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0637df1b

Branch: refs/heads/master
Commit: 0637df1bcc953855f0f003a72b95dd687b21fec9
Parents: 6418bcf 4fa38e2
Author: Kenneth Knowles 
Authored: Wed May 24 13:13:46 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 24 13:13:46 2017 -0700

--
 .../operators/ApexGroupByKeyOperator.java   |   4 +-
 .../core/construction/CoderTranslation.java | 193 ++
 .../beam/runners/core/construction/Coders.java  | 193 --
 .../construction/PCollectionTranslation.java|  98 ++
 .../runners/core/construction/PCollections.java |  97 --
 .../construction/PTransformTranslation.java | 119 +++
 .../runners/core/construction/PTransforms.java  | 119 ---
 .../core/construction/ParDoTranslation.java | 349 +++
 .../beam/runners/core/construction/ParDos.java  | 348 --
 .../core/construction/ReadTranslation.java  | 127 +++
 .../core/construction/ReadTranslator.java   | 127 ---
 .../core/construction/SdkComponents.java|  10 +-
 .../TransformPayloadTranslatorRegistrar.java|   2 +-
 .../core/construction/TriggerTranslation.java   | 336 ++
 .../runners/core/construction/Triggers.java | 336 --
 .../construction/WindowIntoTranslation.java |  61 
 .../core/construction/WindowIntoTranslator.java |  61 
 .../core/construction/WindowingStrategies.java  | 278 ---
 .../WindowingStrategyTranslation.java   | 278 +++
 .../core/construction/CoderTranslationTest.java | 165 +
 .../runners/core/construction/CodersTest.java   | 164 -
 .../PCollectionTranslationTest.java | 203 +++
 .../core/construction/PCollectionsTest.java | 201 ---
 .../construction/PTransformTranslationTest.java | 189 ++
 .../core/construction/PTransformsTest.java  | 188 --
 .../core/construction/ParDoTranslationTest.java | 234 +
 .../runners/core/construction/ParDosTest.java   | 233 -
 .../core/construction/ReadTranslationTest.java  | 179 ++
 .../core/construction/ReadTranslatorTest.java   | 179 --
 .../construction/TriggerTranslationTest.java| 112 ++
 .../runners/core/construction/TriggersTest.java | 111 --
 .../construction/WindowIntoTranslationTest.java | 127 +++
 .../construction/WindowIntoTranslatorTest.java  | 127 ---
 .../construction/WindowingStrategiesTest.java   | 110 --
 .../WindowingStrategyTranslationTest.java   | 111 ++
 .../GroupAlsoByWindowViaWindowSetNewDoFn.java   |   4 +-
 .../beam/runners/core/ReduceFnTester.java   |  10 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   4 +-
 .../dataflow/DataflowPipelineTranslator.java|   4 +-
 .../SparkGroupAlsoByWindowViaWindowSet.java |   4 +-
 ...SparkGroupAlsoByWindowViaOutputBufferFn.java |   4 +-
 41 files changed, 2906 insertions(+), 2893 deletions(-)
--

[4/9] beam git commit: Rename PCollections to PCollectionTranslation

[kenn]

Rename PCollections to PCollectionTranslation

This is to give a standard and obvious suffix for all our helper classes
for translating to/from Runner API protos.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/940819e2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/940819e2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/940819e2

Branch: refs/heads/master
Commit: 940819e2acccb8d77b88aaee821ee972aca02eb3
Parents: 4ec3366
Author: Kenneth Knowles 
Authored: Tue May 23 15:26:44 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue May 23 15:53:41 2017 -0700

--
 .../construction/PCollectionTranslation.java|  97 +
 .../runners/core/construction/PCollections.java |  97 -
 .../core/construction/SdkComponents.java|   3 +-
 .../PCollectionTranslationTest.java | 203 +++
 .../core/construction/PCollectionsTest.java | 201 --
 5 files changed, 302 insertions(+), 299 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/940819e2/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionTranslation.java
new file mode 100644
index 000..cad7b97
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionTranslation.java
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollection.IsBounded;
+import org.apache.beam.sdk.values.WindowingStrategy;
+
+/**
+ * Utility methods for translating {@link PCollection PCollections} to and 
from Runner API protos.
+ */
+public class PCollectionTranslation {
+  private PCollectionTranslation() {}
+
+  public static RunnerApi.PCollection toProto(PCollection pCollection, 
SdkComponents components)
+  throws IOException {
+String coderId = components.registerCoder(pCollection.getCoder());
+String windowingStrategyId =
+
components.registerWindowingStrategy(pCollection.getWindowingStrategy());
+// TODO: Display Data
+
+return RunnerApi.PCollection.newBuilder()
+.setUniqueName(pCollection.getName())
+.setCoderId(coderId)
+.setIsBounded(toProto(pCollection.isBounded()))
+.setWindowingStrategyId(windowingStrategyId)
+.build();
+  }
+
+  public static IsBounded isBounded(RunnerApi.PCollection pCollection) {
+return fromProto(pCollection.getIsBounded());
+  }
+
+  public static Coder getCoder(
+  RunnerApi.PCollection pCollection, RunnerApi.Components components) 
throws IOException {
+return 
Coders.fromProto(components.getCodersOrThrow(pCollection.getCoderId()), 
components);
+  }
+
+  public static WindowingStrategy getWindowingStrategy(
+  RunnerApi.PCollection pCollection, RunnerApi.Components components)
+  throws InvalidProtocolBufferException {
+return WindowingStrategies.fromProto(
+
components.getWindowingStrategiesOrThrow(pCollection.getWindowingStrategyId()), 
components);
+  }
+
+  private static RunnerApi.IsBounded toProto(IsBounded bounded) {
+switch (bounded) {
+  case BOUNDED:
+return RunnerApi.IsBounded.BOUNDED;
+  case UNBOUNDED:
+return RunnerApi.IsBounded.UNBOUNDED;
+  default:
+throw new IllegalArgumentException(
+String.format("Unknown %s %s", IsBounded.class.getSimpleName(), 

[3/9] beam git commit: Rename ParDos to ParDoTranslation

[kenn]

Rename ParDos to ParDoTranslation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/44609383
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/44609383
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/44609383

Branch: refs/heads/master
Commit: 446093836016dabf021d34ca0a858e313f493e2f
Parents: 9b6728e
Author: Kenneth Knowles 
Authored: Tue May 23 15:28:49 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue May 23 15:53:41 2017 -0700

--
 .../core/construction/ParDoTranslation.java | 348 +++
 .../beam/runners/core/construction/ParDos.java  | 348 ---
 .../core/construction/ParDoTranslationTest.java | 234 +
 .../runners/core/construction/ParDosTest.java   | 233 -
 4 files changed, 582 insertions(+), 581 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/44609383/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
new file mode 100644
index 000..baed246
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
@@ -0,0 +1,348 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Sets;
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.BytesValue;
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import 
org.apache.beam.runners.core.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.IterableCoder;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.Components;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.ParDoPayload;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.Parameter.Type;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.SideInput;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.SideInput.Builder;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.StateSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.TimerSpec;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Materializations;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.ParDo.MultiOutput;
+import org.apache.beam.sdk.transforms.ViewFn;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.Cases;
+import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionTrackerParameter;
+import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature.StateDeclaration;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature.TimerDeclaration;
+import 

[1/9] beam git commit: Rename Triggers to TriggerTranslation

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 6418bcfcb -> 0637df1bc


Rename Triggers to TriggerTranslation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4fa38e2d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4fa38e2d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4fa38e2d

Branch: refs/heads/master
Commit: 4fa38e2d590cc4472c119b57790f7a724a700e43
Parents: c8b2119
Author: Kenneth Knowles 
Authored: Tue May 23 15:33:33 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue May 23 15:53:41 2017 -0700

--
 .../operators/ApexGroupByKeyOperator.java   |   4 +-
 .../core/construction/TriggerTranslation.java   | 336 +++
 .../runners/core/construction/Triggers.java | 336 ---
 .../WindowingStrategyTranslation.java   |   4 +-
 .../construction/TriggerTranslationTest.java| 112 +++
 .../runners/core/construction/TriggersTest.java | 111 --
 .../GroupAlsoByWindowViaWindowSetNewDoFn.java   |   4 +-
 .../beam/runners/core/ReduceFnTester.java   |  10 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   4 +-
 .../SparkGroupAlsoByWindowViaWindowSet.java |   4 +-
 ...SparkGroupAlsoByWindowViaOutputBufferFn.java |   4 +-
 11 files changed, 466 insertions(+), 463 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4fa38e2d/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 1d48e20..39f681f 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -41,7 +41,7 @@ import org.apache.beam.runners.core.StateInternalsFactory;
 import org.apache.beam.runners.core.SystemReduceFn;
 import org.apache.beam.runners.core.TimerInternals;
 import org.apache.beam.runners.core.TimerInternals.TimerData;
-import org.apache.beam.runners.core.construction.Triggers;
+import org.apache.beam.runners.core.construction.TriggerTranslation;
 import org.apache.beam.runners.core.triggers.ExecutableTriggerStateMachine;
 import org.apache.beam.runners.core.triggers.TriggerStateMachines;
 import org.apache.beam.sdk.coders.Coder;
@@ -163,7 +163,7 @@ public class ApexGroupByKeyOperator implements 
Operator,
 windowingStrategy,
 ExecutableTriggerStateMachine.create(
 TriggerStateMachines.stateMachineForTrigger(
-Triggers.toProto(windowingStrategy.getTrigger(,
+TriggerTranslation.toProto(windowingStrategy.getTrigger(,
 stateInternalsFactory.stateInternalsForKey(key),
 timerInternals,
 new OutputWindowedValue>() {

http://git-wip-us.apache.org/repos/asf/beam/blob/4fa38e2d/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TriggerTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TriggerTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TriggerTranslation.java
new file mode 100644
index 000..777b165
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TriggerTranslation.java
@@ -0,0 +1,336 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.io.Serializable;
+import 

[7/9] beam git commit: Rename PTransforms to PTransformTranslation

[kenn]

Rename PTransforms to PTransformTranslation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9b6728e2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9b6728e2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9b6728e2

Branch: refs/heads/master
Commit: 9b6728e24748791b7181b20183df3ada31f45682
Parents: 940819e
Author: Kenneth Knowles 
Authored: Tue May 23 15:28:08 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue May 23 15:53:41 2017 -0700

--
 .../construction/PTransformTranslation.java | 119 
 .../runners/core/construction/PTransforms.java  | 119 
 .../beam/runners/core/construction/ParDos.java  |   4 +-
 .../core/construction/SdkComponents.java|   3 +-
 .../TransformPayloadTranslatorRegistrar.java|   2 +-
 .../core/construction/WindowIntoTranslator.java |   2 +-
 .../construction/PTransformTranslationTest.java | 189 +++
 .../core/construction/PTransformsTest.java  | 188 --
 8 files changed, 314 insertions(+), 312 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9b6728e2/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
new file mode 100644
index 000..86638de
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.ImmutableMap;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.ServiceLoader;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+
+/**
+ * Utilities for converting {@link PTransform PTransforms} to and from {@link 
RunnerApi Runner API
+ * protocol buffers}.
+ */
+public class PTransformTranslation {
+  private static final Map
+  KNOWN_PAYLOAD_TRANSLATORS = loadTransformPayloadTranslators();
+
+  private static Map
+  loadTransformPayloadTranslators() {
+ImmutableMap.Builder builder =
+ImmutableMap.builder();
+for (TransformPayloadTranslatorRegistrar registrar :
+ServiceLoader.load(TransformPayloadTranslatorRegistrar.class)) {
+  builder.putAll(registrar.getTransformPayloadTranslators());
+}
+return builder.build();
+  }
+
+  private PTransformTranslation() {}
+
+  /**
+   * Translates an {@link AppliedPTransform} into a runner API proto.
+   *
+   * Does not register the {@code appliedPTransform} within the provided 
{@link SdkComponents}.
+   */
+  static RunnerApi.PTransform toProto(
+  AppliedPTransform appliedPTransform,
+  List subtransforms,
+  SdkComponents components)
+  throws IOException {
+RunnerApi.PTransform.Builder transformBuilder = 
RunnerApi.PTransform.newBuilder();
+for (Map.Entry taggedInput : 
appliedPTransform.getInputs().entrySet()) {
+  checkArgument(
+  taggedInput.getValue() instanceof PCollection,
+  "Unexpected input type %s",
+  taggedInput.getValue().getClass());
+  

[8/9] beam git commit: Rename ReadTranslator to ReadTranslation

[kenn]

Rename ReadTranslator to ReadTranslation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b35e91d4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b35e91d4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b35e91d4

Branch: refs/heads/master
Commit: b35e91d4ed99b74d37a08a1385018b4ca326b3a0
Parents: bc4f44f
Author: Kenneth Knowles 
Authored: Tue May 23 15:30:22 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue May 23 15:53:41 2017 -0700

--
 .../core/construction/ReadTranslation.java  | 127 +
 .../core/construction/ReadTranslator.java   | 127 -
 .../core/construction/ReadTranslationTest.java  | 179 +++
 .../core/construction/ReadTranslatorTest.java   | 179 ---
 4 files changed, 306 insertions(+), 306 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b35e91d4/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslation.java
new file mode 100644
index 000..d6c3400
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslation.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.BytesValue;
+import com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.IsBounded;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.ReadPayload;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
+import org.apache.beam.sdk.io.BoundedSource;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.Source;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.SerializableUtils;
+
+/**
+ * Methods for translating {@link Read.Bounded} and {@link Read.Unbounded}
+ * {@link PTransform PTransforms} into {@link ReadPayload} protos.
+ */
+public class ReadTranslation {
+  private static final String JAVA_SERIALIZED_BOUNDED_SOURCE = 
"urn:beam:java:boundedsource:v1";
+  private static final String JAVA_SERIALIZED_UNBOUNDED_SOURCE = 
"urn:beam:java:unboundedsource:v1";
+
+  public static ReadPayload toProto(Read.Bounded read) {
+return ReadPayload.newBuilder()
+.setIsBounded(IsBounded.BOUNDED)
+.setSource(toProto(read.getSource()))
+.build();
+  }
+
+  public static ReadPayload toProto(Read.Unbounded read) {
+return ReadPayload.newBuilder()
+.setIsBounded(IsBounded.UNBOUNDED)
+.setSource(toProto(read.getSource()))
+.build();
+  }
+
+  public static SdkFunctionSpec toProto(Source source) {
+if (source instanceof BoundedSource) {
+  return toProto((BoundedSource) source);
+} else if (source instanceof UnboundedSource) {
+  return toProto((UnboundedSource) source);
+} else {
+  throw new IllegalArgumentException(
+  String.format("Unknown %s type %s", Source.class.getSimpleName(), 
source.getClass()));
+}
+  }
+
+  private static SdkFunctionSpec toProto(BoundedSource source) {
+return SdkFunctionSpec.newBuilder()
+.setSpec(
+FunctionSpec.newBuilder()
+.setUrn(JAVA_SERIALIZED_BOUNDED_SOURCE)
+.setParameter(
+Any.pack(
+BytesValue.newBuilder()
+.setValue(
+ 

[6/9] beam git commit: Rename Coders to CoderTranslation

[kenn]

Rename Coders to CoderTranslation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7e37b703
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7e37b703
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7e37b703

Branch: refs/heads/master
Commit: 7e37b70317bd06f300a5423cc4cb76a06c3955c3
Parents: b35e91d
Author: Kenneth Knowles 
Authored: Tue May 23 15:31:49 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue May 23 15:53:41 2017 -0700

--
 .../core/construction/CoderTranslation.java | 193 +++
 .../beam/runners/core/construction/Coders.java  | 193 ---
 .../construction/PCollectionTranslation.java|   3 +-
 .../core/construction/ParDoTranslation.java |   3 +-
 .../core/construction/SdkComponents.java|   2 +-
 .../core/construction/CoderTranslationTest.java | 165 
 .../runners/core/construction/CodersTest.java   | 164 
 7 files changed, 363 insertions(+), 360 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7e37b703/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
new file mode 100644
index 000..470db6a
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableBiMap;
+import com.google.common.collect.ImmutableMap;
+import com.google.protobuf.Any;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.BytesValue;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.IterableCoder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.LengthPrefixCoder;
+import org.apache.beam.sdk.coders.StructuredCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.Components;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import 
org.apache.beam.sdk.transforms.windowing.IntervalWindow.IntervalWindowCoder;
+import org.apache.beam.sdk.util.SerializableUtils;
+import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
+
+/** Converts to and from Beam Runner API representations of {@link Coder 
Coders}. */
+public class CoderTranslation {
+  // This URN says that the coder is just a UDF blob this SDK understands
+  // TODO: standardize such things
+  public static final String JAVA_SERIALIZED_CODER_URN = 
"urn:beam:coders:javasdk:0.1";
+
+  // The URNs for coders which are shared across languages
+  @VisibleForTesting
+  static final BiMap 
KNOWN_CODER_URNS =
+  ImmutableBiMap.builder()
+  .put(ByteArrayCoder.class, "urn:beam:coders:bytes:0.1")
+  .put(KvCoder.class, "urn:beam:coders:kv:0.1")
+  .put(VarLongCoder.class, "urn:beam:coders:varint:0.1")
+  .put(IntervalWindowCoder.class, 
"urn:beam:coders:interval_window:0.1")
+  .put(IterableCoder.class, 

[2/9] beam git commit: Rename WindowingStrategies to WindowingStrategyTranslation

[kenn]

Rename WindowingStrategies to WindowingStrategyTranslation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c8b2119a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c8b2119a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c8b2119a

Branch: refs/heads/master
Commit: c8b2119ab9a75c7f781ce73ea9352734640a6f46
Parents: 7e37b70
Author: Kenneth Knowles 
Authored: Tue May 23 15:32:47 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue May 23 15:53:41 2017 -0700

--
 .../construction/PCollectionTranslation.java|   2 +-
 .../core/construction/ParDoTranslation.java |   2 +-
 .../core/construction/SdkComponents.java|   2 +-
 .../construction/WindowIntoTranslation.java |   4 +-
 .../core/construction/WindowingStrategies.java  | 278 ---
 .../WindowingStrategyTranslation.java   | 278 +++
 .../construction/WindowingStrategiesTest.java   | 110 
 .../WindowingStrategyTranslationTest.java   | 111 
 .../dataflow/DataflowPipelineTranslator.java|   4 +-
 9 files changed, 396 insertions(+), 395 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c8b2119a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionTranslation.java
index 46f714e..303c02d 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionTranslation.java
@@ -60,7 +60,7 @@ public class PCollectionTranslation {
   public static WindowingStrategy getWindowingStrategy(
   RunnerApi.PCollection pCollection, RunnerApi.Components components)
   throws InvalidProtocolBufferException {
-return WindowingStrategies.fromProto(
+return WindowingStrategyTranslation.fromProto(
 
components.getWindowingStrategiesOrThrow(pCollection.getWindowingStrategyId()), 
components);
   }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/c8b2119a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
index bc5bb0e..28d577f 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
@@ -265,7 +265,7 @@ public class ParDoTranslation {
 RunnerApi.PCollection inputCollection =
 components.getPcollectionsOrThrow(parDoTransform.getInputsOrThrow(id));
 WindowingStrategy windowingStrategy =
-WindowingStrategies.fromProto(
+WindowingStrategyTranslation.fromProto(
 
components.getWindowingStrategiesOrThrow(inputCollection.getWindowingStrategyId()),
 components);
 Coder elemCoder =

http://git-wip-us.apache.org/repos/asf/beam/blob/c8b2119a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
index 5c81875..b0f164f 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
@@ -200,7 +200,7 @@ class SdkComponents {
 String name = uniqify(baseName, windowingStrategyIds.values());
 windowingStrategyIds.put(windowingStrategy, name);
 RunnerApi.WindowingStrategy windowingStrategyProto =
-WindowingStrategies.toProto(windowingStrategy, this);
+WindowingStrategyTranslation.toProto(windowingStrategy, this);
 componentsBuilder.putWindowingStrategies(name, windowingStrategyProto);
 return name;
   }

[5/9] beam git commit: Rename WindowIntoTranslator to WindowIntoTranslation

[kenn]

Rename WindowIntoTranslator to WindowIntoTranslation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bc4f44f4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bc4f44f4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bc4f44f4

Branch: refs/heads/master
Commit: bc4f44f46ca6afc0018834eb467b1112763c3323
Parents: 4460938
Author: Kenneth Knowles 
Authored: Tue May 23 15:29:16 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue May 23 15:53:41 2017 -0700

--
 .../construction/WindowIntoTranslation.java |  61 +
 .../core/construction/WindowIntoTranslator.java |  61 -
 .../construction/WindowIntoTranslationTest.java | 127 +++
 .../construction/WindowIntoTranslatorTest.java  | 127 ---
 4 files changed, 188 insertions(+), 188 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bc4f44f4/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
new file mode 100644
index 000..69793b5
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import com.google.protobuf.Any;
+import com.google.protobuf.InvalidProtocolBufferException;
+import 
org.apache.beam.runners.core.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi.WindowIntoPayload;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+
+/**
+ * Utility methods for translating a {@link Window.Assign} to and from {@link 
RunnerApi}
+ * representations.
+ */
+public class WindowIntoTranslation {
+
+  static class WindowAssignTranslator implements 
TransformPayloadTranslator {
+@Override
+public FunctionSpec translate(
+AppliedPTransform> transform, SdkComponents 
components) {
+  return FunctionSpec.newBuilder()
+  .setUrn("urn:beam:transform:window:v1")
+  .setParameter(
+  Any.pack(WindowIntoTranslation.toProto(transform.getTransform(), 
components)))
+  .build();
+}
+  }
+
+  public static WindowIntoPayload toProto(Window.Assign transform, 
SdkComponents components) {
+return WindowIntoPayload.newBuilder()
+.setWindowFn(WindowingStrategies.toProto(transform.getWindowFn(), 
components))
+.build();
+  }
+
+  public static WindowFn getWindowFn(WindowIntoPayload payload)
+  throws InvalidProtocolBufferException {
+SdkFunctionSpec spec = payload.getWindowFn();
+return WindowingStrategies.windowFnFromProto(spec);
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/bc4f44f4/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslator.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslator.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslator.java
deleted file mode 100644
index 7ed2a49..000
--- 

[jira] [Commented] (BEAM-2352) Enable elasticsearch IO ITs

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023585#comment-16023585
 ] 

Stephen Sisk commented on BEAM-2352:


I was ready to get these turned on, but we noticed an issue with the JDBC tests 
- the IP addresses of the k8s pods change more than I'd hoped they would.

I'd like to get the PKB integration working (to automatically pick up the IP 
address from the k8s cluster) before we turn the tests on since they'll require 
frequent manual fixes otherwise.

We've disabled the JDBC tests as well until we get this fixed.

> Enable elasticsearch IO ITs
> ---
>
> Key: BEAM-2352
> URL: https://issues.apache.org/jira/browse/BEAM-2352
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> We want the IO ITs to be run in jenkins CI, and this is the second one we'll 
> be getting going.
> I'm following the instructions in the IO testing guide -
>  
> https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit#heading=h.7jl9ndblih6c
>  - this is the first time I'm doing so, and this is a test to see if the 
> instructions are correct. 
> Once I've done this, we'll have good instructions for others to follow and do 
> the same.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2141) beam_PerformanceTests_JDBC have not passed in weeks

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023572#comment-16023572
 ] 

Stephen Sisk commented on BEAM-2141:


I've been watching the behavior of kubernetes, and it changes the pod IPs 
fairly frequently (every time the VM goes down for maintenance, which can be 
weekly.) 

I'm going to investigate how long it's going to take me to get perfkit 
benchmarker reading IP addresses from the kubernetes cluster (I'm hoping it's 
not that long) and just do that work - that way we don't have to worry about 
the pod IP changing, breaking the tests, and then doing a manual fix. 

> beam_PerformanceTests_JDBC have not passed in weeks
> ---
>
> Key: BEAM-2141
> URL: https://issues.apache.org/jira/browse/BEAM-2141
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Daniel Halperin
>Assignee: Stephen Sisk
> Fix For: Not applicable
>
>
> https://builds.apache.org/job/beam_PerformanceTests_JDBC/
> Disabling them, as no one seems to be maintaining them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2141) beam_PerformanceTests_JDBC have not passed in weeks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023569#comment-16023569
 ] 

ASF GitHub Bot commented on BEAM-2141:
--

Github user ssisk closed the pull request at:

https://github.com/apache/beam/pull/3213


> beam_PerformanceTests_JDBC have not passed in weeks
> ---
>
> Key: BEAM-2141
> URL: https://issues.apache.org/jira/browse/BEAM-2141
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Daniel Halperin
>Assignee: Stephen Sisk
> Fix For: Not applicable
>
>
> https://builds.apache.org/job/beam_PerformanceTests_JDBC/
> Disabling them, as no one seems to be maintaining them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3213: [BEAM-2141] Fix postgres address in jdbc jenkins jd...

[ssisk]

Github user ssisk closed the pull request at:

https://github.com/apache/beam/pull/3213


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3204

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #2162

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3905

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2225) Add test cases for large keys to GroupByKeyTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023541#comment-16023541
 ] 

ASF GitHub Bot commented on BEAM-2225:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2979


> Add test cases for large keys to GroupByKeyTest
> ---
>
> Key: BEAM-2225
> URL: https://issues.apache.org/jira/browse/BEAM-2225
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Daniel Mills
>Assignee: Daniel Mills
>Priority: Trivial
>
> We should have @ValidatesRunner test cases for large keys (perhaps in a range 
> from 10KB to 10MB)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Adds large key tests to GroupByKeyTest

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 6dd5585b3 -> 6418bcfcb


Adds large key tests to GroupByKeyTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6a792f32
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6a792f32
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6a792f32

Branch: refs/heads/master
Commit: 6a792f3251904f191ff27e58bb9bb78b9c30fdd9
Parents: 6dd5585
Author: Daniel Mills 
Authored: Mon May 8 16:45:44 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 24 12:46:07 2017 -0700

--
 runners/direct-java/pom.xml |  4 +-
 runners/flink/pom.xml   |  2 +
 runners/google-cloud-dataflow-java/pom.xml  |  1 +
 .../org/apache/beam/sdk/testing/LargeKeys.java  | 48 +
 .../beam/sdk/transforms/GroupByKeyTest.java | 74 
 5 files changed, 128 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6a792f32/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index 857dcf4..bec2113 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -144,11 +144,13 @@
 
 
   org.apache.beam.sdk.testing.NeedsRunner
+   
+  
org.apache.beam.sdk.testing.LargeKeys$Above100MB
   none
   true
   
 org.apache.beam:beam-sdks-java-core
-org.apache.beam:beam-runners-core-java
+org.apache.beam:beam-runners-java-core
   
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/6a792f32/runners/flink/pom.xml
--
diff --git a/runners/flink/pom.xml b/runners/flink/pom.xml
index ff73ec1..fb0a67c 100644
--- a/runners/flink/pom.xml
+++ b/runners/flink/pom.xml
@@ -57,6 +57,7 @@
   org.apache.beam.sdk.testing.ValidatesRunner
   
 org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders,
+org.apache.beam.sdk.testing.LargeKeys$Above100MB,
 org.apache.beam.sdk.testing.UsesSplittableParDo,
 org.apache.beam.sdk.testing.UsesCommittedMetrics,
 org.apache.beam.sdk.testing.UsesTestStream
@@ -89,6 +90,7 @@
   org.apache.beam.sdk.testing.ValidatesRunner
   
 org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders,
+org.apache.beam.sdk.testing.LargeKeys$Above100MB,
 org.apache.beam.sdk.testing.UsesSetState,
 org.apache.beam.sdk.testing.UsesMapState,
 org.apache.beam.sdk.testing.UsesCommittedMetrics,

http://git-wip-us.apache.org/repos/asf/beam/blob/6a792f32/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 895a8e6..16e1266 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -217,6 +217,7 @@
 validates-runner-tests
 
   
+org.apache.beam.sdk.testing.LargeKeys$Above10MB,
 org.apache.beam.sdk.testing.UsesDistributionMetrics,
 org.apache.beam.sdk.testing.UsesGaugeMetrics,
 org.apache.beam.sdk.testing.UsesSetState,

http://git-wip-us.apache.org/repos/asf/beam/blob/6a792f32/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/LargeKeys.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/LargeKeys.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/LargeKeys.java
new file mode 100644
index 000..384b298
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/LargeKeys.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is 

[2/2] beam git commit: This closes #2979: Adds large key tests to GroupByKeyTest

[kenn]

This closes #2979: Adds large key tests to GroupByKeyTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6418bcfc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6418bcfc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6418bcfc

Branch: refs/heads/master
Commit: 6418bcfcb7793ebd42213c03ab7ad65944960ab9
Parents: 6dd5585 6a792f3
Author: Kenneth Knowles 
Authored: Wed May 24 12:46:17 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 24 12:46:17 2017 -0700

--
 runners/direct-java/pom.xml |  4 +-
 runners/flink/pom.xml   |  2 +
 runners/google-cloud-dataflow-java/pom.xml  |  1 +
 .../org/apache/beam/sdk/testing/LargeKeys.java  | 48 +
 .../beam/sdk/transforms/GroupByKeyTest.java | 74 
 5 files changed, 128 insertions(+), 1 deletion(-)
--

[jira] [Resolved] (BEAM-163) When a PCollection (via Window.into) has smaller allowedLateness than upstream, might lose data

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-163.
--
   Resolution: Fixed
 Assignee: Borisa Zivkovic
Fix Version/s: 2.1.0

> When a PCollection (via Window.into) has smaller allowedLateness than 
> upstream, might lose data
> ---
>
> Key: BEAM-163
> URL: https://issues.apache.org/jira/browse/BEAM-163
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Borisa Zivkovic
>  Labels: windowing
> Fix For: 2.1.0
>
>
> In this situation, data may be dropped downstream even though it was allowed 
> upstream. It is probably quite rare that this is desirable.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2308

[Apache Jenkins Server]

See 


Changes:

[klk] Do not ever shrink allowed lateness

--
[...truncated 506.52 KB...]
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-35.0.2.zip
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/pyparsing-2.2.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/pyparsing-2.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging 
appdirs pyparsing
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging 
appdirs pyparsing
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/pyparsing-2.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging 
appdirs pyparsing
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-35.0.2.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2161

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Flink #2920

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #49

[Apache Jenkins Server]

See 


--
[...truncated 1.99 MB...]
2017-05-24T18:37:32.775 [INFO] Excluding it.unimi.dsi:fastutil:jar:6.5.7 from 
the shaded jar.
2017-05-24T18:37:32.775 [INFO] Excluding org.apache.thrift:libthrift:jar:0.9.2 
from the shaded jar.
2017-05-24T18:37:32.775 [INFO] Excluding io.netty:netty-all:jar:4.0.39.Final 
from the shaded jar.
2017-05-24T18:37:32.775 [INFO] Excluding org.ow2.asm:asm:jar:5.0.3 from the 
shaded jar.
2017-05-24T18:37:32.775 [INFO] Excluding commons-io:commons-io:jar:2.4 from the 
shaded jar.
2017-05-24T18:37:32.775 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.1.0-SNAPSHOT from the shaded jar.
2017-05-24T18:37:32.775 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-05-24T18:37:32.775 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-05-24T18:37:32.775 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2017-05-24T18:37:32.775 [INFO] Excluding 
org.apache.logging.log4j:log4j-api:jar:2.6.2 from the shaded jar.
2017-05-24T18:37:35.450 [INFO] Replacing original artifact with shaded artifact.
2017-05-24T18:37:35.450 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded.jar
2017-05-24T18:37:35.450 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-05-24T18:37:35.450 [INFO] Replacing 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-tests.jar
 with 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-shaded-tests.jar
2017-05-24T18:37:35.969 [INFO] 
2017-05-24T18:37:35.969 [INFO] --- maven-javadoc-plugin:2.10.4:jar 
(attach-javadocs) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-24T18:37:36.143 [INFO] 
2017-05-24T18:37:36.143 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-24T18:37:36.150 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-sources.jar
2017-05-24T18:37:36.330 [INFO] 
2017-05-24T18:37:36.330 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-24T18:37:36.337 [INFO] Building jar: 
F:\jenkins\jenkins-slave\workspace\beam_PostCommit_Java_MavenInstall_Windows\sdks\java\io\hadoop\jdk1.8-tests\target\beam-sdks-java-io-hadoop-jdk1.8-tests-2.1.0-SNAPSHOT-test-sources.jar
2017-05-24T18:37:36.528 [INFO] 
2017-05-24T18:37:36.528 [INFO] --- maven-dependency-plugin:3.0.0:analyze-only 
(default) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-24T18:37:37.191 [INFO] No dependency problems found
2017-05-24T18:37:38.282 [INFO] 
2017-05-24T18:37:38.282 [INFO] --- apache-rat-plugin:0.12:check (default) @ 
beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-05-24T18:37:38.301 [INFO] Enabled default license matchers.
2017-05-24T18:37:38.301 [INFO] Will parse SCM ignores for exclusions...
2017-05-24T18:37:38.301 [INFO] Finished adding exclusions from SCM ignore files.
2017-05-24T18:37:38.301 [INFO] 61 implicit excludes (use -debug for more 
details).
2017-05-24T18:37:38.302 [INFO] Exclude: **/target/**/*
2017-05-24T18:37:38.302 [INFO] Exclude: **/dependency-reduced-pom.xml
2017-05-24T18:37:38.302 [INFO] Exclude: **/hs_err_pid*.log
2017-05-24T18:37:38.302 [INFO] Exclude: .github/**/*
2017-05-24T18:37:38.302 [INFO] Exclude: **/*.iml
2017-05-24T18:37:38.302 [INFO] Exclude: **/.idea/**/*
2017-05-24T18:37:38.302 [INFO] Exclude: **/*.egg-info/**/*
2017-05-24T18:37:38.302 [INFO] Exclude: **/package-list
2017-05-24T18:37:38.302 [INFO] Exclude: **/user.avsc
2017-05-24T18:37:38.302 [INFO] Exclude: **/test/resources/**/*.txt
2017-05-24T18:37:38.302 [INFO] Exclude: **/test/**/.placeholder
2017-05-24T18:37:38.302 [INFO] Exclude: .repository/**/*
2017-05-24T18:37:38.302 [INFO] Exclude: **/nose-*.egg/**/*
2017-05-24T18:37:38.302 [INFO] Exclude: **/.eggs/**/*
2017-05-24T18:37:38.302 [INFO] Exclude: **/.tox/**/*
2017-05-24T18:37:38.302 [INFO] Exclude: **/.checkstyle
2017-05-24T18:37:38.302 [INFO] Exclude: **/.classpath
2017-05-24T18:37:38.302 [INFO] Exclude: **/.factorypath
2017-05-24T18:37:38.302 [INFO] Exclude: **/.project
2017-05-24T18:37:38.303 [INFO] Exclude: **/.settings/**/*

[GitHub] beam pull request #3163: [BEAM-163] Warn the user when a PCollection (via Wi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3163


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3163: [BEAM-163] Warn the user when a PCollection (via Window.into) has smaller allowe...

[kenn]

This closes #3163: [BEAM-163] Warn the user when a PCollection (via 
Window.into) has smaller allowe...

  Do not ever shrink allowed lateness


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6dd5585b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6dd5585b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6dd5585b

Branch: refs/heads/master
Commit: 6dd5585b3826b879b0c4a23d728b9cff2c2c1a70
Parents: 924dd63 7fb2938
Author: Kenneth Knowles 
Authored: Wed May 24 11:16:05 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed May 24 11:16:05 2017 -0700

--
 .../beam/sdk/transforms/windowing/Window.java   |  6 +++-
 .../sdk/transforms/windowing/WindowTest.java| 36 
 2 files changed, 41 insertions(+), 1 deletion(-)
--

[1/2] beam git commit: Do not ever shrink allowed lateness

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 924dd6335 -> 6dd5585b3


Do not ever shrink allowed lateness

This does not seem to have any use except to cause data loss.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7fb2938e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7fb2938e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7fb2938e

Branch: refs/heads/master
Commit: 7fb2938e66e48160e9df878ccf9d0c2a67790151
Parents: 924dd63
Author: Borisa Zivkovic 
Authored: Tue May 16 14:16:07 2017 +0100
Committer: Kenneth Knowles 
Committed: Wed May 24 11:15:39 2017 -0700

--
 .../beam/sdk/transforms/windowing/Window.java   |  6 +++-
 .../sdk/transforms/windowing/WindowTest.java| 36 
 2 files changed, 41 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7fb2938e/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
index dc4863b..105ebfb 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
@@ -19,6 +19,7 @@ package org.apache.beam.sdk.transforms.windowing;
 
 import com.google.auto.value.AutoValue;
 import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Ordering;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
@@ -141,6 +142,7 @@ import org.joda.time.Duration;
  */
 @AutoValue
 public abstract class Window extends PTransform  {
+
   /**
* Specifies the conditions under which a final pane will be created when a 
window is permanently
* closed.
@@ -313,7 +315,8 @@ public abstract class Window extends 
PTransform

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #2919

[Apache Jenkins Server]

See 


--
[...truncated 319.93 KB...]
2017-05-24T18:19:48.365 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.8.8/jackson-core-2.8.8.pom
 (6 KB at 77.8 KB/sec)
2017-05-24T18:19:48.368 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/jackson-parent/2.8/jackson-parent-2.8.pom
2017-05-24T18:19:48.421 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/jackson-parent/2.8/jackson-parent-2.8.pom
 (8 KB at 146.9 KB/sec)
2017-05-24T18:19:48.423 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/oss-parent/27/oss-parent-27.pom
2017-05-24T18:19:48.480 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/oss-parent/27/oss-parent-27.pom
 (20 KB at 336.5 KB/sec)
2017-05-24T18:19:48.485 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.8.8/jackson-annotations-2.8.8.pom
2017-05-24T18:19:48.548 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.8.8/jackson-annotations-2.8.8.pom
 (2 KB at 28.6 KB/sec)
2017-05-24T18:19:48.554 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.8.8/jackson-databind-2.8.8.pom
2017-05-24T18:19:48.606 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.8.8/jackson-databind-2.8.8.pom
 (6 KB at 100.0 KB/sec)
2017-05-24T18:19:48.618 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.14/slf4j-api-1.7.14.pom
2017-05-24T18:19:48.672 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.14/slf4j-api-1.7.14.pom
 (3 KB at 50.2 KB/sec)
2017-05-24T18:19:48.674 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.14/slf4j-parent-1.7.14.pom
2017-05-24T18:19:48.729 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.14/slf4j-parent-1.7.14.pom
 (14 KB at 239.0 KB/sec)
2017-05-24T18:19:48.733 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.pom
2017-05-24T18:19:48.787 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.pom
 (12 KB at 213.1 KB/sec)
2017-05-24T18:19:48.789 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy-parent/1.6.8/byte-buddy-parent-1.6.8.pom
2017-05-24T18:19:48.849 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy-parent/1.6.8/byte-buddy-parent-1.6.8.pom
 (26 KB at 420.1 KB/sec)
2017-05-24T18:19:48.853 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro/1.8.1/avro-1.8.1.pom
2017-05-24T18:19:48.909 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro/1.8.1/avro-1.8.1.pom 
(7 KB at 123.6 KB/sec)
2017-05-24T18:19:48.929 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.pom
2017-05-24T18:19:48.989 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.pom
 (2 KB at 20.8 KB/sec)
2017-05-24T18:19:48.991 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.pom
2017-05-24T18:19:49.053 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.pom
 (2 KB at 23.7 KB/sec)
2017-05-24T18:19:49.055 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.pom
2017-05-24T18:19:49.108 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.pom
 (6 KB at 97.1 KB/sec)
2017-05-24T18:19:49.129 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/thoughtworks/paranamer/paranamer-parent/2.7/paranamer-parent-2.7.pom
2017-05-24T18:19:49.197 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/thoughtworks/paranamer/paranamer-parent/2.7/paranamer-parent-2.7.pom
 (12 KB at 165.2 KB/sec)
2017-05-24T18:19:49.201 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.4-M3/snappy-java-1.1.4-M3.pom
2017-05-24T18:19:49.254 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.4-M3/snappy-java-1.1.4-M3.pom
 (4 KB at 60.6 KB/sec)
2017-05-24T18:19:49.256 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.pom
2017-05-24T18:19:49.315 [INFO] Downloaded: 

[jira] [Created] (BEAM-2359) SparkTimerInternals inputWatermarkTime does not get updated in cluster mode

[Aviem Zur (JIRA)]

Aviem Zur created BEAM-2359:
---

 Summary: SparkTimerInternals inputWatermarkTime does not get 
updated in cluster mode
 Key: BEAM-2359
 URL: https://issues.apache.org/jira/browse/BEAM-2359
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Aviem Zur
Assignee: Amit Sela


{{SparkTimerInternals#inputWatermarkTime}} does not get updated in cluster mode.

This causes windows to not get closed and state to increase forever in memory 
and processing time to increase leading to eventual application crash (also, 
triggers based on the watermark do not fire).

The root cause is 
a call from within the {{updateStateByKey}} operation in 
[SparkGroupAlsoByWindowViaWindowSet|https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java#L241-L242]
 which tries to access a static reference to a {{GlobalWatermarkHolder}} 
broadcast variable, however, in cluster mode this static reference would be a 
different one in the executor's JVM and is null (this works in local mode since 
the executor and driver are on the same JVM).

The fix is not trivial since even if we use the broadcast correctly, broadcast 
variables can't be used in this case (from within {{updateStateByKey}}) since  
{{updateStateByKey}} is a {{DStream}} operator and not an {{RDD}} operator so 
it will not be updated every micro-batch but rather will retain the same 
initial value.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2347) Elasticsearch IO - test ES 5?

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023281#comment-16023281
 ] 

Stephen Sisk commented on BEAM-2347:


thanks for clarifying!

I converted this to a sub-task of that issue so that we can track it correctly 
- if you don't think this is a useful issue, I'm fine with you closing it as 
well.

> Elasticsearch IO - test ES 5? 
> --
>
> Key: BEAM-2347
> URL: https://issues.apache.org/jira/browse/BEAM-2347
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Etienne Chauchot
>
> Currently, the ES IO transform uses the v5 REST API client, but the 
> integration tests only work against ES 2x
> I'm guessing this is because the ES 5x java client only supports java 8 - 
> https://discuss.elastic.co/t/elasticsearch-5-0-java-api-5-0-on-jdk-7/68222
> I don't know ES particularly well, but I'm guessing it'd be nice to make the 
> ITs work against ES 5 as well as ES 2. It seems like there are two options 
> for this: 
> 1. Make the ITs java-8 only (as is done in HadoopInputFormat ITs)
> 2. Switch the ITs to use the ES5 REST API client, which supports java 7.
> [~echauchot] am I missing anything here?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2347) Elasticsearch IO - test ES 5?

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk updated BEAM-2347:
---
Issue Type: Sub-task  (was: Bug)
Parent: BEAM-1637

> Elasticsearch IO - test ES 5? 
> --
>
> Key: BEAM-2347
> URL: https://issues.apache.org/jira/browse/BEAM-2347
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Etienne Chauchot
>
> Currently, the ES IO transform uses the v5 REST API client, but the 
> integration tests only work against ES 2x
> I'm guessing this is because the ES 5x java client only supports java 8 - 
> https://discuss.elastic.co/t/elasticsearch-5-0-java-api-5-0-on-jdk-7/68222
> I don't know ES particularly well, but I'm guessing it'd be nice to make the 
> ITs work against ES 5 as well as ES 2. It seems like there are two options 
> for this: 
> 1. Make the ITs java-8 only (as is done in HadoopInputFormat ITs)
> 2. Switch the ITs to use the ES5 REST API client, which supports java 7.
> [~echauchot] am I missing anything here?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2328) Introduce Apache Tika Input component

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk updated BEAM-2328:
---
Component/s: sdk-java-extensions

> Introduce Apache Tika Input component
> -
>
> Key: BEAM-2328
> URL: https://issues.apache.org/jira/browse/BEAM-2328
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas, sdk-java-extensions
>Reporter: Sergey Beryozkin
>Assignee: Sergey Beryozkin
> Fix For: 2.1.0
>
>
> Apache Tika is a popular project that offers an extensive support for parsing 
> the variety of file formats. It is used in many projects including Lucene and 
> Elastic Search. 
> Supporting a Tika Input (Read) at the Beam level would be of major interest 
> to many users.
> PR is to follow



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2357) Create HCatalogIO (Hive)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023208#comment-16023208
 ] 

ASF GitHub Bot commented on BEAM-2357:
--

GitHub user seshadri-cr opened a pull request:

https://github.com/apache/beam/pull/3220

[BEAM-2357]HiveIO implementation using native API - HCatalogReaderWriter

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/seshadri-cr/beam hiveio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3220.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3220


commit 50b676b9a3595b584206d449f6ce2dd83c264b2f
Author: Seshadri Chakkravarthy 
Date:   2017-05-18T19:07:01Z

HiveIO sink

commit 0008f772a989c8cd817a99987a145fbf2f7fc795
Author: Seshadri Chakkravarthy 
Date:   2017-05-18T19:07:01Z

HiveIO sink

commit 027fc67be208fac5898bb55d8702c3eb404fab79
Author: Seshadri Chakkravarthy 
Date:   2017-05-18T19:12:45Z

Merge branch 'hiveio' of https://github.com/seshadri-cr/beam into hiveio

# Conflicts:
#   sdks/java/io/hive/src/main/java/org/apache/beam/sdk/io/hive/HiveIO.java

commit 2c8d2622fcd9a26658dafd9698602858955872c9
Author: Seshadri Chakkravarthy 
Date:   2017-05-18T19:12:45Z

Merge branch 'hiveio' of https://github.com/seshadri-cr/beam into hiveio

# Conflicts:
#   sdks/java/io/hive/src/main/java/org/apache/beam/sdk/io/hive/HiveIO.java

commit 257d2243eb4e84c5c4a9fec349c39a7b7c293398
Author: Seshadri Chakkravarthy 
Date:   2017-05-18T19:14:57Z

Merge branch 'hiveio' of https://github.com/seshadri-cr/beam into hiveio

commit b74523c13e03dc70038bc1e348ce270fbb3fd99b
Author: Seshadri Chakkravarthy 
Date:   2017-05-22T22:25:03Z

HiveIO Source




> Create HCatalogIO (Hive)
> 
>
> Key: BEAM-2357
> URL: https://issues.apache.org/jira/browse/BEAM-2357
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Seshadri Raghunathan
>Priority: Minor
>
> Support for reading and writing from Hive's catalog HCatalog. HCatalogIO will 
> allow access to the Hive records so they can be processed with Beam.
> This JIRA is to create a native implementation using the HCatalog 
> ReaderWriter API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3220: [BEAM-2357]HiveIO implementation using native API -...

[seshadri-cr]

GitHub user seshadri-cr opened a pull request:

https://github.com/apache/beam/pull/3220

[BEAM-2357]HiveIO implementation using native API - HCatalogReaderWriter

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/seshadri-cr/beam hiveio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3220.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3220


commit 50b676b9a3595b584206d449f6ce2dd83c264b2f
Author: Seshadri Chakkravarthy 
Date:   2017-05-18T19:07:01Z

HiveIO sink

commit 0008f772a989c8cd817a99987a145fbf2f7fc795
Author: Seshadri Chakkravarthy 
Date:   2017-05-18T19:07:01Z

HiveIO sink

commit 027fc67be208fac5898bb55d8702c3eb404fab79
Author: Seshadri Chakkravarthy 
Date:   2017-05-18T19:12:45Z

Merge branch 'hiveio' of https://github.com/seshadri-cr/beam into hiveio

# Conflicts:
#   sdks/java/io/hive/src/main/java/org/apache/beam/sdk/io/hive/HiveIO.java

commit 2c8d2622fcd9a26658dafd9698602858955872c9
Author: Seshadri Chakkravarthy 
Date:   2017-05-18T19:12:45Z

Merge branch 'hiveio' of https://github.com/seshadri-cr/beam into hiveio

# Conflicts:
#   sdks/java/io/hive/src/main/java/org/apache/beam/sdk/io/hive/HiveIO.java

commit 257d2243eb4e84c5c4a9fec349c39a7b7c293398
Author: Seshadri Chakkravarthy 
Date:   2017-05-18T19:14:57Z

Merge branch 'hiveio' of https://github.com/seshadri-cr/beam into hiveio

commit b74523c13e03dc70038bc1e348ce270fbb3fd99b
Author: Seshadri Chakkravarthy 
Date:   2017-05-22T22:25:03Z

HiveIO Source




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2358) "/test-your-pipeline" example code results in an exception

[Nicholas Ursa (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023187#comment-16023187
 ] 

Nicholas Ursa commented on BEAM-2358:
-

@Rule only works on methods and fields, so this means you wouldn't be able to 
create a pipeline in a method, which seems overly restrictive.

{code}
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD, ElementType.METHOD})
public @interface Rule {

}
{code}

The change in code came out of this ticket: 
https://issues.apache.org/jira/browse/BEAM-1205


> "/test-your-pipeline" example code results in an exception
> --
>
> Key: BEAM-2358
> URL: https://issues.apache.org/jira/browse/BEAM-2358
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Nicholas Ursa
>Assignee: Davor Bonaci
>  Labels: documentation, easyfix
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> https://beam.apache.org/documentation/pipelines/test-your-pipeline/ has
> {code}
>  public void testCountWords() throws Exception {
>   Pipeline p = TestPipeline.create();
> {code}
> but this results in 
> {code}
> Exception in thread "main" java.lang.IllegalStateException: Is your 
> TestPipeline declaration missing a @Rule annotation? Usage: @Rule public 
> final transient TestPipeline pipeline = TestPipeline.Create();
>   at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:299)
>   at BasicPipelineTest.run(BasicPipelineTest.java:42)
>   at Main.main(Main.java:25)
> {code}
> In the [github 
> example|https://github.com/apache/beam/blob/master/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java#L56]
>  it's written as:
> {code}
> public TestPipeline p = 
> TestPipeline.create().enableAbandonedNodeEnforcement(false);
> {code}
> I'm using 2.0.0 from the maven repo.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-2358) "/test-your-pipeline" example code results in an exception

[Nicholas Ursa (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023187#comment-16023187
 ] 

Nicholas Ursa edited comment on BEAM-2358 at 5/24/17 4:36 PM:
--

@Rule only works on methods and fields, so this means you wouldn't be able to 
create a pipeline in a local variable, which seems overly restrictive.

{code}
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD, ElementType.METHOD})
public @interface Rule {

}
{code}

The change in code came out of this ticket: 
https://issues.apache.org/jira/browse/BEAM-1205



was (Author: nicku33):
@Rule only works on methods and fields, so this means you wouldn't be able to 
create a pipeline in a method, which seems overly restrictive.

{code}
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.FIELD, ElementType.METHOD})
public @interface Rule {

}
{code}

The change in code came out of this ticket: 
https://issues.apache.org/jira/browse/BEAM-1205


> "/test-your-pipeline" example code results in an exception
> --
>
> Key: BEAM-2358
> URL: https://issues.apache.org/jira/browse/BEAM-2358
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Nicholas Ursa
>Assignee: Davor Bonaci
>  Labels: documentation, easyfix
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> https://beam.apache.org/documentation/pipelines/test-your-pipeline/ has
> {code}
>  public void testCountWords() throws Exception {
>   Pipeline p = TestPipeline.create();
> {code}
> but this results in 
> {code}
> Exception in thread "main" java.lang.IllegalStateException: Is your 
> TestPipeline declaration missing a @Rule annotation? Usage: @Rule public 
> final transient TestPipeline pipeline = TestPipeline.Create();
>   at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:299)
>   at BasicPipelineTest.run(BasicPipelineTest.java:42)
>   at Main.main(Main.java:25)
> {code}
> In the [github 
> example|https://github.com/apache/beam/blob/master/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java#L56]
>  it's written as:
> {code}
> public TestPipeline p = 
> TestPipeline.create().enableAbandonedNodeEnforcement(false);
> {code}
> I'm using 2.0.0 from the maven repo.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2358) "/test-your-pipeline" example code results in an exception

[Nicholas Ursa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicholas Ursa updated BEAM-2358:

Description: 
https://beam.apache.org/documentation/pipelines/test-your-pipeline/ has

{code}
 public void testCountWords() throws Exception {
  Pipeline p = TestPipeline.create();
{code}

but this results in 

{code}
Exception in thread "main" java.lang.IllegalStateException: Is your 
TestPipeline declaration missing a @Rule annotation? Usage: @Rule public final 
transient TestPipeline pipeline = TestPipeline.Create();
at 
org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:299)
at BasicPipelineTest.run(BasicPipelineTest.java:42)
at Main.main(Main.java:25)
{code}

In the [github 
example|https://github.com/apache/beam/blob/master/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java#L56]
 it's written as:



{code}
public TestPipeline p = 
TestPipeline.create().enableAbandonedNodeEnforcement(false);
{code}

I'm using 2.0.0 from the maven repo.


  was:
https://beam.apache.org/documentation/pipelines/test-your-pipeline/ has

{code}
 public void testCountWords() throws Exception {
  Pipeline p = TestPipeline.create();
{code}

but this results in 

{{monospaced}}
Exception in thread "main" java.lang.IllegalStateException: Is your 
TestPipeline declaration missing a @Rule annotation? Usage: @Rule public final 
transient TestPipeline pipeline = TestPipeline.Create();
at 
org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:299)
at BasicPipelineTest.run(BasicPipelineTest.java:42)
at Main.main(Main.java:25)

{{monospaced}}

In the [github 
example|https://github.com/apache/beam/blob/master/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java#L56]
 it's written as:



{code}
public TestPipeline p = 
TestPipeline.create().enableAbandonedNodeEnforcement(false);
{code}

I'm using 2.0.0 from the maven repo.



> "/test-your-pipeline" example code results in an exception
> --
>
> Key: BEAM-2358
> URL: https://issues.apache.org/jira/browse/BEAM-2358
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Nicholas Ursa
>Assignee: Davor Bonaci
>  Labels: documentation, easyfix
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> https://beam.apache.org/documentation/pipelines/test-your-pipeline/ has
> {code}
>  public void testCountWords() throws Exception {
>   Pipeline p = TestPipeline.create();
> {code}
> but this results in 
> {code}
> Exception in thread "main" java.lang.IllegalStateException: Is your 
> TestPipeline declaration missing a @Rule annotation? Usage: @Rule public 
> final transient TestPipeline pipeline = TestPipeline.Create();
>   at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:299)
>   at BasicPipelineTest.run(BasicPipelineTest.java:42)
>   at Main.main(Main.java:25)
> {code}
> In the [github 
> example|https://github.com/apache/beam/blob/master/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java#L56]
>  it's written as:
> {code}
> public TestPipeline p = 
> TestPipeline.create().enableAbandonedNodeEnforcement(false);
> {code}
> I'm using 2.0.0 from the maven repo.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2358) "/test-your-pipeline" example code results in an exception

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023132#comment-16023132
 ] 

Luke Cwik commented on BEAM-2358:
-

It seems as though docs should be telling people to being using TestPipeline as 
a JUnit @Rule as described int he exception.

> "/test-your-pipeline" example code results in an exception
> --
>
> Key: BEAM-2358
> URL: https://issues.apache.org/jira/browse/BEAM-2358
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Nicholas Ursa
>Assignee: Davor Bonaci
>  Labels: documentation, easyfix
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> https://beam.apache.org/documentation/pipelines/test-your-pipeline/ has
> {code}
>  public void testCountWords() throws Exception {
>   Pipeline p = TestPipeline.create();
> {code}
> but this results in 
> {{monospaced}}
> Exception in thread "main" java.lang.IllegalStateException: Is your 
> TestPipeline declaration missing a @Rule annotation? Usage: @Rule public 
> final transient TestPipeline pipeline = TestPipeline.Create();
>   at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:299)
>   at BasicPipelineTest.run(BasicPipelineTest.java:42)
>   at Main.main(Main.java:25)
> {{monospaced}}
> In the [github 
> example|https://github.com/apache/beam/blob/master/examples/java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java#L56]
>  it's written as:
> {code}
> public TestPipeline p = 
> TestPipeline.create().enableAbandonedNodeEnforcement(false);
> {code}
> I'm using 2.0.0 from the maven repo.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)