Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3444

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3616

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2645) Implement DisplayData translation to/from protos

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2645:
-

 Summary: Implement DisplayData translation to/from protos
 Key: BEAM-2645
 URL: https://issues.apache.org/jira/browse/BEAM-2645
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-core
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4414

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #2024

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2667

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3443

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #120

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-2642] Update Google Auth to 0.7.1

[lcwik] Increase the gRPC message size to max value

[altay] [BEAM-2636] Make sure we only override the correct class

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 2e51bde5bd3fc2589b0e04f2ced8bd7c24d1046a (origin/master)
Commit message: "This closes #3595"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2e51bde5bd3fc2589b0e04f2ced8bd7c24d1046a
 > git rev-list a6f460fe3b760aafbc748ae18956f0f2c1fedfad # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3525465491584401032.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins357130076759171685.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7432431993199167983.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5108974089092526254.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3615

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2575) ApexRunner doesn't emit watermarks for additional outputs

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise resolved BEAM-2575.

   Resolution: Fixed
Fix Version/s: 2.2.0

> ApexRunner doesn't emit watermarks for additional outputs 
> --
>
> Key: BEAM-2575
> URL: https://issues.apache.org/jira/browse/BEAM-2575
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Thomas Weise
> Fix For: 2.2.0
>
>
> https://lists.apache.org/thread.html/51113a207f96d0522fb81adb65e35e134a0c52cf4bbe1cfc46508d83@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3599: ReduceFnRunner: test when watermark leapfrogs EOW a...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3599

ReduceFnRunner: test when watermark leapfrogs EOW and GC

This is known to fail in older versions; forward porting regression test.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam ReduceFnRunner-wut

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3599.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3599


commit 4345b7ab5c1d28a40d57cd9978b73051975b0827
Author: Kenneth Knowles 
Date:   2017-07-19T22:27:20Z

ReduceFnRunner: test when watermark leapfrogs EOW and GC

This is known to fail in older versions; forward porting regression test.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4413

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4412

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #2023

[Apache Jenkins Server]

See 


Changes:

[altay] [BEAM-2636] Make sure we only override the correct class

--
[...truncated 471.31 KB...]
2017-07-19T21:45:30.607 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.pom
2017-07-19T21:45:30.634 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.pom
 (2 KB at 59.8 KB/sec)
2017-07-19T21:45:30.636 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.6.1/slf4j-parent-1.6.1.pom
2017-07-19T21:45:30.673 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.6.1/slf4j-parent-1.6.1.pom
 (10 KB at 246.1 KB/sec)
2017-07-19T21:45:30.676 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/log4j/log4j/1.2.16/log4j-1.2.16.pom
2017-07-19T21:45:30.705 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/log4j/log4j/1.2.16/log4j-1.2.16.pom (20 KB 
at 684.9 KB/sec)
2017-07-19T21:45:30.710 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/jline/jline/0.9.94/jline-0.9.94.pom
2017-07-19T21:45:30.738 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/jline/jline/0.9.94/jline-0.9.94.pom (7 KB 
at 221.8 KB/sec)
2017-07-19T21:45:30.741 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.pom
2017-07-19T21:45:30.770 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.pom
 (26 KB at 882.4 KB/sec)
2017-07-19T21:45:30.781 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-framework/2.7.1/curator-framework-2.7.1.pom
2017-07-19T21:45:30.813 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-framework/2.7.1/curator-framework-2.7.1.pom
 (3 KB at 62.2 KB/sec)
2017-07-19T21:45:30.815 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.7.1/apache-curator-2.7.1.pom
2017-07-19T21:45:30.846 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.7.1/apache-curator-2.7.1.pom
 (32 KB at 1002.6 KB/sec)
2017-07-19T21:45:30.852 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.pom
2017-07-19T21:45:30.879 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.pom
 (3 KB at 81.7 KB/sec)
2017-07-19T21:45:30.884 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.42/jsch-0.1.42.pom
2017-07-19T21:45:30.911 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.42/jsch-0.1.42.pom 
(967 B at 35.0 KB/sec)
2017-07-19T21:45:30.917 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-recipes/2.7.1/curator-recipes-2.7.1.pom
2017-07-19T21:45:30.944 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-recipes/2.7.1/curator-recipes-2.7.1.pom
 (3 KB at 85.5 KB/sec)
2017-07-19T21:45:30.948 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.pom
2017-07-19T21:45:30.976 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.pom
 (4 KB at 142.6 KB/sec)
2017-07-19T21:45:30.978 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/htrace/htrace/3.1.0-incubating/htrace-3.1.0-incubating.pom
2017-07-19T21:45:31.013 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/htrace/htrace/3.1.0-incubating/htrace-3.1.0-incubating.pom
 (12 KB at 324.2 KB/sec)
2017-07-19T21:45:31.015 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/12/apache-12.pom
2017-07-19T21:45:31.062 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/12/apache-12.pom (16 KB 
at 322.5 KB/sec)
2017-07-19T21:45:31.065 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/datatorrent/netlet/1.2.1/netlet-1.2.1.pom
2017-07-19T21:45:31.094 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/datatorrent/netlet/1.2.1/netlet-1.2.1.pom
 (19 KB at 629.8 KB/sec)
2017-07-19T21:45:31.099 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.pom
2017-07-19T21:45:31.125 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.pom
 (2 KB at 60.1 KB/sec)
2017-07-19T21:45:31.128 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.pom
2017-07-19T21:45:31.163 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.pom
 (7 KB at 173.5 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4411

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2636) user_score on DataflowRunner is broken

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-2636.
---
   Resolution: Fixed
Fix Version/s: 2.2.0

> user_score on DataflowRunner is broken
> --
>
> Key: BEAM-2636
> URL: https://issues.apache.org/jira/browse/BEAM-2636
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.1.0
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
> Fix For: 2.2.0
>
>
> UserScore has a custom transform named {{WriteToBigQuery}}, dataflow runner 
> has a special code handling transforms with that name, this will break for 
> all user transforms that has this name.
> We can either:
> - Handle this correctly
> - Or document this as a reserved keyword and change the example.
> cc: [~chamikara]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3442

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2638) Built-in I/O Transforms page needs to be updated with the new IOs

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-2638:
---
Issue Type: Improvement  (was: Bug)

> Built-in I/O Transforms page needs to be updated with the new IOs
> -
>
> Key: BEAM-2638
> URL: https://issues.apache.org/jira/browse/BEAM-2638
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Minor
>
> The page lacks elasticsearch and mqtt that are in master.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2638) Add missing Elasticsearch and MQTT to the Built-in I/O Transforms page

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-2638.

   Resolution: Fixed
Fix Version/s: Not applicable

> Add missing Elasticsearch and MQTT to the Built-in I/O Transforms page
> --
>
> Key: BEAM-2638
> URL: https://issues.apache.org/jira/browse/BEAM-2638
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Minor
> Fix For: Not applicable
>
>
> The page lacks elasticsearch and mqtt that are in master.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2638) Add missing Elasticsearch and MQTT IOs to the Built-in I/O Transforms page

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-2638:
---
Summary: Add missing Elasticsearch and MQTT IOs to the Built-in I/O 
Transforms page  (was: Add missing Elasticsearch and MQTT to the Built-in I/O 
Transforms page)

> Add missing Elasticsearch and MQTT IOs to the Built-in I/O Transforms page
> --
>
> Key: BEAM-2638
> URL: https://issues.apache.org/jira/browse/BEAM-2638
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Minor
> Fix For: Not applicable
>
>
> The page lacks elasticsearch and mqtt that are in master.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2638) Add missing Elasticsearch and MQTT to the Built-in I/O Transforms page

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-2638:
---
Summary: Add missing Elasticsearch and MQTT to the Built-in I/O Transforms 
page  (was: Built-in I/O Transforms page needs to be updated with the new IOs)

> Add missing Elasticsearch and MQTT to the Built-in I/O Transforms page
> --
>
> Key: BEAM-2638
> URL: https://issues.apache.org/jira/browse/BEAM-2638
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Minor
> Fix For: Not applicable
>
>
> The page lacks elasticsearch and mqtt that are in master.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2636) user_score on DataflowRunner is broken

[Sourabh Bajaj (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093792#comment-16093792
 ] 

Sourabh Bajaj commented on BEAM-2636:
-

This can be resolved now.

> user_score on DataflowRunner is broken
> --
>
> Key: BEAM-2636
> URL: https://issues.apache.org/jira/browse/BEAM-2636
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.1.0
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>
> UserScore has a custom transform named {{WriteToBigQuery}}, dataflow runner 
> has a special code handling transforms with that name, this will break for 
> all user transforms that has this name.
> We can either:
> - Handle this correctly
> - Or document this as a reserved keyword and change the example.
> cc: [~chamikara]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2638) Built-in I/O Transforms page needs to be updated with the new IOs

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-2638:
---
Priority: Minor  (was: Major)

> Built-in I/O Transforms page needs to be updated with the new IOs
> -
>
> Key: BEAM-2638
> URL: https://issues.apache.org/jira/browse/BEAM-2638
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Minor
>
> The page lacks elasticsearch and mqtt that are in master.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit e21c3d4eddfc1006566fcb237937e45d71ce8402
Author: Mergebot 
AuthorDate: Wed Jul 19 21:11:34 2017 +

Prepare repository for deployment.
---
 content/documentation/io/built-in/index.html | 32 +---
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/content/documentation/io/built-in/index.html 
b/content/documentation/io/built-in/index.html
index 25368d0..8902b12 100644
--- a/content/documentation/io/built-in/index.html
+++ b/content/documentation/io/built-in/index.html
@@ -155,28 +155,30 @@
 
   Java
   
-https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java;>AvroIO
 https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system;>Apache
 Hadoop File System
+https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java;>AvroIO
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java;>TextIO
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/;>XML
   
   
+https://github.com/apache/beam/tree/master/sdks/java/io/kinesis;>Amazon 
Kinesis
 https://github.com/apache/beam/tree/master/sdks/java/io/amqp;>AMQP
-https://github.com/apache/beam/tree/master/sdks/java/io/jms;>JMS
 https://github.com/apache/beam/tree/master/sdks/java/io/kafka;>Apache 
Kafka
-https://github.com/apache/beam/tree/master/sdks/java/io/kinesis;>Amazon 
Kinesis
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io;>Google
 Cloud PubSub
+https://github.com/apache/beam/tree/master/sdks/java/io/jms;>JMS
+https://github.com/apache/beam/tree/master/sdks/java/io/mqtt;>MQTT
   
   
 https://github.com/apache/beam/tree/master/sdks/java/io/cassandra;>Apache 
Cassandra
 Apache Hadoop 
InputFormat
 https://github.com/apache/beam/tree/master/sdks/java/io/hbase;>Apache 
HBase
 https://github.com/apache/beam/tree/master/sdks/java/io/hcatalog;>Apache 
Hive (HCatalog)
-https://github.com/apache/beam/tree/master/sdks/java/io/mongodb;>MongoDB
-https://github.com/apache/beam/tree/master/sdks/java/io/jdbc;>JDBC
+https://github.com/apache/beam/tree/master/sdks/java/io/elasticsearch;>Elasticsearch
 (v2.x)
 https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery;>Google
 BigQuery
 https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable;>Google
 Cloud Bigtable
 https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore;>Google
 Cloud Datastore
+https://github.com/apache/beam/tree/master/sdks/java/io/jdbc;>JDBC
+https://github.com/apache/beam/tree/master/sdks/java/io/mongodb;>MongoDB
   
 
 
@@ -228,6 +230,14 @@
 https://issues.apache.org/jira/browse/BEAM-1893;>BEAM-1893
   
   
+Elasticsearch (v5.x)Java
+https://issues.apache.org/jira/browse/BEAM-1637;>BEAM-1637
+  
+  
+Google Cloud SpannerJava
+https://issues.apache.org/jira/browse/BEAM-1542;>BEAM-1542
+  
+  
 InfluxDBJava
 https://issues.apache.org/jira/browse/BEAM-2546;>BEAM-2546
   
@@ -244,14 +254,14 @@
 https://issues.apache.org/jira/browse/BEAM-1857;>BEAM-1857
   
   
-RedisJava
-https://issues.apache.org/jira/browse/BEAM-1017;>BEAM-1017
-  
-  
 RabbitMQJava
 https://issues.apache.org/jira/browse/BEAM-1240;>BEAM-1240
   
   
+RedisJava
+https://issues.apache.org/jira/browse/BEAM-1017;>BEAM-1017
+  
+  
 RestIOJava
 https://issues.apache.org/jira/browse/BEAM-1946;>BEAM-1946
   
@@ -259,10 +269,6 @@
 TikaIOJava
 https://issues.apache.org/jira/browse/BEAM-2328;>BEAM-2328
   
-  
-Cloud SpannerJava
-https://issues.apache.org/jira/browse/BEAM-1542;>BEAM-1542
-  
 
 
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch asf-site updated (e65a405 -> e21c3d4)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from e65a405  Prepare repository for deployment.
 add 232dfe3  [BEAM-2638] Add Elasticsearch (v2 and upcoming v5) and MQTT 
to Built-in I/O Transforms page
 add 4665b04  This closes #276
 new e21c3d4  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/io/built-in/index.html | 32 +---
 src/documentation/io/built-in.md | 32 +---
 2 files changed, 38 insertions(+), 26 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2665

[Apache Jenkins Server]

See 

[beam-site] 01/02: [BEAM-2638] Add Elasticsearch (v2 and upcoming v5) and MQTT to Built-in I/O Transforms page

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 232dfe33813e30320cd46c87bb7d74b8242d502a
Author: echauchot 
AuthorDate: Wed Jul 19 11:44:16 2017 +0200

[BEAM-2638] Add Elasticsearch (v2 and upcoming v5) and MQTT to Built-in I/O 
Transforms page
---
 src/documentation/io/built-in.md | 32 +++-
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/src/documentation/io/built-in.md b/src/documentation/io/built-in.md
index 2db36e5..778209f 100644
--- a/src/documentation/io/built-in.md
+++ b/src/documentation/io/built-in.md
@@ -23,28 +23,30 @@ Consult the [Programming Guide I/O section]({{site.baseurl 
}}/documentation/prog
 
   Java
   
-https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java;>AvroIO
 https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system;>Apache
 Hadoop File System
+https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java;>AvroIO
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java;>TextIO
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/;>XML
   
   
+https://github.com/apache/beam/tree/master/sdks/java/io/kinesis;>Amazon 
Kinesis
 https://github.com/apache/beam/tree/master/sdks/java/io/amqp;>AMQP
-https://github.com/apache/beam/tree/master/sdks/java/io/jms;>JMS
 https://github.com/apache/beam/tree/master/sdks/java/io/kafka;>Apache 
Kafka
-https://github.com/apache/beam/tree/master/sdks/java/io/kinesis;>Amazon 
Kinesis
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io;>Google
 Cloud PubSub
+https://github.com/apache/beam/tree/master/sdks/java/io/jms;>JMS
+https://github.com/apache/beam/tree/master/sdks/java/io/mqtt;>MQTT
   
   
 https://github.com/apache/beam/tree/master/sdks/java/io/cassandra;>Apache 
Cassandra
 Apache 
Hadoop InputFormat
 https://github.com/apache/beam/tree/master/sdks/java/io/hbase;>Apache 
HBase
 https://github.com/apache/beam/tree/master/sdks/java/io/hcatalog;>Apache 
Hive (HCatalog)
-https://github.com/apache/beam/tree/master/sdks/java/io/mongodb;>MongoDB
-https://github.com/apache/beam/tree/master/sdks/java/io/jdbc;>JDBC
+https://github.com/apache/beam/tree/master/sdks/java/io/elasticsearch;>Elasticsearch
 (v2.x)
 https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery;>Google
 BigQuery
 https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable;>Google
 Cloud Bigtable
 https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore;>Google
 Cloud Datastore
+https://github.com/apache/beam/tree/master/sdks/java/io/jdbc;>JDBC
+https://github.com/apache/beam/tree/master/sdks/java/io/mongodb;>MongoDB
   
 
 
@@ -96,6 +98,14 @@ This table contains I/O transforms that are currently 
planned or in-progress. St
 https://issues.apache.org/jira/browse/BEAM-1893;>BEAM-1893
   
   
+Elasticsearch (v5.x)Java
+https://issues.apache.org/jira/browse/BEAM-1637;>BEAM-1637
+  
+  
+Google Cloud SpannerJava
+https://issues.apache.org/jira/browse/BEAM-1542;>BEAM-1542
+  
+  
 InfluxDBJava
 https://issues.apache.org/jira/browse/BEAM-2546;>BEAM-2546
   
@@ -112,14 +122,14 @@ This table contains I/O transforms that are currently 
planned or in-progress. St
 https://issues.apache.org/jira/browse/BEAM-1857;>BEAM-1857
   
   
-RedisJava
-https://issues.apache.org/jira/browse/BEAM-1017;>BEAM-1017
-  
-  
 RabbitMQJava
 https://issues.apache.org/jira/browse/BEAM-1240;>BEAM-1240
   
   
+RedisJava
+https://issues.apache.org/jira/browse/BEAM-1017;>BEAM-1017
+  
+  
 RestIOJava
 https://issues.apache.org/jira/browse/BEAM-1946;>BEAM-1946
   
@@ -127,8 +137,4 @@ This table contains I/O transforms that are currently 
planned or in-progress. St
 TikaIOJava
 https://issues.apache.org/jira/browse/BEAM-2328;>BEAM-2328
   
-  
-Cloud SpannerJava
-https://issues.apache.org/jira/browse/BEAM-1542;>BEAM-1542
-  
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (f6175fa -> 4665b04)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from f6175fa  This closes #274
 add e65a405  Prepare repository for deployment.
 new 232dfe3  [BEAM-2638] Add Elasticsearch (v2 and upcoming v5) and MQTT 
to Built-in I/O Transforms page
 new 4665b04  This closes #276

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/io/io-toc/index.html  |   3 +-
 content/documentation/io/testing/index.html | 113 +++-
 src/documentation/io/built-in.md|  32 
 3 files changed, 130 insertions(+), 18 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 02/02: This closes #276

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 4665b0487f0a29556157f564c1121e6451fe85fe
Merge: e65a405 232dfe3
Author: Mergebot 
AuthorDate: Wed Jul 19 21:09:26 2017 +

This closes #276

 src/documentation/io/built-in.md | 32 +++-
 1 file changed, 19 insertions(+), 13 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[jira] [Commented] (BEAM-2636) user_score on DataflowRunner is broken

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093785#comment-16093785
 ] 

ASF GitHub Bot commented on BEAM-2636:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3595


> user_score on DataflowRunner is broken
> --
>
> Key: BEAM-2636
> URL: https://issues.apache.org/jira/browse/BEAM-2636
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.1.0
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>
> UserScore has a custom transform named {{WriteToBigQuery}}, dataflow runner 
> has a special code handling transforms with that name, this will break for 
> all user transforms that has this name.
> We can either:
> - Handle this correctly
> - Or document this as a reserved keyword and change the example.
> cc: [~chamikara]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: [BEAM-2636] Make sure we only override the correct class

[altay]

Repository: beam
Updated Branches:
  refs/heads/master eb0850ef8 -> 2e51bde5b


[BEAM-2636] Make sure we only override the correct class


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d128c3b3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d128c3b3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d128c3b3

Branch: refs/heads/master
Commit: d128c3b378a58b0c2c31c2d30fd29e211e118324
Parents: eb0850e
Author: Sourabh Bajaj 
Authored: Wed Jul 19 10:08:14 2017 -0700
Committer: Ahmet Altay 
Committed: Wed Jul 19 14:07:54 2017 -0700

--
 sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d128c3b3/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index 89c18d4..aec7d00 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -418,6 +418,9 @@ class DataflowRunner(PipelineRunner):
   PropertyNames.OUTPUT_NAME: PropertyNames.OUT}])
 
   def apply_WriteToBigQuery(self, transform, pcoll):
+# Make sure this is the WriteToBigQuery class that we expected
+if not isinstance(transform, beam.io.WriteToBigQuery):
+  return self.apply_PTransform(transform, pcoll)
 standard_options = pcoll.pipeline._options.view_as(StandardOptions)
 if standard_options.streaming:
   if (transform.write_disposition ==

[GitHub] beam pull request #3595: [BEAM-2636] Make sure we only override the correct ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3595


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3595

[altay]

This closes #3595


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2e51bde5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2e51bde5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2e51bde5

Branch: refs/heads/master
Commit: 2e51bde5bd3fc2589b0e04f2ced8bd7c24d1046a
Parents: eb0850e d128c3b
Author: Ahmet Altay 
Authored: Wed Jul 19 14:08:01 2017 -0700
Committer: Ahmet Altay 
Committed: Wed Jul 19 14:08:01 2017 -0700

--
 sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 3 +++
 1 file changed, 3 insertions(+)
--

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3613

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2664

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3441

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4410

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3440

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3586: Increase the gRPC message size to max value

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3586


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1234) Consider a hint ParDo.withHighFanout()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093728#comment-16093728
 ] 

ASF GitHub Bot commented on BEAM-1234:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3586


> Consider a hint ParDo.withHighFanout()
> --
>
> Key: BEAM-1234
> URL: https://issues.apache.org/jira/browse/BEAM-1234
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Priority: Minor
>
> I'm finding myself again and again suggesting users on StackOverflow to 
> insert fusion breaks after high-fanout ParDo's.
> I think we should just implement this as a hint on ParDo and MapElements 
> transforms, like we have on GroupByKey.fewKeys() or 
> Combine.withHotKeyFanout().
> E.g.: c.apply(ParDo.of(some high-fanout DoFn).withHighFanout()), and a runner 
> that implements fusion could decide to insert a runner-specific fusion break. 
> This somewhat sidesteps the issues in 
> https://issues.apache.org/jira/browse/BEAM-730 and 
> https://lists.apache.org/thread.html/ac34c9ac665a8d9f67b0254015e44c59ea65ecc1360d4014b95d3b2e@%3Cdev.beam.apache.org%3E
>  because every runner can decide how to do the right thing, or is free to 
> ignore the hint.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: Increase the gRPC message size to max value

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 4d1db2265 -> eb0850ef8


Increase the gRPC message size to max value


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b424aa04
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b424aa04
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b424aa04

Branch: refs/heads/master
Commit: b424aa0409b507fe1c0c56a5f652d9be6458de66
Parents: 4d1db22
Author: Vikas Kedigehalli 
Authored: Tue Jul 18 10:06:46 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jul 19 13:17:37 2017 -0700

--
 .../beam/fn/harness/channel/ManagedChannelFactory.java   | 6 ++
 sdks/python/apache_beam/runners/worker/data_plane.py | 8 +++-
 2 files changed, 13 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b424aa04/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/channel/ManagedChannelFactory.java
--
diff --git 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/channel/ManagedChannelFactory.java
 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/channel/ManagedChannelFactory.java
index d26f4a5..3138bab 100644
--- 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/channel/ManagedChannelFactory.java
+++ 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/channel/ManagedChannelFactory.java
@@ -61,6 +61,9 @@ public abstract class ManagedChannelFactory {
   ? EpollDomainSocketChannel.class : EpollSocketChannel.class)
   .eventLoopGroup(new EpollEventLoopGroup())
   .usePlaintext(true)
+  // Set the message size to max value here. The actual size is 
governed by the
+  // buffer size in the layers above.
+  .maxInboundMessageSize(Integer.MAX_VALUE)
   .build();
 }
   }
@@ -74,6 +77,9 @@ public abstract class ManagedChannelFactory {
 public ManagedChannel forDescriptor(ApiServiceDescriptor 
apiServiceDescriptor) {
   return ManagedChannelBuilder.forTarget(apiServiceDescriptor.getUrl())
   .usePlaintext(true)
+  // Set the message size to max value here. The actual size is 
governed by the
+  // buffer size in the layers above.
+  .maxInboundMessageSize(Integer.MAX_VALUE)
   .build();
 }
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/b424aa04/sdks/python/apache_beam/runners/worker/data_plane.py
--
diff --git a/sdks/python/apache_beam/runners/worker/data_plane.py 
b/sdks/python/apache_beam/runners/worker/data_plane.py
index 26f65ee..e713041 100644
--- a/sdks/python/apache_beam/runners/worker/data_plane.py
+++ b/sdks/python/apache_beam/runners/worker/data_plane.py
@@ -269,7 +269,13 @@ class GrpcClientDataChannelFactory(DataChannelFactory):
 url = remote_grpc_port.api_service_descriptor.url
 if url not in self._data_channel_cache:
   logging.info('Creating channel for %s', url)
-  grpc_channel = grpc.insecure_channel(url)
+  grpc_channel = grpc.insecure_channel(
+  url,
+  # Options to have no limits (-1) on the size of the messages
+  # received or sent over the data plane. The actual buffer size is
+  # controlled in a layer above.
+  options=[("grpc.max_receive_message_length", -1),
+   ("grpc.max_send_message_length", -1)])
   self._data_channel_cache[url] = GrpcClientDataChannel(
   beam_fn_api_pb2.BeamFnDataStub(grpc_channel))
 return self._data_channel_cache[url]

[2/2] beam git commit: Increase the gRPC message size to max value

[lcwik]

Increase the gRPC message size to max value

This closes #3586


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/eb0850ef
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/eb0850ef
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/eb0850ef

Branch: refs/heads/master
Commit: eb0850ef889891eac7a2c96d744aa0ef18afb8f8
Parents: 4d1db22 b424aa0
Author: Luke Cwik 
Authored: Wed Jul 19 13:18:09 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jul 19 13:18:09 2017 -0700

--
 .../beam/fn/harness/channel/ManagedChannelFactory.java   | 6 ++
 sdks/python/apache_beam/runners/worker/data_plane.py | 8 +++-
 2 files changed, 13 insertions(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-2642) Upgrade to Google Auth 0.7.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093719#comment-16093719
 ] 

ASF GitHub Bot commented on BEAM-2642:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3596


> Upgrade to Google Auth 0.7.1
> 
>
> Key: BEAM-2642
> URL: https://issues.apache.org/jira/browse/BEAM-2642
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
> Fix For: 2.2.0
>
>
> Looking up application default credentials on a GCE VM can fail due to VM 
> metadata server being unavailable during VM launch. This is a rare event but 
> Google Cloud Dataflow customers hit this rare case one or two times a month 
> due to the sheer number of VMs. GCE attempted to mitigate VM metadata server 
> unavailability but were only able to reduce it be an order of magnitude thus 
> we need support from the client to retry. Additionally, when contacting the 
> GCE VM metadata server, we should be using the fixed IP address avoiding the 
> nameserver lookup (another potential point of failure).
> Problem area in the code:
> https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261
> Note that the code in this library and the Apiary auth support code are very 
> similar. The fix was done within the Apiary auth code (note the use of the 
> static IP address and also the presence of a fixed number of retries):
> https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74
> It turned out that the fixes resulted in zero future customer contacts about 
> this issue.
> Google Auth 0.7.1 was released containing these fixes mentioned in 
> https://github.com/google/google-auth-library-java/issues/109



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2642) Upgrade to Google Auth 0.7.1

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-2642.
-
   Resolution: Fixed
Fix Version/s: 2.2.0

> Upgrade to Google Auth 0.7.1
> 
>
> Key: BEAM-2642
> URL: https://issues.apache.org/jira/browse/BEAM-2642
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
> Fix For: 2.2.0
>
>
> Looking up application default credentials on a GCE VM can fail due to VM 
> metadata server being unavailable during VM launch. This is a rare event but 
> Google Cloud Dataflow customers hit this rare case one or two times a month 
> due to the sheer number of VMs. GCE attempted to mitigate VM metadata server 
> unavailability but were only able to reduce it be an order of magnitude thus 
> we need support from the client to retry. Additionally, when contacting the 
> GCE VM metadata server, we should be using the fixed IP address avoiding the 
> nameserver lookup (another potential point of failure).
> Problem area in the code:
> https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261
> Note that the code in this library and the Apiary auth support code are very 
> similar. The fix was done within the Apiary auth code (note the use of the 
> static IP address and also the presence of a fixed number of retries):
> https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74
> It turned out that the fixes resulted in zero future customer contacts about 
> this issue.
> Google Auth 0.7.1 was released containing these fixes mentioned in 
> https://github.com/google/google-auth-library-java/issues/109



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3596: [BEAM-2642] Update Google Auth to 0.7.1

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3596


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: [BEAM-2642] Update Google Auth to 0.7.1

[lcwik]

[BEAM-2642] Update Google Auth to 0.7.1

This closes #3596


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4d1db226
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4d1db226
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4d1db226

Branch: refs/heads/master
Commit: 4d1db2265298af324372e5212ec06cd10b4f4908
Parents: a6f460f 51427a6
Author: Luke Cwik 
Authored: Wed Jul 19 13:09:13 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jul 19 13:09:13 2017 -0700

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] beam git commit: [BEAM-2642] Update Google Auth to 0.7.1

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master a6f460fe3 -> 4d1db2265


[BEAM-2642] Update Google Auth to 0.7.1

Google auth libraries 0.6.1 and 0.7.1 have the same versions of dependencies.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/51427a6e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/51427a6e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/51427a6e

Branch: refs/heads/master
Commit: 51427a6e3fd9fc3f311ee4df076745db9d67af7a
Parents: a6f460f
Author: Luke Cwik 
Authored: Wed Jul 19 11:36:35 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jul 19 11:36:35 2017 -0700

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/51427a6e/pom.xml
--
diff --git a/pom.xml b/pom.xml
index d27d367..f2d0dde 100644
--- a/pom.xml
+++ b/pom.xml
@@ -119,7 +119,7 @@
 1.0.0-rc2
 1.0-rc2
 1.4.1
-0.6.1
+0.7.1
 1.22.0
 1.4.5
 1.0.2

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2663

[Apache Jenkins Server]

See 

[beam-site] branch asf-site updated (7ccef23 -> e65a405)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 7ccef23  Prepare repository for deployment.
 add 7d3fc98  Port of Google doc
 add b6fdf18  IO Testing, unit tests: update after readthrough
 add cd40290  fixup! IO Testing, unit tests: update after readthrough
 add 970991b  fixup! fixup! IO Testing, unit tests: update after readthrough
 add f6175fa  This closes #274
 new e65a405  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/io/io-toc/index.html  |   3 +-
 content/documentation/io/testing/index.html | 113 +++-
 src/documentation/io/io-toc.md  |   3 +-
 src/documentation/io/testing.md | 101 -
 4 files changed, 213 insertions(+), 7 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit e65a4057c9f5431ef63e1bfc8dde92e51d82
Author: Mergebot 
AuthorDate: Wed Jul 19 19:19:44 2017 +

Prepare repository for deployment.
---
 content/documentation/io/io-toc/index.html  |   3 +-
 content/documentation/io/testing/index.html | 113 +++-
 2 files changed, 111 insertions(+), 5 deletions(-)

diff --git a/content/documentation/io/io-toc/index.html 
b/content/documentation/io/io-toc/index.html
index 1cd94ea..1c2002a 100644
--- a/content/documentation/io/io-toc/index.html
+++ b/content/documentation/io/io-toc/index.html
@@ -153,12 +153,13 @@
 
 
   Authoring I/O Transforms 
- Overview
+  Testing I/O Transforms
 
 
 
 
diff --git a/content/documentation/io/testing/index.html 
b/content/documentation/io/testing/index.html
index 86d132a..e8173ff 100644
--- a/content/documentation/io/testing/index.html
+++ b/content/documentation/io/testing/index.html
@@ -139,17 +139,122 @@
 
   Pipeline I/O Table of 
Contents
 
-Testing I/O Transforms
+Testing I/O Transforms in Apache 
Beam
+
+Examples and design patterns for testing Apache Beam I/O 
transforms
+
+
+  Adapt for:
+  
+Java SDK
+Python SDK
+  
+
 
 
   Note: This guide is still in progress. There is an open issue to finish 
the guide: https://issues.apache.org/jira/browse/BEAM-1025;>BEAM-1025.
 
 
-Next steps
+Introduction
+
+This document explains the set of tests that the Beam community recommends 
based on our past experience writing I/O transforms. If you wish to contribute 
your I/O transform to the Beam community, we’ll ask you to implement these 
tests.
+
+While it is standard to write unit tests and integration tests, there are 
many possible definitions. Our definitions are:
+
+
+  Unit Tests:
+
+  Goal: verifying correctness of the transform only - core behavior, 
corner cases, etc.
+  Data store used: an in-memory version of the data store (if 
available), otherwise you’ll need to write a fake
+  Data set size: tiny (10s to 100s of rows)
+
+  
+  Integration Tests:
+
+  Goal: catch problems that occur when interacting with real versions 
of the runners/data store
+  Data store used: an actual instance, pre-configured before the 
test
+  Data set size: small to medium (1000 rows to 10s of GBs)
+
+  
+
+
+A note on performance 
benchmarking
+
+We do not advocate writing a separate test specifically for performance 
benchmarking. Instead, we recommend setting up integration tests that can 
accept the necessary parameters to cover many different testing scenarios.
+
+For example, if integration tests are written according to the guidelines 
below, the integration tests can be run on different runners (either local or 
in a cluster configuration) and against a data store that is a small instance 
with a small data set, or a large production-ready cluster with larger data 
set. This can provide coverage for a variety of scenarios - one of them is 
performance benchmarking.
+
+Test Balance - Unit vs 
Integration
+
+It’s easy to cover a large amount of code with an integration test, but it 
is then hard to find a cause for test failures and the test is flakier.
+
+However, there is a valuable set of bugs found by tests that exercise 
multiple workers reading/writing to data store instances that have multiple 
nodes (eg, read replicas, etc.).  Those scenarios are hard to find with unit 
tests and we find they commonly cause bugs in I/O transforms.
+
+Our test strategy is a balance of those 2 contradictory needs. We recommend 
doing as much testing as possible in unit tests, and writing a single, small 
integration test that can be run in various configurations.
+
+Examples
+
+Java:
+
+  https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java;>BigtableIO’s
 testing implementation is considered the best example of current best 
practices for unit testing Sources
+  https://github.com/apache/beam/blob/master/sdks/java/io/jdbc;>JdbcIO 
has the current best practice examples for writing integration tests.
+  https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch;>ElasticsearchIO
 demonstrates testing for bounded read/write
+  https://github.com/apache/beam/tree/master/sdks/java/io/mqtt;>MqttIO 
and https://github.com/apache/beam/tree/master/sdks/java/io/amqp;>AmpqpIO 
demonstrate unbounded read/write
+
+
+Python:
+
+  https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio_test.py;>avroio_test
 for examples of testing liquid sharding, source_test_utils, assert_that and equal_to
+
+
+Unit Tests
+
+Goals
+
+
+  Validate the correctness of the code in your I/O transform.
+  Validate that the I/O transform works correctly when used in concert 

[beam-site] 03/05: fixup! IO Testing, unit tests: update after readthrough

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit cd402901509c4a9fdb369011014eec2094c1514c
Author: Stephen Sisk 
AuthorDate: Tue Jul 18 15:08:50 2017 -0700

fixup! IO Testing, unit tests: update after readthrough
---
 src/documentation/io/io-toc.md  |  1 -
 src/documentation/io/testing.md | 48 +++--
 2 files changed, 27 insertions(+), 22 deletions(-)

diff --git a/src/documentation/io/io-toc.md b/src/documentation/io/io-toc.md
index 170321e..49c1d12 100644
--- a/src/documentation/io/io-toc.md
+++ b/src/documentation/io/io-toc.md
@@ -16,7 +16,6 @@ permalink: /documentation/io/io-toc/
 > Note: This guide is still in progress. There is an open issue to finish the 
 > guide: [BEAM-1025](https://issues.apache.org/jira/browse/BEAM-1025).
 
 * [Authoring I/O Transforms - Overview]({{site.baseurl 
}}/documentation/io/authoring-overview/)
-
 * [Testing I/O Transforms]({{site.baseurl }}/documentation/io/testing/)
 
 

[beam-site] 04/05: fixup! fixup! IO Testing, unit tests: update after readthrough

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 970991b7d0052244a15a07d409771f3721ca233e
Author: Stephen Sisk 
AuthorDate: Wed Jul 19 10:20:59 2017 -0700

fixup! fixup! IO Testing, unit tests: update after readthrough
---
 src/documentation/io/testing.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/documentation/io/testing.md b/src/documentation/io/testing.md
index 924bbe0..adfde04 100644
--- a/src/documentation/io/testing.md
+++ b/src/documentation/io/testing.md
@@ -99,7 +99,7 @@ Instead of using mocks in your unit tests (pre-programming 
exact responses to ea
 
 ### Network failure
 
-To help with testing and separation of concerns, **code that interacts across 
a network should be handled in a separate class from your I/O transform**. The 
suggested design pattern is that your I/O transform throws exceptions once it 
determines that a read is no longer possible.
+To help with testing and separation of concerns, **code that interacts across 
a network should be handled in a separate class from your I/O transform**. The 
suggested design pattern is that your I/O transform throws exceptions once it 
determines that a read or write is no longer possible.
 
 This allows the I/O transform's unit tests to act as if they have a perfect 
network connection, and they do not need to retry/otherwise handle network 
connection problems.
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (6a6b93a -> f6175fa)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 6a6b93a  This closes #257
 add 7ccef23  Prepare repository for deployment.
 new 7d3fc98  Port of Google doc
 new b6fdf18  IO Testing, unit tests: update after readthrough
 new cd40290  fixup! IO Testing, unit tests: update after readthrough
 new 970991b  fixup! fixup! IO Testing, unit tests: update after readthrough
 new f6175fa  This closes #274

The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/io/built-in/index.html |   4 ++
 src/documentation/io/io-toc.md   |   3 +-
 src/documentation/io/testing.md  | 101 ++-
 3 files changed, 106 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 05/05: This closes #274

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit f6175fa4591a9e0d564da3943509f60fcd8d59e9
Merge: 7ccef23 970991b
Author: Mergebot 
AuthorDate: Wed Jul 19 19:17:10 2017 +

This closes #274

 src/documentation/io/io-toc.md  |   3 +-
 src/documentation/io/testing.md | 101 +++-
 2 files changed, 102 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[jira] [Commented] (BEAM-2643) Add TextIO.read_all() to Python SDK

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093633#comment-16093633
 ] 

Eugene Kirpichov commented on BEAM-2643:


See also https://github.com/apache/beam/pull/3598

> Add TextIO.read_all() to Python SDK
> ---
>
> Key: BEAM-2643
> URL: https://issues.apache.org/jira/browse/BEAM-2643
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>
> Java SDK now has TextIO.read_all() API that allows reading a massive number 
> of files by moving from using the BoundedSource API (which may perform 
> expensive source operations on the control plane) to using ParDo operations.
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L170
> This API should be added for Python SDK as well.
> This form of reading files does not support dynamic work rebalancing for now. 
> But this should not matter much when reading a massive number of relatively 
> small files. In the future this API can support dynamic work rebalancing 
> through Splittable DoFn.
> cc: [~jkff]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam-site] 01/05: Port of Google doc

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 7d3fc98a1371baa38fddcf7a821766efa4b6ad00
Author: Stephen Sisk 
AuthorDate: Fri Jul 14 09:59:59 2017 -0700

Port of Google doc
---
 src/documentation/io/io-toc.md  |  4 +-
 src/documentation/io/testing.md | 94 +
 2 files changed, 97 insertions(+), 1 deletion(-)

diff --git a/src/documentation/io/io-toc.md b/src/documentation/io/io-toc.md
index 811f70a..170321e 100644
--- a/src/documentation/io/io-toc.md
+++ b/src/documentation/io/io-toc.md
@@ -17,9 +17,11 @@ permalink: /documentation/io/io-toc/
 
 * [Authoring I/O Transforms - Overview]({{site.baseurl 
}}/documentation/io/authoring-overview/)
 
+* [Testing I/O Transforms]({{site.baseurl }}/documentation/io/testing/)
+
 
diff --git a/src/documentation/io/testing.md b/src/documentation/io/testing.md
index e43c628..ddab572 100644
--- a/src/documentation/io/testing.md
+++ b/src/documentation/io/testing.md
@@ -11,6 +11,100 @@ permalink: /documentation/io/testing/
 > Note: This guide is still in progress. There is an open issue to finish the 
 > guide: [BEAM-1025](https://issues.apache.org/jira/browse/BEAM-1025).
 
 
+## Testing IO Transforms in Apache Beam 
+
+*Examples and design patterns for testing Apache Beam I/O transforms*
+
+
+## Introduction {#introduction}
+
+This document explains the set of tests that the Beam community recommends 
based on our past experience writing I/O transforms. If you wish to contribute 
your I/O transform to the Beam community, we'll ask you to implement these 
tests.
+
+While it is standard to write unit tests and integration tests, there are many 
possible definitions. Our definitions are:
+
+*   **Unit Tests:**
+*   Goal: verifying correctness of the transform itself - core behavior, 
corner cases, etc.
+*   Data store used: an in-memory version of the data store (if 
available), otherwise you'll need to write a [fake](#setting-up-mocks-fakes)
+*   Data set size: tiny (10s to 100s of rows)
+*   **Integration Tests:**
+*   Goal: catch problems that occur when interacting with real versions of 
the runners/data store
+*   Data store used: an actual instance, pre-configured before the test
+*   Data set size: small to medium (1000 rows to 10s of GBs)
+
+
+## A note on performance benchmarking
+
+Doing performance benchmarking is definitely useful and would provide value to 
the beam community. However, we do not advocate writing a separate performance 
test specifically for this purpose. Instead, we advocate setting up integration 
tests so that they be used with different runners and data set sizes. 
+
+For example, if integration tests are written according to the guidelines 
below, the integration tests can be run on different runners (either local or 
in a cluster configuration) and against a data store that is a small instance 
with a small data set, or a large production-ready cluster with larger data 
set. This can provide coverage for a variety of scenarios - one of them is 
performance benchmarking.
+
+See the Integration Testing section for more information.
+
+
+## Test Balance - Unit vs Integration {#test-balance-unit-vs-integration}
+
+It's easy to cover a large amount of code with an integration test, but it is 
then hard to find a cause for failures and the test is flakier. 
+
+However, there is a valuable set of bugs found by tests that exercise multiple 
workers reading/writing to data store instances that have multiple nodes (eg, 
read replicas, etc.).  Those scenarios are hard to find with unit tests and we 
find they commonly cause bugs in I/O transforms.
+
+Our test strategy is a balance of those 2 contradictory needs. We recommend 
doing as much testing as possible in unit tests, and writing a single, small 
integration test that can be run in various configurations.
+
+
+## Examples {#examples}
+
+
+
+*   `BigtableIO`'s testing implementation is considered the best example of 
current best practices for unit testing `Source`s. 
+*   `DatastoreIO` best demonstrates usage of the Service interface design 
pattern.
+*   `JdbcIO` has the current best practice examples for writing integration 
tests.
+
+
+## Unit Tests {#unit-tests}
+
+
+### Goals {#goals}
+
+
+
+*   Validate the correctness of the code in your I/O transform.
+*   Validate that the I/O transform works correctly when used in concert with 
reference implementations of the data store it connects with (where "reference 
implementation" means a fake or in-memory version).
+*   Be able to run quickly (< 1 sec) and need only one machine, with a 
reasonably small memory/disk footprint and no non-local network access 
(preferably none at all).
+*   Validate that the I/O transform can handle network failures. 
+
+
+### Non-goals
+
+
+
+*   Test problems in the external data store - 

[beam-site] 02/05: IO Testing, unit tests: update after readthrough

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit b6fdf18846cb0fdb829f41b5c5d5a8338160fc47
Author: Stephen Sisk 
AuthorDate: Fri Jul 14 10:22:03 2017 -0700

IO Testing, unit tests: update after readthrough
---
 src/documentation/io/testing.md | 17 -
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/src/documentation/io/testing.md b/src/documentation/io/testing.md
index ddab572..a1c1f46 100644
--- a/src/documentation/io/testing.md
+++ b/src/documentation/io/testing.md
@@ -23,8 +23,8 @@ This document explains the set of tests that the Beam 
community recommends based
 While it is standard to write unit tests and integration tests, there are many 
possible definitions. Our definitions are:
 
 *   **Unit Tests:**
-*   Goal: verifying correctness of the transform itself - core behavior, 
corner cases, etc.
-*   Data store used: an in-memory version of the data store (if 
available), otherwise you'll need to write a [fake](#setting-up-mocks-fakes)
+*   Goal: verifying correctness of the transform only - core behavior, 
corner cases, etc.
+*   Data store used: an in-memory version of the data store (if 
available), otherwise you'll need to write a [fake](#use-fakes)
 *   Data set size: tiny (10s to 100s of rows)
 *   **Integration Tests:**
 *   Goal: catch problems that occur when interacting with real versions of 
the runners/data store
@@ -34,16 +34,14 @@ While it is standard to write unit tests and integration 
tests, there are many p
 
 ## A note on performance benchmarking
 
-Doing performance benchmarking is definitely useful and would provide value to 
the beam community. However, we do not advocate writing a separate performance 
test specifically for this purpose. Instead, we advocate setting up integration 
tests so that they be used with different runners and data set sizes. 
+We do not advocate writing a separate test specifically for performance 
benchmarking. Instead, we advocate setting up integration tests so that they 
can be parameterized in a way that allows for covering many different testing 
scenarios.
 
 For example, if integration tests are written according to the guidelines 
below, the integration tests can be run on different runners (either local or 
in a cluster configuration) and against a data store that is a small instance 
with a small data set, or a large production-ready cluster with larger data 
set. This can provide coverage for a variety of scenarios - one of them is 
performance benchmarking.
 
-See the Integration Testing section for more information.
-
 
 ## Test Balance - Unit vs Integration {#test-balance-unit-vs-integration}
 
-It's easy to cover a large amount of code with an integration test, but it is 
then hard to find a cause for failures and the test is flakier. 
+It's easy to cover a large amount of code with an integration test, but it is 
then hard to find a cause for test failures and the test is flakier.
 
 However, there is a valuable set of bugs found by tests that exercise multiple 
workers reading/writing to data store instances that have multiple nodes (eg, 
read replicas, etc.).  Those scenarios are hard to find with unit tests and we 
find they commonly cause bugs in I/O transforms.
 
@@ -83,12 +81,12 @@ Our test strategy is a balance of those 2 contradictory 
needs. We recommend doin
 
 A general guide to writing Unit Tests for all transforms can be found in the 
[PTransform Style 
Guide](https://beam.apache.org/contribute/ptransform-style-guide/#testing ). We 
have expanded on a few important points below.
 
-If you are implementing a `Source`/`Reader` class, make sure to exhaustively 
unit-test your code. A minor implementation error can lead to data corruption 
or data loss (such as skipping or duplicating records) that can be hard for 
your users to detect. Also look into using SourceTestUtils - it is a key piece 
of test `Source` implementations.
+If you are implementing a `Source`/`Reader` class, make sure to exhaustively 
unit-test your code. A minor implementation error can lead to data corruption 
or data loss (such as skipping or duplicating records) that can be hard for 
your users to detect. Also look into using `SourceTestUtils` - it is a key 
piece of test `Source` implementations.
 
 If you are not using the `Source` API, you can use DoFnTester to help with 
your testing. Datastore's I/O transforms have some good examples of how to use 
it in testing I/O transforms.
 
 
-### Use mocks/fakes
+### Use fakes {#use-fakes}
 
 Instead of using mocks in your unit tests (pre-programming exact responses to 
each call for each test), use fakes (a lightweight implementation of the 
service that behaves the same way at a very small scale) or an in-memory 
version of the service you're testing. This has proven to be the right mix of 
"you 

[jira] [Commented] (BEAM-2640) Introduce Create.ofProvider(ValueProvider)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093632#comment-16093632
 ] 

ASF GitHub Bot commented on BEAM-2640:
--

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3598

[BEAM-2640, BEAM-2641] Introduces TextIO.read().withHintMatchesManyFiles()

In that case it expands to TextIO.readAll(). Implementing this when the 
filepattern is a ValueProvider nudged me to also implement BEAM-2640 - 
Create.ofProvider(ValueProvider).

Links:
https://issues.apache.org/jira/browse/BEAM-2640
https://issues.apache.org/jira/browse/BEAM-2641

R: @reuvenlax 
CC: @sammcveety 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam textio-readall

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3598.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3598


commit d26416d142808d8279c40cd57a659e103b296f3f
Author: Eugene Kirpichov 
Date:   2017-07-19T18:50:58Z

[BEAM-2640] Introduces Create.ofProvider(ValueProvider)

I also converted DatastoreV1 to use this overload, and, as an
exercise, added a withQuery(ValueProvider) overload to JdbcIO.

commit 872f0df5c227385bc95a42ecc02934a4082565ba
Author: Eugene Kirpichov 
Date:   2017-07-19T18:51:33Z

[BEAM-2641] Introduces TextIO.read().withHintMatchesManyFiles()

In that case it expands to TextIO.readAll().




> Introduce Create.ofProvider(ValueProvider)
> --
>
> Key: BEAM-2640
> URL: https://issues.apache.org/jira/browse/BEAM-2640
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> When you have a ValueProvider that may or may not be accessible at 
> construction time, a common task is to wrap it into a single-element 
> PCollection. This is especially common when migrating an IO connector that 
> used something like Create.of(query) followed by a ParDo, to having query be 
> a ValueProvider.
> Currently this is done in an icky way (e.g. 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L615)
> We should have a convenience helper for that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3598: [BEAM-2640, BEAM-2641] Introduces TextIO.read().wit...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3598

[BEAM-2640, BEAM-2641] Introduces TextIO.read().withHintMatchesManyFiles()

In that case it expands to TextIO.readAll(). Implementing this when the 
filepattern is a ValueProvider nudged me to also implement BEAM-2640 - 
Create.ofProvider(ValueProvider).

Links:
https://issues.apache.org/jira/browse/BEAM-2640
https://issues.apache.org/jira/browse/BEAM-2641

R: @reuvenlax 
CC: @sammcveety 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam textio-readall

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3598.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3598


commit d26416d142808d8279c40cd57a659e103b296f3f
Author: Eugene Kirpichov 
Date:   2017-07-19T18:50:58Z

[BEAM-2640] Introduces Create.ofProvider(ValueProvider)

I also converted DatastoreV1 to use this overload, and, as an
exercise, added a withQuery(ValueProvider) overload to JdbcIO.

commit 872f0df5c227385bc95a42ecc02934a4082565ba
Author: Eugene Kirpichov 
Date:   2017-07-19T18:51:33Z

[BEAM-2641] Introduces TextIO.read().withHintMatchesManyFiles()

In that case it expands to TextIO.readAll().




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3439

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #119

[Apache Jenkins Server]

See 


Changes:

[klk] [BEAM-2306] Add checkstyle check to fail the build when @Deprecated is

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a6f460fe3b760aafbc748ae18956f0f2c1fedfad (origin/master)
Commit message: "This closes #3531: [BEAM-2306] Fail build when @Deprecated is 
used without @deprecated javadoc"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a6f460fe3b760aafbc748ae18956f0f2c1fedfad
 > git rev-list 7fde976d14fe697dd88d2b161540c73d5cb01517 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1387453180019638117.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2079599808115030904.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1188456120242871175.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8214915576474820332.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed 

[jira] [Updated] (BEAM-2642) Upgrade to Google Auth 0.7.1

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-2642:

Description: 
Looking up application default credentials on a GCE VM can fail due to VM 
metadata server being unavailable during VM launch. This is a rare event but 
Google Cloud Dataflow customers hit this rare case one or two times a month due 
to the sheer number of VMs. GCE attempted to mitigate VM metadata server 
unavailability but were only able to reduce it be an order of magnitude thus we 
need support from the client to retry. Additionally, when contacting the GCE VM 
metadata server, we should be using the fixed IP address avoiding the 
nameserver lookup (another potential point of failure).

Problem area in the code:
https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261

Note that the code in this library and the Apiary auth support code are very 
similar. The fix was done within the Apiary auth code (note the use of the 
static IP address and also the presence of a fixed number of retries):
https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74

It turned out that the fixes resulted in zero future customer contacts about 
this issue.

Google Auth 0.7.1 was released containing these fixes mentioned in 
https://github.com/google/google-auth-library-java/issues/109

  was:
Looking up application default credentials on a GCE VM can fail due to VM 
metadata server being unavailable during VM launch. This is a rare event but 
Google Cloud Dataflow customers hit this rare case one or two times a month due 
to the sheer number of VMs. GCE attempted to mitigate VM metadata server 
unavailability but were only able to reduce it be an order of magnitude thus we 
need support from the client to retry. Additionally, when contacting the GCE VM 
metadata server, we should be using the fixed IP address avoiding the 
nameserver lookup (another potential point of failure).

Problem area in the code:
https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261

Note that the code in this library and the Apiary auth support code are very 
similar. The fix was done within the Apiary auth code (note the use of the 
static IP address and also the presence of a fixed number of retries):
https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74

It turned out that the fixes resulted in zero future customer contacts about 
this issue.

Google Auth 0.7.1 was released containing these fixes.


> Upgrade to Google Auth 0.7.1
> 
>
> Key: BEAM-2642
> URL: https://issues.apache.org/jira/browse/BEAM-2642
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Looking up application default credentials on a GCE VM can fail due to VM 
> metadata server being unavailable during VM launch. This is a rare event but 
> Google Cloud Dataflow customers hit this rare case one or two times a month 
> due to the sheer number of VMs. GCE attempted to mitigate VM metadata server 
> unavailability but were only able to reduce it be an order of magnitude thus 
> we need support from the client to retry. Additionally, when contacting the 
> GCE VM metadata server, we should be using the fixed IP address avoiding the 
> nameserver lookup (another potential point of failure).
> Problem area in the code:
> https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261
> Note that the code in this library and the Apiary auth support code are very 
> similar. The fix was done within the Apiary auth code (note the use of the 
> static IP address and also the presence of a fixed number of retries):
> https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74
> It turned out that the fixes resulted in zero future customer contacts about 
> this issue.
> Google Auth 0.7.1 was released containing these fixes mentioned in 
> https://github.com/google/google-auth-library-java/issues/109



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3597: Register a PTransformTranslator for Combine

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3597

Register a PTransformTranslator for Combine

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
Include the Combine Payload in the Runner API Graph.

Add getCombineFn(AppliedPTransform) to extract the CombineFn from an
arbitrary transform.

Update Pipeline Translation tests to include accumulator coders.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam combine_translation_get_fn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3597.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3597


commit 60e840595b56cbc81538307231468a6fd2ba8059
Author: Thomas Groh 
Date:   2017-07-19T17:55:33Z

Register a PTransformTranslator for Combine

Include the Combine Payload in the Runner API Graph.

Add getCombineFn(AppliedPTransform) to extract the CombineFn from an
arbitrary transform.

Update Pipeline Translation tests to include accumulator coders.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2640) Introduce Create.ofProvider(ValueProvider)

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093588#comment-16093588
 ] 

Eugene Kirpichov commented on BEAM-2640:


Coder inference for this is complicated by the fact that ValueProvider does not 
expose a TypeDescriptor for the value being provided. Even though it could. I'm 
gonna start with requiring to explicitly provide a coder.

> Introduce Create.ofProvider(ValueProvider)
> --
>
> Key: BEAM-2640
> URL: https://issues.apache.org/jira/browse/BEAM-2640
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> When you have a ValueProvider that may or may not be accessible at 
> construction time, a common task is to wrap it into a single-element 
> PCollection. This is especially common when migrating an IO connector that 
> used something like Create.of(query) followed by a ParDo, to having query be 
> a ValueProvider.
> Currently this is done in an icky way (e.g. 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L615)
> We should have a convenience helper for that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2643) Add TextIO.read_all() to Python SDK

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-2643:


 Summary: Add TextIO.read_all() to Python SDK
 Key: BEAM-2643
 URL: https://issues.apache.org/jira/browse/BEAM-2643
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py
Reporter: Chamikara Jayalath


Java SDK now has TextIO.read_all() API that allows reading a massive number of 
files by moving from using the BoundedSource API (which may perform expensive 
source operations on the control plane) to using ParDo operations.

https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L170

This API should be added for Python SDK as well.

This form of reading files does not support dynamic work rebalancing for now. 
But this should not matter much when reading a massive number of relatively 
small files. In the future this API can support dynamic work rebalancing 
through Splittable DoFn.

cc: [~jkff]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2642) Upgrade to Google Auth 0.7.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093578#comment-16093578
 ] 

ASF GitHub Bot commented on BEAM-2642:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3596

[BEAM-2642] Update Google Auth to 0.7.1

Google auth libraries 0.6.1 and 0.7.1 have the same versions of 
dependencies.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam2642

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3596.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3596






> Upgrade to Google Auth 0.7.1
> 
>
> Key: BEAM-2642
> URL: https://issues.apache.org/jira/browse/BEAM-2642
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Looking up application default credentials on a GCE VM can fail due to VM 
> metadata server being unavailable during VM launch. This is a rare event but 
> Google Cloud Dataflow customers hit this rare case one or two times a month 
> due to the sheer number of VMs. GCE attempted to mitigate VM metadata server 
> unavailability but were only able to reduce it be an order of magnitude thus 
> we need support from the client to retry. Additionally, when contacting the 
> GCE VM metadata server, we should be using the fixed IP address avoiding the 
> nameserver lookup (another potential point of failure).
> Problem area in the code:
> https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261
> Note that the code in this library and the Apiary auth support code are very 
> similar. The fix was done within the Apiary auth code (note the use of the 
> static IP address and also the presence of a fixed number of retries):
> https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74
> It turned out that the fixes resulted in zero future customer contacts about 
> this issue.
> Google Auth 0.7.1 was released containing these fixes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3596: [BEAM-2642] Update Google Auth to 0.7.1

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/3596

[BEAM-2642] Update Google Auth to 0.7.1

Google auth libraries 0.6.1 and 0.7.1 have the same versions of 
dependencies.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam2642

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3596.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3596






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2642) Upgrade to Google Auth 0.7.1

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-2642:
---

 Summary: Upgrade to Google Auth 0.7.1
 Key: BEAM-2642
 URL: https://issues.apache.org/jira/browse/BEAM-2642
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Affects Versions: 2.0.0, 2.1.0
Reporter: Luke Cwik
Assignee: Luke Cwik


Looking up application default credentials on a GCE VM can fail due to VM 
metadata server being unavailable during VM launch. This is a rare event but 
Google Cloud Dataflow customers hit this rare case one or two times a month due 
to the sheer number of VMs. GCE attempted to mitigate VM metadata server 
unavailability but were only able to reduce it be an order of magnitude thus we 
need support from the client to retry. Additionally, when contacting the GCE VM 
metadata server, we should be using the fixed IP address avoiding the 
nameserver lookup (another potential point of failure).

Problem area in the code:
https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261

Note that the code in this library and the Apiary auth support code are very 
similar. The fix was done within the Apiary auth code (note the use of the 
static IP address and also the presence of a fixed number of retries):
https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74

It turned out that the fixes resulted in zero future customer contacts about 
this issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2642) Upgrade to Google Auth 0.7.1

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-2642:

Description: 
Looking up application default credentials on a GCE VM can fail due to VM 
metadata server being unavailable during VM launch. This is a rare event but 
Google Cloud Dataflow customers hit this rare case one or two times a month due 
to the sheer number of VMs. GCE attempted to mitigate VM metadata server 
unavailability but were only able to reduce it be an order of magnitude thus we 
need support from the client to retry. Additionally, when contacting the GCE VM 
metadata server, we should be using the fixed IP address avoiding the 
nameserver lookup (another potential point of failure).

Problem area in the code:
https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261

Note that the code in this library and the Apiary auth support code are very 
similar. The fix was done within the Apiary auth code (note the use of the 
static IP address and also the presence of a fixed number of retries):
https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74

It turned out that the fixes resulted in zero future customer contacts about 
this issue.

Google Auth 0.7.1 was released containing these fixes.

  was:
Looking up application default credentials on a GCE VM can fail due to VM 
metadata server being unavailable during VM launch. This is a rare event but 
Google Cloud Dataflow customers hit this rare case one or two times a month due 
to the sheer number of VMs. GCE attempted to mitigate VM metadata server 
unavailability but were only able to reduce it be an order of magnitude thus we 
need support from the client to retry. Additionally, when contacting the GCE VM 
metadata server, we should be using the fixed IP address avoiding the 
nameserver lookup (another potential point of failure).

Problem area in the code:
https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261

Note that the code in this library and the Apiary auth support code are very 
similar. The fix was done within the Apiary auth code (note the use of the 
static IP address and also the presence of a fixed number of retries):
https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74

It turned out that the fixes resulted in zero future customer contacts about 
this issue.


> Upgrade to Google Auth 0.7.1
> 
>
> Key: BEAM-2642
> URL: https://issues.apache.org/jira/browse/BEAM-2642
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Looking up application default credentials on a GCE VM can fail due to VM 
> metadata server being unavailable during VM launch. This is a rare event but 
> Google Cloud Dataflow customers hit this rare case one or two times a month 
> due to the sheer number of VMs. GCE attempted to mitigate VM metadata server 
> unavailability but were only able to reduce it be an order of magnitude thus 
> we need support from the client to retry. Additionally, when contacting the 
> GCE VM metadata server, we should be using the fixed IP address avoiding the 
> nameserver lookup (another potential point of failure).
> Problem area in the code:
> https://github.com/google/google-auth-library-java/blob/b94f8e4d02bf6917af2e2f7ef8d7114a51dbcfa8/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L261
> Note that the code in this library and the Apiary auth support code are very 
> similar. The fix was done within the Apiary auth code (note the use of the 
> static IP address and also the presence of a fixed number of retries):
> https://github.com/google/google-api-java-client/blob/4fc8c099d9db5646770868cc1bc9a33c9225b3c7/google-api-client/src/main/java/com/google/api/client/googleapis/auth/oauth2/OAuth2Utils.java#L74
> It turned out that the fixes resulted in zero future customer contacts about 
> this issue.
> Google Auth 0.7.1 was released containing these fixes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3612

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2641) Improve discoverability of TextIO.readAll() as a replacement of TextIO.read() for large globs

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-2641:
--

 Summary: Improve discoverability of TextIO.readAll() as a 
replacement of TextIO.read() for large globs
 Key: BEAM-2641
 URL: https://issues.apache.org/jira/browse/BEAM-2641
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov


TextIO.readAll() dramatically outperforms TextIO.read() when reading very large 
numbers of files (hundreds of thousands or millions or more).

However, it is not obvious that this is what you should use if you have such a 
filepattern in TextIO.read().

We should take a variety of measures to make it more discoverable, e.g.:

* Add a parameter to TextIO.read(), like "withHintManyFiles()"
* Log something suggesting the use of that hint when splitting TextIO if the 
filepattern is very large
* Improve documentation
* Post something on StackOverflow about this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2640) Introduce Create.ofProvider(ValueProvider)

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-2640:
---
Issue Type: New Feature  (was: Bug)

> Introduce Create.ofProvider(ValueProvider)
> --
>
> Key: BEAM-2640
> URL: https://issues.apache.org/jira/browse/BEAM-2640
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> When you have a ValueProvider that may or may not be accessible at 
> construction time, a common task is to wrap it into a single-element 
> PCollection. This is especially common when migrating an IO connector that 
> used something like Create.of(query) followed by a ParDo, to having query be 
> a ValueProvider.
> Currently this is done in an icky way (e.g. 
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L615)
> We should have a convenience helper for that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2640) Introduce Create.ofProvider(ValueProvider)

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-2640:
--

 Summary: Introduce Create.ofProvider(ValueProvider)
 Key: BEAM-2640
 URL: https://issues.apache.org/jira/browse/BEAM-2640
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov


When you have a ValueProvider that may or may not be accessible at 
construction time, a common task is to wrap it into a single-element 
PCollection. This is especially common when migrating an IO connector that 
used something like Create.of(query) followed by a ParDo, to having query be a 
ValueProvider.

Currently this is done in an icky way (e.g. 
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L615)

We should have a convenience helper for that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4409

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2636) user_score on DataflowRunner is broken

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093477#comment-16093477
 ] 

ASF GitHub Bot commented on BEAM-2636:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3595

[BEAM-2636] Make sure we only override the correct class for WriteToBigQuery

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-2636-bq-override

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3595.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3595


commit 9056783c962cd1093eee3eca718bdbc5b42e554a
Author: Sourabh Bajaj 
Date:   2017-07-19T17:08:14Z

[BEAM-2636] Make sure we only override the correct class




> user_score on DataflowRunner is broken
> --
>
> Key: BEAM-2636
> URL: https://issues.apache.org/jira/browse/BEAM-2636
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.1.0
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>
> UserScore has a custom transform named {{WriteToBigQuery}}, dataflow runner 
> has a special code handling transforms with that name, this will break for 
> all user transforms that has this name.
> We can either:
> - Handle this correctly
> - Or document this as a reserved keyword and change the example.
> cc: [~chamikara]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3595: [BEAM-2636] Make sure we only override the correct ...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3595

[BEAM-2636] Make sure we only override the correct class for WriteToBigQuery

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-2636-bq-override

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3595.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3595


commit 9056783c962cd1093eee3eca718bdbc5b42e554a
Author: Sourabh Bajaj 
Date:   2017-07-19T17:08:14Z

[BEAM-2636] Make sure we only override the correct class




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #2019

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2662

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3594: Do not submit: scratch work

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/beam/pull/3594

Do not submit: scratch work

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/beam test-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3594.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3594


commit 200b55c1a9cacddf501869d3127f34e735e68f37
Author: Charles Chen 
Date:   2017-07-19T16:50:42Z

Test commit




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3438

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2306) @Deprecated without @deprecated javadoc explanation should cause build failure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093368#comment-16093368
 ] 

ASF GitHub Bot commented on BEAM-2306:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3531


> @Deprecated without @deprecated javadoc explanation should cause build failure
> --
>
> Key: BEAM-2306
> URL: https://issues.apache.org/jira/browse/BEAM-2306
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.0.0
>Reporter: Kenneth Knowles
>  Labels: starter
>
> We have a number of places with {{@Deprecated}} annotations on seemingly 
> innocuous methods, for example in {{CoderRegistry}}, with no accompanying 
> {{@deprecated}} javadoc.
>  - If there is a preferred alternative, it should be explicitly linked.
>  - If there is no alternative, that should be explained.
>  - The deprecation should indicate whether it is for removal at version 3.0.0 
> or whether it was deprecated prior to 2.0.0 and may be removed at some 
> increment 2.x.y.
> I believe javadoc or findbugs has the ability to enforce proper policy. This 
> ticket tracks getting that policy in place.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: [BEAM-2306] Add checkstyle check to fail the build when @Deprecated is used without @deprecated javadoc (or vice versa).

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 7fde976d1 -> a6f460fe3


[BEAM-2306] Add checkstyle check to fail the build when @Deprecated is used 
without @deprecated javadoc (or vice versa).

The check is disabled for existing violations where reason for deprecation 
and/or alternative is not clear.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d2901145
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d2901145
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d2901145

Branch: refs/heads/master
Commit: d290114549c0b379774dbabe119a79d3ee1b2b56
Parents: 7fde976
Author: Alex Filatov 
Authored: Mon Jul 10 13:20:49 2017 +0300
Committer: Kenneth Knowles 
Committed: Wed Jul 19 09:03:31 2017 -0700

--
 .../construction/CreatePCollectionViewTranslation.java  | 11 ++-
 .../core/construction/PTransformTranslation.java|  4 
 .../beam/runners/core/InMemoryTimerInternals.java   |  9 +
 .../java/org/apache/beam/runners/core/StateTags.java|  3 +++
 .../beam/runners/direct/DirectTimerInternals.java   |  9 +
 .../translation/wrappers/streaming/DoFnOperator.java|  9 +
 .../apache/beam/runners/dataflow/DataflowRunner.java|  3 ++-
 .../options/DataflowPipelineWorkerPoolOptions.java  |  3 +++
 .../build-tools/src/main/resources/beam/checkstyle.xml  |  8 
 .../src/main/java/org/apache/beam/sdk/coders/Coder.java | 12 +++-
 .../java/org/apache/beam/sdk/coders/CoderRegistry.java  |  9 +
 .../main/java/org/apache/beam/sdk/io/AvroSource.java|  6 --
 .../main/java/org/apache/beam/sdk/testing/PAssert.java  |  5 +++--
 .../java/org/apache/beam/sdk/testing/StreamingIT.java   |  4 
 .../java/org/apache/beam/sdk/transforms/Combine.java|  1 -
 .../main/java/org/apache/beam/sdk/transforms/DoFn.java  |  3 +++
 .../main/java/org/apache/beam/sdk/transforms/View.java  |  2 +-
 .../beam/sdk/transforms/reflect/DoFnInvokers.java   |  9 -
 .../java/org/apache/beam/sdk/util/IdentityWindowFn.java |  1 -
 .../org/apache/beam/sdk/values/PCollectionViews.java|  1 -
 .../main/java/org/apache/beam/sdk/values/PValue.java|  4 ++--
 .../org/apache/beam/sdk/coders/DefaultCoderTest.java|  3 ++-
 .../org/apache/beam/fn/harness/BoundedSourceRunner.java |  6 +++---
 23 files changed, 95 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d2901145/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
index 8fc99b9..c67d688 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
@@ -86,6 +86,10 @@ public class CreatePCollectionViewTranslation {
 PCollectionView.class.getSimpleName());
   }
 
+  /**
+   * @deprecated runners should move away from translating 
`CreatePCollectionView` and treat this
+   * as part of the translation for a `ParDo` side input.
+   */
   @Deprecated
   static class CreatePCollectionViewTranslator
   implements TransformPayloadTranslator {
@@ -112,7 +116,12 @@ public class CreatePCollectionViewTranslation {
 }
   }
 
-  /** Registers {@link CreatePCollectionViewTranslator}. */
+  /**
+   * Registers {@link CreatePCollectionViewTranslator}.
+   *
+   * @deprecated runners should move away from translating 
`CreatePCollectionView` and treat this
+   * as part of the translation for a `ParDo` side input.
+   */
   @AutoService(TransformPayloadTranslatorRegistrar.class)
   @Deprecated
   public static class Registrar implements TransformPayloadTranslatorRegistrar 
{

http://git-wip-us.apache.org/repos/asf/beam/blob/d2901145/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
index bae7b05..0b4a2ab 100644
--- 

[2/2] beam git commit: This closes #3531: [BEAM-2306] Fail build when @Deprecated is used without @deprecated javadoc

[kenn]

This closes #3531: [BEAM-2306] Fail build when @Deprecated is used without 
@deprecated javadoc

  [BEAM-2306] Add checkstyle check to fail the build when @Deprecated is used 
without @deprecated javadoc (or vice versa).


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a6f460fe
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a6f460fe
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a6f460fe

Branch: refs/heads/master
Commit: a6f460fe3b760aafbc748ae18956f0f2c1fedfad
Parents: 7fde976 d290114
Author: Kenneth Knowles 
Authored: Wed Jul 19 09:03:48 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed Jul 19 09:03:48 2017 -0700

--
 .../construction/CreatePCollectionViewTranslation.java  | 11 ++-
 .../core/construction/PTransformTranslation.java|  4 
 .../beam/runners/core/InMemoryTimerInternals.java   |  9 +
 .../java/org/apache/beam/runners/core/StateTags.java|  3 +++
 .../beam/runners/direct/DirectTimerInternals.java   |  9 +
 .../translation/wrappers/streaming/DoFnOperator.java|  9 +
 .../apache/beam/runners/dataflow/DataflowRunner.java|  3 ++-
 .../options/DataflowPipelineWorkerPoolOptions.java  |  3 +++
 .../build-tools/src/main/resources/beam/checkstyle.xml  |  8 
 .../src/main/java/org/apache/beam/sdk/coders/Coder.java | 12 +++-
 .../java/org/apache/beam/sdk/coders/CoderRegistry.java  |  9 +
 .../main/java/org/apache/beam/sdk/io/AvroSource.java|  6 --
 .../main/java/org/apache/beam/sdk/testing/PAssert.java  |  5 +++--
 .../java/org/apache/beam/sdk/testing/StreamingIT.java   |  4 
 .../java/org/apache/beam/sdk/transforms/Combine.java|  1 -
 .../main/java/org/apache/beam/sdk/transforms/DoFn.java  |  3 +++
 .../main/java/org/apache/beam/sdk/transforms/View.java  |  2 +-
 .../beam/sdk/transforms/reflect/DoFnInvokers.java   |  9 -
 .../java/org/apache/beam/sdk/util/IdentityWindowFn.java |  1 -
 .../org/apache/beam/sdk/values/PCollectionViews.java|  1 -
 .../main/java/org/apache/beam/sdk/values/PValue.java|  4 ++--
 .../org/apache/beam/sdk/coders/DefaultCoderTest.java|  3 ++-
 .../org/apache/beam/fn/harness/BoundedSourceRunner.java |  6 +++---
 23 files changed, 95 insertions(+), 30 deletions(-)
--

[GitHub] beam pull request #3531: [BEAM-2306] Fail build when @Deprecated is used wit...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3531


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2453) The Java DirectRunner should exercise all parts of a CombineFn

[Neville Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093359#comment-16093359
 ] 

Neville Li commented on BEAM-2453:
--

Here's an example of incorrect use of {{Combine.perKey}} that could be 
identified by this fix:
https://github.com/spotify/scio/issues/729

> The Java DirectRunner should exercise all parts of a CombineFn
> --
>
> Key: BEAM-2453
> URL: https://issues.apache.org/jira/browse/BEAM-2453
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Specifically it should:
> Create some number of accumulators; add elements to these accumulators, merge 
> the created accumulators, and extract the output.
> This can be performed by replacing the {{Combine.perKey}} composite transform 
> with a multi-step combine {{CombineBundles -> GroupByKey -> 
> MergeAccumulators}}
> Where {{CombineBundles}} is a {{ParDo}} which takes input {{KV}} 
> and produces {{KV}}, outputting in {{FinishBundle}} (this can only 
> be performed if the Combine takes no side inputs or does not have merging 
> windows). {{MergeAccumulators}} takes in {{KV}} and 
> produces {{KV}} by merging all of the accumulators and extracting 
> the output.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2384) CEIL, FLOOR, TRUNCATE, PI, ATAN2 math function

[Tarush Grover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tarush Grover resolved BEAM-2384.
-
   Resolution: Fixed
Fix Version/s: 2.2.0

> CEIL, FLOOR, TRUNCATE, PI, ATAN2 math function
> --
>
> Key: BEAM-2384
> URL: https://issues.apache.org/jira/browse/BEAM-2384
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2515) BeamSql: refactor the MockedBeamSqlTable and related tests

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu resolved BEAM-2515.

   Resolution: Fixed
Fix Version/s: 2.2.0

> BeamSql: refactor the MockedBeamSqlTable and related tests
> --
>
> Key: BEAM-2515
> URL: https://issues.apache.org/jira/browse/BEAM-2515
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: 2.2.0
>
>
> MockedBeamSqlTable is only for Bounded data sources, after another Unbounded 
> mock added, some refactor will be needed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2563) Add integration test for math operators

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu resolved BEAM-2563.

   Resolution: Fixed
Fix Version/s: 2.2.0

> Add integration test for math operators
> ---
>
> Key: BEAM-2563
> URL: https://issues.apache.org/jira/browse/BEAM-2563
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2559) Add integration test for the sql functions

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu resolved BEAM-2559.

   Resolution: Fixed
Fix Version/s: 2.2.0

> Add integration test for the sql functions
> --
>
> Key: BEAM-2559
> URL: https://issues.apache.org/jira/browse/BEAM-2559
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: James Xu
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2613) Add integration test for comparison operators

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu resolved BEAM-2613.

   Resolution: Fixed
Fix Version/s: 2.2.0

> Add integration test for comparison operators
> -
>
> Key: BEAM-2613
> URL: https://issues.apache.org/jira/browse/BEAM-2613
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2621) rename BeamSqlRecordType to BeamSqlRowType

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu resolved BEAM-2621.

   Resolution: Fixed
Fix Version/s: 2.2.0

> rename BeamSqlRecordType to BeamSqlRowType
> --
>
> Key: BEAM-2621
> URL: https://issues.apache.org/jira/browse/BEAM-2621
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>  Labels: dsl_sql_merge
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3611

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2355) support comparison operator

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu resolved BEAM-2355.

   Resolution: Fixed
Fix Version/s: 2.2.0

> support comparison operator
> ---
>
> Key: BEAM-2355
> URL: https://issues.apache.org/jira/browse/BEAM-2355
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
> Fix For: 2.2.0
>
>
> https://calcite.apache.org/docs/reference.html#comparison-operators



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2621) rename BeamSqlRecordType to BeamSqlRowType

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093165#comment-16093165
 ] 

Xu Mingmin commented on BEAM-2621:
--

[~xumingming] can you close it?

> rename BeamSqlRecordType to BeamSqlRowType
> --
>
> Key: BEAM-2621
> URL: https://issues.apache.org/jira/browse/BEAM-2621
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2563) Add integration test for math operators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093157#comment-16093157
 ] 

ASF GitHub Bot commented on BEAM-2563:
--

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3569


> Add integration test for math operators
> ---
>
> Key: BEAM-2563
> URL: https://issues.apache.org/jira/browse/BEAM-2563
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3569: [BEAM-2563] Add integration test for math operators

[XuMingmin]

Github user XuMingmin closed the pull request at:

https://github.com/apache/beam/pull/3569


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #4408

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2613) Add integration test for comparison operators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093140#comment-16093140
 ] 

ASF GitHub Bot commented on BEAM-2613:
--

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/3583


> Add integration test for comparison operators
> -
>
> Key: BEAM-2613
> URL: https://issues.apache.org/jira/browse/BEAM-2613
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3583: [BEAM-2613] add integration test for comparison ope...

[xumingming]

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/3583


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #2018

[Apache Jenkins Server]

See 


--
[...truncated 536.99 KB...]
2017-07-19T13:07:26.058 [INFO] Excluding 
org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding 
org.apache.commons:commons-math3:jar:3.1.1 from the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded 
jar.
2017-07-19T13:07:26.058 [INFO] Excluding 
commons-httpclient:commons-httpclient:jar:3.1 from the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding commons-io:commons-io:jar:2.4 from the 
shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding commons-net:commons-net:jar:3.1 from 
the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding 
commons-collections:commons-collections:jar:3.2.2 from the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding javax.servlet:servlet-api:jar:2.5 from 
the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 
from the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding 
org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from 
the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding com.sun.jersey:jersey-json:jar:1.9 
from the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding org.codehaus.jettison:jettison:jar:1.1 
from the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 
from the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding net.java.dev.jets3t:jets3t:jar:0.9.0 
from the shaded jar.
2017-07-19T13:07:26.058 [INFO] Excluding 
com.jamesmurty.utils:java-xmlbuilder:jar:0.4 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
commons-configuration:commons-configuration:jar:1.6 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
commons-digester:commons-digester:jar:1.8 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 
from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.curator:curator-framework:jar:2.7.1 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding com.jcraft:jsch:jar:0.1.42 from the 
shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.curator:curator-client:jar:2.7.1 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.curator:curator-recipes:jar:2.7.1 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.htrace:htrace-core:jar:3.1.0-incubating from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding io.netty:netty:jar:3.7.0.Final from 
the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding com.datatorrent:netlet:jar:1.3.0 from 
the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
com.esotericsoftware.kryo:kryo:jar:2.24.0 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
com.esotericsoftware.minlog:minlog:jar:1.2 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding org.objenesis:objenesis:jar:2.1 from 
the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
javax.validation:validation-api:jar:1.1.0.Final from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 
from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.apache.apex:malhar-library:jar:3.4.0 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.eclipse.jetty:jetty-servlet:jar:8.1.10.v20130312 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.eclipse.jetty:jetty-security:jar:8.1.10.v20130312 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.eclipse.jetty:jetty-server:jar:8.1.10.v20130312 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding 
org.eclipse.jetty:jetty-continuation:jar:8.1.10.v20130312 from the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding com.sun.mail:javax.mail:jar:1.5.0 from 
the shaded jar.
2017-07-19T13:07:26.059 [INFO] Excluding javax.activation:activation:jar:1.1 
from