Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #111

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple 
IO

--
[...truncated 2.64 MB...]
2017-06-09T06:34:18.594 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 804.0 KB/sec)
2017-06-09T06:34:18.598 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-09T06:34:18.610 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1069.5 KB/sec)
2017-06-09T06:34:18.614 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-09T06:34:18.626 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 181.1 KB/sec)
2017-06-09T06:34:18.630 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-09T06:34:18.640 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 157.3 KB/sec)
2017-06-09T06:34:18.645 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-09T06:34:18.658 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 403.8 KB/sec)
2017-06-09T06:34:18.662 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-09T06:34:18.671 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 277.2 KB/sec)
2017-06-09T06:34:18.675 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-09T06:34:18.685 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 342.7 KB/sec)
2017-06-09T06:34:18.690 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-09T06:34:18.699 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 111.4 KB/sec)
2017-06-09T06:34:18.704 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-09T06:34:18.711 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 242.3 KB/sec)
2017-06-09T06:34:18.717 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-09T06:34:18.726 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 196.0 KB/sec)
2017-06-09T06:34:18.733 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-09T06:34:18.746 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 204.2 KB/sec)
2017-06-09T06:34:18.751 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-09T06:34:18.765 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 1258.3 KB/sec)
2017-06-09T06:34:18.772 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-09T06:34:18.782 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 942.8 KB/sec)
2017-06-09T06:34:18.785 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-09T06:34:18.796 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
 (5 KB at 404.9 KB/sec)
2017-06-09T06:34:18.800 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
2017-06-09T06:34:18.810 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
 (3 KB at 230.6 KB/sec)
2017-06-09T06:34:18.814 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom
2017-06-09T06:34:18.827 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.11.pom

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #63

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple 
IO

--
[...truncated 1.36 MB...]
2017-06-09\T\06:29:09.770 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 65.4 KB/sec)
2017-06-09\T\06:29:09.772 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-09\T\06:29:09.797 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 92.7 KB/sec)
2017-06-09\T\06:29:09.799 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-09\T\06:29:09.824 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 79.8 KB/sec)
2017-06-09\T\06:29:09.826 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-09\T\06:29:09.861 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 56.8 KB/sec)
2017-06-09\T\06:29:09.863 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-09\T\06:29:09.888 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 168.7 KB/sec)
2017-06-09\T\06:29:09.895 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-09\T\06:29:09.920 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 241.2 KB/sec)
2017-06-09\T\06:29:09.923 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-09\T\06:29:09.948 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 126.6 KB/sec)
2017-06-09\T\06:29:09.949 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-09\T\06:29:09.975 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 964.1 KB/sec)
2017-06-09\T\06:29:09.977 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-09\T\06:29:10.002 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
549.0 KB/sec)
2017-06-09\T\06:29:10.015 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-09\T\06:29:10.040 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 185.2 KB/sec)
2017-06-09\T\06:29:10.041 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-09\T\06:29:10.066 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 56.3 KB/sec)
2017-06-09\T\06:29:10.069 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-09\T\06:29:10.095 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 123.4 KB/sec)
2017-06-09\T\06:29:10.099 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
2017-06-09\T\06:29:10.126 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
 (5 KB at 150.0 KB/sec)
2017-06-09\T\06:29:10.127 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
2017-06-09\T\06:29:10.154 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
 (32 KB at 1169.3 KB/sec)
2017-06-09\T\06:29:10.157 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.8/zookeeper-3.4.8.pom
2017-06-09\T\06:29:10.182 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.8/zookeeper-3.4.8.pom
 (4 KB at 157.2 KB/sec)
2017-06-09\T\06:29:10.183 [INFO] Do

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #63

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple 
IO

--
[...truncated 1.37 MB...]
2017-06-09\T\06:29:20.290 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
 (2 KB at 73.9 KB/sec)
2017-06-09\T\06:29:20.292 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-09\T\06:29:20.319 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 60.6 KB/sec)
2017-06-09\T\06:29:20.321 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-09\T\06:29:20.348 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 85.8 KB/sec)
2017-06-09\T\06:29:20.350 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-09\T\06:29:20.377 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 73.9 KB/sec)
2017-06-09\T\06:29:20.378 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-09\T\06:29:20.405 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 73.6 KB/sec)
2017-06-09\T\06:29:20.406 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-09\T\06:29:20.451 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 93.7 KB/sec)
2017-06-09\T\06:29:20.455 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-09\T\06:29:20.482 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 223.3 KB/sec)
2017-06-09\T\06:29:20.486 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-09\T\06:29:20.514 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 113.0 KB/sec)
2017-06-09\T\06:29:20.516 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-09\T\06:29:20.546 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 835.5 KB/sec)
2017-06-09\T\06:29:20.547 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-09\T\06:29:20.575 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
508.3 KB/sec)
2017-06-09\T\06:29:20.589 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-09\T\06:29:20.616 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 171.5 KB/sec)
2017-06-09\T\06:29:20.618 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-09\T\06:29:20.645 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 50.3 KB/sec)
2017-06-09\T\06:29:20.649 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-09\T\06:29:20.676 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 118.9 KB/sec)
2017-06-09\T\06:29:20.682 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
2017-06-09\T\06:29:20.709 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
 (5 KB at 150.0 KB/sec)
2017-06-09\T\06:29:20.710 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
2017-06-09\T\06:29:20.738 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
 (32 KB at 1127.5 KB/sec)
2017-06-09\T\06:29:20

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #63

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple 
IO

--
[...truncated 1.37 MB...]
2017-06-09\T\06:26:34.726 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
 (2 KB at 79.8 KB/sec)
2017-06-09\T\06:26:34.728 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-09\T\06:26:34.753 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 65.4 KB/sec)
2017-06-09\T\06:26:34.755 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-09\T\06:26:34.782 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 85.8 KB/sec)
2017-06-09\T\06:26:34.783 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-09\T\06:26:34.809 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 76.8 KB/sec)
2017-06-09\T\06:26:34.810 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-09\T\06:26:34.837 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 73.6 KB/sec)
2017-06-09\T\06:26:34.839 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-09\T\06:26:34.864 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 168.7 KB/sec)
2017-06-09\T\06:26:34.868 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-09\T\06:26:34.893 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 241.2 KB/sec)
2017-06-09\T\06:26:34.897 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-09\T\06:26:34.923 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 121.7 KB/sec)
2017-06-09\T\06:26:34.925 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-09\T\06:26:34.952 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 928.3 KB/sec)
2017-06-09\T\06:26:34.953 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-09\T\06:26:34.981 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
490.2 KB/sec)
2017-06-09\T\06:26:34.994 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-09\T\06:26:35.023 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 159.7 KB/sec)
2017-06-09\T\06:26:35.024 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-09\T\06:26:35.050 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 54.1 KB/sec)
2017-06-09\T\06:26:35.053 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-09\T\06:26:35.079 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 123.4 KB/sec)
2017-06-09\T\06:26:35.084 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
2017-06-09\T\06:26:35.110 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
 (5 KB at 155.8 KB/sec)
2017-06-09\T\06:26:35.112 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
2017-06-09\T\06:26:35.141 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
 (32 KB at 1052.4 KB/sec)
2017-06-09\T\06:26:3

Build failed in Jenkins: beam_PostCommit_Python_Verify #2451

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple 
IO

--
[...truncated 503.45 KB...]
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[TaggedOutput, int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /t

[GitHub] beam pull request #3328: [BEAM-2410] Remove TransportClient from ElasticSear...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3328


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2410) Remove TransportClient from ElasticSearchIO to decouple IO and ES versions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043960#comment-16043960
 ] 

ASF GitHub Bot commented on BEAM-2410:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3328


> Remove TransportClient from ElasticSearchIO to decouple IO and ES versions
> --
>
> Key: BEAM-2410
> URL: https://issues.apache.org/jira/browse/BEAM-2410
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
> Fix For: 2.1.0
>
>
> Elasticsearch IO uses TransportClient in tests (UT and IT) but uses 
> RestClient in the production code. TransportClient version is coupled with 
> Elasticsearch server version. Use RestClient everywhere to lower the coupling.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-2410) Remove TransportClient from ElasticSearchIO to decouple IO and ES versions

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-2410.

   Resolution: Fixed
Fix Version/s: 2.1.0

> Remove TransportClient from ElasticSearchIO to decouple IO and ES versions
> --
>
> Key: BEAM-2410
> URL: https://issues.apache.org/jira/browse/BEAM-2410
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
> Fix For: 2.1.0
>
>
> Elasticsearch IO uses TransportClient in tests (UT and IT) but uses 
> RestClient in the production code. TransportClient version is coupled with 
> Elasticsearch server version. Use RestClient everywhere to lower the coupling.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: [BEAM-2410] This closes #3328

[jbonofre]

[BEAM-2410] This closes #3328


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1c6861f2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1c6861f2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1c6861f2

Branch: refs/heads/master
Commit: 1c6861f22249dadacb34994564d33b14f1d57d3d
Parents: e980ae9 7caea7a
Author: Jean-Baptiste Onofré 
Authored: Fri Jun 9 07:31:27 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Fri Jun 9 07:31:27 2017 +0200

--
 .../sdk/io/common/IOTestPipelineOptions.java|  6 +-
 .../sdk/io/elasticsearch/ElasticsearchIO.java   |  4 +-
 .../elasticsearch/ElasticSearchIOTestUtils.java | 81 +++-
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java | 14 ++--
 .../io/elasticsearch/ElasticsearchIOTest.java   | 36 +
 .../elasticsearch/ElasticsearchTestDataSet.java | 37 -
 6 files changed, 87 insertions(+), 91 deletions(-)
--

[1/2] beam git commit: [BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple IO and ES server versions

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/master e980ae921 -> 1c6861f22


[BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple IO and ES 
server versions


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7caea7a8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7caea7a8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7caea7a8

Branch: refs/heads/master
Commit: 7caea7a845eff072a647baf69b9b004db4523652
Parents: e980ae9
Author: Etienne Chauchot 
Authored: Mon Jun 5 16:21:58 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Fri Jun 9 07:31:06 2017 +0200

--
 .../sdk/io/common/IOTestPipelineOptions.java|  6 +-
 .../sdk/io/elasticsearch/ElasticsearchIO.java   |  4 +-
 .../elasticsearch/ElasticSearchIOTestUtils.java | 81 +++-
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java | 14 ++--
 .../io/elasticsearch/ElasticsearchIOTest.java   | 36 +
 .../elasticsearch/ElasticsearchTestDataSet.java | 37 -
 6 files changed, 87 insertions(+), 91 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7caea7a8/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
--
diff --git 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
index 387fd22..25ab929 100644
--- 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
+++ 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
@@ -71,11 +71,7 @@ public interface IOTestPipelineOptions extends 
TestPipelineOptions {
   Integer getElasticsearchHttpPort();
   void setElasticsearchHttpPort(Integer value);
 
-  @Description("Tcp port for elasticsearch server")
-  @Default.Integer(9300)
-  Integer getElasticsearchTcpPort();
-  void setElasticsearchTcpPort(Integer value);
-
+  /* Cassandra */
   @Description("Host for Cassandra server (host name/ip address)")
   @Default.String("cassandra-host")
   String getCassandraHost();

http://git-wip-us.apache.org/repos/asf/beam/blob/7caea7a8/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
--
diff --git 
a/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
 
b/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
index f6ceef2..e3965dc 100644
--- 
a/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
+++ 
b/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
@@ -139,7 +139,7 @@ public class ElasticsearchIO {
 
   private static final ObjectMapper mapper = new ObjectMapper();
 
-  private static JsonNode parseResponse(Response response) throws IOException {
+  static JsonNode parseResponse(Response response) throws IOException {
 return mapper.readValue(response.getEntity().getContent(), JsonNode.class);
   }
 
@@ -264,7 +264,7 @@ public class ElasticsearchIO {
   builder.addIfNotNull(DisplayData.item("username", getUsername()));
 }
 
-private RestClient createClient() throws MalformedURLException {
+RestClient createClient() throws MalformedURLException {
   HttpHost[] hosts = new HttpHost[getAddresses().size()];
   int i = 0;
   for (String address : getAddresses()) {

http://git-wip-us.apache.org/repos/asf/beam/blob/7caea7a8/sdks/java/io/elasticsearch/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
--
diff --git 
a/sdks/java/io/elasticsearch/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
 
b/sdks/java/io/elasticsearch/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
index b0d161f..203963d 100644
--- 
a/sdks/java/io/elasticsearch/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
+++ 
b/sdks/java/io/elasticsearch/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
@@ -17,19 +17,17 @@
  */
 package org.apache.beam.sdk.io.elasticsearch;
 
+import com.fasterxml.jackson.databind.JsonNode;
 import java.io.IOException;
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
-import 
org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsRequest;
-import 
org.elasticsearch.action.admin.indices.exists.indices.IndicesExistsResponse;
-import org.elasticsearch.action

Build failed in Jenkins: beam_PostCommit_Python_Verify #2450

[Apache Jenkins Server]

See 


--
[...truncated 503.14 KB...]
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[TaggedOutput, int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.t

[jira] [Commented] (BEAM-2326) Verbose INFO logging with stateful DoFns and Dataflow

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043784#comment-16043784
 ] 

Kenneth Knowles commented on BEAM-2326:
---

This was fixed right when it was reported, so should be g2g for the released 
version.

> Verbose INFO logging with stateful DoFns and Dataflow 
> --
>
> Key: BEAM-2326
> URL: https://issues.apache.org/jira/browse/BEAM-2326
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 0.6.0
>Reporter: peay
>Assignee: Kenneth Knowles
>  Labels: log
> Fix For: 2.0.0
>
>
> I am seeing a lot of INFO level logging:
> {code}
>  jsonPayload: {
>   logger: 
> "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" 
>   message: "Processing timers for key {} for stateful DoFn"
>  }
>  jsonPayload: {
>   message: "Processing key KV{one of my keys} for stateful DoFn"
>   logger: 
> "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn"  
>   
> }
> {code}
> out of one of my stateful DoFn. There is one such group of logs for each key 
> I process, which leads to a very large amount of logs and possibly to a 
> significant slowdown.
> Also, not sure if the {{Processing timers}} log message is missing some 
> string interpolation or if the empty key is on purpose.
> At any rate, this seems more like something for {{DEBUG}} than {{INFO}} given 
> the large volume.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-2326) Verbose INFO logging with stateful DoFns and Dataflow

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2326.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

> Verbose INFO logging with stateful DoFns and Dataflow 
> --
>
> Key: BEAM-2326
> URL: https://issues.apache.org/jira/browse/BEAM-2326
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 0.6.0
>Reporter: peay
>Assignee: Kenneth Knowles
>  Labels: log
> Fix For: 2.0.0
>
>
> I am seeing a lot of INFO level logging:
> {code}
>  jsonPayload: {
>   logger: 
> "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" 
>   message: "Processing timers for key {} for stateful DoFn"
>  }
>  jsonPayload: {
>   message: "Processing key KV{one of my keys} for stateful DoFn"
>   logger: 
> "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn"  
>   
> }
> {code}
> out of one of my stateful DoFn. There is one such group of logs for each key 
> I process, which leads to a very large amount of logs and possibly to a 
> significant slowdown.
> Also, not sure if the {{Processing timers}} log message is missing some 
> string interpolation or if the empty key is on purpose.
> At any rate, this seems more like something for {{DEBUG}} than {{INFO}} given 
> the large volume.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #110

[Apache Jenkins Server]

See 


Changes:

[klk] Add Runner API oriented PTransformMatchers for DirectRunner overrides

[klk] DirectRunner override matchers using Runner API

[robertwb] Use inner module for non-public coders.

[kirpichov] Implements HCatalogIO

[kirpichov] Slight debuggability improvements in BigtableIO

[chamikara] Move Runner API protos to portability/runners/api

--
[...truncated 2.65 MB...]
2017-06-09T00:38:16.342 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 938.0 KB/sec)
2017-06-09T00:38:16.346 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-09T00:38:16.355 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1426.0 KB/sec)
2017-06-09T00:38:16.359 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-09T00:38:16.368 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 241.4 KB/sec)
2017-06-09T00:38:16.373 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-09T00:38:16.381 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 196.7 KB/sec)
2017-06-09T00:38:16.385 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-09T00:38:16.393 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 656.2 KB/sec)
2017-06-09T00:38:16.396 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-09T00:38:16.405 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 277.2 KB/sec)
2017-06-09T00:38:16.409 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-09T00:38:16.417 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 428.3 KB/sec)
2017-06-09T00:38:16.422 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-09T00:38:16.430 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 125.4 KB/sec)
2017-06-09T00:38:16.436 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-09T00:38:16.445 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 188.5 KB/sec)
2017-06-09T00:38:16.450 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-09T00:38:16.459 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 196.0 KB/sec)
2017-06-09T00:38:16.464 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-09T00:38:16.472 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 331.8 KB/sec)
2017-06-09T00:38:16.477 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-09T00:38:16.486 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 1957.4 KB/sec)
2017-06-09T00:38:16.490 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-09T00:38:16.500 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 942.8 KB/sec)
2017-06-09T00:38:16.504 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-09T00:38:16.513 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
 (5 KB at 494.9 KB/sec)
2017-06-09T00:38:16.517 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
2017-06-09T00:38:16.526 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
 (3 KB at 256.2 KB/sec

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #62

[Apache Jenkins Server]

See 


Changes:

[klk] Add Runner API oriented PTransformMatchers for DirectRunner overrides

[klk] DirectRunner override matchers using Runner API

[robertwb] Use inner module for non-public coders.

[kirpichov] Implements HCatalogIO

[kirpichov] Slight debuggability improvements in BigtableIO

[chamikara] Move Runner API protos to portability/runners/api

--
[...truncated 1.37 MB...]
2017-06-09\T\00:35:11.376 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
 (2 KB at 73.9 KB/sec)
2017-06-09\T\00:35:11.377 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-09\T\00:35:11.404 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 60.6 KB/sec)
2017-06-09\T\00:35:11.406 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-09\T\00:35:11.432 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 89.1 KB/sec)
2017-06-09\T\00:35:11.434 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-09\T\00:35:11.460 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 76.8 KB/sec)
2017-06-09\T\00:35:11.462 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-09\T\00:35:11.488 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 76.4 KB/sec)
2017-06-09\T\00:35:11.489 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-09\T\00:35:11.516 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 156.2 KB/sec)
2017-06-09\T\00:35:11.521 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-09\T\00:35:11.547 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 231.9 KB/sec)
2017-06-09\T\00:35:11.551 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-09\T\00:35:11.578 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 117.2 KB/sec)
2017-06-09\T\00:35:11.580 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-09\T\00:35:11.606 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 964.1 KB/sec)
2017-06-09\T\00:35:11.608 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-09\T\00:35:11.633 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
549.0 KB/sec)
2017-06-09\T\00:35:11.646 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-09\T\00:35:11.673 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 171.5 KB/sec)
2017-06-09\T\00:35:11.674 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-09\T\00:35:11.700 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 54.1 KB/sec)
2017-06-09\T\00:35:11.704 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-09\T\00:35:11.730 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 123.4 KB/sec)
2017-06-09\T\00:35:11.735 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
2017-06-09\T\00:35:11.761 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
 (5 KB at 155.8 KB/sec)
2017-06-09\T\00:35:11.771 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #62

[Apache Jenkins Server]

See 


Changes:

[klk] Add Runner API oriented PTransformMatchers for DirectRunner overrides

[klk] DirectRunner override matchers using Runner API

[robertwb] Use inner module for non-public coders.

[kirpichov] Implements HCatalogIO

[kirpichov] Slight debuggability improvements in BigtableIO

[chamikara] Move Runner API protos to portability/runners/api

--
[...truncated 1.37 MB...]
2017-06-09\T\00:34:43.413 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
 (2 KB at 68.8 KB/sec)
2017-06-09\T\00:34:43.415 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-09\T\00:34:43.445 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 54.5 KB/sec)
2017-06-09\T\00:34:43.446 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-09\T\00:34:43.472 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 89.1 KB/sec)
2017-06-09\T\00:34:43.474 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-09\T\00:34:43.506 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 62.4 KB/sec)
2017-06-09\T\00:34:43.508 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-09\T\00:34:43.535 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 73.6 KB/sec)
2017-06-09\T\00:34:43.537 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-09\T\00:34:43.563 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 162.2 KB/sec)
2017-06-09\T\00:34:43.569 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-09\T\00:34:43.595 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 231.9 KB/sec)
2017-06-09\T\00:34:43.599 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-09\T\00:34:43.626 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 117.2 KB/sec)
2017-06-09\T\00:34:43.627 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-09\T\00:34:43.655 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 895.2 KB/sec)
2017-06-09\T\00:34:43.657 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-09\T\00:34:43.683 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
527.9 KB/sec)
2017-06-09\T\00:34:43.695 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-09\T\00:34:43.721 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 178.1 KB/sec)
2017-06-09\T\00:34:43.723 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-09\T\00:34:43.749 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 56.3 KB/sec)
2017-06-09\T\00:34:43.752 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-09\T\00:34:43.778 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 123.4 KB/sec)
2017-06-09\T\00:34:43.783 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
2017-06-09\T\00:34:43.809 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
 (5 KB at 155.8 KB/sec)
2017-06-09\T\00:34:43.810 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #62

[Apache Jenkins Server]

See 


Changes:

[klk] Add Runner API oriented PTransformMatchers for DirectRunner overrides

[klk] DirectRunner override matchers using Runner API

[robertwb] Use inner module for non-public coders.

[kirpichov] Implements HCatalogIO

[kirpichov] Slight debuggability improvements in BigtableIO

[chamikara] Move Runner API protos to portability/runners/api

--
[...truncated 1.36 MB...]
2017-06-09\T\00:29:55.520 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 65.4 KB/sec)
2017-06-09\T\00:29:55.522 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-09\T\00:29:55.547 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 92.7 KB/sec)
2017-06-09\T\00:29:55.549 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-09\T\00:29:55.575 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 76.8 KB/sec)
2017-06-09\T\00:29:55.576 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-09\T\00:29:55.602 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 76.4 KB/sec)
2017-06-09\T\00:29:55.603 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-09\T\00:29:55.630 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 156.2 KB/sec)
2017-06-09\T\00:29:55.635 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-09\T\00:29:55.662 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 223.3 KB/sec)
2017-06-09\T\00:29:55.666 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-09\T\00:29:55.692 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 121.7 KB/sec)
2017-06-09\T\00:29:55.694 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-09\T\00:29:55.722 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 895.2 KB/sec)
2017-06-09\T\00:29:55.725 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-09\T\00:29:55.751 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
527.9 KB/sec)
2017-06-09\T\00:29:55.766 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-09\T\00:29:55.792 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 178.1 KB/sec)
2017-06-09\T\00:29:55.794 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-09\T\00:29:55.819 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 56.3 KB/sec)
2017-06-09\T\00:29:55.823 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-09\T\00:29:55.849 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 123.4 KB/sec)
2017-06-09\T\00:29:55.854 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
2017-06-09\T\00:29:55.880 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
 (5 KB at 155.8 KB/sec)
2017-06-09\T\00:29:55.882 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
2017-06-09\T\00:29:55.910 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
 (32 KB at 1127.5 KB/sec)
2017-06-09\T\00:29:55.913 [INFO] Downloading: 
https://repo.maven.apache.org/mave

[GitHub] beam pull request #3335: Tests for reading windowed side input from resumed ...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3335

Tests for reading windowed side input from resumed SDF call

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam sdf-cp-side

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3335.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3335


commit 970cff1c06289ebc8519fb69a72cb77546c7af70
Author: Eugene Kirpichov 
Date:   2017-06-08T23:54:12Z

Tests for reading windowed side input from resumed SDF call




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Python_Verify #2449

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Slight debuggability improvements in BigtableIO

[chamikara] Move Runner API protos to portability/runners/api

--
[...truncated 502.92 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[TaggedOutput, int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' %

[jira] [Commented] (BEAM-2333) Rehydrate Pipeline from Runner API proto

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043628#comment-16043628
 ] 

ASF GitHub Bot commented on BEAM-2333:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3334

[BEAM-2333] Go to proto and back before running a pipeline in Java 
DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

Not ready for review. Opening this to get ongoing Jenkins feedback.

 - this includes #3330 adjusting DirectRunner overrides
 - this requires #3329 to get additional inputs correctly set up I think

Notably, we don't need to port all evaluators because replacements occur 
after deserialization, so things like `GroupByKey` are already mapped to 
`DirectGroupByKey` so its evaluator remains unchanged.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam Pipeline-rehydrate

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3334.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3334


commit 6b515e560aed423ea3e934046fa2098a21fe7ddb
Author: Kenneth Knowles 
Date:   2017-05-26T21:26:55Z

Port ViewOverrideFactory to SDK-agnostic APIs

commit be1a4e84ea08d1f950e5035624f7147a61376506
Author: Kenneth Knowles 
Date:   2017-05-26T21:27:23Z

Port DirectGroupByKey to SDK-agnostic APIs

commit 876cfa55502f9980066032ba9d6c458d41437bb2
Author: Kenneth Knowles 
Date:   2017-05-26T23:07:45Z

Port DirectRunner WriteFiles override to SDK-agnostic APIs

commit 183db6b1df0a2935a5323d6106d9e64096898de2
Author: Kenneth Knowles 
Date:   2017-06-08T20:39:32Z

Port DirectRunner TestStream override to SDK-agnostic APIs

commit c9efad0247b96ad89c4a32723742974bb3224c11
Author: Kenneth Knowles 
Date:   2017-06-08T20:44:52Z

Port DirectRunner ParDo overrides to SDK-agnostic APIs

commit 9b1c5ef125a9ab4929e4167d32a372a54cce7eb1
Author: Kenneth Knowles 
Date:   2017-06-08T20:46:18Z

Add more utilities to ParDoTranslation

commit 88c8df8228e40b4563bf15619f6a3a8f8d90f2f2
Author: Kenneth Knowles 
Date:   2017-06-08T21:27:02Z

Enable SplittableParDo on rehydrated ParDo transform

commit 7a206389a30449d97d3b40b30ef7d3d7e812971b
Author: Kenneth Knowles 
Date:   2017-06-08T23:21:57Z

fixup! Add more utilities to ParDoTranslation

commit e4191f9733d6185c43263516f831854919e53832
Author: Kenneth Knowles 
Date:   2017-05-26T18:22:50Z

Add Pipeline rehydration from proto

commit c106f1600d5dd5596f3e64fd35c699045ead4cad
Author: Kenneth Knowles 
Date:   2017-05-26T18:23:05Z

Dehydrate then rehydrate Pipeline before DirectRunner.run()

commit c76aafca18008e34dd16d292e02358326d888938
Author: Kenneth Knowles 
Date:   2017-06-07T20:58:11Z

Use dehydration-insensitive APIs in WindowEvaluatorFactory

commit 328790f280bf633801cd36aa55e3b3b2e1220666
Author: Kenneth Knowles 
Date:   2017-06-07T21:35:09Z

Use dehydration-insensitive APIs in ParDoEvaluatorFactory




> Rehydrate Pipeline from Runner API proto
> 
>
> Key: BEAM-2333
> URL: https://issues.apache.org/jira/browse/BEAM-2333
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3334: [BEAM-2333] Go to proto and back before running a p...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3334

[BEAM-2333] Go to proto and back before running a pipeline in Java 
DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

Not ready for review. Opening this to get ongoing Jenkins feedback.

 - this includes #3330 adjusting DirectRunner overrides
 - this requires #3329 to get additional inputs correctly set up I think

Notably, we don't need to port all evaluators because replacements occur 
after deserialization, so things like `GroupByKey` are already mapped to 
`DirectGroupByKey` so its evaluator remains unchanged.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam Pipeline-rehydrate

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3334.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3334


commit 6b515e560aed423ea3e934046fa2098a21fe7ddb
Author: Kenneth Knowles 
Date:   2017-05-26T21:26:55Z

Port ViewOverrideFactory to SDK-agnostic APIs

commit be1a4e84ea08d1f950e5035624f7147a61376506
Author: Kenneth Knowles 
Date:   2017-05-26T21:27:23Z

Port DirectGroupByKey to SDK-agnostic APIs

commit 876cfa55502f9980066032ba9d6c458d41437bb2
Author: Kenneth Knowles 
Date:   2017-05-26T23:07:45Z

Port DirectRunner WriteFiles override to SDK-agnostic APIs

commit 183db6b1df0a2935a5323d6106d9e64096898de2
Author: Kenneth Knowles 
Date:   2017-06-08T20:39:32Z

Port DirectRunner TestStream override to SDK-agnostic APIs

commit c9efad0247b96ad89c4a32723742974bb3224c11
Author: Kenneth Knowles 
Date:   2017-06-08T20:44:52Z

Port DirectRunner ParDo overrides to SDK-agnostic APIs

commit 9b1c5ef125a9ab4929e4167d32a372a54cce7eb1
Author: Kenneth Knowles 
Date:   2017-06-08T20:46:18Z

Add more utilities to ParDoTranslation

commit 88c8df8228e40b4563bf15619f6a3a8f8d90f2f2
Author: Kenneth Knowles 
Date:   2017-06-08T21:27:02Z

Enable SplittableParDo on rehydrated ParDo transform

commit 7a206389a30449d97d3b40b30ef7d3d7e812971b
Author: Kenneth Knowles 
Date:   2017-06-08T23:21:57Z

fixup! Add more utilities to ParDoTranslation

commit e4191f9733d6185c43263516f831854919e53832
Author: Kenneth Knowles 
Date:   2017-05-26T18:22:50Z

Add Pipeline rehydration from proto

commit c106f1600d5dd5596f3e64fd35c699045ead4cad
Author: Kenneth Knowles 
Date:   2017-05-26T18:23:05Z

Dehydrate then rehydrate Pipeline before DirectRunner.run()

commit c76aafca18008e34dd16d292e02358326d888938
Author: Kenneth Knowles 
Date:   2017-06-07T20:58:11Z

Use dehydration-insensitive APIs in WindowEvaluatorFactory

commit 328790f280bf633801cd36aa55e3b3b2e1220666
Author: Kenneth Knowles 
Date:   2017-06-07T21:35:09Z

Use dehydration-insensitive APIs in ParDoEvaluatorFactory




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-685) TextIO path pattern error on Windows OS

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-685.

   Resolution: Duplicate
 Assignee: Luke Cwik
Fix Version/s: Not applicable

Since Java TextIO is now using FileSystems, this a duplicate of BEAM-1045

> TextIO path pattern error on Windows OS
> ---
>
> Key: BEAM-685
> URL: https://issues.apache.org/jira/browse/BEAM-685
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Luke Cwik
> Fix For: Not applicable
>
>
> It seems that the pattern {{some-output-file-*}} is rejected on Windows, or 
> at least by nio's Windows path code. This could particularly affect Windows 
> users getting started with the DirectRunner.
> Apologies for lack of steps to certainly repro - this is a quick write up of 
> an issue a user had in person.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2448

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Implements HCatalogIO

--
[...truncated 502.46 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[TaggedOutput, int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that shou

[jira] [Created] (BEAM-2428) Enforce all inputs and outputs are properly declared in Transform Hierarchy

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-2428:
-

 Summary: Enforce all inputs and outputs are properly declared in 
Transform Hierarchy
 Key: BEAM-2428
 URL: https://issues.apache.org/jira/browse/BEAM-2428
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Thomas Groh


Within a composite transform, each child node may only consume values declared 
as the input to the composite and values produced as output by other direct 
children of the composite node. This can be enforced automatically via pipeline 
traversal, or by the {{TransformHierarchy}} directly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3315: Move Runner API protos to portability/runners/api

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3315


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3315

[chamikara]

This closes #3315


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e980ae92
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e980ae92
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e980ae92

Branch: refs/heads/master
Commit: e980ae921f48b180ffb7fd7dba7eea10ef0f304a
Parents: bc2f97c 7689e43
Author: chamik...@google.com 
Authored: Thu Jun 8 15:30:01 2017 -0700
Committer: chamik...@google.com 
Committed: Thu Jun 8 15:30:01 2017 -0700

--
 .gitignore  |  2 +-
 sdks/python/apache_beam/coders/coders.py|  2 +-
 sdks/python/apache_beam/pipeline.py |  4 ++--
 sdks/python/apache_beam/portability/__init__.py | 18 +
 .../apache_beam/portability/runners/__init__.py | 18 +
 .../portability/runners/api/__init__.py | 21 
 sdks/python/apache_beam/pvalue.py   |  2 +-
 sdks/python/apache_beam/runners/api/__init__.py | 21 
 .../runners/dataflow/dataflow_runner.py |  4 ++--
 .../apache_beam/runners/pipeline_context.py |  2 +-
 .../runners/portability/fn_api_runner.py|  2 +-
 .../apache_beam/runners/worker/data_plane.py|  2 +-
 .../runners/worker/data_plane_test.py   |  2 +-
 .../apache_beam/runners/worker/log_handler.py   |  2 +-
 .../runners/worker/log_handler_test.py  |  2 +-
 .../apache_beam/runners/worker/sdk_worker.py|  2 +-
 .../runners/worker/sdk_worker_main.py   |  2 +-
 .../runners/worker/sdk_worker_test.py   |  2 +-
 sdks/python/apache_beam/transforms/core.py  |  2 +-
 .../python/apache_beam/transforms/ptransform.py |  2 +-
 sdks/python/apache_beam/transforms/trigger.py   |  2 +-
 sdks/python/apache_beam/transforms/window.py|  4 ++--
 sdks/python/apache_beam/utils/urns.py   |  2 +-
 sdks/python/gen_protos.py   |  2 +-
 sdks/python/run_pylint.sh   |  2 +-
 25 files changed, 81 insertions(+), 45 deletions(-)
--

[1/2] beam git commit: Move Runner API protos to portability/runners/api

[chamikara]

Repository: beam
Updated Branches:
  refs/heads/master bc2f97c77 -> e980ae921


Move Runner API protos to portability/runners/api

This fixes a circular import issue between transforms/ and runners/


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7689e43a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7689e43a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7689e43a

Branch: refs/heads/master
Commit: 7689e43ac4b88c85962cea14d65d788ad27dbe93
Parents: bc2f97c
Author: Charles Chen 
Authored: Wed Jun 7 16:08:43 2017 -0700
Committer: chamik...@google.com 
Committed: Thu Jun 8 15:29:39 2017 -0700

--
 .gitignore  |  2 +-
 sdks/python/apache_beam/coders/coders.py|  2 +-
 sdks/python/apache_beam/pipeline.py |  4 ++--
 sdks/python/apache_beam/portability/__init__.py | 18 +
 .../apache_beam/portability/runners/__init__.py | 18 +
 .../portability/runners/api/__init__.py | 21 
 sdks/python/apache_beam/pvalue.py   |  2 +-
 sdks/python/apache_beam/runners/api/__init__.py | 21 
 .../runners/dataflow/dataflow_runner.py |  4 ++--
 .../apache_beam/runners/pipeline_context.py |  2 +-
 .../runners/portability/fn_api_runner.py|  2 +-
 .../apache_beam/runners/worker/data_plane.py|  2 +-
 .../runners/worker/data_plane_test.py   |  2 +-
 .../apache_beam/runners/worker/log_handler.py   |  2 +-
 .../runners/worker/log_handler_test.py  |  2 +-
 .../apache_beam/runners/worker/sdk_worker.py|  2 +-
 .../runners/worker/sdk_worker_main.py   |  2 +-
 .../runners/worker/sdk_worker_test.py   |  2 +-
 sdks/python/apache_beam/transforms/core.py  |  2 +-
 .../python/apache_beam/transforms/ptransform.py |  2 +-
 sdks/python/apache_beam/transforms/trigger.py   |  2 +-
 sdks/python/apache_beam/transforms/window.py|  4 ++--
 sdks/python/apache_beam/utils/urns.py   |  2 +-
 sdks/python/gen_protos.py   |  2 +-
 sdks/python/run_pylint.sh   |  2 +-
 25 files changed, 81 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7689e43a/.gitignore
--
diff --git a/.gitignore b/.gitignore
index bd419a7..631d7f3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -25,7 +25,7 @@ sdks/python/**/*.egg
 sdks/python/LICENSE
 sdks/python/NOTICE
 sdks/python/README.md
-sdks/python/apache_beam/runners/api/*pb2*.*
+sdks/python/apache_beam/portability/runners/api/*pb2*.*
 
 # Ignore IntelliJ files.
 .idea/

http://git-wip-us.apache.org/repos/asf/beam/blob/7689e43a/sdks/python/apache_beam/coders/coders.py
--
diff --git a/sdks/python/apache_beam/coders/coders.py 
b/sdks/python/apache_beam/coders/coders.py
index f3e0b43..1be1f3c 100644
--- a/sdks/python/apache_beam/coders/coders.py
+++ b/sdks/python/apache_beam/coders/coders.py
@@ -25,6 +25,7 @@ import cPickle as pickle
 import google.protobuf
 
 from apache_beam.coders import coder_impl
+from apache_beam.portability.runners.api import beam_runner_api_pb2
 from apache_beam.utils import urns
 from apache_beam.utils import proto_utils
 
@@ -205,7 +206,6 @@ class Coder(object):
 """For internal use only; no backwards-compatibility guarantees.
 """
 # TODO(BEAM-115): Use specialized URNs and components.
-from apache_beam.runners.api import beam_runner_api_pb2
 return beam_runner_api_pb2.Coder(
 spec=beam_runner_api_pb2.SdkFunctionSpec(
 spec=beam_runner_api_pb2.FunctionSpec(

http://git-wip-us.apache.org/repos/asf/beam/blob/7689e43a/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index 9093abf..cea7215 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -339,7 +339,7 @@ class Pipeline(object):
   def to_runner_api(self):
 """For internal use only; no backwards-compatibility guarantees."""
 from apache_beam.runners import pipeline_context
-from apache_beam.runners.api import beam_runner_api_pb2
+from apache_beam.portability.runners.api import beam_runner_api_pb2
 context = pipeline_context.PipelineContext()
 # Mutates context; placing inline would force dependence on
 # argument evaluation order.
@@ -525,7 +525,7 @@ class AppliedPTransform(object):
 if isinstance(output, pvalue.PCollection)}
 
   def to_runner_api(self, context):
-from apache_beam.runners.api import beam_runner_api_pb2
+from apache_beam.portability.runners

[GitHub] beam pull request #3331: Slight debuggability improvements in BigtableIO

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3331


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3331: Slight debuggability improvements in BigtableIO

[jkff]

This closes #3331: Slight debuggability improvements in BigtableIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bc2f97c7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bc2f97c7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bc2f97c7

Branch: refs/heads/master
Commit: bc2f97c7769bb9b81bc13a181c6e68e5fe219726
Parents: 911bfbd 82a6cb6
Author: Eugene Kirpichov 
Authored: Thu Jun 8 14:57:55 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Jun 8 14:57:55 2017 -0700

--
 .../beam/sdk/io/range/ByteKeyRangeTracker.java  | 22 
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java|  2 +-
 2 files changed, 14 insertions(+), 10 deletions(-)
--

[1/2] beam git commit: Slight debuggability improvements in BigtableIO

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 911bfbdef -> bc2f97c77


Slight debuggability improvements in BigtableIO

- ByteKeyRangeTracker.splitAtPosition logs the "insane" case first.
- BigtableIO logs the split position at INFO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/82a6cb61
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/82a6cb61
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/82a6cb61

Branch: refs/heads/master
Commit: 82a6cb6104c0bb179832dabf4433c183743ea983
Parents: 911bfbd
Author: Eugene Kirpichov 
Authored: Thu Jun 8 14:51:15 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Jun 8 14:51:15 2017 -0700

--
 .../beam/sdk/io/range/ByteKeyRangeTracker.java  | 22 
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java|  2 +-
 2 files changed, 14 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/82a6cb61/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
index 99717a4..b889ec7 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
@@ -71,6 +71,10 @@ public final class ByteKeyRangeTracker implements 
RangeTracker {
 "Trying to return record which is before the last-returned record");
 
 if (position == null) {
+  LOG.info(
+  "Adjusting range start from {} to {} as position of first returned 
record",
+  range.getStartKey(),
+  recordStart);
   range = range.withStartKey(recordStart);
 }
 position = recordStart;
@@ -87,6 +91,15 @@ public final class ByteKeyRangeTracker implements 
RangeTracker {
 
   @Override
   public synchronized boolean trySplitAtPosition(ByteKey splitPosition) {
+// Sanity check.
+if (!range.containsKey(splitPosition)) {
+  LOG.warn(
+  "{}: Rejecting split request at {} because it is not within the 
range.",
+  this,
+  splitPosition);
+  return false;
+}
+
 // Unstarted.
 if (position == null) {
   LOG.warn(
@@ -106,15 +119,6 @@ public final class ByteKeyRangeTracker implements 
RangeTracker {
   return false;
 }
 
-// Sanity check.
-if (!range.containsKey(splitPosition)) {
-  LOG.warn(
-  "{}: Rejecting split request at {} because it is not within the 
range.",
-  this,
-  splitPosition);
-  return false;
-}
-
 range = range.withEndKey(splitPosition);
 return true;
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/82a6cb61/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 1692cda..62679bb 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -1027,7 +1027,7 @@ public class BigtableIO {
 "{}: Failed to interpolate key for fraction {}.", 
rangeTracker.getRange(), fraction, e);
 return null;
   }
-  LOG.debug(
+  LOG.info(
   "Proposing to split {} at fraction {} (key {})", rangeTracker, 
fraction, splitKey);
   BigtableSource primary;
   BigtableSource residual;

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4061

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2427) ParDo should only accept DoFns

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-2427:
-

 Summary: ParDo should only accept DoFns
 Key: BEAM-2427
 URL: https://issues.apache.org/jira/browse/BEAM-2427
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay


for version 3.0

Remove support for accepting user fns directly as ParDo input.

Remove this function: 
https://github.com/apache/beam/blob/911bfbdef62a7bb750426da9cc59537c4ffd9e1f/sdks/python/apache_beam/transforms/core.py#L639

Before version 3.0 the above function could be deprecated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2426) Remove imports from runner/init

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-2426:
-

 Summary: Remove imports from runner/init
 Key: BEAM-2426
 URL: https://issues.apache.org/jira/browse/BEAM-2426
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay


for Version 3.0

Remove imports from runner/__init__, they cause circular dependencies. In 
general reduce init imports in the sdk.

https://github.com/apache/beam/blob/911bfbdef62a7bb750426da9cc59537c4ffd9e1f/sdks/python/apache_beam/runners/__init__.py



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1630) Add Splittable DoFn to Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043527#comment-16043527
 ] 

ASF GitHub Bot commented on BEAM-1630:
--

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/

[BEAM-1630] Adds ability to dynamically replace PTransforms during runtime.

Adds two new interfaces, PTransformMatcher and PTransformOverride.

Currently only supports replacements where input and output types are an 
exact match (we have to address complexities due to type hints before 
supporting replacements with different types).

This can be used to dynamically update a populated pipeline at runtime. 
Each runner can configure it's own overrides.

This will be used by SplittableDoFn where matching ParDo transforms will be 
dynamically replaced by SplittableParDo.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam 
sdf_direct_runner_ptransform_override

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #






> Add Splittable DoFn to Python SDK
> -
>
> Key: BEAM-1630
> URL: https://issues.apache.org/jira/browse/BEAM-1630
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Splittable DoFn [1] is currently being implemented for Java SDK [2]. We 
> should add this to Python SDK as well.
> Following document proposes an API for this.
> https://docs.google.com/document/d/1h_zprJrOilivK2xfvl4L42vaX4DMYGfH1YDmi-s_ozM/edit?usp=sharing
> [1] https://s.apache.org/splittable-do-fn
> [2] 
> https://lists.apache.org/thread.html/0ce61ac162460a149d5c93cdface37cc383f8030fe86ca09e5699b18@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3333: [BEAM-1630] Adds ability to dynamically replace PTr...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/

[BEAM-1630] Adds ability to dynamically replace PTransforms during runtime.

Adds two new interfaces, PTransformMatcher and PTransformOverride.

Currently only supports replacements where input and output types are an 
exact match (we have to address complexities due to type hints before 
supporting replacements with different types).

This can be used to dynamically update a populated pipeline at runtime. 
Each runner can configure it's own overrides.

This will be used by SplittableDoFn where matching ParDo transforms will be 
dynamically replaced by SplittableParDo.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam 
sdf_direct_runner_ptransform_override

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3332: Use beam.Map rather than beam.ParDo for PubSub enco...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3332

Use beam.Map rather than beam.ParDo for PubSub encoding.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam pubsub-encode

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3332.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3332


commit 24fe39660d6931346839c181bca89b5d8fb11e91
Author: Robert Bradshaw 
Date:   2017-06-08T22:01:53Z

Use beam.Map rather than beam.ParDo for PubSub encoding.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043517#comment-16043517
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles reopened a pull request:

https://github.com/apache/beam/pull/3330

[BEAM-2371] Port DirectRunner overrides to SDK-agnostic APIs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh 

This is peeled off my branch on the DirectRunner, which is now on switching 
up the evaluator factories to support dehydrated/rehydrated pipelines. That 
thread is blocked on getting all the inputs recorded but this little bit can go 
in sooner to avoid a mega PR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-overrides

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3330


commit 9f380c61065632c959d135a0f3c6a6ae977c699e
Author: Kenneth Knowles 
Date:   2017-05-26T21:26:55Z

Port ViewOverrideFactory to SDK-agnostic APIs

commit 3cb6237085065cf7e9bd9dcf92953e409d422cd5
Author: Kenneth Knowles 
Date:   2017-05-26T21:27:23Z

Port DirectGroupByKey to SDK-agnostic APIs

commit d88b058aba3f7e5ca8a92f5e9fc61aa5b252f3f5
Author: Kenneth Knowles 
Date:   2017-05-26T23:07:45Z

Port DirectRunner WriteFiles override to SDK-agnostic APIs

commit 22b06906159b1e74ee68b9a57073126129264885
Author: Kenneth Knowles 
Date:   2017-06-08T20:39:32Z

Port DirectRunner TestStream override to SDK-agnostic APIs

commit 8b4630d4a97638e902dc5db8bcd79f65b366e558
Author: Kenneth Knowles 
Date:   2017-06-08T20:46:18Z

Add more utilities to ParDoTranslation

commit e141e4a48f321a04e2f4dd404ff650826da2f07c
Author: Kenneth Knowles 
Date:   2017-06-08T21:27:02Z

Enable SplittableParDo on rehydrated ParDo transform

commit ce657ab34c6f9b11e453d58981fb4183a1358b5f
Author: Kenneth Knowles 
Date:   2017-06-08T20:44:52Z

Port DirectRunner ParDo overrides to SDK-agnostic APIs




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3330: [BEAM-2371] Port DirectRunner overrides to SDK-agno...

[kennknowles]

GitHub user kennknowles reopened a pull request:

https://github.com/apache/beam/pull/3330

[BEAM-2371] Port DirectRunner overrides to SDK-agnostic APIs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh 

This is peeled off my branch on the DirectRunner, which is now on switching 
up the evaluator factories to support dehydrated/rehydrated pipelines. That 
thread is blocked on getting all the inputs recorded but this little bit can go 
in sooner to avoid a mega PR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-overrides

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3330


commit 9f380c61065632c959d135a0f3c6a6ae977c699e
Author: Kenneth Knowles 
Date:   2017-05-26T21:26:55Z

Port ViewOverrideFactory to SDK-agnostic APIs

commit 3cb6237085065cf7e9bd9dcf92953e409d422cd5
Author: Kenneth Knowles 
Date:   2017-05-26T21:27:23Z

Port DirectGroupByKey to SDK-agnostic APIs

commit d88b058aba3f7e5ca8a92f5e9fc61aa5b252f3f5
Author: Kenneth Knowles 
Date:   2017-05-26T23:07:45Z

Port DirectRunner WriteFiles override to SDK-agnostic APIs

commit 22b06906159b1e74ee68b9a57073126129264885
Author: Kenneth Knowles 
Date:   2017-06-08T20:39:32Z

Port DirectRunner TestStream override to SDK-agnostic APIs

commit 8b4630d4a97638e902dc5db8bcd79f65b366e558
Author: Kenneth Knowles 
Date:   2017-06-08T20:46:18Z

Add more utilities to ParDoTranslation

commit e141e4a48f321a04e2f4dd404ff650826da2f07c
Author: Kenneth Knowles 
Date:   2017-06-08T21:27:02Z

Enable SplittableParDo on rehydrated ParDo transform

commit ce657ab34c6f9b11e453d58981fb4183a1358b5f
Author: Kenneth Knowles 
Date:   2017-06-08T20:44:52Z

Port DirectRunner ParDo overrides to SDK-agnostic APIs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Python_Verify #2447

[Apache Jenkins Server]

See 


Changes:

[robertwb] Use inner module for non-public coders.

--
[...truncated 502.99 KB...]
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[TaggedOutput, int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransfo

[GitHub] beam pull request #3331: Slight debuggability improvements in BigtableIO

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/3331

Slight debuggability improvements in BigtableIO

- ByteKeyRangeTracker.splitAtPosition logs the "insane" case first.
- BigtableIO logs the split position at INFO
- Log when adjusting range for first position

R: @sduskis

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam bt-range

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3331.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3331


commit 82a6cb6104c0bb179832dabf4433c183743ea983
Author: Eugene Kirpichov 
Date:   2017-06-08T21:51:15Z

Slight debuggability improvements in BigtableIO

- ByteKeyRangeTracker.splitAtPosition logs the "insane" case first.
- BigtableIO logs the split position at INFO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2425) Package does not exit when building beam-sdks-java-javadoc in JDK1.7

[Mark Liu (JIRA)]

Mark Liu created BEAM-2425:
--

 Summary: Package does not exit when building 
beam-sdks-java-javadoc in JDK1.7
 Key: BEAM-2425
 URL: https://issues.apache.org/jira/browse/BEAM-2425
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Mark Liu
Assignee: Jean-Baptiste Onofré


Error from [Jenkins 
build|https://builds.apache.org/job/beam_PostCommit_Java_JDK_Versions_Test/57/jdk=JDK%201.7%20(latest),label=beam/console]:
{code}
[javadoc] Constructing Javadoc information...
  [javadoc] 
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/JDK
 1.7 
(latest)/label/beam/sdks/java/javadoc/target/collected/org/apache/beam/sdk/io/cassandra/CassandraServiceImpl.java:20:
 error: package com.datastax.driver.core does not exist
  [javadoc] import com.datastax.driver.core.Cluster;
  [javadoc]^
...
  [javadoc] 
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/JDK
 1.7 
(latest)/label/beam/sdks/java/javadoc/target/collected/org/apache/beam/sdk/io/cassandra/CassandraServiceImpl.java:27:
 error: package com.datastax.driver.core.policies does not exist
  [javadoc] import com.datastax.driver.core.policies.DCAwareRoundRobinPolicy;
  [javadoc] ^
...
{code}

This is maven Postcommit build in JDK1.7

We have exclude many packages for Java8 in [this ant build 
script|https://github.com/apache/beam/blob/master/sdks/java/javadoc/ant.xml], 
but not org.apache.beam.sdk.io.cassandra. Probably need to add there?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-2357) Add HCatalogIO (Hive)

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-2357.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

> Add HCatalogIO (Hive)
> -
>
> Key: BEAM-2357
> URL: https://issues.apache.org/jira/browse/BEAM-2357
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Seshadri Raghunathan
>Priority: Minor
> Fix For: 2.1.0
>
>
> Support for reading and writing from Hive's catalog HCatalog. HCatalogIO will 
> allow access to the Hive records so they can be processed with Beam.
> This JIRA is to create a native implementation using the HCatalog 
> ReaderWriter API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2357) Add HCatalogIO (Hive)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043440#comment-16043440
 ] 

ASF GitHub Bot commented on BEAM-2357:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3220


> Add HCatalogIO (Hive)
> -
>
> Key: BEAM-2357
> URL: https://issues.apache.org/jira/browse/BEAM-2357
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Seshadri Raghunathan
>Priority: Minor
>
> Support for reading and writing from Hive's catalog HCatalog. HCatalogIO will 
> allow access to the Hive records so they can be processed with Beam.
> This JIRA is to create a native implementation using the HCatalog 
> ReaderWriter API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3220: [BEAM-2357]HiveIO implementation using native API -...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3220


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3220: Implements HCatalogIO

[jkff]

This closes #3220: Implements HCatalogIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/911bfbde
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/911bfbde
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/911bfbde

Branch: refs/heads/master
Commit: 911bfbdef62a7bb750426da9cc59537c4ffd9e1f
Parents: 7c5a70e 349898c
Author: Eugene Kirpichov 
Authored: Thu Jun 8 14:07:47 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Jun 8 14:07:47 2017 -0700

--
 sdks/java/io/hcatalog/pom.xml   | 163 ++
 .../apache/beam/sdk/io/hcatalog/HCatalogIO.java | 511 +++
 .../beam/sdk/io/hcatalog/package-info.java  |  22 +
 .../io/hcatalog/EmbeddedMetastoreService.java   |  88 
 .../beam/sdk/io/hcatalog/HCatalogIOTest.java| 271 ++
 .../sdk/io/hcatalog/HCatalogIOTestUtils.java| 106 
 .../hcatalog/src/test/resources/hive-site.xml   | 301 +++
 sdks/java/io/pom.xml|   1 +
 8 files changed, 1463 insertions(+)
--

[1/2] beam git commit: Implements HCatalogIO

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 7c5a70edb -> 911bfbdef


Implements HCatalogIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/349898c4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/349898c4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/349898c4

Branch: refs/heads/master
Commit: 349898c4702fc3e52d8c0cd1c5a04f14cd40fd27
Parents: 7c5a70e
Author: Seshadri Chakkravarthy 
Authored: Thu May 18 12:07:01 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Jun 8 14:05:38 2017 -0700

--
 sdks/java/io/hcatalog/pom.xml   | 163 ++
 .../apache/beam/sdk/io/hcatalog/HCatalogIO.java | 511 +++
 .../beam/sdk/io/hcatalog/package-info.java  |  22 +
 .../io/hcatalog/EmbeddedMetastoreService.java   |  88 
 .../beam/sdk/io/hcatalog/HCatalogIOTest.java| 271 ++
 .../sdk/io/hcatalog/HCatalogIOTestUtils.java| 106 
 .../hcatalog/src/test/resources/hive-site.xml   | 301 +++
 sdks/java/io/pom.xml|   1 +
 8 files changed, 1463 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/349898c4/sdks/java/io/hcatalog/pom.xml
--
diff --git a/sdks/java/io/hcatalog/pom.xml b/sdks/java/io/hcatalog/pom.xml
new file mode 100644
index 000..19b62a5
--- /dev/null
+++ b/sdks/java/io/hcatalog/pom.xml
@@ -0,0 +1,163 @@
+
+
+http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+
+  4.0.0
+
+  
+org.apache.beam
+beam-sdks-java-io-parent
+2.1.0-SNAPSHOT
+../pom.xml
+  
+
+  beam-sdks-java-io-hcatalog
+  Apache Beam :: SDKs :: Java :: IO :: HCatalog
+  IO to read and write for HCatalog source.
+
+  
+2.1.0
+2.5
+  
+
+  
+
+  
+org.apache.maven.plugins
+maven-surefire-plugin
+
+  true
+
+  
+
+  
+  
+  
+
+  org.apache.beam
+  beam-sdks-java-core
+
+
+
+  org.apache.beam
+  beam-sdks-java-io-hadoop-common
+
+
+
+  org.apache.hadoop
+  hadoop-common
+
+
+
+  commons-io
+  commons-io
+  ${apache.commons.version}
+
+
+
+  org.slf4j
+  slf4j-api
+
+
+
+  com.google.guava
+  guava
+
+
+
+  com.google.code.findbugs
+  jsr305
+
+
+
+  org.apache.hive
+  hive-exec
+  ${hive.version}
+
+
+
+  com.google.auto.value
+  auto-value
+  provided
+
+
+
+  org.apache.hive.hcatalog
+  hive-hcatalog-core
+  ${hive.version}
+  
+
+  org.apache.hive
+  hive-exec
+
+
+  com.google.protobuf
+  protobuf-java
+
+  
+
+
+
+  org.apache.hive.hcatalog
+  hive-hcatalog-core
+  tests
+  test
+  ${hive.version}
+
+
+
+  junit
+  junit
+  test
+
+
+
+  org.apache.hive
+  hive-exec
+  ${hive.version}
+  test-jar
+  test
+
+
+
+  org.apache.hive
+  hive-common
+  ${hive.version}
+  test
+
+
+
+  org.apache.hive
+  hive-cli
+  ${hive.version}
+  test
+
+
+
+  org.apache.beam
+  beam-runners-direct-java
+  test
+
+
+
+  org.hamcrest
+  hamcrest-all
+  test
+
+  
+
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/beam/blob/349898c4/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
--
diff --git 
a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
 
b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
new file mode 100644
index 000..07b56e3
--- /dev/null
+++ 
b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied

[GitHub] beam pull request #3325: Use inner module for non-public coders.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3325


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Use inner module for non-public coders.

[robertwb]

Use inner module for non-public coders.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/36aea2d2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/36aea2d2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/36aea2d2

Branch: refs/heads/master
Commit: 36aea2d26d7c8ea3299d7a25d617a6ba99794e18
Parents: 4aef938
Author: Robert Bradshaw 
Authored: Wed Jun 7 23:35:11 2017 -0700
Committer: Robert Bradshaw 
Committed: Thu Jun 8 13:58:31 2017 -0700

--
 sdks/python/apache_beam/runners/worker/operation_specs.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/36aea2d2/sdks/python/apache_beam/runners/worker/operation_specs.py
--
diff --git a/sdks/python/apache_beam/runners/worker/operation_specs.py 
b/sdks/python/apache_beam/runners/worker/operation_specs.py
index b8d19a1..bdafbea 100644
--- a/sdks/python/apache_beam/runners/worker/operation_specs.py
+++ b/sdks/python/apache_beam/runners/worker/operation_specs.py
@@ -326,11 +326,12 @@ def get_coder_from_spec(coder_spec):
 assert len(coder_spec['component_encodings']) == 2
 value_coder, window_coder = [
 get_coder_from_spec(c) for c in coder_spec['component_encodings']]
-return coders.WindowedValueCoder(value_coder, window_coder=window_coder)
+return coders.coders.WindowedValueCoder(
+value_coder, window_coder=window_coder)
   elif coder_spec['@type'] == 'kind:interval_window':
 assert ('component_encodings' not in coder_spec
 or not coder_spec['component_encodings'])
-return coders.IntervalWindowCoder()
+return coders.coders.IntervalWindowCoder()
   elif coder_spec['@type'] == 'kind:global_window':
 assert ('component_encodings' not in coder_spec
 or not coder_spec['component_encodings'])

[1/2] beam git commit: Closes #3325

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 4aef93854 -> 7c5a70edb


Closes #3325


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7c5a70ed
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7c5a70ed
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7c5a70ed

Branch: refs/heads/master
Commit: 7c5a70edb2d19f22b0a6b81ae516762703358986
Parents: 4aef938 36aea2d
Author: Robert Bradshaw 
Authored: Thu Jun 8 13:58:31 2017 -0700
Committer: Robert Bradshaw 
Committed: Thu Jun 8 13:58:31 2017 -0700

--
 sdks/python/apache_beam/runners/worker/operation_specs.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--

Build failed in Jenkins: beam_PostCommit_Python_Verify #2446

[Apache Jenkins Server]

See 


Changes:

[klk] Add Runner API oriented PTransformMatchers for DirectRunner overrides

[klk] DirectRunner override matchers using Runner API

--
[...truncated 502.68 KB...]
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[TaggedOutput, int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #4060

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043348#comment-16043348
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/3330


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3330: [BEAM-2371] Port DirectRunner overrides to SDK-agno...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/3330


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043347#comment-16043347
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3330

[BEAM-2371] Port DirectRunner overrides to SDK-agnostic APIs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh 

This is peeled off my branch on the DirectRunner, which is now on switching 
up the evaluator factories to support dehydrated/rehydrated pipelines. That 
thread is blocked on getting all the inputs recorded but this little bit can go 
in sooner to avoid a mega PR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-overrides

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3330


commit 9f380c61065632c959d135a0f3c6a6ae977c699e
Author: Kenneth Knowles 
Date:   2017-05-26T21:26:55Z

Port ViewOverrideFactory to SDK-agnostic APIs

commit 3cb6237085065cf7e9bd9dcf92953e409d422cd5
Author: Kenneth Knowles 
Date:   2017-05-26T21:27:23Z

Port DirectGroupByKey to SDK-agnostic APIs

commit d88b058aba3f7e5ca8a92f5e9fc61aa5b252f3f5
Author: Kenneth Knowles 
Date:   2017-05-26T23:07:45Z

Port DirectRunner WriteFiles override to SDK-agnostic APIs




> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3330: [BEAM-2371] Port DirectRunner overrides to SDK-agno...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3330

[BEAM-2371] Port DirectRunner overrides to SDK-agnostic APIs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`.
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh 

This is peeled off my branch on the DirectRunner, which is now on switching 
up the evaluator factories to support dehydrated/rehydrated pipelines. That 
thread is blocked on getting all the inputs recorded but this little bit can go 
in sooner to avoid a mega PR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DirectRunner-overrides

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3330


commit 9f380c61065632c959d135a0f3c6a6ae977c699e
Author: Kenneth Knowles 
Date:   2017-05-26T21:26:55Z

Port ViewOverrideFactory to SDK-agnostic APIs

commit 3cb6237085065cf7e9bd9dcf92953e409d422cd5
Author: Kenneth Knowles 
Date:   2017-05-26T21:27:23Z

Port DirectGroupByKey to SDK-agnostic APIs

commit d88b058aba3f7e5ca8a92f5e9fc61aa5b252f3f5
Author: Kenneth Knowles 
Date:   2017-05-26T23:07:45Z

Port DirectRunner WriteFiles override to SDK-agnostic APIs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Python_Verify #2445

[Apache Jenkins Server]

See 


Changes:

[altay] Raise entity limit per RPC to 9MB.

--
[...truncated 503.79 KB...]
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[TaggedOutput, int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))

[jira] [Created] (BEAM-2424) CAST operator supporting numeric types

[Tarush Grover (JIRA)]

Tarush Grover created BEAM-2424:
---

 Summary: CAST operator supporting numeric types
 Key: BEAM-2424
 URL: https://issues.apache.org/jira/browse/BEAM-2424
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: Tarush Grover
Assignee: Tarush Grover
 Fix For: 2.1.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2282) logic in function accept in BeamSqlExpression

[Tarush Grover (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043254#comment-16043254
 ] 

Tarush Grover commented on BEAM-2282:
-

[~mingmxu] currently accept function in BeamSqlExpression is not called during 
a sql query but it directs to calcite validation and in that it calls its 
`accept` function. Is my understanding correct?

> logic in function accept in BeamSqlExpression
> -
>
> Key: BEAM-2282
> URL: https://issues.apache.org/jira/browse/BEAM-2282
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>
> Method {{accept}} is used to make sure the SQL operator is supported in Beam 
> SQL. IMO it should only check the case that some scenarios are not supported 
> in BeamSQL even it's valid in Calcite SQL grammar.
> It's not necessary to my understand to validate SQL grammar as that's done 
> already in the parsing phase.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3239: [BEAM-2371] Port DirectRunner override matchers to ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3239


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2371) Make Java DirectRunner demonstrate language-agnostic Runner API translation wrappers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043249#comment-16043249
 ] 

ASF GitHub Bot commented on BEAM-2371:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3239


> Make Java DirectRunner demonstrate language-agnostic Runner API translation 
> wrappers
> 
>
> Key: BEAM-2371
> URL: https://issues.apache.org/jira/browse/BEAM-2371
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This will complete the PoC for runners-core-construction-java and the Runner 
> API and show other runners the easy path to executing non-Java pipelines, 
> modulo Fn API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/3] beam git commit: This closes #3239: [BEAM-2371] Port DirectRunner override matchers to language-agnostic construction APIs

[kenn]

This closes #3239: [BEAM-2371] Port DirectRunner override matchers to 
language-agnostic construction APIs

  DirectRunner override matchers using Runner API
  Add Runner API oriented PTransformMatchers for DirectRunner overrides


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4aef9385
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4aef9385
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4aef9385

Branch: refs/heads/master
Commit: 4aef9385477f0d13bdaa345cb5a629b2fd00b867
Parents: b53e6f0 d8d9087
Author: Kenneth Knowles 
Authored: Thu Jun 8 11:37:15 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 8 11:37:15 2017 -0700

--
 .../core/construction/PTransformMatchers.java   | 94 +++-
 .../construction/PTransformTranslation.java |  7 +-
 .../construction/PTransformMatchersTest.java| 32 +++
 .../beam/runners/direct/DirectRunner.java   | 20 ++---
 4 files changed, 137 insertions(+), 16 deletions(-)
--

[2/3] beam git commit: Add Runner API oriented PTransformMatchers for DirectRunner overrides

[kenn]

Add Runner API oriented PTransformMatchers for DirectRunner overrides


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ca7b9c28
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ca7b9c28
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ca7b9c28

Branch: refs/heads/master
Commit: ca7b9c288151d318898ab000b91d26fcf62046ca
Parents: b53e6f0
Author: Kenneth Knowles 
Authored: Thu May 25 06:29:09 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 8 11:36:28 2017 -0700

--
 .../core/construction/PTransformMatchers.java   | 94 +++-
 .../construction/PTransformTranslation.java |  7 +-
 .../construction/PTransformMatchersTest.java| 32 +++
 3 files changed, 128 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ca7b9c28/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
index bfe24a0..c339891 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
@@ -18,6 +18,7 @@
 package org.apache.beam.runners.core.construction;
 
 import com.google.common.base.MoreObjects;
+import java.io.IOException;
 import java.util.HashSet;
 import java.util.Set;
 import org.apache.beam.sdk.annotations.Experimental;
@@ -50,6 +51,34 @@ public class PTransformMatchers {
   private PTransformMatchers() {}
 
   /**
+   * Returns a {@link PTransformMatcher} that matches a {@link PTransform} if 
the URN of the
+   * {@link PTransform} is equal to the URN provided ot this matcher.
+   */
+  public static PTransformMatcher urnEqualTo(String urn) {
+return new EqualUrnPTransformMatcher(urn);
+  }
+
+  private static class EqualUrnPTransformMatcher implements PTransformMatcher {
+private final String urn;
+
+private EqualUrnPTransformMatcher(String urn) {
+  this.urn = urn;
+}
+
+@Override
+public boolean matches(AppliedPTransform application) {
+  return 
urn.equals(PTransformTranslation.urnForTransformOrNull(application.getTransform()));
+}
+
+@Override
+public String toString() {
+  return MoreObjects.toStringHelper(this)
+  .add("urn", urn)
+  .toString();
+}
+  }
+
+  /**
* Returns a {@link PTransformMatcher} that matches a {@link PTransform} if 
the class of the
* {@link PTransform} is equal to the {@link Class} provided ot this matcher.
*/
@@ -151,6 +180,68 @@ public class PTransformMatchers {
   }
 
   /**
+   * A {@link PTransformMatcher} that matches a {@link ParDo} by URN if it has 
a splittable {@link
+   * DoFn}.
+   */
+  public static PTransformMatcher splittableParDo() {
+return new PTransformMatcher() {
+  @Override
+  public boolean matches(AppliedPTransform application) {
+if (PTransformTranslation.PAR_DO_TRANSFORM_URN.equals(
+
PTransformTranslation.urnForTransformOrNull(application.getTransform( {
+
+  try {
+return ParDoTranslation.isSplittable(application);
+  } catch (IOException e) {
+throw new RuntimeException(
+String.format(
+"Transform with URN %s could not be translated",
+PTransformTranslation.PAR_DO_TRANSFORM_URN),
+e);
+  }
+}
+return false;
+  }
+
+  @Override
+  public String toString() {
+return 
MoreObjects.toStringHelper("SplittableParDoMultiMatcher").toString();
+  }
+};
+  }
+
+  /**
+   * A {@link PTransformMatcher} that matches a {@link ParDo} transform by URN
+   * and whether it contains state or timers as specified by {@link 
ParDoTranslation}.
+   */
+  public static PTransformMatcher stateOrTimerParDo() {
+return new PTransformMatcher() {
+  @Override
+  public boolean matches(AppliedPTransform application) {
+if (PTransformTranslation.PAR_DO_TRANSFORM_URN.equals(
+
PTransformTranslation.urnForTransformOrNull(application.getTransform( {
+
+  try {
+return ParDoTranslation.usesStateOrTimers(application);
+  } catch (IOException e) {
+throw new RuntimeException(
+String.format(
+"Transform with URN %s could not be translated",
+PTransformTranslation.PAR_DO_TRANSFORM_URN),
+

[1/3] beam git commit: DirectRunner override matchers using Runner API

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master b53e6f0dc -> 4aef93854


DirectRunner override matchers using Runner API


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d8d90878
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d8d90878
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d8d90878

Branch: refs/heads/master
Commit: d8d9087877c01f1786271726a541fb3eeda7f939
Parents: ca7b9c2
Author: Kenneth Knowles 
Authored: Thu May 25 06:31:16 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 8 11:36:28 2017 -0700

--
 .../beam/runners/direct/DirectRunner.java   | 20 +---
 1 file changed, 9 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d8d90878/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index dbd1ec4..136ccf3 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -30,6 +30,7 @@ import java.util.Map;
 import java.util.Set;
 import org.apache.beam.runners.core.SplittableParDoViaKeyedWorkItems;
 import org.apache.beam.runners.core.construction.PTransformMatchers;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
 import org.apache.beam.runners.core.construction.SplittableParDo;
 import org.apache.beam.runners.direct.DirectRunner.DirectPipelineResult;
 import 
org.apache.beam.runners.direct.TestStreamEvaluatorFactory.DirectTestStreamFactory;
@@ -42,12 +43,9 @@ import org.apache.beam.sdk.metrics.MetricResults;
 import org.apache.beam.sdk.metrics.MetricsEnvironment;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.runners.PTransformOverride;
-import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.ParDo.MultiOutput;
-import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.values.PCollection;
 import org.joda.time.Duration;
@@ -230,33 +228,33 @@ public class DirectRunner extends 
PipelineRunner {
 new WriteWithShardingFactory())) /* Uses a view internally. */
 .add(
 PTransformOverride.of(
-PTransformMatchers.classEqualTo(CreatePCollectionView.class),
+
PTransformMatchers.urnEqualTo(PTransformTranslation.CREATE_VIEW_TRANSFORM_URN),
 new ViewOverrideFactory())) /* Uses pardos and GBKs */
 .add(
 PTransformOverride.of(
-PTransformMatchers.classEqualTo(TestStream.class),
+
PTransformMatchers.urnEqualTo(PTransformTranslation.TEST_STREAM_TRANSFORM_URN),
 new DirectTestStreamFactory(this))) /* primitive */
 // SplittableParMultiDo is implemented in terms of nonsplittable 
simple ParDos and extra
 // primitives
 .add(
 PTransformOverride.of(
-PTransformMatchers.splittableParDoMulti(), new 
ParDoMultiOverrideFactory()))
+PTransformMatchers.splittableParDo(), new 
ParDoMultiOverrideFactory()))
 // state and timer pardos are implemented in terms of simple ParDos 
and extra primitives
 .add(
 PTransformOverride.of(
-PTransformMatchers.stateOrTimerParDoMulti(), new 
ParDoMultiOverrideFactory()))
+PTransformMatchers.stateOrTimerParDo(), new 
ParDoMultiOverrideFactory()))
 .add(
 PTransformOverride.of(
-
PTransformMatchers.classEqualTo(SplittableParDo.ProcessKeyedElements.class),
+PTransformMatchers.urnEqualTo(
+SplittableParDo.SPLITTABLE_PROCESS_KEYED_ELEMENTS_URN),
 new SplittableParDoViaKeyedWorkItems.OverrideFactory()))
 .add(
 PTransformOverride.of(
-PTransformMatchers.classEqualTo(
-
SplittableParDoViaKeyedWorkItems.GBKIntoKeyedWorkItems.class),
+
PTransformMatchers.urnEqualTo(SplittableParDo.SPLITTABLE_GBKIKWI_URN),
 new DirectGBKIntoKeyedWorkItemsOverrideFactory())) /* Returns 
a GBKO */
 .add(
 PTransformOverride.of(
-PTransformMatchers.classEqualTo(GroupByKey.class),
+
PTransformMatchers.urnEqualTo(

Build failed in Jenkins: beam_PostCommit_Python_Verify #2444

[Apache Jenkins Server]

See 


Changes:

[altay] Refine Python DirectRunner watermark advancement behavior

--
[...truncated 502.49 KB...]
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #109

[Apache Jenkins Server]

See 


Changes:

[tgroh] Expand all PValues to component PCollections always

[lcwik] Avoid flakiness in data channel for empty streams.

[altay] Refine Python DirectRunner watermark advancement behavior

[altay] Raise entity limit per RPC to 9MB.

--
[...truncated 2.55 MB...]
2017-06-08T18:38:04.888 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 45.2 KB/sec)
2017-06-08T18:38:04.892 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-08T18:38:04.933 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 274.5 KB/sec)
2017-06-08T18:38:04.937 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-08T18:38:04.973 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 356.5 KB/sec)
2017-06-08T18:38:04.978 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-08T18:38:05.009 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 70.1 KB/sec)
2017-06-08T18:38:05.014 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-08T18:38:05.048 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 46.3 KB/sec)
2017-06-08T18:38:05.053 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-08T18:38:05.086 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 164.1 KB/sec)
2017-06-08T18:38:05.089 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-08T18:38:05.121 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 78.0 KB/sec)
2017-06-08T18:38:05.126 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-08T18:38:05.160 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 100.8 KB/sec)
2017-06-08T18:38:05.165 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-08T18:38:05.198 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 30.4 KB/sec)
2017-06-08T18:38:05.203 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-08T18:38:05.235 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 53.0 KB/sec)
2017-06-08T18:38:05.240 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-08T18:38:05.272 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 55.1 KB/sec)
2017-06-08T18:38:05.277 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-08T18:38:05.308 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 85.6 KB/sec)
2017-06-08T18:38:05.313 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-08T18:38:05.345 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 550.5 KB/sec)
2017-06-08T18:38:05.349 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-08T18:38:05.380 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 304.1 KB/sec)
2017-06-08T18:38:05.384 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-08T18:38:05.414 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
 (5 KB at 148.5 KB/sec)
2017-06-08T18:38:05.418 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #61

[Apache Jenkins Server]

See 


Changes:

[tgroh] Expand all PValues to component PCollections always

[lcwik] Avoid flakiness in data channel for empty streams.

[altay] Refine Python DirectRunner watermark advancement behavior

[altay] Raise entity limit per RPC to 9MB.

--
[...truncated 1.26 MB...]
2017-06-08\T\18:29:56.037 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/clapper/grizzled-slf4j_2.10/1.0.2/grizzled-slf4j_2.10-1.0.2.pom
2017-06-08\T\18:29:56.066 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/clapper/grizzled-slf4j_2.10/1.0.2/grizzled-slf4j_2.10-1.0.2.pom
 (2 KB at 64.5 KB/sec)
2017-06-08\T\18:29:56.068 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
2017-06-08\T\18:29:56.095 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
 (2 KB at 73.9 KB/sec)
2017-06-08\T\18:29:56.097 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-08\T\18:29:56.125 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 58.4 KB/sec)
2017-06-08\T\18:29:56.127 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-08\T\18:29:56.155 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 82.8 KB/sec)
2017-06-08\T\18:29:56.156 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-08\T\18:29:56.182 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 76.8 KB/sec)
2017-06-08\T\18:29:56.184 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-08\T\18:29:56.212 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 70.9 KB/sec)
2017-06-08\T\18:29:56.215 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-08\T\18:29:56.241 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 162.2 KB/sec)
2017-06-08\T\18:29:56.248 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-08\T\18:29:56.275 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 223.3 KB/sec)
2017-06-08\T\18:29:56.281 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-08\T\18:29:56.308 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 117.2 KB/sec)
2017-06-08\T\18:29:56.310 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-08\T\18:29:56.342 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 783.3 KB/sec)
2017-06-08\T\18:29:56.344 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-08\T\18:29:56.373 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
473.3 KB/sec)
2017-06-08\T\18:29:56.395 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-08\T\18:29:56.424 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 159.7 KB/sec)
2017-06-08\T\18:29:56.426 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-08\T\18:29:56.452 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 54.1 KB/sec)
2017-06-08\T\18:29:56.458 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-08\T\18:29:56.486 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 114.6 KB/sec)
2017-06-08\T\18:29:56.494 [INFO] Downloading:

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #61

[Apache Jenkins Server]

See 


Changes:

[tgroh] Expand all PValues to component PCollections always

[lcwik] Avoid flakiness in data channel for empty streams.

[altay] Refine Python DirectRunner watermark advancement behavior

[altay] Raise entity limit per RPC to 9MB.

--
[...truncated 1.25 MB...]
2017-06-08\T\18:29:33.781 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-08\T\18:29:33.809 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 58.4 KB/sec)
2017-06-08\T\18:29:33.811 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-08\T\18:29:33.837 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 89.1 KB/sec)
2017-06-08\T\18:29:33.839 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-08\T\18:29:33.867 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 68.8 KB/sec)
2017-06-08\T\18:29:33.868 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-08\T\18:29:33.894 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 76.4 KB/sec)
2017-06-08\T\18:29:33.895 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-08\T\18:29:33.921 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 162.2 KB/sec)
2017-06-08\T\18:29:33.926 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-08\T\18:29:33.952 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 231.9 KB/sec)
2017-06-08\T\18:29:33.957 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-08\T\18:29:33.983 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 121.7 KB/sec)
2017-06-08\T\18:29:33.984 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-08\T\18:29:34.012 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 895.2 KB/sec)
2017-06-08\T\18:29:34.014 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-08\T\18:29:34.041 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
508.3 KB/sec)
2017-06-08\T\18:29:34.055 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-08\T\18:29:34.081 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 178.1 KB/sec)
2017-06-08\T\18:29:34.083 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-08\T\18:29:34.108 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 56.3 KB/sec)
2017-06-08\T\18:29:34.112 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-08\T\18:29:34.138 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 123.4 KB/sec)
2017-06-08\T\18:29:34.144 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
2017-06-08\T\18:29:34.170 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
 (5 KB at 150.0 KB/sec)
2017-06-08\T\18:29:34.171 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
2017-06-08\T\18:29:34.198 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
 (32 KB at 1169.3 KB/sec)
2017-06-08\T\18:29:34.201 [INFO] Downloading: 
https:

[GitHub] beam pull request #3329: Visit a Transform Hierarchy in Topological Order

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3329

Visit a Transform Hierarchy in Topological Order

This reverts commit 6ad6433ec0c02aec8656e9e3b27f6e0f974f8ece.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam topo_order_only

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3329.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3329


commit 8b436faa4c844e3aa1a98f5c9db356f7c0f4ba2d
Author: Thomas Groh 
Date:   2017-06-06T23:15:19Z

Visit a Transform Hierarchy in Topological Order

This reverts commit 6ad6433ec0c02aec8656e9e3b27f6e0f974f8ece.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #61

[Apache Jenkins Server]

See 


Changes:

[tgroh] Expand all PValues to component PCollections always

[lcwik] Avoid flakiness in data channel for empty streams.

[altay] Refine Python DirectRunner watermark advancement behavior

[altay] Raise entity limit per RPC to 9MB.

--
[...truncated 520.65 KB...]
2017-06-08\T\18:07:23.474 [INFO] Running 
org.apache.beam.sdk.transforms.SampleTest$PickAnyTest
2017-06-08\T\18:07:24.715 [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 1.239 s - in 
org.apache.beam.sdk.transforms.SampleTest$PickAnyTest
2017-06-08\T\18:07:24.721 [INFO] Running 
org.apache.beam.sdk.transforms.SampleTest$MiscTest
2017-06-08\T\18:07:25.712 [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.989 s - in org.apache.beam.sdk.transforms.SampleTest$MiscTest
2017-06-08\T\18:07:25.712 [INFO] Running 
org.apache.beam.sdk.transforms.WithKeysTest
2017-06-08\T\18:07:26.122 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.409 s - in org.apache.beam.sdk.transforms.WithKeysTest
2017-06-08\T\18:07:26.123 [INFO] Running 
org.apache.beam.sdk.transforms.ToStringTest
2017-06-08\T\18:07:26.669 [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.544 s - in org.apache.beam.sdk.transforms.ToStringTest
2017-06-08\T\18:07:26.669 [INFO] Running 
org.apache.beam.sdk.transforms.ReshuffleTest
2017-06-08\T\18:07:28.044 [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 1.373 s - in org.apache.beam.sdk.transforms.ReshuffleTest
2017-06-08\T\18:07:28.044 [INFO] Running org.apache.beam.sdk.transforms.ViewTest
2017-06-08\T\18:07:33.396 [INFO] Tests run: 37, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 5.35 s - in org.apache.beam.sdk.transforms.ViewTest
2017-06-08\T\18:07:33.396 [INFO] Running 
org.apache.beam.sdk.transforms.CombineFnsTest
2017-06-08\T\18:07:33.958 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.561 s - in org.apache.beam.sdk.transforms.CombineFnsTest
2017-06-08\T\18:07:33.958 [INFO] Running 
org.apache.beam.sdk.transforms.GroupIntoBatchesTest
2017-06-08\T\18:07:34.565 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.605 s - in 
org.apache.beam.sdk.transforms.GroupIntoBatchesTest
2017-06-08\T\18:07:34.565 [INFO] Running 
org.apache.beam.sdk.transforms.ParDoLifecycleTest
2017-06-08\T\18:07:34.972 [INFO] Tests run: 12, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 0.405 s - in 
org.apache.beam.sdk.transforms.ParDoLifecycleTest
2017-06-08\T\18:07:34.972 [INFO] Running 
org.apache.beam.sdk.transforms.ApproximateQuantilesTest
2017-06-08\T\18:07:35.445 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.471 s - in 
org.apache.beam.sdk.transforms.ApproximateQuantilesTest
2017-06-08\T\18:07:35.445 [INFO] Running 
org.apache.beam.sdk.transforms.join.CoGroupByKeyTest
2017-06-08\T\18:07:36.025 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.579 s - in 
org.apache.beam.sdk.transforms.join.CoGroupByKeyTest
2017-06-08\T\18:07:36.025 [INFO] Running 
org.apache.beam.sdk.transforms.PartitionTest
2017-06-08\T\18:07:36.391 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.365 s - in org.apache.beam.sdk.transforms.PartitionTest
2017-06-08\T\18:07:36.391 [INFO] Running 
org.apache.beam.sdk.transforms.ReifyTimestampsTest
2017-06-08\T\18:07:36.673 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.28 s - in org.apache.beam.sdk.transforms.ReifyTimestampsTest
2017-06-08\T\18:07:36.673 [INFO] Running 
org.apache.beam.sdk.transforms.DistinctTest
2017-06-08\T\18:07:37.243 [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.569 s - in org.apache.beam.sdk.transforms.DistinctTest
2017-06-08\T\18:07:37.244 [INFO] Running 
org.apache.beam.sdk.transforms.KvSwapTest
2017-06-08\T\18:07:37.514 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.269 s - in org.apache.beam.sdk.transforms.KvSwapTest
2017-06-08\T\18:07:37.514 [INFO] Running 
org.apache.beam.sdk.options.ProxyInvocationHandlerTest
2017-06-08\T\18:07:37.569 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.054 s - in 
org.apache.beam.sdk.options.ProxyInvocationHandlerTest
2017-06-08\T\18:07:37.570 [INFO] Running 
org.apache.beam.sdk.io.CountingSourceTest
2017-06-08\T\18:07:52.368 [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 14.796 s - in org.apache.beam.sdk.io.CountingSourceTest
2017-06-08\T\18:07:52.368 [INFO] Running 
org.apache.beam.sdk.io.GenerateSequenceTest
2017-06-08\T\18:08:01.412 [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 9.042 s - in org.apache.beam.sdk.io.GenerateSequenceTest
2017-06-08\T\18:08:01.412 [INFO] Running 
org.apac

[GitHub] beam pull request #3302: [BEAM-991] Raise entities limit per RPC to 9MB.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3302


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-991) DatastoreIO Write should flush early for large batches

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043144#comment-16043144
 ] 

ASF GitHub Bot commented on BEAM-991:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3302


> DatastoreIO Write should flush early for large batches
> --
>
> Key: BEAM-991
> URL: https://issues.apache.org/jira/browse/BEAM-991
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>
> If entities are large (avg size > 20KB) then the a single batched write (500 
> entities) would exceed the Datastore size limit of a single request (10MB) 
> from https://cloud.google.com/datastore/docs/concepts/limits.
> First reported in: 
> http://stackoverflow.com/questions/40156400/why-does-dataflow-erratically-fail-in-datastore-access



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Raise entity limit per RPC to 9MB.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master fdfc70eec -> b53e6f0dc


Raise entity limit per RPC to 9MB.

This is closer to the API limit, while still leaving room for overhead. Brings
the Java SDK into line with the Python SDK.

Switch the unit test to use the size of each entity, which is what the
connector is actually using, rather than the property size (which is slightly
smaller and would cause the test to fail for some values).


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/156f326a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/156f326a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/156f326a

Branch: refs/heads/master
Commit: 156f326a16e15b4e22a189a2a263d11d7b273656
Parents: fdfc70e
Author: Colin Phipps 
Authored: Mon Jun 5 12:12:49 2017 +
Committer: Ahmet Altay 
Committed: Thu Jun 8 10:57:09 2017 -0700

--
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java  |  2 +-
 .../beam/sdk/io/gcp/datastore/DatastoreV1Test.java  | 16 +---
 2 files changed, 10 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/156f326a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
index b198a6f..06b9c8a 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
@@ -213,7 +213,7 @@ public class DatastoreV1 {
* the mutations themselves and not the CommitRequest wrapper around them.
*/
   @VisibleForTesting
-  static final int DATASTORE_BATCH_UPDATE_BYTES_LIMIT = 5_000_000;
+  static final int DATASTORE_BATCH_UPDATE_BYTES_LIMIT = 9_000_000;
 
   /**
* Returns an empty {@link DatastoreV1.Read} builder. Configure the source 
{@code projectId},

http://git-wip-us.apache.org/repos/asf/beam/blob/156f326a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
index 460049e..229b1fb 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
@@ -651,12 +651,14 @@ public class DatastoreV1Test {
   @Test
   public void testDatatoreWriterFnWithLargeEntities() throws Exception {
 List mutations = new ArrayList<>();
-int propertySize = 900_000;
+int entitySize = 0;
 for (int i = 0; i < 12; ++i) {
-  Entity.Builder entity = Entity.newBuilder().setKey(makeKey("key" + i, i 
+ 1));
-  entity.putProperties("long", makeValue(new String(new char[propertySize])
-).setExcludeFromIndexes(true).build());
-  mutations.add(makeUpsert(entity.build()).build());
+  Entity entity = Entity.newBuilder().setKey(makeKey("key" + i, i + 1))
+.putProperties("long", makeValue(new String(new char[900_000])
+  ).setExcludeFromIndexes(true).build())
+.build();
+  entitySize = entity.getSerializedSize(); // Take the size of any one 
entity.
+  mutations.add(makeUpsert(entity).build());
 }
 
 DatastoreWriterFn datastoreWriter = new 
DatastoreWriterFn(StaticValueProvider.of(PROJECT_ID),
@@ -667,10 +669,10 @@ public class DatastoreV1Test {
 
 // This test is over-specific currently; it requires that we split the 12 
entity writes into 3
 // requests, but we only need each CommitRequest to be less than 10MB in 
size.
-int propertiesPerRpc = DATASTORE_BATCH_UPDATE_BYTES_LIMIT / propertySize;
+int entitiesPerRpc = DATASTORE_BATCH_UPDATE_BYTES_LIMIT / entitySize;
 int start = 0;
 while (start < mutations.size()) {
-  int end = Math.min(mutations.size(), start + propertiesPerRpc);
+  int end = Math.min(mutations.size(), start + entitiesPerRpc);
   CommitRequest.Builder commitRequest = CommitRequest.newBuilder();
   commitRequest.setMode(CommitRequest.Mode.NON_TRANSACTIONAL);
   commitRequest.addAllMutations(mutations.subList(start, end));

[2/2] beam git commit: This closes #3302

[altay]

This closes #3302


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b53e6f0d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b53e6f0d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b53e6f0d

Branch: refs/heads/master
Commit: b53e6f0dc91948040364d1e742b023d5a2be8433
Parents: fdfc70e 156f326
Author: Ahmet Altay 
Authored: Thu Jun 8 10:57:11 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Jun 8 10:57:11 2017 -0700

--
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java  |  2 +-
 .../beam/sdk/io/gcp/datastore/DatastoreV1Test.java  | 16 +---
 2 files changed, 10 insertions(+), 8 deletions(-)
--

[2/2] beam git commit: This closes #3322

[altay]

This closes #3322


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fdfc70ee
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fdfc70ee
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fdfc70ee

Branch: refs/heads/master
Commit: fdfc70eecb8a3833c155e8bb7b6e128d956a61de
Parents: d81ed21 3e04902
Author: Ahmet Altay 
Authored: Thu Jun 8 10:55:47 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Jun 8 10:55:47 2017 -0700

--
 .../runners/direct/watermark_manager.py | 20 +---
 sdks/python/apache_beam/utils/timestamp.py  |  5 +
 2 files changed, 22 insertions(+), 3 deletions(-)
--

[GitHub] beam pull request #3322: Refine Python DirectRunner watermark advancement be...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3322


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Python_Verify #2443

[Apache Jenkins Server]

See 


Changes:

[lcwik] Avoid flakiness in data channel for empty streams.

--
[...truncated 501.99 KB...]
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[TaggedOutput, int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback cod

[1/2] beam git commit: Refine Python DirectRunner watermark advancement behavior

[altay]

Repository: beam
Updated Branches:
  refs/heads/master d81ed2172 -> fdfc70eec


Refine Python DirectRunner watermark advancement behavior

This change helps prepare for streaming pipeline execution.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3e049020
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3e049020
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3e049020

Branch: refs/heads/master
Commit: 3e04902008b410269b23179dc2146623ff1fbd0a
Parents: d81ed21
Author: Charles Chen 
Authored: Wed Jun 7 17:46:36 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Jun 8 10:55:44 2017 -0700

--
 .../runners/direct/watermark_manager.py | 20 +---
 sdks/python/apache_beam/utils/timestamp.py  |  5 +
 2 files changed, 22 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3e049020/sdks/python/apache_beam/runners/direct/watermark_manager.py
--
diff --git a/sdks/python/apache_beam/runners/direct/watermark_manager.py 
b/sdks/python/apache_beam/runners/direct/watermark_manager.py
index 3a13539..0d7cd4f 100644
--- a/sdks/python/apache_beam/runners/direct/watermark_manager.py
+++ b/sdks/python/apache_beam/runners/direct/watermark_manager.py
@@ -25,6 +25,7 @@ from apache_beam import pipeline
 from apache_beam import pvalue
 from apache_beam.utils.timestamp import MAX_TIMESTAMP
 from apache_beam.utils.timestamp import MIN_TIMESTAMP
+from apache_beam.utils.timestamp import TIME_GRANULARITY
 
 
 class WatermarkManager(object):
@@ -193,9 +194,22 @@ class _TransformWatermarks(object):
 
   def refresh(self):
 with self._lock:
-  pending_holder = (WatermarkManager.WATERMARK_NEG_INF
-if self._pending else
-WatermarkManager.WATERMARK_POS_INF)
+  min_pending_timestamp = WatermarkManager.WATERMARK_POS_INF
+  has_pending_elements = False
+  for input_bundle in self._pending:
+# TODO(ccy): we can have the Bundle class keep track of the minimum
+# timestamp so we don't have to do an iteration here.
+for wv in input_bundle.get_elements_iterable():
+  has_pending_elements = True
+  if wv.timestamp < min_pending_timestamp:
+min_pending_timestamp = wv.timestamp
+
+  # If there is a pending element with a certain timestamp, we can at most
+  # advance our watermark to the maximum timestamp less than that
+  # timestamp.
+  pending_holder = WatermarkManager.WATERMARK_POS_INF
+  if has_pending_elements:
+pending_holder = min_pending_timestamp - TIME_GRANULARITY
 
   input_watermarks = [
   tw.output_watermark for tw in self._input_transform_watermarks]

http://git-wip-us.apache.org/repos/asf/beam/blob/3e049020/sdks/python/apache_beam/utils/timestamp.py
--
diff --git a/sdks/python/apache_beam/utils/timestamp.py 
b/sdks/python/apache_beam/utils/timestamp.py
index 5d1b48c..b3e840e 100644
--- a/sdks/python/apache_beam/utils/timestamp.py
+++ b/sdks/python/apache_beam/utils/timestamp.py
@@ -208,3 +208,8 @@ class Duration(object):
   def __mod__(self, other):
 other = Duration.of(other)
 return Duration(micros=self.micros % other.micros)
+
+
+# The minimum granularity / interval expressible in a Timestamp / Duration
+# object.
+TIME_GRANULARITY = Duration(micros=1)

[jira] [Commented] (BEAM-2404) BigQueryIO reading stalls if no data is returned by query

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043133#comment-16043133
 ] 

Stephen Sisk commented on BEAM-2404:


I wonder if we're not correctly sending the cleanup signal since there's 
nothing in the collection? 

Looking at PassThroughThenCleanup, I don't immediately see why that'd be the 
case, but I do wonder. 

cc [~bchambers] [~reuvenlax] to see if they have thoughts.

> BigQueryIO reading stalls if no data is returned by query
> -
>
> Key: BEAM-2404
> URL: https://issues.apache.org/jira/browse/BEAM-2404
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Andre
>Assignee: Stephen Sisk
>
> When running a BigQueryIO query that doesn't return any rows (e.g. nothing 
> has changed in a delta job) the job seems to stall and nothing happens as no 
> temp files are being written which I think might be what it is waiting for. 
> Just adding one row to the source table will make the job run through 
> successfully.
> Code:
> {code:java}
> PCollection  rows = p.apply("ReadFromBQ",
>  BigQueryIO.read()
>  .fromQuery("SELECT * FROM `myproject.dataset.table`")
>  .withoutResultFlattening().usingStandardSql());
> {code}
>   
> Log:
> {code:java}   
> Jun 02, 2017 9:00:36 AM 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl 
> startJob
> INFO: Started BigQuery job: {jobId=beam_job_batch-query, 
> projectId=my-project}.
> bq show -j --format=prettyjson --project_id=my-project beam_job_batch-query
> Jun 02, 2017 9:03:11 AM 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase executeExtract
> INFO: Starting BigQuery extract job: beam_job_batch-extract
> Jun 02, 2017 9:03:12 AM 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl 
> startJob
> INFO: Started BigQuery job: {jobId=beam_job_batch-extract, 
> projectId=my-project}.
> bq show -j --format=prettyjson --project_id=my-project beam_job_batch-extract
> Jun 02, 2017 9:04:06 AM 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase executeExtract
> INFO: BigQuery extract job completed: beam_job_batch-extract
> Jun 02, 2017 9:04:08 AM org.apache.beam.sdk.io.FileBasedSource 
> expandFilePattern
> INFO: Matched 1 files for pattern 
> gs://my-bucket/tmp/BigQueryExtractTemp/ff594d003c6440a1ad84b9e02858b5c6/.avro
> Jun 02, 2017 9:04:09 AM org.apache.beam.sdk.io.FileBasedSource 
> getEstimatedSizeBytes
> INFO: Filepattern 
> gs://my-bucket/tmp/BigQueryExtractTemp/ff594d003c6440a1ad84b9e02858b5c6/.avro
>  matched 1 files with total size 9750
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3324: Avoid flakiness in data channel for empty streams.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3324


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Avoid flakiness in data channel for empty streams.

[lcwik]

Avoid flakiness in data channel for empty streams.

This closes #3324


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d81ed217
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d81ed217
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d81ed217

Branch: refs/heads/master
Commit: d81ed21726cb50962bc4fcfb57d50e11eacdcf3b
Parents: e066a9d 4ebebfd
Author: Luke Cwik 
Authored: Thu Jun 8 10:09:12 2017 -0700
Committer: Luke Cwik 
Committed: Thu Jun 8 10:09:12 2017 -0700

--
 .../python/apache_beam/runners/worker/data_plane.py | 16 +++-
 1 file changed, 11 insertions(+), 5 deletions(-)
--

[1/2] beam git commit: Avoid flakiness in data channel for empty streams.

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master e066a9d6d -> d81ed2172


Avoid flakiness in data channel for empty streams.

As empty stream is used as end-of-stream marker, don't ever send
it as the data itself.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4ebebfdb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4ebebfdb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4ebebfdb

Branch: refs/heads/master
Commit: 4ebebfdb34de3e209c033de15e32cf67ab346d44
Parents: e066a9d
Author: Robert Bradshaw 
Authored: Wed Jun 7 23:00:43 2017 -0700
Committer: Luke Cwik 
Committed: Thu Jun 8 10:06:17 2017 -0700

--
 .../python/apache_beam/runners/worker/data_plane.py | 16 +++-
 1 file changed, 11 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4ebebfdb/sdks/python/apache_beam/runners/worker/data_plane.py
--
diff --git a/sdks/python/apache_beam/runners/worker/data_plane.py 
b/sdks/python/apache_beam/runners/worker/data_plane.py
index 5edd0b4..7365db6 100644
--- a/sdks/python/apache_beam/runners/worker/data_plane.py
+++ b/sdks/python/apache_beam/runners/worker/data_plane.py
@@ -167,12 +167,18 @@ class _GrpcDataChannel(DataChannel):
 yield data
 
   def output_stream(self, instruction_id, target):
+# TODO: Return an output stream that sends data
+# to the Runner once a fixed size buffer is full.
+# Currently we buffer all the data before sending
+# any messages.
 def add_to_send_queue(data):
-  self._to_send.put(
-  beam_fn_api_pb2.Elements.Data(
-  instruction_reference=instruction_id,
-  target=target,
-  data=data))
+  if data:
+self._to_send.put(
+beam_fn_api_pb2.Elements.Data(
+instruction_reference=instruction_id,
+target=target,
+data=data))
+  # End of stream marker.
   self._to_send.put(
   beam_fn_api_pb2.Elements.Data(
   instruction_reference=instruction_id,

Build failed in Jenkins: beam_PostCommit_Python_Verify #2442

[Apache Jenkins Server]

See 


Changes:

[tgroh] Expand all PValues to component PCollections always

--
[...truncated 502.93 KB...]
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[TaggedOutput, int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3

[jira] [Closed] (BEAM-2418) Datastore IO does not work out of the box

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk closed BEAM-2418.
--

> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: Not applicable
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2364) PCollectionView should not be a PValue, it should expand to its underlying PCollection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042947#comment-16042947
 ] 

ASF GitHub Bot commented on BEAM-2364:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3303


> PCollectionView should not be a PValue, it should expand to its underlying 
> PCollection
> --
>
> Key: BEAM-2364
> URL: https://issues.apache.org/jira/browse/BEAM-2364
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>  Labels: backward-incompatible
>
> This bug has been present for a very long time.
> It is a change to {{@Internal}} details but is going to be 
> backwards-incompatible in a couple ways there, because backwards compatible 
> behavior is incorrect. But we need to be very careful with the surrounding 
> logic.
> The particular motivating need is that as long as a PCollectionView expands 
> to only itself, the outputs for surgery and serialization are not correct. 
> There may be a solution involving hardcoded logic, TBD.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3303: [BEAM-2364] Expand all PValues to component PCollec...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3303


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam git commit: Expand all PValues to component PCollections always

[tgroh]

Expand all PValues to component PCollections always

Update the implementation of WriteView

The PCollectionView is constructed within the composite override, but
WriteView just produces a primitive PCollection which has no consumers.
Track the ViewWriter within the Direct Runner, and utilize that
transform rather than the producer to update PCollection Watermarks.

Remove most Flink View overrides. All of the overrides are materially
identical within the flink runner, so use a single override to replace
all of them.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ccf73448
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ccf73448
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ccf73448

Branch: refs/heads/master
Commit: ccf7344820d6c69ca922aa3176dc141718382629
Parents: 86e1fab
Author: Thomas Groh 
Authored: Thu Jun 1 18:39:58 2017 -0700
Committer: Thomas Groh 
Committed: Thu Jun 8 09:13:57 2017 -0700

--
 .../apache/beam/runners/apex/ApexRunner.java|  59 +--
 .../translation/ApexPipelineTranslator.java |  16 +-
 .../construction/RunnerPCollectionView.java |   8 +
 .../apache/beam/runners/direct/DirectGraph.java |   4 +
 .../beam/runners/direct/DirectGraphVisitor.java |  22 +-
 .../runners/direct/ViewEvaluatorFactory.java|   8 +-
 .../runners/direct/ViewOverrideFactory.java |  29 +-
 .../beam/runners/direct/DirectGraphs.java   |   7 +
 .../runners/direct/EvaluationContextTest.java   |   5 +-
 .../ImmutabilityEnforcementFactoryTest.java |   4 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   1 +
 .../runners/direct/TransformExecutorTest.java   |   1 +
 .../direct/ViewEvaluatorFactoryTest.java|   5 +-
 .../runners/direct/ViewOverrideFactoryTest.java |  16 +-
 .../direct/WatermarkCallbackExecutorTest.java   |   1 +
 .../runners/direct/WatermarkManagerTest.java|   1 +
 .../runners/flink/CreateStreamingFlinkView.java | 154 
 .../flink/FlinkStreamingPipelineTranslator.java |  36 +-
 .../FlinkStreamingTransformTranslators.java |   8 +-
 .../flink/FlinkStreamingViewOverrides.java  | 372 ---
 .../runners/dataflow/BatchViewOverrides.java| 182 +++--
 .../runners/dataflow/CreateDataflowView.java|   8 +-
 .../dataflow/DataflowPipelineTranslator.java|  11 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  17 +-
 .../dataflow/StreamingViewOverrides.java|  10 +-
 .../DataflowPipelineTranslatorTest.java |   6 +-
 .../spark/translation/TransformTranslator.java  |  50 +--
 .../beam/sdk/runners/TransformHierarchy.java|  46 ++-
 .../org/apache/beam/sdk/transforms/Combine.java |  17 +-
 .../org/apache/beam/sdk/transforms/View.java|  38 +-
 .../org/apache/beam/sdk/values/PCollection.java |  12 +
 .../beam/sdk/values/PCollectionViews.java   |  14 +
 .../org/apache/beam/sdk/values/PValueBase.java  |  12 -
 .../sdk/testing/PCollectionViewTesting.java |   8 +
 34 files changed, 458 insertions(+), 730 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ccf73448/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
index c595b3f..95b354a 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
@@ -62,8 +62,6 @@ import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.ParDo.MultiOutput;
-import org.apache.beam.sdk.transforms.View;
-import org.apache.beam.sdk.transforms.View.AsIterable;
 import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionTuple;
@@ -214,7 +212,7 @@ public class ApexRunner extends 
PipelineRunner {
* @param  The type associated with the {@link PCollectionView} used 
as a side input
*/
   public static class CreateApexPCollectionView
-  extends PTransform>, PCollectionView> {
+  extends PTransform, PCollection> {
 private static final long serialVersionUID = 1L;
 private PCollectionView view;
 
@@ -228,7 +226,13 @@ public class ApexRunner extends 
PipelineRunner {
 }
 
 @Override
-public PCollectionView expand(PCollection> input) {
+public PCollection expand(PCollection input) {
+  return PCollection.createPrimitiveOutputInternal(
+  input.getPipeline(), input.getWindowingStrategy(), 
input.isBounded())
+  .setCoder(input.getC

[2/3] beam git commit: Expand all PValues to component PCollections always

[tgroh]

http://git-wip-us.apache.org/repos/asf/beam/blob/ccf73448/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/StreamingViewOverrides.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/StreamingViewOverrides.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/StreamingViewOverrides.java
index 6c385d7..1853248 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/StreamingViewOverrides.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/StreamingViewOverrides.java
@@ -42,12 +42,12 @@ import org.apache.beam.sdk.values.PCollectionView;
 class StreamingViewOverrides {
   static class StreamingCreatePCollectionViewFactory
   extends SingleInputOutputOverrideFactory<
-  PCollection, PCollectionView, 
CreatePCollectionView> {
+  PCollection, PCollection, CreatePCollectionView> {
 @Override
-public PTransformReplacement, PCollectionView>
+public PTransformReplacement, PCollection>
 getReplacementTransform(
 AppliedPTransform<
-PCollection, PCollectionView, 
CreatePCollectionView>
+PCollection, PCollection, 
CreatePCollectionView>
 transform) {
   StreamingCreatePCollectionView streamingView =
   new 
StreamingCreatePCollectionView<>(transform.getTransform().getView());
@@ -56,7 +56,7 @@ class StreamingViewOverrides {
 }
 
 private static class StreamingCreatePCollectionView
-extends PTransform, PCollectionView> {
+extends PTransform, PCollection> {
   private final PCollectionView view;
 
   private StreamingCreatePCollectionView(PCollectionView view) {
@@ -64,7 +64,7 @@ class StreamingViewOverrides {
   }
 
   @Override
-  public PCollectionView expand(PCollection input) {
+  public PCollection expand(PCollection input) {
 return input
 .apply(Combine.globally(new 
Concatenate()).withoutDefaults())
 .apply(ParDo.of(StreamingPCollectionViewWriterFn.create(view, 
input.getCoder(

http://git-wip-us.apache.org/repos/asf/beam/blob/ccf73448/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
index 89dc2d5..53215f6 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
@@ -920,15 +920,15 @@ public class DataflowPipelineTranslatorTest implements 
Serializable {
 assertAllStepOutputsHaveUniqueIds(job);
 
 List steps = job.getSteps();
-assertEquals(5, steps.size());
+assertEquals(9, steps.size());
 
 @SuppressWarnings("unchecked")
 List> toIsmRecordOutputs =
-(List>) 
steps.get(3).getProperties().get(PropertyNames.OUTPUT_INFO);
+(List>) 
steps.get(7).getProperties().get(PropertyNames.OUTPUT_INFO);
 assertTrue(
 Structs.getBoolean(Iterables.getOnlyElement(toIsmRecordOutputs), 
"use_indexed_format"));
 
-Step collectionToSingletonStep = steps.get(4);
+Step collectionToSingletonStep = steps.get(8);
 assertEquals("CollectionToSingleton", collectionToSingletonStep.getKind());
   }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ccf73448/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
index 64aa35a..ac5e0cd 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
@@ -508,50 +508,6 @@ public final class TransformTranslator {
 };
   }
 
-  private static  TransformEvaluator> viewAsSingleton() 
{
-return new TransformEvaluator>() {
-  @Override
-  public void evaluate(View.AsSingleton transform, EvaluationContext 
context) {
-Iterable> iter =
-context.getWindowedValues(context.getInput(transform));
-PCollectionView output = context.getOutput(transform);
-Coder>> coderInternal = 
output.getCoderInte

[1/3] beam git commit: This closes #3303

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 86e1fab69 -> e066a9d6d


This closes #3303


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e066a9d6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e066a9d6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e066a9d6

Branch: refs/heads/master
Commit: e066a9d6dfc4d76f2d7d49de9f744b978982bb81
Parents: 86e1fab ccf7344
Author: Thomas Groh 
Authored: Thu Jun 8 09:13:57 2017 -0700
Committer: Thomas Groh 
Committed: Thu Jun 8 09:13:57 2017 -0700

--
 .../apache/beam/runners/apex/ApexRunner.java|  59 +--
 .../translation/ApexPipelineTranslator.java |  16 +-
 .../construction/RunnerPCollectionView.java |   8 +
 .../apache/beam/runners/direct/DirectGraph.java |   4 +
 .../beam/runners/direct/DirectGraphVisitor.java |  22 +-
 .../runners/direct/ViewEvaluatorFactory.java|   8 +-
 .../runners/direct/ViewOverrideFactory.java |  29 +-
 .../beam/runners/direct/DirectGraphs.java   |   7 +
 .../runners/direct/EvaluationContextTest.java   |   5 +-
 .../ImmutabilityEnforcementFactoryTest.java |   4 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   1 +
 .../runners/direct/TransformExecutorTest.java   |   1 +
 .../direct/ViewEvaluatorFactoryTest.java|   5 +-
 .../runners/direct/ViewOverrideFactoryTest.java |  16 +-
 .../direct/WatermarkCallbackExecutorTest.java   |   1 +
 .../runners/direct/WatermarkManagerTest.java|   1 +
 .../runners/flink/CreateStreamingFlinkView.java | 154 
 .../flink/FlinkStreamingPipelineTranslator.java |  36 +-
 .../FlinkStreamingTransformTranslators.java |   8 +-
 .../flink/FlinkStreamingViewOverrides.java  | 372 ---
 .../runners/dataflow/BatchViewOverrides.java| 182 +++--
 .../runners/dataflow/CreateDataflowView.java|   8 +-
 .../dataflow/DataflowPipelineTranslator.java|  11 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  17 +-
 .../dataflow/StreamingViewOverrides.java|  10 +-
 .../DataflowPipelineTranslatorTest.java |   6 +-
 .../spark/translation/TransformTranslator.java  |  50 +--
 .../beam/sdk/runners/TransformHierarchy.java|  46 ++-
 .../org/apache/beam/sdk/transforms/Combine.java |  17 +-
 .../org/apache/beam/sdk/transforms/View.java|  38 +-
 .../org/apache/beam/sdk/values/PCollection.java |  12 +
 .../beam/sdk/values/PCollectionViews.java   |  14 +
 .../org/apache/beam/sdk/values/PValueBase.java  |  12 -
 .../sdk/testing/PCollectionViewTesting.java |   8 +
 34 files changed, 458 insertions(+), 730 deletions(-)
--

Build failed in Jenkins: beam_PostCommit_Python_Verify #2441

[Apache Jenkins Server]

See 


--
[...truncated 502.58 KB...]
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[TaggedOutput, int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #108

[Apache Jenkins Server]

See 


--
[...truncated 2.55 MB...]
2017-06-08T12:35:55.553 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-06-08T12:35:55.579 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 57.9 KB/sec)
2017-06-08T12:35:55.582 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-08T12:35:55.608 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 432.9 KB/sec)
2017-06-08T12:35:55.612 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-08T12:35:55.635 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 534.7 KB/sec)
2017-06-08T12:35:55.639 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-08T12:35:55.664 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 86.9 KB/sec)
2017-06-08T12:35:55.669 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-08T12:35:55.693 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 65.6 KB/sec)
2017-06-08T12:35:55.697 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-08T12:35:55.722 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 210.0 KB/sec)
2017-06-08T12:35:55.726 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-08T12:35:55.749 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 108.5 KB/sec)
2017-06-08T12:35:55.753 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-08T12:35:55.776 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 149.0 KB/sec)
2017-06-08T12:35:55.780 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-08T12:35:55.806 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 38.6 KB/sec)
2017-06-08T12:35:55.811 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-08T12:35:55.835 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 70.7 KB/sec)
2017-06-08T12:35:55.841 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-08T12:35:55.866 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 70.5 KB/sec)
2017-06-08T12:35:55.871 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-08T12:35:55.895 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 110.6 KB/sec)
2017-06-08T12:35:55.900 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-08T12:35:55.926 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 677.5 KB/sec)
2017-06-08T12:35:55.930 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-08T12:35:55.956 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 362.6 KB/sec)
2017-06-08T12:35:55.960 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-08T12:35:55.986 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
 (5 KB at 171.3 KB/sec)
2017-06-08T12:35:55.992 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.pom
2017-06-08T12:35:56.017 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-lau

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #60

[Apache Jenkins Server]

See 


--
[...truncated 1.26 MB...]
2017-06-08\T\12:31:15.687 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/clapper/grizzled-slf4j_2.10/1.0.2/grizzled-slf4j_2.10-1.0.2.pom
2017-06-08\T\12:31:15.712 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/clapper/grizzled-slf4j_2.10/1.0.2/grizzled-slf4j_2.10-1.0.2.pom
 (2 KB at 74.8 KB/sec)
2017-06-08\T\12:31:15.714 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
2017-06-08\T\12:31:15.740 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
 (2 KB at 76.8 KB/sec)
2017-06-08\T\12:31:15.741 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-08\T\12:31:15.774 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 49.6 KB/sec)
2017-06-08\T\12:31:15.775 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-08\T\12:31:15.802 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 85.8 KB/sec)
2017-06-08\T\12:31:15.804 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-08\T\12:31:15.829 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 76.8 KB/sec)
2017-06-08\T\12:31:15.830 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-08\T\12:31:15.856 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 76.4 KB/sec)
2017-06-08\T\12:31:15.857 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-08\T\12:31:15.885 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 150.6 KB/sec)
2017-06-08\T\12:31:15.889 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-08\T\12:31:15.916 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 223.3 KB/sec)
2017-06-08\T\12:31:15.920 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-08\T\12:31:15.947 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 117.2 KB/sec)
2017-06-08\T\12:31:15.948 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-08\T\12:31:15.975 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 928.3 KB/sec)
2017-06-08\T\12:31:15.977 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-08\T\12:31:16.004 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
508.3 KB/sec)
2017-06-08\T\12:31:16.017 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-08\T\12:31:16.044 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 171.5 KB/sec)
2017-06-08\T\12:31:16.045 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-08\T\12:31:16.074 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 48.5 KB/sec)
2017-06-08\T\12:31:16.078 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-08\T\12:31:16.103 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 128.4 KB/sec)
2017-06-08\T\12:31:16.108 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
2017-06-08\T\12:31:16.135 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
 (5 KB 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #60

[Apache Jenkins Server]

See 


--
[...truncated 1.25 MB...]
2017-06-08\T\12:27:51.552 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-08\T\12:27:51.577 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 65.4 KB/sec)
2017-06-08\T\12:27:51.579 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-08\T\12:27:51.613 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 68.2 KB/sec)
2017-06-08\T\12:27:51.614 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-08\T\12:27:51.640 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 76.8 KB/sec)
2017-06-08\T\12:27:51.641 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-08\T\12:27:51.667 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 76.4 KB/sec)
2017-06-08\T\12:27:51.668 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-08\T\12:27:51.695 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 156.2 KB/sec)
2017-06-08\T\12:27:51.701 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-08\T\12:27:51.728 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 223.3 KB/sec)
2017-06-08\T\12:27:51.731 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-08\T\12:27:51.759 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 113.0 KB/sec)
2017-06-08\T\12:27:51.760 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-08\T\12:27:51.788 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 895.2 KB/sec)
2017-06-08\T\12:27:51.790 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-08\T\12:27:51.817 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
508.3 KB/sec)
2017-06-08\T\12:27:51.828 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-08\T\12:27:51.854 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 178.1 KB/sec)
2017-06-08\T\12:27:51.855 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-08\T\12:27:51.881 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 54.1 KB/sec)
2017-06-08\T\12:27:51.884 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-08\T\12:27:51.910 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 123.4 KB/sec)
2017-06-08\T\12:27:51.915 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
2017-06-08\T\12:27:51.940 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.12.0/curator-test-2.12.0.pom
 (5 KB at 162.0 KB/sec)
2017-06-08\T\12:27:51.941 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
2017-06-08\T\12:27:51.974 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.12.0/apache-curator-2.12.0.pom
 (32 KB at 956.7 KB/sec)
2017-06-08\T\12:27:51.977 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.8/zookeeper-3.4.8.pom
2017-06-08\T\12:27:52.005 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.8/zookeeper-3.4.8.pom
 (4 KB at 140.4 KB/sec)
2017

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #60

[Apache Jenkins Server]

See 


--
[...truncated 895.14 KB...]
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding org.json:json:jar:20160810 from the 
shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding com.google.api:gax:jar:1.1.0 from 
the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding org.threeten:threetenbp:jar:1.3.3 
from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api.grpc:proto-google-iam-v1:jar:0.1.11 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.cloud:google-cloud-spanner:jar:0.16.0-beta from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.cloud:google-cloud-core-grpc:jar:1.0.0-rc3 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding com.google.api:gax-grpc:jar:0.15.0 
from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.9 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.9 from 
the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.9 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.9 from 
the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.9 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.9 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.cloud.bigtable:bigtable-client-core:jar:0.9.6.2 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
commons-logging:commons-logging:jar:1.2 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.auth:google-auth-library-appengine:jar:0.6.0 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.appengine:appengine-api-1.0-sdk:jar:1.9.34 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Including 
org.apache.beam:beam-runners-core-construction-java:jar:2.1.0-SNAPSHOT in the 
shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding com.google.code.gson:gson:jar:2.7 
from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.oauth-client:google-oauth-client:jar:1.22.0 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.http-client:google-http-client:jar:1.22.0 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding commons-codec:commons-codec:jar:1.3 
from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.http-client:google-http-client-jackson2:jar:1.22.0 from the shaded 
jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev196-1.22.0 from the 
shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.auth:google-auth-library-credentials:jar:0.6.1 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.auth:google-auth-library-oauth2-http:jar:0.6.1 from the shaded jar.
2017-06-08\T\12:11:45.809 [INFO] Excluding 
com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
2017-06-08\T\12:11:45.809 

[1/2] beam git commit: upgrade to version 2.1.0-SNAPSHOT

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/DSL_SQL 4c5b7584a -> 5c1f2cbc6


upgrade to version 2.1.0-SNAPSHOT


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/03a913a9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/03a913a9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/03a913a9

Branch: refs/heads/DSL_SQL
Commit: 03a913a95c99474841a175b727925ba7c1eed4c9
Parents: 4c5b758
Author: mingmxu 
Authored: Wed Jun 7 19:27:32 2017 -0700
Committer: mingmxu 
Committed: Wed Jun 7 19:27:32 2017 -0700

--
 dsls/pom.xml|  2 +-
 dsls/sql/pom.xml| 43 +++---
 .../beam/dsls/sql/schema/BeamSqlRowCoder.java   | 59 ++--
 .../dsls/sql/schema/kafka/BeamKafkaTable.java   | 20 +--
 .../dsls/sql/schema/text/BeamTextCSVTable.java  |  2 +-
 .../schema/text/BeamTextCSVTableIOWriter.java   |  2 +-
 6 files changed, 52 insertions(+), 76 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/03a913a9/dsls/pom.xml
--
diff --git a/dsls/pom.xml b/dsls/pom.xml
index 6f9d635..a741563 100644
--- a/dsls/pom.xml
+++ b/dsls/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.7.0-SNAPSHOT
+2.1.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/03a913a9/dsls/sql/pom.xml
--
diff --git a/dsls/sql/pom.xml b/dsls/sql/pom.xml
index bc658e6..39e32c4 100644
--- a/dsls/sql/pom.xml
+++ b/dsls/sql/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-dsls-parent
-0.7.0-SNAPSHOT
+2.1.0-SNAPSHOT
   
 
   beam-dsls-sql
@@ -117,41 +117,6 @@
 
   
 
-  
-
-  
-org.apache.beam
-beam-sdks-java-core
-0.6.0
-  
-  
-org.apache.beam
-beam-runners-direct-java
-0.6.0
-  
-  
-org.apache.beam
-beam-sdks-java-io-kafka
-0.6.0
-  
-  
-org.apache.beam
-beam-runners-core-java
-0.6.0
-  
-  
-org.apache.beam
-beam-sdks-common-runner-api
-0.6.0
-  
-  
-org.apache.beam
-beam-runners-core-construction-java
-0.6.0
-  
-
-  
-
   
 
   junit
@@ -213,5 +178,11 @@
   joda-time
   joda-time
 
+
+  org.apache.kafka
+  kafka-clients
+  0.10.1.0
+  provided
+
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/03a913a9/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRowCoder.java
--
diff --git 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRowCoder.java 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRowCoder.java
index f161d27..14a0f31 100644
--- 
a/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRowCoder.java
+++ 
b/dsls/sql/src/main/java/org/apache/beam/dsls/sql/schema/BeamSqlRowCoder.java
@@ -54,9 +54,8 @@ public class BeamSqlRowCoder extends CustomCoder {
   }
 
   @Override
-  public void encode(BeamSQLRow value, OutputStream outStream,
-  org.apache.beam.sdk.coders.Coder.Context context) throws CoderException, 
IOException {
-listCoder.encode(value.getNullFields(), outStream, context.nested());
+  public void encode(BeamSQLRow value, OutputStream outStream) throws 
CoderException, IOException {
+listCoder.encode(value.getNullFields(), outStream);
 
 for (int idx = 0; idx < value.size(); ++idx) {
   if (value.getNullFields().contains(idx)) {
@@ -65,36 +64,35 @@ public class BeamSqlRowCoder extends 
CustomCoder {
 
   switch (value.getDataType().getFieldsType().get(idx)) {
 case INTEGER:
-  intCoder.encode(value.getInteger(idx), outStream, context.nested());
+  intCoder.encode(value.getInteger(idx), outStream);
   break;
 case SMALLINT:
-  intCoder.encode((int) value.getShort(idx), outStream, 
context.nested());
+  intCoder.encode((int) value.getShort(idx), outStream);
   break;
 case TINYINT:
-  intCoder.encode((int) value.getByte(idx), outStream, 
context.nested());
+  intCoder.encode((int) value.getByte(idx), outStream);
   break;
 case DOUBLE:
-  doubleCoder.encode(value.getDouble(idx), outStream, 
context.nested());
+  doubleCoder.encode(value.getDouble(idx), outStream);
   break;
 case FLOAT:
-  doubleCoder.encode((double) value.getFloat(idx), outStream, 
context.nested());
+  doubleCoder.encode((double) value.getFloat(idx), outStream);
   break;
 case DECIMAL:
-  bigDec

[GitHub] beam pull request #3323: upgrade to 2.1.0-SNAPSHOT

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3323


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3323

[jbonofre]

This closes #3323


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5c1f2cbc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5c1f2cbc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5c1f2cbc

Branch: refs/heads/DSL_SQL
Commit: 5c1f2cbc6c54354612c142415b77a2aded84a06d
Parents: 4c5b758 03a913a
Author: Jean-Baptiste Onofré 
Authored: Thu Jun 8 13:44:18 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Thu Jun 8 13:44:18 2017 +0200

--
 dsls/pom.xml|  2 +-
 dsls/sql/pom.xml| 43 +++---
 .../beam/dsls/sql/schema/BeamSqlRowCoder.java   | 59 ++--
 .../dsls/sql/schema/kafka/BeamKafkaTable.java   | 20 +--
 .../dsls/sql/schema/text/BeamTextCSVTable.java  |  2 +-
 .../schema/text/BeamTextCSVTableIOWriter.java   |  2 +-
 6 files changed, 52 insertions(+), 76 deletions(-)
--

Build failed in Jenkins: beam_PostCommit_Python_Verify #2440

[Apache Jenkins Server]

See 


--
[...truncated 502.14 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
test_read_from_text_metrics 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.0.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:135:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, None], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:135:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Co

[jira] [Commented] (BEAM-2410) Remove TransportClient from ElasticSearchIO to decouple IO and ES versions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042456#comment-16042456
 ] 

ASF GitHub Bot commented on BEAM-2410:
--

GitHub user echauchot opened a pull request:

https://github.com/apache/beam/pull/3328

[BEAM-2410] Remove TransportClient from ElasticSearchIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`.
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This PR is a first step to support Elasticsearch 5.x but it is also very 
useful for the IO that supports Elasticsearch 2.x, hence the separate PR.

R: @jbonofre 
R: @jkff 
CC: @ssisk regarding my comment 
https://issues.apache.org/jira/browse/BEAM-2347?focusedCommentId=16022861&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16022861

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/echauchot/beam 
BEAM-2410-ELASTICSEARCHIO-REMOVE-TRANSPORTCLIENT

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3328.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3328


commit a41afd46dc6f057ec7eea24c3ab2ff996258e5f6
Author: Etienne Chauchot 
Date:   2017-06-05T14:21:58Z

[BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple IO and 
ES server versions




> Remove TransportClient from ElasticSearchIO to decouple IO and ES versions
> --
>
> Key: BEAM-2410
> URL: https://issues.apache.org/jira/browse/BEAM-2410
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>
> Elasticsearch IO uses TransportClient in tests (UT and IT) but uses 
> RestClient in the production code. TransportClient version is coupled with 
> Elasticsearch server version. Use RestClient everywhere to lower the coupling.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3328: [BEAM-2410] Remove TransportClient from ElasticSear...

[echauchot]

GitHub user echauchot opened a pull request:

https://github.com/apache/beam/pull/3328

[BEAM-2410] Remove TransportClient from ElasticSearchIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`.
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This PR is a first step to support Elasticsearch 5.x but it is also very 
useful for the IO that supports Elasticsearch 2.x, hence the separate PR.

R: @jbonofre 
R: @jkff 
CC: @ssisk regarding my comment 
https://issues.apache.org/jira/browse/BEAM-2347?focusedCommentId=16022861&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16022861

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/echauchot/beam 
BEAM-2410-ELASTICSEARCHIO-REMOVE-TRANSPORTCLIENT

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3328.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3328


commit a41afd46dc6f057ec7eea24c3ab2ff996258e5f6
Author: Etienne Chauchot 
Date:   2017-06-05T14:21:58Z

[BEAM-2410] Remove TransportClient from ElasticSearchIO to decouple IO and 
ES server versions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1941) Add Watermark Metrics in Runners

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042397#comment-16042397
 ] 

ASF GitHub Bot commented on BEAM-1941:
--

Github user JingsongLi closed the pull request at:

https://github.com/apache/beam/pull/2856


> Add Watermark Metrics in Runners
> 
>
> Key: BEAM-1941
> URL: https://issues.apache.org/jira/browse/BEAM-1941
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>
> The source watermark metrics show the consumer latency of Source. 
> It allows the user to know the health of the job, or it can be used to 
> monitor and alarm.
> Since each runner is likely already tracking a watermark, another option here 
> is to just have the runner report it appropriately, rather than having the 
> source report it using metrics. This also addresses the fact that even if the 
> source has advanced to 8:00, the runner may still know about buffered 
> elements at 7:00, and so not advance the watermark all the way to 8:00. 
> [~bchambers]
> Includes:
> 1.Source watermark (`min` amongst all splits):
>type = Gauge, namespace = io, name = source_watermark
> 2.Source watermark per split:
>type = Gauge, namespace = io.splits, name = .source_watermark



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)