Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Gearpump #177

[Apache Jenkins Server]

See 


Changes:

[lcwik] Properly convert milliseconds whether there's less than 3/more than 9

[klk] Add support for PipelineOptions parameters

[klk] Fix DoFn javadoc: StateSpec does not require a key

[jbonofre] Make modules that depend on Hadoop and Spark use the same version

[jbonofre] [BEAM-2530] Fix compilation of modules with Java 9 that depend on

[altay] For GCS operations use an http client with a default timeout value.

[jbonofre] [maven-release-plugin] prepare branch release-2.1.0

[jbonofre] [maven-release-plugin] prepare for next development iteration

[jbonofre] Update Python SDK version

[tgroh] Website Mergebot Job

[lcwik] [BEAM-2553] Update Maven exec plugin to 1.6.0 to incorporate messaging

[tgroh] Disallow Combiner Lifting for multi-window WindowFns

[altay] Update SDK dependencies

[altay] Fix PValue input in _PubSubReadEvaluator

[lzljs3620320] [BEAM-2534] Handle offset gaps in Kafka messages.

[klk] Add timeout to initialization of partition in KafkaIO

[klk] Ignore processing time timers in expired windows

[klk] Process timer firings for a window together

[klk] Made DataflowRunner TransformTranslator public

[klk] Fix bad merge

[klk] Simplified ByteBuddyOnTimerInvokerFactory

[jbonofre] Fix javadoc generation for AmqpIO, CassandraIO and HCatalogIO

[owenzhang1990] Upgrade to gearpump 0.8.4

[owenzhang1990] Fix ParDoTest#testPipelineOptionsParameter

--
[...truncated 856.91 KB...]
2017-07-10T04:55:12.036 [INFO] Running 
org.apache.beam.sdk.values.PCollectionTupleTest
2017-07-10T04:55:25.673 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 13.626 s - in org.apache.beam.sdk.values.PCollectionTupleTest
2017-07-10T04:55:25.673 [INFO] Running org.apache.beam.sdk.PipelineTest
2017-07-10T04:55:33.457 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 7.781 s - in org.apache.beam.sdk.PipelineTest
2017-07-10T04:55:33.457 [INFO] Running org.apache.beam.sdk.io.ReadTest
2017-07-10T04:55:33.634 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.172 s - in org.apache.beam.sdk.io.ReadTest
2017-07-10T04:55:33.634 [INFO] Running org.apache.beam.sdk.testing.PAssertTest
2017-07-10T04:57:04.841 [ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 
0, Time elapsed: 91.055 s <<< FAILURE! - in 
org.apache.beam.sdk.testing.PAssertTest
2017-07-10T04:57:04.841 [ERROR] 
testGlobalWindowContainsInAnyOrder(org.apache.beam.sdk.testing.PAssertTest)  
Time elapsed: 1.443 s  <<< ERROR!
org.apache.gearpump.serializer.FastKryoSerializer$KryoSerializationException: 

 Class is not registered: 
org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow
Note: To register this class use: 
kryo.register(org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow.class);
You can also register the class by providing a configuration with serializer
defined,

gearpump{
  serializers {
## Follow this format when adding new serializer for new message types
#"yourpackage.YourClass" = "yourpackage.YourSerializerForThisClass"

## If you intend to use default serializer for this class, then you can 
write this
#"yourpackage.YourClass" = ""
  }
}

If you want to register the serializer globally, you need to change
gear.conf on every worker in the cluster; if you only want to register
the serializer for a single streaming application, you need to create
a file under conf/ named application.conf, and add the above configuration
into application.conf. To verify whether the configuration is effective,
you can browser your UI http://{UI Server 
Host}:8090/api/v1.0/app/{appId}/config,
and check whether your custom serializer is added.

at 
org.apache.gearpump.serializer.FastKryoSerializer.serialize(FastKryoSerializer.scala:71)
at 
org.apache.gearpump.streaming.task.ExpressTransport$$anonfun$transport$1.apply(ExpressTransport.scala:57)
at 
org.apache.gearpump.streaming.task.ExpressTransport$$anonfun$transport$1.apply(ExpressTransport.scala:48)
at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at 
org.apache.gearpump.streaming.task.ExpressTransport$class.transport(ExpressTransport.scala:48)
at 
org.apache.gearpump.streaming.task.TaskActor.transport(TaskActor.scala:47)
at 
org.apache.gearpump.streaming.task.Subscription.sendMessage(Subscription.scala:113)
at 
org.apache.gearpump.streaming.task.Subscription$$anonfun$3.apply(Subscription.scala:76)
at 
org.apache.gearpump.streaming.task.Subscription$$anonfun$3.apply(Subscription.scala:74)
at 
org.apache.gearpump.streaming.task.Subscription.sendMessage(Subscription.scala:95)
at 

[jira] [Commented] (BEAM-2574) test unsupported/invalid cases in DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079886#comment-16079886
 ] 

ASF GitHub Bot commented on BEAM-2574:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3530

[BEAM-2574] test unsupported/invalid cases in DSL

R: @xumingming @takidau 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2574

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3530.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3530


commit 7fcfaa9770f33c7b07fb1cfcea907cfa4aad43c1
Author: mingmxu 
Date:   2017-07-10T05:26:29Z

update unit test to cover exception cases




> test unsupported/invalid cases in DSL
> -
>
> Key: BEAM-2574
> URL: https://issues.apache.org/jira/browse/BEAM-2574
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>
> add test cases to cover the scenarios which are not supported, or have 
> invalid usages.
> Note, the previous failure of test is because of {{TestPipeline}} is marked 
> as {{ClassRule}}, it works with {{Rule}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3530: [BEAM-2574] test unsupported/invalid cases in DSL

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3530

[BEAM-2574] test unsupported/invalid cases in DSL

R: @xumingming @takidau 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2574

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3530.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3530


commit 7fcfaa9770f33c7b07fb1cfcea907cfa4aad43c1
Author: mingmxu 
Date:   2017-07-10T05:26:29Z

update unit test to cover exception cases




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4337

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2231) ApiSurface should be lazy

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079865#comment-16079865
 ] 

Kenneth Knowles commented on BEAM-2231:
---

It has been a while, so I may not have the best recollection of the pieces of 
the code. But I think that {{addExposedTypes}} should skip pruned namespaces.

> ApiSurface should be lazy
> -
>
> Key: BEAM-2231
> URL: https://issues.apache.org/jira/browse/BEAM-2231
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Innocent
>
> Currently, the ApiSurface loads classes recursively, when they should be 
> pruned before loading by the pruning pattern. This has caused crashes because 
> some classes that are never referenced in our code.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[19/43] beam git commit: Disallow Combiner Lifting for multi-window WindowFns

[kenn]

Disallow Combiner Lifting for multi-window WindowFns


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/29c2bca4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/29c2bca4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/29c2bca4

Branch: refs/heads/gearpump-runner
Commit: 29c2bca4649317f2ebb1c89f92bf97fbb27602ca
Parents: 39c80f6
Author: Thomas Groh 
Authored: Wed Jul 5 14:16:50 2017 -0700
Committer: Thomas Groh 
Committed: Wed Jul 5 14:16:50 2017 -0700

--
 .../apache/beam/runners/dataflow/DataflowPipelineTranslator.java| 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/29c2bca4/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 28fd1bb..f1783de 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -793,6 +793,7 @@ public class DataflowPipelineTranslator {
 
context.getPipelineOptions().as(StreamingOptions.class).isStreaming();
 boolean disallowCombinerLifting =
 !windowingStrategy.getWindowFn().isNonMerging()
+|| !windowingStrategy.getWindowFn().assignsToOneWindow()
 || (isStreaming && !transform.fewKeys())
 // TODO: Allow combiner lifting on the non-default 
trigger, as appropriate.
 || !(windowingStrategy.getTrigger() instanceof 
DefaultTrigger);

[26/43] beam git commit: [BEAM-2534] Handle offset gaps in Kafka messages.

[kenn]

[BEAM-2534] Handle offset gaps in Kafka messages.

KafkaIO logged a warning when there is a gap in offstes for messages.
Kafka also support 'KV' store style topics where some of the messages
are deleted leading gaps in offsets. This PR removes the log and
accounts for offset gaps in backlog estimate.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2259c309
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2259c309
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2259c309

Branch: refs/heads/gearpump-runner
Commit: 2259c309c5b81a5d1e32732dd35e1102766401fa
Parents: bf6dda3
Author: Raghu Angadi 
Authored: Wed Jun 28 12:07:06 2017 -0700
Committer: JingsongLi 
Committed: Fri Jul 7 00:44:02 2017 +0800

--
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   | 49 
 1 file changed, 29 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2259c309/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index 702bdd3..e520367 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -904,6 +904,22 @@ public class KafkaIO {
   return name;
 }
 
+// Maintains approximate average over last 1000 elements
+private static class MovingAvg {
+  private static final int MOVING_AVG_WINDOW = 1000;
+  private double avg = 0;
+  private long numUpdates = 0;
+
+  void update(double quantity) {
+numUpdates++;
+avg += (quantity - avg) / Math.min(MOVING_AVG_WINDOW, numUpdates);
+  }
+
+  double get() {
+return avg;
+  }
+}
+
 // maintains state of each assigned partition (buffered records, consumed 
offset, etc)
 private static class PartitionState {
   private final TopicPartition topicPartition;
@@ -911,9 +927,8 @@ public class KafkaIO {
   private long latestOffset;
   private Iterator> recordIter = 
Collections.emptyIterator();
 
-  // simple moving average for size of each record in bytes
-  private double avgRecordSize = 0;
-  private static final int movingAvgWindow = 1000; // very roughly avg of 
last 1000 elements
+  private MovingAvg avgRecordSize = new MovingAvg();
+  private MovingAvg avgOffsetGap = new MovingAvg(); // > 0 only when log 
compaction is enabled.
 
   PartitionState(TopicPartition partition, long nextOffset) {
 this.topicPartition = partition;
@@ -921,17 +936,13 @@ public class KafkaIO {
 this.latestOffset = UNINITIALIZED_OFFSET;
   }
 
-  // update consumedOffset and avgRecordSize
-  void recordConsumed(long offset, int size) {
+  // Update consumedOffset, avgRecordSize, and avgOffsetGap
+  void recordConsumed(long offset, int size, long offsetGap) {
 nextOffset = offset + 1;
 
-// this is always updated from single thread. probably not worth 
making it an AtomicDouble
-if (avgRecordSize <= 0) {
-  avgRecordSize = size;
-} else {
-  // initially, first record heavily contributes to average.
-  avgRecordSize += ((size - avgRecordSize) / movingAvgWindow);
-}
+// This is always updated from single thread. Probably not worth 
making atomic.
+avgRecordSize.update(size);
+avgOffsetGap.update(offsetGap);
   }
 
   synchronized void setLatestOffset(long latestOffset) {
@@ -944,14 +955,15 @@ public class KafkaIO {
 if (backlogMessageCount == UnboundedReader.BACKLOG_UNKNOWN) {
   return UnboundedReader.BACKLOG_UNKNOWN;
 }
-return (long) (backlogMessageCount * avgRecordSize);
+return (long) (backlogMessageCount * avgRecordSize.get());
   }
 
   synchronized long backlogMessageCount() {
 if (latestOffset < 0 || nextOffset < 0) {
   return UnboundedReader.BACKLOG_UNKNOWN;
 }
-return Math.max(0, (latestOffset - nextOffset));
+double remaining = (latestOffset - nextOffset) / (1 + 
avgOffsetGap.get());
+return Math.max(0, (long) Math.ceil(remaining));
   }
 }
 
@@ -1154,14 +1166,11 @@ public class KafkaIO {
 continue;
   }
 
-  // sanity check
-  if (offset != expected) {
-LOG.warn("{}: gap in offsets for {} at {}. {} records missing.",
-this, pState.topicPartition, expected, offset - expected);
-  }
+  

[08/43] beam git commit: [BEAM-2530] Fix compilation of modules with Java 9 that depend on jdk.tools

[kenn]

[BEAM-2530] Fix compilation of modules with Java 9 that depend on jdk.tools


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/75475ef3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/75475ef3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/75475ef3

Branch: refs/heads/gearpump-runner
Commit: 75475ef3dc23a09fa9bbba478d6fdbc468f7dd2e
Parents: 11010cf
Author: Ismaël Mejía 
Authored: Wed Jun 28 16:58:55 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Mon Jul 3 10:10:04 2017 +0200

--
 runners/apex/pom.xml  |  7 +++
 runners/spark/pom.xml |  7 +++
 sdks/java/io/hbase/pom.xml|  7 +++
 sdks/java/io/hcatalog/pom.xml | 12 
 4 files changed, 33 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/75475ef3/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 88ff0f2..20f2d28 100644
--- a/runners/apex/pom.xml
+++ b/runners/apex/pom.xml
@@ -75,6 +75,13 @@
   apex-engine
   ${apex.core.version}
   runtime
+  
+
+
+  jdk.tools
+  jdk.tools
+
+  
 
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/75475ef3/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index 1d93427..8a69496 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -149,6 +149,13 @@
   org.apache.hadoop
   hadoop-common
   provided
+  
+
+
+  jdk.tools
+  jdk.tools
+
+  
 
 
   org.apache.hadoop

http://git-wip-us.apache.org/repos/asf/beam/blob/75475ef3/sdks/java/io/hbase/pom.xml
--
diff --git a/sdks/java/io/hbase/pom.xml b/sdks/java/io/hbase/pom.xml
index 9d5e2aa..40ac8df 100644
--- a/sdks/java/io/hbase/pom.xml
+++ b/sdks/java/io/hbase/pom.xml
@@ -121,6 +121,13 @@
   org.apache.hadoop
   hadoop-common
   test
+  
+
+
+  jdk.tools
+  jdk.tools
+
+  
 
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/75475ef3/sdks/java/io/hcatalog/pom.xml
--
diff --git a/sdks/java/io/hcatalog/pom.xml b/sdks/java/io/hcatalog/pom.xml
index 8af740d..a31ff86 100644
--- a/sdks/java/io/hcatalog/pom.xml
+++ b/sdks/java/io/hcatalog/pom.xml
@@ -61,6 +61,13 @@
 
   org.apache.hadoop
   hadoop-common
+  
+
+
+  jdk.tools
+  jdk.tools
+
+  
 
 
 
@@ -109,6 +116,11 @@
   com.google.protobuf
   protobuf-java
 
+
+
+  jdk.tools
+  jdk.tools
+
   
 
 

[23/43] beam git commit: This closes #3496: Disallow Combiner Lifting for multi-window WindowFns

[kenn]

This closes #3496: Disallow Combiner Lifting for multi-window WindowFns


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e8fdd373
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e8fdd373
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e8fdd373

Branch: refs/heads/gearpump-runner
Commit: e8fdd373c7b86652fda94ff76ef7246357e6288c
Parents: ea9f7fb 29c2bca
Author: Kenneth Knowles 
Authored: Wed Jul 5 16:41:16 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed Jul 5 16:41:16 2017 -0700

--
 .../apache/beam/runners/dataflow/DataflowPipelineTranslator.java| 1 +
 1 file changed, 1 insertion(+)
--

[11/43] beam git commit: This closes #3486

[kenn]

This closes #3486


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b8ac3264
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b8ac3264
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b8ac3264

Branch: refs/heads/gearpump-runner
Commit: b8ac32641183b06a87517a169c8ecc5371996aed
Parents: 0bd47c0 68f1fb6
Author: Ahmet Altay 
Authored: Tue Jul 4 11:07:31 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jul 4 11:07:31 2017 -0700

--
 sdks/python/apache_beam/io/gcp/gcsio.py | 10 +-
 1 file changed, 9 insertions(+), 1 deletion(-)
--

[16/43] beam git commit: Website Mergebot Job

[kenn]

Website Mergebot Job

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/06897b1c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/06897b1c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/06897b1c

Branch: refs/heads/gearpump-runner
Commit: 06897b1cc142f658437ac7779c849e5182e331f1
Parents: 14fa7f7
Author: Jason Kuster 
Authored: Fri Jun 9 01:39:15 2017 -0700
Committer: Thomas Groh 
Committed: Wed Jul 5 09:22:20 2017 -0700

--
 .../jenkins/common_job_properties.groovy|  5 +-
 .../job_beam_PreCommit_Website_Merge.groovy | 59 
 2 files changed, 62 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/06897b1c/.test-infra/jenkins/common_job_properties.groovy
--
diff --git a/.test-infra/jenkins/common_job_properties.groovy 
b/.test-infra/jenkins/common_job_properties.groovy
index 0e047ea..70534c6 100644
--- a/.test-infra/jenkins/common_job_properties.groovy
+++ b/.test-infra/jenkins/common_job_properties.groovy
@@ -23,11 +23,12 @@
 class common_job_properties {
 
   // Sets common top-level job properties for website repository jobs.
-  static void setTopLevelWebsiteJobProperties(context) {
+  static void setTopLevelWebsiteJobProperties(context,
+  String branch = 'asf-site') {
 setTopLevelJobProperties(
 context,
 'beam-site',
-'asf-site',
+branch,
 'beam',
 30)
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/06897b1c/.test-infra/jenkins/job_beam_PreCommit_Website_Merge.groovy
--
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Website_Merge.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Website_Merge.groovy
new file mode 100644
index 000..0e2ae3f
--- /dev/null
+++ b/.test-infra/jenkins/job_beam_PreCommit_Website_Merge.groovy
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import common_job_properties
+
+// Defines a job.
+job('beam_PreCommit_Website_Merge') {
+  description('Runs website tests for mergebot.')
+
+  // Set common parameters.
+  common_job_properties.setTopLevelWebsiteJobProperties(delegate, 'mergebot')
+
+  triggers {
+githubPush()
+  }
+
+  steps {
+// Run the following shell script as a build step.
+shell '''
+# Install RVM per instructions at https://rvm.io/rvm/install.
+RVM_GPG_KEY=409B6B1796C275462A1703113804BB82D39DC0E3
+gpg --keyserver hkp://keys.gnupg.net --recv-keys $RVM_GPG_KEY
+
+\\curl -sSL https://get.rvm.io | bash
+source /home/jenkins/.rvm/scripts/rvm
+
+# Install Ruby.
+RUBY_VERSION_NUM=2.3.0
+rvm install ruby $RUBY_VERSION_NUM --autolibs=read-only
+
+# Install Bundler gem
+PATH=~/.gem/ruby/$RUBY_VERSION_NUM/bin:$PATH
+GEM_PATH=~/.gem/ruby/$RUBY_VERSION_NUM/:$GEM_PATH
+gem install bundler --user-install
+
+# Install all needed gems.
+bundle install --path ~/.gem/
+
+# Build the new site and test it.
+rm -fr ./content/
+bundle exec rake test
+'''.stripIndent().trim()
+  }
+}

[33/43] beam git commit: Made DataflowRunner TransformTranslator public

[kenn]

Made DataflowRunner TransformTranslator public


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/da92256b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/da92256b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/da92256b

Branch: refs/heads/gearpump-runner
Commit: da92256ba64c5f476bd6283db2484bd72293
Parents: 1e16aa2
Author: Jeremie Lenfant-Engelmann 
Authored: Wed Jun 28 16:11:21 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 21:15:05 2017 -0700

--
 .../org/apache/beam/runners/dataflow/TransformTranslator.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/da92256b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
index a7452b2..7f61b6c 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
@@ -36,7 +36,8 @@ import org.apache.beam.sdk.values.TupleTag;
  * A {@link TransformTranslator} knows how to translate a particular subclass 
of {@link PTransform}
  * for the Cloud Dataflow service. It does so by mutating the {@link 
TranslationContext}.
  */
-interface TransformTranslator {
+@Internal
+public interface TransformTranslator {
   void translate(TransformT transform, TranslationContext context);
 
   /**

[20/43] beam git commit: This closes #3489: Fix DoFn javadoc: StateSpec does not require a key

[kenn]

This closes #3489: Fix DoFn javadoc: StateSpec does not require a key


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/63f66d6d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/63f66d6d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/63f66d6d

Branch: refs/heads/gearpump-runner
Commit: 63f66d6d81652dba5a7804648cde0e7debffd0f6
Parents: 39c80f6 f99ab1a
Author: Kenneth Knowles 
Authored: Wed Jul 5 15:00:09 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed Jul 5 15:00:09 2017 -0700

--
 .../core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[37/43] beam git commit: This closes #3470: [BEAM-940] ByteBuddyOnTimerInvokerFactory: key the cache with a (Class, id) tuple or OnTimerMethod

[kenn]

This closes #3470: [BEAM-940] ByteBuddyOnTimerInvokerFactory: key the cache 
with a (Class, id) tuple or OnTimerMethod

  Simplified ByteBuddyOnTimerInvokerFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/440c7d45
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/440c7d45
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/440c7d45

Branch: refs/heads/gearpump-runner
Commit: 440c7d45b21e64d49566f0674a2dc91f6d284257
Parents: 17bc3b1 c8d9833
Author: Kenneth Knowles 
Authored: Thu Jul 6 21:47:11 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 21:47:11 2017 -0700

--
 .../reflect/ByteBuddyOnTimerInvokerFactory.java | 73 
 .../reflect/OnTimerMethodSpecifier.java | 37 ++
 2 files changed, 65 insertions(+), 45 deletions(-)
--

[GitHub] beam pull request #3515: [BEAM-79] Sync gearpump-runner with master and upgr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3515


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[04/43] beam git commit: This closes #3355: Add support for PipelineOptions parameters

[kenn]

This closes #3355: Add support for PipelineOptions parameters


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0e429b33
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0e429b33
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0e429b33

Branch: refs/heads/gearpump-runner
Commit: 0e429b33ff85eba08da5018c9febd0b99b44f720
Parents: f75dfe7 56cb6c5
Author: Kenneth Knowles 
Authored: Fri Jun 30 15:39:20 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 30 15:39:20 2017 -0700

--
 ...eBoundedSplittableProcessElementInvoker.java |  5 ++
 .../beam/runners/core/SimpleDoFnRunner.java | 20 +++
 .../apache/beam/sdk/transforms/DoFnTester.java  |  5 ++
 .../reflect/ByteBuddyDoFnInvokerFactory.java|  6 ++
 .../sdk/transforms/reflect/DoFnInvoker.java | 13 +++-
 .../sdk/transforms/reflect/DoFnSignature.java   | 23 +++
 .../sdk/transforms/reflect/DoFnSignatures.java  | 22 ++-
 .../apache/beam/sdk/transforms/ParDoTest.java   | 63 
 .../transforms/reflect/DoFnSignaturesTest.java  | 14 +
 9 files changed, 169 insertions(+), 2 deletions(-)
--

[06/43] beam git commit: Make modules that depend on Hadoop and Spark use the same version property

[kenn]

Make modules that depend on Hadoop and Spark use the same version property


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ce4e5174
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ce4e5174
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ce4e5174

Branch: refs/heads/gearpump-runner
Commit: ce4e5174750ae2c4b1691c6994bd0f92e161
Parents: 0e429b3
Author: Ismaël Mejía 
Authored: Sun Jun 4 22:55:05 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Mon Jul 3 08:19:02 2017 +0200

--
 examples/java/pom.xml   | 18 --
 examples/java8/pom.xml  | 18 --
 pom.xml | 65 +++-
 runners/apex/pom.xml|  2 +-
 runners/spark/pom.xml   |  7 ---
 sdks/java/extensions/sorter/pom.xml |  6 --
 sdks/java/io/hadoop-file-system/pom.xml | 31 --
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml|  2 -
 sdks/java/io/hbase/pom.xml  |  9 ++-
 sdks/java/io/hcatalog/pom.xml   |  6 +-
 sdks/java/io/jdbc/pom.xml   |  2 -
 sdks/java/io/pom.xml| 31 --
 sdks/java/javadoc/pom.xml   |  2 -
 .../main/resources/archetype-resources/pom.xml  |  1 -
 .../main/resources/archetype-resources/pom.xml  |  1 -
 15 files changed, 98 insertions(+), 103 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ce4e5174/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 701e4fe..7ae4e6a 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -34,10 +34,6 @@
 
   jar
 
-  
-1.6.2
-  
-
   
 
 
+2.7.3
 1.3
 2.8.9
 3.0.1
@@ -139,7 +145,7 @@
 v1-rev10-1.22.0
 1.7.14
 0.20.0-beta
-1.6.2
+1.6.3
 4.3.5.RELEASE
 3.1.4
 v1-rev71-1.22.0
@@ -1075,6 +1081,42 @@
 ${snappy-java.version}
   
 
+  
+org.apache.hadoop
+hadoop-client
+${hadoop.version}
+  
+
+  
+org.apache.hadoop
+hadoop-common
+${hadoop.version}
+  
+
+  
+org.apache.hadoop
+hadoop-mapreduce-client-core
+${hadoop.version}
+  
+
+  
+org.apache.spark
+spark-core_2.10
+${spark.version}
+  
+
+  
+org.apache.spark
+spark-streaming_2.10
+${spark.version}
+  
+
+  
+org.apache.spark
+spark-network-common_2.10
+${spark.version}
+  
+
   
 
   
@@ -1144,6 +1186,27 @@
 test
   
 
+  
+org.apache.hadoop
+hadoop-minicluster
+${hadoop.version}
+test
+  
+
+  
+org.apache.hadoop
+hadoop-hdfs
+${hadoop.version}
+test
+  
+
+  
+org.apache.hadoop
+hadoop-hdfs
+${hadoop.version}
+tests
+test
+  
 
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ce4e5174/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 2c54654..88ff0f2 100644
--- a/runners/apex/pom.xml
+++ b/runners/apex/pom.xml
@@ -261,7 +261,7 @@
 
com.esotericsoftware.kryo:kryo::${apex.kryo.version}
 
com.datatorrent:netlet::1.3.0
 
org.slf4j:slf4j-api:jar:1.7.14
-
org.apache.hadoop:hadoop-common:jar:2.6.0
+
org.apache.hadoop:hadoop-common:jar:${hadoop.version}
 
joda-time:joda-time:jar:2.4
 
com.google.guava:guava:jar:20.0
   

http://git-wip-us.apache.org/repos/asf/beam/blob/ce4e5174/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index ee72dd9..1d93427 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -34,8 +34,6 @@
   
 UTF-8
 UTF-8
-1.6.3
-2.2.0
 0.9.0.1
 2.4.4
 3.1.2
@@ -135,31 +133,26 @@
 
   org.apache.spark
   spark-core_2.10
-  ${spark.version}
   provided
 
 
   org.apache.spark
   spark-streaming_2.10
-  ${spark.version}
   provided
 
 
   org.apache.spark
   spark-network-common_2.10
-  ${spark.version}
   provided
 
 
   org.apache.hadoop
   hadoop-common
-  ${hadoop.version}
   provided
 
 
   org.apache.hadoop
   hadoop-mapreduce-client-core
-  ${hadoop.version}
   provided
   

[25/43] beam git commit: This closes #3499

[kenn]

This closes #3499


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bf6dda32
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bf6dda32
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bf6dda32

Branch: refs/heads/gearpump-runner
Commit: bf6dda32047e0bfba18de5aa27b1b6a593f86fc9
Parents: e8fdd37 a75202f
Author: Ahmet Altay 
Authored: Wed Jul 5 18:09:05 2017 -0700
Committer: Ahmet Altay 
Committed: Wed Jul 5 18:09:05 2017 -0700

--
 sdks/python/apache_beam/runners/direct/transform_evaluator.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--

[02/43] beam git commit: Properly convert milliseconds whether there's less than 3/more than 9 digits.

[kenn]

Properly convert milliseconds whether there's less than 3/more than 9 digits.

This closes #3469


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f75dfe71
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f75dfe71
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f75dfe71

Branch: refs/heads/gearpump-runner
Commit: f75dfe711b357800762a93dc5059d9a19da9ee61
Parents: 893bf42 7b8cd64
Author: Luke Cwik 
Authored: Fri Jun 30 09:00:13 2017 -0700
Committer: Luke Cwik 
Committed: Fri Jun 30 09:00:13 2017 -0700

--
 .../beam/runners/dataflow/util/TimeUtil.java| 24 
 .../runners/dataflow/util/TimeUtilTest.java |  6 +
 2 files changed, 15 insertions(+), 15 deletions(-)
--

[35/43] beam git commit: Fix bad merge

[kenn]

Fix bad merge


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/17bc3b14
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/17bc3b14
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/17bc3b14

Branch: refs/heads/gearpump-runner
Commit: 17bc3b140c7c7315880ce18d4e15d6ac512c35d2
Parents: 17d7e59
Author: Kenneth Knowles 
Authored: Thu Jul 6 21:45:39 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 21:45:39 2017 -0700

--
 .../java/org/apache/beam/runners/dataflow/TransformTranslator.java  | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/17bc3b14/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
index 7f61b6c..06ed1e0 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
@@ -22,6 +22,7 @@ import java.util.Map;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
 import org.apache.beam.runners.dataflow.util.OutputReference;
 import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.annotations.Internal;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;

[42/43] beam git commit: Fix ParDoTest#testPipelineOptionsParameter

[kenn]

Fix ParDoTest#testPipelineOptionsParameter


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/725f547f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/725f547f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/725f547f

Branch: refs/heads/gearpump-runner
Commit: 725f547f5e487dd3e84d5d0f95c0fa3efa853279
Parents: 2206827
Author: manuzhang 
Authored: Sat Jul 8 00:13:19 2017 +0800
Committer: manuzhang 
Committed: Sat Jul 8 00:13:19 2017 +0800

--
 .../gearpump/translators/io/GearpumpSource.java  | 12 ++--
 .../translators/utils/DoFnRunnerFactory.java |  5 +++--
 .../translators/utils/TranslatorUtils.java   | 19 +++
 3 files changed, 24 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/725f547f/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/io/GearpumpSource.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/io/GearpumpSource.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/io/GearpumpSource.java
index daa8c81..2f53139 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/io/GearpumpSource.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/io/GearpumpSource.java
@@ -18,9 +18,6 @@
 
 package org.apache.beam.runners.gearpump.translators.io;
 
-import com.fasterxml.jackson.core.JsonProcessingException;
-import com.fasterxml.jackson.databind.ObjectMapper;
-
 import java.io.IOException;
 import java.time.Instant;
 
@@ -48,11 +45,7 @@ public abstract class GearpumpSource implements 
DataSource {
   private boolean available = false;
 
   GearpumpSource(PipelineOptions options) {
-try {
-  this.serializedOptions = new ObjectMapper().writeValueAsBytes(options);
-} catch (JsonProcessingException e) {
-  throw new RuntimeException(e);
-}
+this.serializedOptions = TranslatorUtils.serializePipelineOptions(options);
   }
 
   protected abstract Source.Reader createReader(PipelineOptions options) 
throws IOException;
@@ -60,8 +53,7 @@ public abstract class GearpumpSource implements DataSource 
{
   @Override
   public void open(TaskContext context, Instant startTime) {
 try {
-  PipelineOptions options = new ObjectMapper()
-  .readValue(serializedOptions, PipelineOptions.class);
+  PipelineOptions options = 
TranslatorUtils.deserializePipelineOptions(serializedOptions);
   this.reader = createReader(options);
   this.available = reader.start();
 } catch (Exception e) {

http://git-wip-us.apache.org/repos/asf/beam/blob/725f547f/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/DoFnRunnerFactory.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/DoFnRunnerFactory.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/DoFnRunnerFactory.java
index 35cf2b5..375b696 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/DoFnRunnerFactory.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/DoFnRunnerFactory.java
@@ -43,7 +43,7 @@ public class DoFnRunnerFactory implements 
Serializable {
 
   private static final long serialVersionUID = -4109539010014189725L;
   private final DoFn fn;
-  private final transient PipelineOptions options;
+  private final byte[] serializedOptions;
   private final Collection sideInputs;
   private final DoFnRunners.OutputManager outputManager;
   private final TupleTag mainOutputTag;
@@ -61,7 +61,7 @@ public class DoFnRunnerFactory implements 
Serializable {
   StepContext stepContext,
   WindowingStrategy windowingStrategy) {
 this.fn = doFn;
-this.options = pipelineOptions;
+this.serializedOptions = 
TranslatorUtils.serializePipelineOptions(pipelineOptions);
 this.sideInputs = sideInputs;
 this.outputManager = outputManager;
 this.mainOutputTag = mainOutputTag;
@@ -72,6 +72,7 @@ public class DoFnRunnerFactory implements 
Serializable {
 
   public PushbackSideInputDoFnRunner createRunner(
   ReadyCheckingSideInputReader sideInputReader) {
+PipelineOptions options = 
TranslatorUtils.deserializePipelineOptions(serializedOptions);
 DoFnRunner underlying = DoFnRunners.simpleRunner(
 options, fn, sideInputReader, outputManager, 

[40/43] beam git commit: Merge branch 'master' of https://github.com/apache/beam into sync-master

[kenn]

Merge branch 'master' of https://github.com/apache/beam into sync-master


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c2d3fbc3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c2d3fbc3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c2d3fbc3

Branch: refs/heads/gearpump-runner
Commit: c2d3fbc3a85a276850a2a56e040eccf2e2079339
Parents: 627ae0b 9e565f2
Author: manuzhang 
Authored: Fri Jul 7 16:33:03 2017 +0800
Committer: manuzhang 
Committed: Fri Jul 7 16:33:03 2017 +0800

--
 .../jenkins/common_job_properties.groovy|   5 +-
 .../job_beam_PreCommit_Website_Merge.groovy |  59 +
 examples/java/pom.xml   |  20 ++-
 examples/java8/pom.xml  |  20 ++-
 .../examples/complete/game/LeaderBoardTest.java |   2 +
 examples/pom.xml|   2 +-
 pom.xml |  87 -
 runners/apex/pom.xml|  11 +-
 runners/core-construction-java/pom.xml  |   2 +-
 runners/core-java/pom.xml   |   2 +-
 ...eBoundedSplittableProcessElementInvoker.java |   5 +
 .../beam/runners/core/ReduceFnRunner.java   | 106 ++-
 .../beam/runners/core/SimpleDoFnRunner.java |  20 +++
 .../beam/runners/core/ReduceFnRunnerTest.java   |  81 +++-
 runners/direct-java/pom.xml |   4 +-
 runners/flink/pom.xml   |   2 +-
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../dataflow/DataflowPipelineTranslator.java|   1 +
 .../runners/dataflow/TransformTranslator.java   |   4 +-
 .../beam/runners/dataflow/util/TimeUtil.java|  24 ++--
 .../runners/dataflow/util/TimeUtilTest.java |   6 +
 runners/pom.xml |   2 +-
 runners/spark/pom.xml   |  16 +--
 sdks/common/fn-api/pom.xml  |   2 +-
 sdks/common/pom.xml |   2 +-
 sdks/common/runner-api/pom.xml  |   2 +-
 sdks/java/build-tools/pom.xml   |   2 +-
 sdks/java/core/pom.xml  |   2 +-
 .../org/apache/beam/sdk/transforms/DoFn.java|   2 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   5 +
 .../reflect/ByteBuddyDoFnInvokerFactory.java|   6 +
 .../reflect/ByteBuddyOnTimerInvokerFactory.java |  73 ---
 .../sdk/transforms/reflect/DoFnInvoker.java |  13 +-
 .../sdk/transforms/reflect/DoFnSignature.java   |  23 
 .../sdk/transforms/reflect/DoFnSignatures.java  |  22 +++-
 .../reflect/OnTimerMethodSpecifier.java |  37 ++
 .../apache/beam/sdk/transforms/ParDoTest.java   |  63 +
 .../transforms/reflect/DoFnSignaturesTest.java  |  14 ++
 .../google-cloud-platform-core/pom.xml  |   2 +-
 sdks/java/extensions/jackson/pom.xml|   2 +-
 sdks/java/extensions/join-library/pom.xml   |   2 +-
 sdks/java/extensions/pom.xml|   2 +-
 sdks/java/extensions/protobuf/pom.xml   |   2 +-
 sdks/java/extensions/sorter/pom.xml |   8 +-
 sdks/java/harness/pom.xml   |   2 +-
 sdks/java/io/amqp/pom.xml   |   2 +-
 sdks/java/io/cassandra/pom.xml  |   2 +-
 sdks/java/io/common/pom.xml |   2 +-
 sdks/java/io/elasticsearch/pom.xml  |   2 +-
 sdks/java/io/google-cloud-platform/pom.xml  |   2 +-
 sdks/java/io/hadoop-common/pom.xml  |   2 +-
 sdks/java/io/hadoop-file-system/pom.xml |  33 +
 sdks/java/io/hadoop/input-format/pom.xml|   2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml|   4 +-
 sdks/java/io/hadoop/pom.xml |   2 +-
 sdks/java/io/hbase/pom.xml  |  18 ++-
 sdks/java/io/hcatalog/pom.xml   |  20 ++-
 sdks/java/io/jdbc/pom.xml   |   4 +-
 sdks/java/io/jms/pom.xml|   2 +-
 sdks/java/io/kafka/pom.xml  |   2 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   | 130 +--
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   |  30 +
 sdks/java/io/kinesis/pom.xml|   2 +-
 sdks/java/io/mongodb/pom.xml|   2 +-
 sdks/java/io/mqtt/pom.xml   |   2 +-
 sdks/java/io/pom.xml|  33 +
 sdks/java/io/xml/pom.xml|   2 +-
 sdks/java/java8tests/pom.xml|   2 +-
 sdks/java/javadoc/pom.xml   |  19 ++-
 .../maven-archetypes/examples-java8/pom.xml |   2 +-
 .../main/resources/archetype-resources/pom.xml  |   1 -
 sdks/java/maven-archetypes/examples/pom.xml |   2 +-
 

[43/43] beam git commit: This closes #3515: Sync gearpump-runner with master and upgrade to gearpump 0.8.4

[kenn]

This closes #3515: Sync gearpump-runner with master and upgrade to gearpump 
0.8.4

  Fix ParDoTest#testPipelineOptionsParameter
  Upgrade to gearpump 0.8.4
  Fix javadoc generation for AmqpIO, CassandraIO and HCatalogIO
  Simplified ByteBuddyOnTimerInvokerFactory
  Fix bad merge
  Made DataflowRunner TransformTranslator public
  Process timer firings for a window together
  Ignore processing time timers in expired windows
  Add timeout to initialization of partition in KafkaIO
  [BEAM-2534] Handle offset gaps in Kafka messages.
  Fix PValue input in _PubSubReadEvaluator
  Update SDK dependencies
  Disallow Combiner Lifting for multi-window WindowFns
  [BEAM-2553] Update Maven exec plugin to 1.6.0 to incorporate messaging 
improvements
  Website Mergebot Job
  Update Python SDK version
  [maven-release-plugin] prepare for next development iteration
  [maven-release-plugin] prepare branch release-2.1.0
  For GCS operations use an http client with a default timeout value.
  [BEAM-2530] Fix compilation of modules with Java 9 that depend on jdk.tools
  Make modules that depend on Hadoop and Spark use the same version property
  Fix DoFn javadoc: StateSpec does not require a key
  Add support for PipelineOptions parameters
  Properly convert milliseconds whether there's less than 3/more than 9 digits. 
TimeUtil did not properly convert (and returned null) when the number of digits 
for fractions of seconds was less than 3 digits or more than 9 digits. The 
solution is to pad with zeros when there is less than 3 digits and to truncate 
when there is more than 3.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1ce60b48
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1ce60b48
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1ce60b48

Branch: refs/heads/gearpump-runner
Commit: 1ce60b488e991562712937aaaf13564e3a909117
Parents: 627ae0b 725f547
Author: Kenneth Knowles 
Authored: Sun Jul 9 21:47:12 2017 -0700
Committer: Kenneth Knowles 
Committed: Sun Jul 9 21:47:12 2017 -0700

--
 .../jenkins/common_job_properties.groovy|   5 +-
 .../job_beam_PreCommit_Website_Merge.groovy |  59 +
 examples/java/pom.xml   |  20 ++-
 examples/java8/pom.xml  |  20 ++-
 .../examples/complete/game/LeaderBoardTest.java |   2 +
 examples/pom.xml|   2 +-
 pom.xml |  87 -
 runners/apex/pom.xml|  11 +-
 runners/core-construction-java/pom.xml  |   2 +-
 runners/core-java/pom.xml   |   2 +-
 ...eBoundedSplittableProcessElementInvoker.java |   5 +
 .../beam/runners/core/ReduceFnRunner.java   | 106 ++-
 .../beam/runners/core/SimpleDoFnRunner.java |  20 +++
 .../beam/runners/core/ReduceFnRunnerTest.java   |  81 +++-
 runners/direct-java/pom.xml |   4 +-
 runners/flink/pom.xml   |   2 +-
 runners/gearpump/pom.xml|   4 +-
 .../gearpump/translators/io/GearpumpSource.java |  12 +-
 .../translators/utils/DoFnRunnerFactory.java|   5 +-
 .../translators/utils/TranslatorUtils.java  |  19 +++
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../dataflow/DataflowPipelineTranslator.java|   1 +
 .../runners/dataflow/TransformTranslator.java   |   4 +-
 .../beam/runners/dataflow/util/TimeUtil.java|  24 ++--
 .../runners/dataflow/util/TimeUtilTest.java |   6 +
 runners/pom.xml |   2 +-
 runners/spark/pom.xml   |  16 +--
 sdks/common/fn-api/pom.xml  |   2 +-
 sdks/common/pom.xml |   2 +-
 sdks/common/runner-api/pom.xml  |   2 +-
 sdks/java/build-tools/pom.xml   |   2 +-
 sdks/java/core/pom.xml  |   2 +-
 .../org/apache/beam/sdk/transforms/DoFn.java|   2 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |   5 +
 .../reflect/ByteBuddyDoFnInvokerFactory.java|   6 +
 .../reflect/ByteBuddyOnTimerInvokerFactory.java |  73 ---
 .../sdk/transforms/reflect/DoFnInvoker.java |  13 +-
 .../sdk/transforms/reflect/DoFnSignature.java   |  23 
 .../sdk/transforms/reflect/DoFnSignatures.java  |  22 +++-
 .../reflect/OnTimerMethodSpecifier.java |  37 ++
 .../apache/beam/sdk/transforms/ParDoTest.java   |  63 +
 .../transforms/reflect/DoFnSignaturesTest.java  |  14 ++
 .../google-cloud-platform-core/pom.xml  |   2 +-
 sdks/java/extensions/jackson/pom.xml|   2 +-
 sdks/java/extensions/join-library/pom.xml   |   2 +-
 sdks/java/extensions/pom.xml|   2 +-
 sdks/java/extensions/protobuf/pom.xml   |   2 +-
 

[10/43] beam git commit: For GCS operations use an http client with a default timeout value.

[kenn]

For GCS operations use an http client with a default timeout value.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/68f1fb64
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/68f1fb64
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/68f1fb64

Branch: refs/heads/gearpump-runner
Commit: 68f1fb64fd2565e287e322d715ca778d01e7137b
Parents: 0bd47c0
Author: Ahmet Altay 
Authored: Fri Jun 30 17:37:33 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jul 4 11:07:25 2017 -0700

--
 sdks/python/apache_beam/io/gcp/gcsio.py | 10 +-
 1 file changed, 9 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/68f1fb64/sdks/python/apache_beam/io/gcp/gcsio.py
--
diff --git a/sdks/python/apache_beam/io/gcp/gcsio.py 
b/sdks/python/apache_beam/io/gcp/gcsio.py
index d43c8ba..643fbc7 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio.py
@@ -31,6 +31,7 @@ import re
 import threading
 import time
 import traceback
+import httplib2
 
 from apache_beam.utils import retry
 
@@ -68,6 +69,10 @@ except ImportError:
 # +---++-+-+-+
 DEFAULT_READ_BUFFER_SIZE = 16 * 1024 * 1024
 
+# This is the number of seconds the library will wait for GCS operations to
+# complete.
+DEFAULT_HTTP_TIMEOUT_SECONDS = 60
+
 # This is the number of seconds the library will wait for a partial-file read
 # operation from GCS to complete before retrying.
 DEFAULT_READ_SEGMENT_TIMEOUT_SECONDS = 60
@@ -99,6 +104,7 @@ class GcsIO(object):
 
   def __new__(cls, storage_client=None):
 if storage_client:
+  # This path is only used for testing.
   return super(GcsIO, cls).__new__(cls, storage_client)
 else:
   # Create a single storage client for each thread.  We would like to avoid
@@ -108,7 +114,9 @@ class GcsIO(object):
   local_state = threading.local()
   if getattr(local_state, 'gcsio_instance', None) is None:
 credentials = auth.get_service_credentials()
-storage_client = storage.StorageV1(credentials=credentials)
+storage_client = storage.StorageV1(
+credentials=credentials,
+http=httplib2.Http(timeout=DEFAULT_HTTP_TIMEOUT_SECONDS))
 local_state.gcsio_instance = (
 super(GcsIO, cls).__new__(cls, storage_client))
 local_state.gcsio_instance.client = storage_client

[17/43] beam git commit: [BEAM-2553] Update Maven exec plugin to 1.6.0 to incorporate messaging improvements

[kenn]

[BEAM-2553] Update Maven exec plugin to 1.6.0 to incorporate messaging 
improvements


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6ca410a9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6ca410a9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6ca410a9

Branch: refs/heads/gearpump-runner
Commit: 6ca410a908f1f4e7ac1e141ee1335f7a537bb150
Parents: f0549b4
Author: Luke Cwik 
Authored: Wed Jul 5 10:38:44 2017 -0700
Committer: Luke Cwik 
Committed: Wed Jul 5 10:38:44 2017 -0700

--
 pom.xml| 2 +-
 .../starter/src/test/resources/projects/basic/reference/pom.xml| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6ca410a9/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 49e6edd..01474c1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -159,7 +159,7 @@
 2.20
 3.6.1
 3.0.1
-1.4.0
+1.6.0
 3.0.2
 3.0.2
 3.0.0

http://git-wip-us.apache.org/repos/asf/beam/blob/6ca410a9/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
index 60405e6..6056fb0 100644
--- 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
@@ -28,7 +28,7 @@
 @project.version@
 
 3.6.1
-1.4.0
+1.6.0
 1.7.14
   
 

[21/43] beam git commit: Update SDK dependencies

[kenn]

Update SDK dependencies


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/23e385fa
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/23e385fa
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/23e385fa

Branch: refs/heads/gearpump-runner
Commit: 23e385faa193a00f9b10e3f8f0afe832087bff06
Parents: 63f66d6
Author: Ahmet Altay 
Authored: Wed Jul 5 14:34:07 2017 -0700
Committer: Ahmet Altay 
Committed: Wed Jul 5 16:11:37 2017 -0700

--
 sdks/python/setup.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/23e385fa/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 6646a58..8a0c9ae 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -120,9 +120,9 @@ GCP_REQUIREMENTS = [
   'google-apitools>=0.5.10,<=0.5.11',
   'proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4',
   'googledatastore==7.0.1',
-  'google-cloud-pubsub==0.25.0',
+  'google-cloud-pubsub==0.26.0',
   # GCP packages required by tests
-  'google-cloud-bigquery>=0.23.0,<0.25.0',
+  'google-cloud-bigquery>=0.23.0,<0.26.0',
 ]
 
 

[31/43] beam git commit: Process timer firings for a window together

[kenn]

Process timer firings for a window together


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/935c0773
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/935c0773
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/935c0773

Branch: refs/heads/gearpump-runner
Commit: 935c077341de5804b29ffee3926795acf403
Parents: bd631b8
Author: Kenneth Knowles 
Authored: Thu Jun 22 18:43:39 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 14:12:39 2017 -0700

--
 .../examples/complete/game/LeaderBoardTest.java |  2 +
 .../beam/runners/core/ReduceFnRunner.java   | 98 +---
 .../beam/runners/core/ReduceFnRunnerTest.java   | 49 +-
 3 files changed, 115 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/935c0773/examples/java8/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
--
diff --git 
a/examples/java8/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
 
b/examples/java8/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
index 745c210..611e2b3 100644
--- 
a/examples/java8/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
+++ 
b/examples/java8/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
@@ -276,6 +276,8 @@ public class LeaderBoardTest implements Serializable {
 .addElements(event(TestUser.RED_ONE, 4, Duration.standardMinutes(2)),
 event(TestUser.BLUE_TWO, 3, Duration.ZERO),
 event(TestUser.BLUE_ONE, 3, Duration.standardMinutes(3)))
+// Move the watermark to the end of the window to output on time
+.advanceWatermarkTo(baseTime.plus(TEAM_WINDOW_DURATION))
 // Move the watermark past the end of the allowed lateness plus the 
end of the window
 .advanceWatermarkTo(baseTime.plus(ALLOWED_LATENESS)
 .plus(TEAM_WINDOW_DURATION).plus(Duration.standardMinutes(1)))

http://git-wip-us.apache.org/repos/asf/beam/blob/935c0773/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
index 0632c05..634a2d1 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
@@ -29,7 +29,6 @@ import java.util.Collection;
 import java.util.Collections;
 import java.util.HashMap;
 import java.util.HashSet;
-import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
@@ -638,11 +637,9 @@ public class ReduceFnRunner {
   }
 
   /**
-   * Enriches TimerData with state necessary for processing a timer as well as
-   * common queries about a timer.
+   * A descriptor of the activation for a window based on a timer.
*/
-  private class EnrichedTimerData {
-public final Instant timestamp;
+  private class WindowActivation {
 public final ReduceFn.Context directContext;
 public final ReduceFn.Context renamedContext;
 // If this is an end-of-window timer then we may need to set a garbage 
collection timer
@@ -653,19 +650,34 @@ public class ReduceFnRunner {
 // end-of-window time to be a signal to garbage collect.
 public final boolean isGarbageCollection;
 
-EnrichedTimerData(
-TimerData timer,
+WindowActivation(
 ReduceFn.Context directContext,
 ReduceFn.Context renamedContext) {
-  this.timestamp = timer.getTimestamp();
   this.directContext = directContext;
   this.renamedContext = renamedContext;
   W window = directContext.window();
-  this.isEndOfWindow = TimeDomain.EVENT_TIME == timer.getDomain()
-  && timer.getTimestamp().equals(window.maxTimestamp());
-  Instant cleanupTime = LateDataUtils.garbageCollectionTime(window, 
windowingStrategy);
+
+  // The output watermark is before the end of the window if it is either 
unknown
+  // or it is known to be before it. If it is unknown, that means that 
there hasn't been
+  // enough data to advance it.
+  boolean outputWatermarkBeforeEOW =
+  timerInternals.currentOutputWatermarkTime() == null
+  || 
!timerInternals.currentOutputWatermarkTime().isAfter(window.maxTimestamp());
+
+  // The 

[07/43] beam git commit: This closes #3291

[kenn]

This closes #3291


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/11010cf0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/11010cf0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/11010cf0

Branch: refs/heads/gearpump-runner
Commit: 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab
Parents: 0e429b3 ce4e517
Author: Jean-Baptiste Onofré 
Authored: Mon Jul 3 09:26:40 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Mon Jul 3 09:26:40 2017 +0200

--
 examples/java/pom.xml   | 18 --
 examples/java8/pom.xml  | 18 --
 pom.xml | 65 +++-
 runners/apex/pom.xml|  2 +-
 runners/spark/pom.xml   |  7 ---
 sdks/java/extensions/sorter/pom.xml |  6 --
 sdks/java/io/hadoop-file-system/pom.xml | 31 --
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml|  2 -
 sdks/java/io/hbase/pom.xml  |  9 ++-
 sdks/java/io/hcatalog/pom.xml   |  6 +-
 sdks/java/io/jdbc/pom.xml   |  2 -
 sdks/java/io/pom.xml| 31 --
 sdks/java/javadoc/pom.xml   |  2 -
 .../main/resources/archetype-resources/pom.xml  |  1 -
 .../main/resources/archetype-resources/pom.xml  |  1 -
 15 files changed, 98 insertions(+), 103 deletions(-)
--

[41/43] beam git commit: Upgrade to gearpump 0.8.4

[kenn]

Upgrade to gearpump 0.8.4


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/22068274
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/22068274
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/22068274

Branch: refs/heads/gearpump-runner
Commit: 22068274428666da93c793a170810ddb42755704
Parents: c2d3fbc
Author: manuzhang 
Authored: Fri Jul 7 16:41:13 2017 +0800
Committer: manuzhang 
Committed: Fri Jul 7 22:08:37 2017 +0800

--
 runners/gearpump/pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/22068274/runners/gearpump/pom.xml
--
diff --git a/runners/gearpump/pom.xml b/runners/gearpump/pom.xml
index 3c98d5e..54c8d5c 100644
--- a/runners/gearpump/pom.xml
+++ b/runners/gearpump/pom.xml
@@ -43,7 +43,7 @@
   
 UTF-8
 UTF-8
-0.8.4-SNAPSHOT
+0.8.4
   
 
   
@@ -58,7 +58,7 @@
 maven-surefire-plugin
 
   
-runnable-on-service-tests
+validates-runner-tests
 integration-test
 
   test

[05/43] beam git commit: Fix DoFn javadoc: StateSpec does not require a key

[kenn]

Fix DoFn javadoc: StateSpec does not require a key


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f99ab1a4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f99ab1a4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f99ab1a4

Branch: refs/heads/gearpump-runner
Commit: f99ab1a472868e4ce175a86e5d76823b1c09c10b
Parents: 0e429b3
Author: Kenneth Knowles 
Authored: Fri Jun 30 21:42:17 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 30 21:42:17 2017 -0700

--
 .../core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f99ab1a4/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
index fb6d0ee..a2e5c16 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
@@ -385,7 +385,7 @@ public abstract class DoFn implements 
Serializable, HasDisplayD
* {@literal new DoFn, Baz>()} {
*
*  {@literal @StateId("my-state-id")}
-   *  {@literal private final StateSpec} myStateSpec =
+   *  {@literal private final StateSpec} myStateSpec =
*   StateSpecs.value(new MyStateCoder());
*
*  {@literal @ProcessElement}

[27/43] beam git commit: This closes #3461

[kenn]

This closes #3461


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/85a99e29
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/85a99e29
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/85a99e29

Branch: refs/heads/gearpump-runner
Commit: 85a99e29448670ae6728a8ee2e4cd3ef95877c3e
Parents: bf6dda3 2259c30
Author: JingsongLi 
Authored: Fri Jul 7 00:48:36 2017 +0800
Committer: JingsongLi 
Committed: Fri Jul 7 00:48:36 2017 +0800

--
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   | 49 
 1 file changed, 29 insertions(+), 20 deletions(-)
--

[15/43] beam git commit: This closes #3485

[kenn]

This closes #3485


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f0549b46
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f0549b46
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f0549b46

Branch: refs/heads/gearpump-runner
Commit: f0549b464737d0d11ae1da8e0534622b10247208
Parents: 14fa7f7 06897b1
Author: Thomas Groh 
Authored: Wed Jul 5 09:22:20 2017 -0700
Committer: Thomas Groh 
Committed: Wed Jul 5 09:22:20 2017 -0700

--
 .../jenkins/common_job_properties.groovy|  5 +-
 .../job_beam_PreCommit_Website_Merge.groovy | 59 
 2 files changed, 62 insertions(+), 2 deletions(-)
--

[34/43] beam git commit: This closes #3467: Made DataflowRunner TransformTranslator public

[kenn]

This closes #3467: Made DataflowRunner TransformTranslator public


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/17d7e598
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/17d7e598
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/17d7e598

Branch: refs/heads/gearpump-runner
Commit: 17d7e5980e58a6ee668f41f1518a09bed274b5b2
Parents: 628dace da92256
Author: Kenneth Knowles 
Authored: Thu Jul 6 21:18:05 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 21:18:05 2017 -0700

--
 .../org/apache/beam/runners/dataflow/TransformTranslator.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--

[28/43] beam git commit: Add timeout to initialization of partition in KafkaIO

[kenn]

Add timeout to initialization of partition in KafkaIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/526037b6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/526037b6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/526037b6

Branch: refs/heads/gearpump-runner
Commit: 526037b6786315b9f9fdca6edb636baeb6f83e3f
Parents: 85a99e2
Author: Raghu Angadi 
Authored: Mon Jul 3 23:54:10 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 11:58:41 2017 -0700

--
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   | 81 +++-
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   | 30 
 2 files changed, 92 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/526037b6/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index e520367..026313a 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -49,9 +49,11 @@ import java.util.Random;
 import java.util.Set;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.SynchronousQueue;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicBoolean;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
@@ -1061,8 +1063,32 @@ public class KafkaIO {
   curBatch = Iterators.cycle(nonEmpty);
 }
 
+private void setupInitialOffset(PartitionState pState) {
+  Read spec = source.spec;
+
+  if (pState.nextOffset != UNINITIALIZED_OFFSET) {
+consumer.seek(pState.topicPartition, pState.nextOffset);
+  } else {
+// nextOffset is unininitialized here, meaning start reading from 
latest record as of now
+// ('latest' is the default, and is configurable) or 'look up offset 
by startReadTime.
+// Remember the current position without waiting until the first 
record is read. This
+// ensures checkpoint is accurate even if the reader is closed before 
reading any records.
+Instant startReadTime = spec.getStartReadTime();
+if (startReadTime != null) {
+  pState.nextOffset =
+  consumerSpEL.offsetForTime(consumer, pState.topicPartition, 
spec.getStartReadTime());
+  consumer.seek(pState.topicPartition, pState.nextOffset);
+} else {
+  pState.nextOffset = consumer.position(pState.topicPartition);
+}
+  }
+}
+
 @Override
 public boolean start() throws IOException {
+  final int defaultPartitionInitTimeout = 60 * 1000;
+  final int kafkaRequestTimeoutMultiple = 2;
+
   Read spec = source.spec;
   consumer = spec.getConsumerFactoryFn().apply(spec.getConsumerConfig());
   consumerSpEL.evaluateAssign(consumer, spec.getTopicPartitions());
@@ -1077,25 +1103,38 @@ public class KafkaIO {
   keyDeserializerInstance.configure(spec.getConsumerConfig(), true);
   valueDeserializerInstance.configure(spec.getConsumerConfig(), false);
 
-  for (PartitionState p : partitionStates) {
-if (p.nextOffset != UNINITIALIZED_OFFSET) {
-  consumer.seek(p.topicPartition, p.nextOffset);
-} else {
-  // nextOffset is unininitialized here, meaning start reading from 
latest record as of now
-  // ('latest' is the default, and is configurable) or 'look up offset 
by startReadTime.
-  // Remember the current position without waiting until the first 
record is read. This
-  // ensures checkpoint is accurate even if the reader is closed 
before reading any records.
-  Instant startReadTime = spec.getStartReadTime();
-  if (startReadTime != null) {
-p.nextOffset =
-consumerSpEL.offsetForTime(consumer, p.topicPartition, 
spec.getStartReadTime());
-consumer.seek(p.topicPartition, p.nextOffset);
-  } else {
-p.nextOffset = consumer.position(p.topicPartition);
+  // Seek to start offset for each partition. This is the first 
interaction with the server.
+  // Unfortunately it can block forever in case of network issues like 
incorrect ACLs.
+  // Initialize partition in a separate thread and cancel it if takes 
longer than a minute.
+  for (final PartitionState pState : 

[29/43] beam git commit: This closes #3492: [BEAM-2551] KafkaIO reader blocks indefinitely in case of network issues

[kenn]

This closes #3492: [BEAM-2551] KafkaIO reader blocks indefinitely in case of 
network issues

  Add timeout to initialization of partition in KafkaIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b8f8d18a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b8f8d18a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b8f8d18a

Branch: refs/heads/gearpump-runner
Commit: b8f8d18ae2cdbb4874d9a0f45038037ecc2381d1
Parents: 85a99e2 526037b
Author: Kenneth Knowles 
Authored: Thu Jul 6 11:59:18 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 11:59:18 2017 -0700

--
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   | 81 +++-
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   | 30 
 2 files changed, 92 insertions(+), 19 deletions(-)
--

[32/43] beam git commit: This closes #3429: [BEAM-2502, BEAM-2505] More fixes to ReduceFnRunner.onTimers

[kenn]

This closes #3429: [BEAM-2502, BEAM-2505] More fixes to ReduceFnRunner.onTimers

  Process timer firings for a window together
  Ignore processing time timers in expired windows


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/628dace9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/628dace9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/628dace9

Branch: refs/heads/gearpump-runner
Commit: 628dace9c173182299f322fa6ed974c75dbff4b5
Parents: b8f8d18 935c077
Author: Kenneth Knowles 
Authored: Thu Jul 6 14:13:56 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 14:13:56 2017 -0700

--
 .../examples/complete/game/LeaderBoardTest.java |   2 +
 .../beam/runners/core/ReduceFnRunner.java   | 106 +--
 .../beam/runners/core/ReduceFnRunnerTest.java   |  81 +-
 3 files changed, 156 insertions(+), 33 deletions(-)
--

[24/43] beam git commit: Fix PValue input in _PubSubReadEvaluator

[kenn]

Fix PValue input in _PubSubReadEvaluator


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a75202f3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a75202f3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a75202f3

Branch: refs/heads/gearpump-runner
Commit: a75202f344f22be5c5fdf62b3eb54a151ad29af6
Parents: e8fdd37
Author: Charles Chen 
Authored: Wed Jul 5 16:18:51 2017 -0700
Committer: Ahmet Altay 
Committed: Wed Jul 5 18:08:59 2017 -0700

--
 sdks/python/apache_beam/runners/direct/transform_evaluator.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a75202f3/sdks/python/apache_beam/runners/direct/transform_evaluator.py
--
diff --git a/sdks/python/apache_beam/runners/direct/transform_evaluator.py 
b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
index 641291d..cb2ace2 100644
--- a/sdks/python/apache_beam/runners/direct/transform_evaluator.py
+++ b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
@@ -436,8 +436,9 @@ class _PubSubReadEvaluator(_TransformEvaluator):
   bundles = [bundle]
 else:
   bundles = []
-input_pvalue = self._applied_ptransform.inputs
-if not input_pvalue:
+if self._applied_ptransform.inputs:
+  input_pvalue = self._applied_ptransform.inputs[0]
+else:
   input_pvalue = pvalue.PBegin(self._applied_ptransform.transform.pipeline)
 unprocessed_bundle = self._evaluation_context.create_bundle(
 input_pvalue)

[01/43] beam git commit: Properly convert milliseconds whether there's less than 3/more than 9 digits. TimeUtil did not properly convert (and returned null) when the number of digits for fractions of

[kenn]

Repository: beam
Updated Branches:
  refs/heads/gearpump-runner 627ae0bc3 -> 1ce60b488


Properly convert milliseconds whether there's less than 3/more than 9 digits.
TimeUtil did not properly convert (and returned null) when the number of
digits for fractions of seconds was less than 3 digits or more than 9 digits.
The solution is to pad with zeros when there is less than 3 digits and to
truncate when there is more than 3.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7b8cd640
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7b8cd640
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7b8cd640

Branch: refs/heads/gearpump-runner
Commit: 7b8cd6401cb5ed6e184ed36571a89d3ae324dd5f
Parents: 893bf42
Author: Jeremie Lenfant-Engelmann 
Authored: Wed Jun 28 18:32:56 2017 -0700
Committer: Luke Cwik 
Committed: Fri Jun 30 08:59:19 2017 -0700

--
 .../beam/runners/dataflow/util/TimeUtil.java| 24 
 .../runners/dataflow/util/TimeUtilTest.java |  6 +
 2 files changed, 15 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7b8cd640/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/TimeUtil.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/TimeUtil.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/TimeUtil.java
index bff379f..172dc6e 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/TimeUtil.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/TimeUtil.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.dataflow.util;
 
+import com.google.common.base.Strings;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 import javax.annotation.Nullable;
@@ -98,26 +99,19 @@ public final class TimeUtil {
 int hour = Integer.valueOf(matcher.group(4));
 int minute = Integer.valueOf(matcher.group(5));
 int second = Integer.valueOf(matcher.group(6));
-int millis = 0;
-
-String frac = matcher.group(7);
-if (frac != null) {
-  int fracs = Integer.valueOf(frac);
-  if (frac.length() == 3) {  // millisecond resolution
-millis = fracs;
-  } else if (frac.length() == 6) {  // microsecond resolution
-millis = fracs / 1000;
-  } else if (frac.length() == 9) {  // nanosecond resolution
-millis = fracs / 100;
-  } else {
-return null;
-  }
-}
+int millis = computeMillis(matcher.group(7));
 
 return new DateTime(year, month, day, hour, minute, second, millis,
 ISOChronology.getInstanceUTC()).toInstant();
   }
 
+  private static int computeMillis(String frac) {
+if (frac == null) {
+  return 0;
+}
+return Integer.valueOf(frac.length() > 3 ? frac.substring(0, 3) : 
Strings.padEnd(frac, 3, '0'));
+  }
+
   /**
* Converts a {@link ReadableDuration} into a Dataflow API duration string.
*/

http://git-wip-us.apache.org/repos/asf/beam/blob/7b8cd640/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/TimeUtilTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/TimeUtilTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/TimeUtilTest.java
index e0785d4..1ac9fab 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/TimeUtilTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/TimeUtilTest.java
@@ -47,8 +47,14 @@ public final class TimeUtilTest {
 assertEquals(new Instant(1), fromCloudTime("1970-01-01T00:00:00.001001Z"));
 assertEquals(new Instant(1), 
fromCloudTime("1970-01-01T00:00:00.00100Z"));
 assertEquals(new Instant(1), 
fromCloudTime("1970-01-01T00:00:00.00101Z"));
+assertEquals(new Instant(0), fromCloudTime("1970-01-01T00:00:00.0Z"));
+assertEquals(new Instant(0), fromCloudTime("1970-01-01T00:00:00.00Z"));
+assertEquals(new Instant(420), fromCloudTime("1970-01-01T00:00:00.42Z"));
+assertEquals(new Instant(300), fromCloudTime("1970-01-01T00:00:00.3Z"));
+assertEquals(new Instant(20), fromCloudTime("1970-01-01T00:00:00.02Z"));
 assertNull(fromCloudTime(""));
 assertNull(fromCloudTime("1970-01-01T00:00:00"));
+assertNull(fromCloudTime("1970-01-01T00:00:00.1e3Z"));
   }
 
   @Test

[38/43] beam git commit: Fix javadoc generation for AmqpIO, CassandraIO and HCatalogIO

[kenn]

Fix javadoc generation for AmqpIO, CassandraIO and HCatalogIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/35061e88
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/35061e88
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/35061e88

Branch: refs/heads/gearpump-runner
Commit: 35061e88066589d1dbfa81aa37fbb270274d70c5
Parents: 440c7d4
Author: Ismaël Mejía 
Authored: Thu Jul 6 17:37:33 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Fri Jul 7 10:31:29 2017 +0200

--
 pom.xml   | 18 ++
 sdks/java/javadoc/pom.xml | 15 +++
 2 files changed, 33 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/35061e88/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 01474c1..d9ab9ae 100644
--- a/pom.xml
+++ b/pom.xml
@@ -428,6 +428,18 @@
 
   
 org.apache.beam
+beam-sdks-java-io-amqp
+${project.version}
+  
+
+  
+org.apache.beam
+beam-sdks-java-io-cassandra
+${project.version}
+  
+
+  
+org.apache.beam
 beam-sdks-java-io-elasticsearch
 ${project.version}
   
@@ -465,6 +477,12 @@
 
   
 org.apache.beam
+beam-sdks-java-io-hcatalog
+${project.version}
+  
+
+  
+org.apache.beam
 beam-sdks-java-io-jdbc
 ${project.version}
   

http://git-wip-us.apache.org/repos/asf/beam/blob/35061e88/sdks/java/javadoc/pom.xml
--
diff --git a/sdks/java/javadoc/pom.xml b/sdks/java/javadoc/pom.xml
index ddb92cf..51109fb 100644
--- a/sdks/java/javadoc/pom.xml
+++ b/sdks/java/javadoc/pom.xml
@@ -99,6 +99,16 @@
 
 
   org.apache.beam
+  beam-sdks-java-io-amqp
+
+
+
+  org.apache.beam
+  beam-sdks-java-io-cassandra
+
+
+
+  org.apache.beam
   beam-sdks-java-io-elasticsearch
 
 
@@ -124,6 +134,11 @@
 
 
   org.apache.beam
+  beam-sdks-java-io-hcatalog
+
+
+
+  org.apache.beam
   beam-sdks-java-io-jdbc
 
 

[12/43] beam git commit: [maven-release-plugin] prepare branch release-2.1.0

[kenn]

[maven-release-plugin] prepare branch release-2.1.0


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/51877a34
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/51877a34
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/51877a34

Branch: refs/heads/gearpump-runner
Commit: 51877a3405dbf778c3bb88f19bb194e54c3b3def
Parents: b8ac326
Author: Jean-Baptiste Onofré 
Authored: Wed Jul 5 16:47:29 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Wed Jul 5 16:47:29 2017 +0200

--
 pom.xml | 2 +-
 runners/direct-java/pom.xml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/51877a34/pom.xml
--
diff --git a/pom.xml b/pom.xml
index c0207ef..057954a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 https://git-wip-us.apache.org/repos/asf?p=beam.git;a=summary
-HEAD
+release-2.1.0
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/51877a34/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index 6346575..5b5aec2 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -117,7 +117,7 @@
   
 
 
-  
+  
 
   
 

[36/43] beam git commit: Simplified ByteBuddyOnTimerInvokerFactory

[kenn]

Simplified ByteBuddyOnTimerInvokerFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c8d98336
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c8d98336
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c8d98336

Branch: refs/heads/gearpump-runner
Commit: c8d983363efd3f3d93825ecc8e8abae2dfa4e008
Parents: 17bc3b1
Author: Innocent Djiofack 
Authored: Wed Jun 28 22:15:11 2017 -0400
Committer: Kenneth Knowles 
Committed: Thu Jul 6 21:46:53 2017 -0700

--
 .../reflect/ByteBuddyOnTimerInvokerFactory.java | 73 
 .../reflect/OnTimerMethodSpecifier.java | 37 ++
 2 files changed, 65 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c8d98336/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
index e031337..5e31f2e 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.sdk.transforms.reflect;
 
+
 import com.google.common.base.CharMatcher;
 import com.google.common.cache.CacheBuilder;
 import com.google.common.cache.CacheLoader;
@@ -61,13 +62,14 @@ class ByteBuddyOnTimerInvokerFactory implements 
OnTimerInvokerFactory {
 
 @SuppressWarnings("unchecked")
 Class> fnClass = (Class>) 
fn.getClass();
-
 try {
-  Constructor constructor = constructorCache.get(fnClass).get(timerId);
-  @SuppressWarnings("unchecked")
-  OnTimerInvoker invoker =
+OnTimerMethodSpecifier onTimerMethodSpecifier =
+OnTimerMethodSpecifier.forClassAndTimerId(fnClass, timerId);
+Constructor constructor = 
constructorCache.get(onTimerMethodSpecifier);
+
+OnTimerInvoker invoker =
   (OnTimerInvoker) constructor.newInstance(fn);
-  return invoker;
+return invoker;
 } catch (InstantiationException
 | IllegalAccessException
 | IllegalArgumentException
@@ -97,50 +99,31 @@ class ByteBuddyOnTimerInvokerFactory implements 
OnTimerInvokerFactory {
   private static final String FN_DELEGATE_FIELD_NAME = "delegate";
 
   /**
-   * A cache of constructors of generated {@link OnTimerInvoker} classes, 
keyed by {@link DoFn}
-   * class and then by {@link TimerId}.
+   * A cache of constructors of generated {@link OnTimerInvoker} classes,
+   * keyed by {@link OnTimerMethodSpecifier}.
*
* Needed because generating an invoker class is expensive, and to avoid 
generating an
* excessive number of classes consuming PermGen memory in Java's that still 
have PermGen.
*/
-  private final LoadingCache, LoadingCache>
-  constructorCache =
-  CacheBuilder.newBuilder()
-  .build(
-  new CacheLoader<
-  Class>, LoadingCache>() {
-@Override
-public LoadingCache load(
-final Class> fnClass) throws 
Exception {
-  return CacheBuilder.newBuilder().build(new 
OnTimerConstructorLoader(fnClass));
-}
-  });
-
-  /**
-   * A cache loader fixed to a particular {@link DoFn} class that loads 
constructors for the
-   * invokers for its {@link OnTimer @OnTimer} methods.
-   */
-  private static class OnTimerConstructorLoader extends CacheLoader {
-
-private final DoFnSignature signature;
-
-public OnTimerConstructorLoader(Class> clazz) {
-  this.signature = DoFnSignatures.getSignature(clazz);
-}
-
-@Override
-public Constructor load(String timerId) throws Exception {
-  Class> invokerClass =
-  generateOnTimerInvokerClass(signature, timerId);
-  try {
-return invokerClass.getConstructor(signature.fnClass());
-  } catch (IllegalArgumentException | NoSuchMethodException | 
SecurityException e) {
-throw new RuntimeException(e);
-  }
-}
-  }
-
-  /**
+  private final LoadingCache 
constructorCache =
+  CacheBuilder.newBuilder().build(
+  new CacheLoader() {
+  @Override
+  public Constructor 

[14/43] beam git commit: Update Python SDK version

[kenn]

Update Python SDK version


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/14fa7f79
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/14fa7f79
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/14fa7f79

Branch: refs/heads/gearpump-runner
Commit: 14fa7f79f0830739122b7573e032ad0aea172a98
Parents: 7f0723c
Author: Jean-Baptiste Onofré 
Authored: Wed Jul 5 16:52:48 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Wed Jul 5 16:52:48 2017 +0200

--
 sdks/python/apache_beam/version.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/14fa7f79/sdks/python/apache_beam/version.py
--
diff --git a/sdks/python/apache_beam/version.py 
b/sdks/python/apache_beam/version.py
index ae92a23..8b0a430 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,4 +18,4 @@
 """Apache Beam SDK version information and utilities."""
 
 
-__version__ = '2.1.0.dev'
+__version__ = '2.2.0.dev'

[30/43] beam git commit: Ignore processing time timers in expired windows

[kenn]

Ignore processing time timers in expired windows


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bd631b89
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bd631b89
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bd631b89

Branch: refs/heads/gearpump-runner
Commit: bd631b89a8434f0756e1596875e89013fb623ab5
Parents: b8f8d18
Author: Kenneth Knowles 
Authored: Thu Jun 22 18:09:11 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 14:12:39 2017 -0700

--
 .../beam/runners/core/ReduceFnRunner.java   | 10 ++
 .../beam/runners/core/ReduceFnRunnerTest.java   | 32 
 2 files changed, 42 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bd631b89/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
index ef33bef..0632c05 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
@@ -693,6 +693,11 @@ public class ReduceFnRunner {
   @SuppressWarnings("unchecked")
 WindowNamespace windowNamespace = (WindowNamespace) 
timer.getNamespace();
   W window = windowNamespace.getWindow();
+
+  if (TimeDomain.PROCESSING_TIME == timer.getDomain() && 
windowIsExpired(window)) {
+continue;
+  }
+
   ReduceFn.Context directContext =
   contextFactory.base(window, StateStyle.DIRECT);
   ReduceFn.Context renamedContext =
@@ -1090,4 +1095,9 @@ public class ReduceFnRunner {
 }
   }
 
+  private boolean windowIsExpired(BoundedWindow w) {
+return timerInternals
+.currentInputWatermarkTime()
+
.isAfter(w.maxTimestamp().plus(windowingStrategy.getAllowedLateness()));
+  }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/bd631b89/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
index 3a2c220..79ee91b 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
@@ -286,6 +286,38 @@ public class ReduceFnRunnerTest {
 
   /**
* Tests that when a processing time timer comes in after a window is expired
+   * it is just ignored.
+   */
+  @Test
+  public void testLateProcessingTimeTimer() throws Exception {
+WindowingStrategy strategy =
+WindowingStrategy.of((WindowFn) 
FixedWindows.of(Duration.millis(100)))
+.withTimestampCombiner(TimestampCombiner.EARLIEST)
+.withMode(AccumulationMode.ACCUMULATING_FIRED_PANES)
+.withAllowedLateness(Duration.ZERO)
+.withTrigger(
+Repeatedly.forever(
+
AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.millis(10;
+
+ReduceFnTester tester =
+ReduceFnTester.combining(strategy, Sum.ofIntegers(), VarIntCoder.of());
+
+tester.advanceProcessingTime(new Instant(5000));
+injectElement(tester, 2); // processing timer @ 5000 + 10; EOW timer @ 100
+injectElement(tester, 5);
+
+// After this advancement, the window is expired and only the GC process
+// should be allowed to touch it
+tester.advanceInputWatermarkNoTimers(new Instant(100));
+
+// This should not output
+tester.advanceProcessingTime(new Instant(6000));
+
+assertThat(tester.extractOutput(), emptyIterable());
+  }
+
+  /**
+   * Tests that when a processing time timer comes in after a window is expired
* but in the same bundle it does not cause a spurious output.
*/
   @Test

[03/43] beam git commit: Add support for PipelineOptions parameters

[kenn]

Add support for PipelineOptions parameters

This is a step towards eliminating catch-all context parameters and
making DoFns express their fine-grained data needs.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/56cb6c51
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/56cb6c51
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/56cb6c51

Branch: refs/heads/gearpump-runner
Commit: 56cb6c51748fde6ad56522733ab10edca062e802
Parents: f75dfe7
Author: Kenneth Knowles 
Authored: Tue Jun 13 10:29:50 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 30 15:38:47 2017 -0700

--
 ...eBoundedSplittableProcessElementInvoker.java |  5 ++
 .../beam/runners/core/SimpleDoFnRunner.java | 20 +++
 .../apache/beam/sdk/transforms/DoFnTester.java  |  5 ++
 .../reflect/ByteBuddyDoFnInvokerFactory.java|  6 ++
 .../sdk/transforms/reflect/DoFnInvoker.java | 13 +++-
 .../sdk/transforms/reflect/DoFnSignature.java   | 23 +++
 .../sdk/transforms/reflect/DoFnSignatures.java  | 22 ++-
 .../apache/beam/sdk/transforms/ParDoTest.java   | 63 
 .../transforms/reflect/DoFnSignaturesTest.java  | 14 +
 9 files changed, 169 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/56cb6c51/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
index 2db6531..475abf2 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
@@ -118,6 +118,11 @@ public class 
OutputAndTimeBoundedSplittableProcessElementInvoker<
   }
 
   @Override
+  public PipelineOptions pipelineOptions() {
+return pipelineOptions;
+  }
+
+  @Override
   public StartBundleContext startBundleContext(DoFn 
doFn) {
 throw new IllegalStateException(
 "Should not access startBundleContext() from @"

http://git-wip-us.apache.org/repos/asf/beam/blob/56cb6c51/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index 7d7babd..c3bfef6 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -233,6 +233,11 @@ public class SimpleDoFnRunner implements 
DoFnRunner.StartBundleContext 
startBundleContext(DoFn doFn) {
   return this;
 }
@@ -298,6 +303,11 @@ public class SimpleDoFnRunner implements 
DoFnRunner.StartBundleContext 
startBundleContext(DoFn doFn) {
   throw new UnsupportedOperationException(
   "Cannot access StartBundleContext outside of @StartBundle method.");
@@ -467,6 +477,11 @@ public class SimpleDoFnRunner implements 
DoFnRunner.StartBundleContext 
startBundleContext(DoFn doFn) {
   throw new UnsupportedOperationException("StartBundleContext parameters 
are not supported.");
 }
@@ -568,6 +583,11 @@ public class SimpleDoFnRunner implements 
DoFnRunner.StartBundleContext 
startBundleContext(DoFn doFn) {
   throw new UnsupportedOperationException("StartBundleContext parameters 
are not supported.");
 

[jira] [Commented] (BEAM-2572) Implement an S3 filesystem for Python SDK

[Sourabh Bajaj (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079846#comment-16079846
 ] 

Sourabh Bajaj commented on BEAM-2572:
-

+1 to not changing the signature of the ReadFromText.


> Implement an S3 filesystem for Python SDK
> -
>
> Key: BEAM-2572
> URL: https://issues.apache.org/jira/browse/BEAM-2572
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py
>Reporter: Dmitry Demeshchuk
>Assignee: Ahmet Altay
>Priority: Minor
>
> There are two paths worth exploring, to my understanding:
> 1. Sticking to the HDFS-based approach (like it's done in Java).
> 2. Using boto/boto3 for accessing S3 through its common API endpoints.
> I personally prefer the second approach, for a few reasons:
> 1. In real life, HDFS and S3 have different consistency guarantees, therefore 
> their behaviors may contradict each other in some edge cases (say, we write 
> something to S3, but it's not immediately accessible for reading from another 
> end).
> 2. There are other AWS-based sources and sinks we may want to create in the 
> future: DynamoDB, Kinesis, SQS, etc.
> 3. boto3 already provides somewhat good logic for basic things like 
> reattempting.
> Whatever path we choose, there's another problem related to this: we 
> currently cannot pass any global settings (say, pipeline options, or just an 
> arbitrary kwarg) to a filesystem. Because of that, we'd have to setup the 
> runner nodes to have AWS keys set up in the environment, which is not trivial 
> to achieve and doesn't look too clean either (I'd rather see one single place 
> for configuring the runner options).
> Also, it's worth mentioning that I already have a janky S3 filesystem 
> implementation that only supports DirectRunner at the moment (because of the 
> previous paragraph). I'm perfectly fine finishing it myself, with some 
> guidance from the maintainers.
> Where should I move on from here, and whose input should I be looking for?
> Thanks!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2593

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3359

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2486) Should throws some useful messages when statefulParDo use non-KV input

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2486.
---
Resolution: Fixed

> Should throws some useful messages when statefulParDo use non-KV input
> --
>
> Key: BEAM-2486
> URL: https://issues.apache.org/jira/browse/BEAM-2486
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, runner-flink
>Reporter: Jingsong Lee
>Assignee: Kenneth Knowles
> Fix For: 2.1.0
>
>
> Now Flink runner will throws a ClassCastException without detail messages 
> when a statefulParDo use non-KV input. It is not easy for users to find 
> errors and causes. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2486) Should throws some useful messages when statefulParDo use non-KV input

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079826#comment-16079826
 ] 

ASF GitHub Bot commented on BEAM-2486:
--

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/3519


> Should throws some useful messages when statefulParDo use non-KV input
> --
>
> Key: BEAM-2486
> URL: https://issues.apache.org/jira/browse/BEAM-2486
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, runner-flink
>Reporter: Jingsong Lee
>Assignee: Kenneth Knowles
> Fix For: 2.1.0
>
>
> Now Flink runner will throws a ClassCastException without detail messages 
> when a statefulParDo use non-KV input. It is not easy for users to find 
> errors and causes. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: BEAM-2575 ApexRunner doesn't emit watermarks for additional outputs

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master f467d6bbb -> 9f904dc00


BEAM-2575 ApexRunner doesn't emit watermarks for additional outputs


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cb5061e7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cb5061e7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cb5061e7

Branch: refs/heads/master
Commit: cb5061e7149519cb18673f4c572757dce3cc7bd1
Parents: f467d6b
Author: Thomas Weise 
Authored: Sun Jul 9 11:57:43 2017 -0700
Committer: Thomas Weise 
Committed: Sun Jul 9 12:12:52 2017 -0700

--
 .../operators/ApexParDoOperator.java| 21 +---
 .../runners/apex/examples/WordCountTest.java|  8 ++--
 2 files changed, 20 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cb5061e7/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
index 809ca2a..c3cbab2 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
@@ -359,10 +359,7 @@ public class ApexParDoOperator extends 
BaseOperator implements
   }
 }
 if (sideInputs.isEmpty()) {
-  if (traceTuples) {
-LOG.debug("\nemitting watermark {}\n", mark);
-  }
-  output.emit(mark);
+  outputWatermark(mark);
   return;
 }
 
@@ -370,10 +367,20 @@ public class ApexParDoOperator extends 
BaseOperator implements
 Math.min(pushedBackWatermark.get(), currentInputWatermark);
 if (potentialOutputWatermark > currentOutputWatermark) {
   currentOutputWatermark = potentialOutputWatermark;
-  if (traceTuples) {
-LOG.debug("\nemitting watermark {}\n", currentOutputWatermark);
+  
outputWatermark(ApexStreamTuple.WatermarkTuple.of(currentOutputWatermark));
+}
+  }
+
+  private void outputWatermark(ApexStreamTuple.WatermarkTuple mark) {
+if (traceTuples) {
+  LOG.debug("\nemitting {}\n", mark);
+}
+output.emit(mark);
+if (!additionalOutputPortMapping.isEmpty()) {
+  for (DefaultOutputPort additionalOutput :
+  additionalOutputPortMapping.values()) {
+additionalOutput.emit(mark);
   }
-  output.emit(ApexStreamTuple.WatermarkTuple.of(currentOutputWatermark));
 }
   }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/cb5061e7/runners/apex/src/test/java/org/apache/beam/runners/apex/examples/WordCountTest.java
--
diff --git 
a/runners/apex/src/test/java/org/apache/beam/runners/apex/examples/WordCountTest.java
 
b/runners/apex/src/test/java/org/apache/beam/runners/apex/examples/WordCountTest.java
index e76096e..ba75746 100644
--- 
a/runners/apex/src/test/java/org/apache/beam/runners/apex/examples/WordCountTest.java
+++ 
b/runners/apex/src/test/java/org/apache/beam/runners/apex/examples/WordCountTest.java
@@ -123,11 +123,15 @@ public class WordCountTest {
 options.setInputFile(new File(inputFile).getAbsolutePath());
 String outputFilePrefix = "target/wordcountresult.txt";
 options.setOutput(outputFilePrefix);
-WordCountTest.main(TestPipeline.convertToArgs(options));
 
 File outFile1 = new File(outputFilePrefix + "-0-of-2");
 File outFile2 = new File(outputFilePrefix + "-1-of-2");
-Assert.assertTrue(outFile1.exists() && outFile2.exists());
+Assert.assertTrue(!outFile1.exists() || outFile1.delete());
+Assert.assertTrue(!outFile2.exists() || outFile2.delete());
+
+WordCountTest.main(TestPipeline.convertToArgs(options));
+
+Assert.assertTrue("result files exist", outFile1.exists() && 
outFile2.exists());
 HashSet results = new HashSet<>();
 results.addAll(FileUtils.readLines(outFile1));
 results.addAll(FileUtils.readLines(outFile2));

[GitHub] beam pull request #3529: BEAM-2575 ApexRunner doesn't emit watermarks for ad...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3529


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3529: BEAM-2575 ApexRunner doesn't emit watermarks for additional outputs

[kenn]

This closes #3529: BEAM-2575 ApexRunner doesn't emit watermarks for additional 
outputs


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9f904dc0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9f904dc0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9f904dc0

Branch: refs/heads/master
Commit: 9f904dc00552349b535752e751ce82a7e484b471
Parents: f467d6b cb5061e
Author: Kenneth Knowles 
Authored: Sun Jul 9 20:18:55 2017 -0700
Committer: Kenneth Knowles 
Committed: Sun Jul 9 20:18:55 2017 -0700

--
 .../operators/ApexParDoOperator.java| 21 +---
 .../runners/apex/examples/WordCountTest.java|  8 ++--
 2 files changed, 20 insertions(+), 9 deletions(-)
--

[jira] [Commented] (BEAM-2557) DirectRunner should not have a hard dependency on Hamcrest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079816#comment-16079816
 ] 

ASF GitHub Bot commented on BEAM-2557:
--

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/3518


> DirectRunner should not have a hard dependency on Hamcrest
> --
>
> Key: BEAM-2557
> URL: https://issues.apache.org/jira/browse/BEAM-2557
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 2.1.0
>
>
> It seems that https://github.com/apache/beam/pull/3161/files introduced a 
> hard dependency from the DirectRunner to TestPipelineOptions. But those 
> options are only really appropriate for test scope - they incur a hard 
> dependency on Hamcrest. From what I can tell, you really just want to tweak 
> the DirectRunner behavior so the one flag could go in DirectPipelineOptions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2203) Arithmetic operators: support DATETIME & DATETIME_INTERVAL

[James Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079805#comment-16079805
 ] 

James Xu commented on BEAM-2203:


I have assigned this one to you [~vectorijk]

> Arithmetic operators: support DATETIME & DATETIME_INTERVAL
> --
>
> Key: BEAM-2203
> URL: https://issues.apache.org/jira/browse/BEAM-2203
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Kai Jiang
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3518: [BEAM-2557] Move DirectRunner knob out of core SDK ...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/3518


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[beam-site] branch asf-site updated (2db733b -> 3a9fd07)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 2db733b  Prepare repository for deployment.
 add 3b4bf9c  update apex runner properties example
 add e759c9e  This closes #269
 new 3a9fd07  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/runners/apex/index.html | 6 +++---
 src/documentation/runners/apex.md | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[jira] [Assigned] (BEAM-2203) Arithmetic operators: support DATETIME & DATETIME_INTERVAL

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2203:
--

Assignee: Kai Jiang

> Arithmetic operators: support DATETIME & DATETIME_INTERVAL
> --
>
> Key: BEAM-2203
> URL: https://issues.apache.org/jira/browse/BEAM-2203
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Kai Jiang
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4336

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2574) test unsupported/invalid cases in DSL

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2574:
-
Description: 
add test cases to cover the scenarios which are not supported, or have invalid 
usages.

Note, the previous failure of test is because of {{TestPipeline}} is marked as 
{{ClassRule}}, it works with {{Rule}}.

  was:add test cases to cover the scenarios which are not supported, or have 
invalid usages.


> test unsupported/invalid cases in DSL
> -
>
> Key: BEAM-2574
> URL: https://issues.apache.org/jira/browse/BEAM-2574
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>
> add test cases to cover the scenarios which are not supported, or have 
> invalid usages.
> Note, the previous failure of test is because of {{TestPipeline}} is marked 
> as {{ClassRule}}, it works with {{Rule}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2592

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3358

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2557) DirectRunner should not have a hard dependency on Hamcrest

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-2557.

Resolution: Fixed

> DirectRunner should not have a hard dependency on Hamcrest
> --
>
> Key: BEAM-2557
> URL: https://issues.apache.org/jira/browse/BEAM-2557
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 2.1.0
>
>
> It seems that https://github.com/apache/beam/pull/3161/files introduced a 
> hard dependency from the DirectRunner to TestPipelineOptions. But those 
> options are only really appropriate for test scope - they incur a hard 
> dependency on Hamcrest. From what I can tell, you really just want to tweak 
> the DirectRunner behavior so the one flag could go in DirectPipelineOptions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[2/2] beam git commit: This close #3518

[jbonofre]

This close #3518


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e0acb18b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e0acb18b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e0acb18b

Branch: refs/heads/release-2.1.0
Commit: e0acb18b0457c19b8cb4d2f2c1156b30074f4025
Parents: 4e281f6 ec6ccae
Author: Jean-Baptiste Onofré 
Authored: Sun Jul 9 21:26:32 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Sun Jul 9 21:26:32 2017 +0200

--
 runners/direct-java/pom.xml |  2 +-
 .../beam/runners/direct/DirectRegistrar.java|  2 +-
 .../beam/runners/direct/DirectRunner.java   |  5 +--
 .../beam/runners/direct/DirectTestOptions.java  | 42 
 .../runners/direct/DirectRegistrarTest.java |  2 +-
 .../beam/sdk/testing/TestPipelineOptions.java   | 10 -
 6 files changed, 47 insertions(+), 16 deletions(-)
--

[jira] [Commented] (BEAM-2575) ApexRunner doesn't emit watermarks for additional outputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079696#comment-16079696
 ] 

ASF GitHub Bot commented on BEAM-2575:
--

GitHub user tweise opened a pull request:

https://github.com/apache/beam/pull/3529

BEAM-2575 ApexRunner doesn't emit watermarks for additional outputs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @reuvenlax @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tweise/beam BEAM-2575_additonalOutputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3529.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3529


commit cb5061e7149519cb18673f4c572757dce3cc7bd1
Author: Thomas Weise 
Date:   2017-07-09T18:57:43Z

BEAM-2575 ApexRunner doesn't emit watermarks for additional outputs




> ApexRunner doesn't emit watermarks for additional outputs 
> --
>
> Key: BEAM-2575
> URL: https://issues.apache.org/jira/browse/BEAM-2575
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>
> https://lists.apache.org/thread.html/51113a207f96d0522fb81adb65e35e134a0c52cf4bbe1cfc46508d83@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3529: BEAM-2575 ApexRunner doesn't emit watermarks for ad...

[tweise]

GitHub user tweise opened a pull request:

https://github.com/apache/beam/pull/3529

BEAM-2575 ApexRunner doesn't emit watermarks for additional outputs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @reuvenlax @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tweise/beam BEAM-2575_additonalOutputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3529.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3529


commit cb5061e7149519cb18673f4c572757dce3cc7bd1
Author: Thomas Weise 
Date:   2017-07-09T18:57:43Z

BEAM-2575 ApexRunner doesn't emit watermarks for additional outputs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3528: [trivial] pom.xml: add license-maven-plugin and som...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/3528

[trivial] pom.xml: add license-maven-plugin and some default license merges

When reviewing the license of third-party dependencies, it's a mess because 
the
output is determined by strings people put in their `pom.xml` files rather 
than
a standard list.

Use `license-maven-plugin` to clean this up a bit -- each merge defines a 
standard name
for the license (the first in the list) followed by a list of variants. 
This makes
the output of `mvn license:aggregate-add-third-party` much cleaner.

I was conservative about which licenses I merged - for example, I left 
'Apache License'
alone rather than necessarily mapping it to 'The Apache Software License, 
Version 2.0',
since there are other versions.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam licensing-fixups

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3528.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3528


commit 4560022cd0223cf2ace5df1e278ced4b139ac687
Author: Daniel Halperin 
Date:   2017-07-09T19:02:22Z

pom.xml: add license-maven-plugin and some default license merges

When reviewing the license of third-party dependencies, it's a mess because 
the
output is determined by strings people put in their pom.xml files rather 
than
a standard list.

Use license-maven-plugin to clean this up a bit -- each merge defines a 
standard name
for the license (the first in the list) followed by a list of variants. 
This makes
the output of mvn license:aggregate-add-third-party much cleaner.

I was conservatie about which licenses I merged - for example, I left 
'Apache License'
alone rather than necessarily mapping it to 'The Apache Software License, 
Version 2.0',
since there are other versions.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-2575) ApexRunner doesn't emit watermarks for additional outputs

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise reassigned BEAM-2575:
--

Assignee: Thomas Weise

> ApexRunner doesn't emit watermarks for additional outputs 
> --
>
> Key: BEAM-2575
> URL: https://issues.apache.org/jira/browse/BEAM-2575
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>
> https://lists.apache.org/thread.html/51113a207f96d0522fb81adb65e35e134a0c52cf4bbe1cfc46508d83@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2575) ApexRunner doesn't emit watermarks for additional outputs

[Thomas Weise (JIRA)]

Thomas Weise created BEAM-2575:
--

 Summary: ApexRunner doesn't emit watermarks for additional outputs 
 Key: BEAM-2575
 URL: https://issues.apache.org/jira/browse/BEAM-2575
 Project: Beam
  Issue Type: Bug
  Components: runner-apex
Reporter: Thomas Weise


https://lists.apache.org/thread.html/51113a207f96d0522fb81adb65e35e134a0c52cf4bbe1cfc46508d83@%3Cdev.beam.apache.org%3E




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2574) test unsupported/invalid cases in DSL

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-2574:


 Summary: test unsupported/invalid cases in DSL
 Key: BEAM-2574
 URL: https://issues.apache.org/jira/browse/BEAM-2574
 Project: Beam
  Issue Type: Task
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


add test cases to cover the scenarios which are not supported, or have invalid 
usages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2591

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #3357

[Apache Jenkins Server]

See 


--
[...truncated 479.99 KB...]
2017-07-09T18:07:51.858 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/log4j/log4j/1.2.16/log4j-1.2.16.pom (20 KB 
at 684.9 KB/sec)
2017-07-09T18:07:51.861 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.pom
2017-07-09T18:07:51.887 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.pom
 (10 KB at 344.4 KB/sec)
2017-07-09T18:07:51.900 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-clients_2.10/1.3.0/flink-clients_2.10-1.3.0.jar
2017-07-09T18:07:51.900 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.jar
2017-07-09T18:07:51.901 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.3.0/flink-core-1.3.0.jar
2017-07-09T18:07:51.901 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/force-shading/1.3.0/force-shading-1.3.0.jar
2017-07-09T18:07:51.904 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-annotations/1.3.0/flink-annotations-1.3.0.jar
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
2017-07-09T18:07:51.935 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/force-shading/1.3.0/force-shading-1.3.0.jar
 (8 KB at 203.2 KB/sec)
2017-07-09T18:07:51.935 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar
2017-07-09T18:07:51.939 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-annotations/1.3.0/flink-annotations-1.3.0.jar
 (8 KB at 184.9 KB/sec)
2017-07-09T18:07:51.939 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/objenesis/objenesis/2.1/objenesis-2.1.jar
2017-07-09T18:07:51.972 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/objenesis/objenesis/2.1/objenesis-2.1.jar
 (41 KB at 591.0 KB/sec)
2017-07-09T18:07:51.972 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-metrics-core/1.3.0/flink-metrics-core-1.3.0.jar
2017-07-09T18:07:52.004 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar
 (332 KB at 3216.8 KB/sec)
2017-07-09T18:07:52.004 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-java/1.3.0/flink-java-1.3.0.jar
2017-07-09T18:07:52.009 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-metrics-core/1.3.0/flink-metrics-core-1.3.0.jar
 (16 KB at 147.3 KB/sec)
2017-07-09T18:07:52.010 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.3.0/flink-shaded-hadoop2-1.3.0.jar
2017-07-09T18:07:52.015 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-clients_2.10/1.3.0/flink-clients_2.10-1.3.0.jar
 (89 KB at 764.7 KB/sec)
2017-07-09T18:07:52.015 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/xmlenc/xmlenc/0.52/xmlenc-0.52.jar
2017-07-09T18:07:52.051 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/xmlenc/xmlenc/0.52/xmlenc-0.52.jar (15 KB 
at 99.7 KB/sec)
2017-07-09T18:07:52.051 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.4/commons-codec-1.4.jar
2017-07-09T18:07:52.094 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.4/commons-codec-1.4.jar
 (57 KB at 298.9 KB/sec)
2017-07-09T18:07:52.094 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-net/commons-net/3.1/commons-net-3.1.jar
2017-07-09T18:07:52.145 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-java/1.3.0/flink-java-1.3.0.jar
 (752 KB at 3119.8 KB/sec)
2017-07-09T18:07:52.145 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar
2017-07-09T18:07:52.162 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.3.0/flink-core-1.3.0.jar
 (1362 KB at 5196.6 KB/sec)
2017-07-09T18:07:52.162 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar
2017-07-09T18:07:52.175 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-net/commons-net/3.1/commons-net-3.1.jar
 (267 KB at 985.1 KB/sec)
2017-07-09T18:07:52.175 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-el/commons-el/1.0/commons-el-1.0.jar
2017-07-09T18:07:52.207 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar
 (173 KB 

[beam-site] branch mergebot updated (09c53eb -> 2db733b)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 09c53eb  This closes #266
 add 2db733b  Prepare repository for deployment.

No new revisions were added by this update.

Summary of changes:
 content/contribute/team/index.html | 9 +
 1 file changed, 9 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] branch asf-site updated (8447470 -> 2db733b)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 8447470  Prepare repository for deployment.
 add b8438a7  Add JingsongLee as committer
 add 153a095  Restore index.html and fix order
 add 09c53eb  This closes #266
 new 2db733b  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/team/index.html | 9 +
 src/_beam_team/team.md | 6 ++
 2 files changed, 15 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 2db733bc7ff6aaf7021c37dca8be4b3f62fc306a
Author: Mergebot 
AuthorDate: Sun Jul 9 15:49:48 2017 +

Prepare repository for deployment.
---
 content/contribute/team/index.html | 9 +
 1 file changed, 9 insertions(+)

diff --git a/content/contribute/team/index.html 
b/content/contribute/team/index.html
index ade8a0c..76ec22e 100644
--- a/content/contribute/team/index.html
+++ b/content/contribute/team/index.html
@@ -294,6 +294,15 @@
 
   
 
+  Jingsong Lee
+  lzljs3620320
+  lzljs3620320 [at] apache [dot] org
+  Alibaba
+  committer
+  +8
+
+  
+
   Stas Levin
   staslevin
   staslevin [at] apache [dot] org

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (c0c7e54 -> 09c53eb)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from c0c7e54  This closes #264
 add 8447470  Prepare repository for deployment.
 new b8438a7  Add JingsongLee as committer
 new 153a095  Restore index.html and fix order
 new 09c53eb  This closes #266

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/io/built-in/index.html | 9 +
 src/_beam_team/team.md   | 6 ++
 2 files changed, 11 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 03/03: This closes #266

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 09c53ebee38383ddb828f92da2e6280bdbc9ae90
Merge: 8447470 153a095
Author: Mergebot 
AuthorDate: Sun Jul 9 15:47:27 2017 +

This closes #266

 src/_beam_team/team.md | 6 ++
 1 file changed, 6 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 01/03: Add JingsongLee as committer

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit b8438a7ca717d1c659e0883b5e197dc45de6c9f0
Author: JingsongLi 
AuthorDate: Fri Jul 7 00:05:19 2017 +0800

Add JingsongLee as committer
---
 content/contribute/team/index.html | 9 +
 src/_beam_team/team.md | 6 ++
 2 files changed, 15 insertions(+)

diff --git a/content/contribute/team/index.html 
b/content/contribute/team/index.html
index ade8a0c..7794d38 100644
--- a/content/contribute/team/index.html
+++ b/content/contribute/team/index.html
@@ -373,6 +373,15 @@
   committer
   +2
 
+
+
+  Jingsong Lee
+  lzljs3620320
+  lzljs3620320 [at] apache [dot] org
+  Alibaba
+  committer
+  +8
+
   
 
   
diff --git a/src/_beam_team/team.md b/src/_beam_team/team.md
index 77e5e63..9a1f7e2 100644
--- a/src/_beam_team/team.md
+++ b/src/_beam_team/team.md
@@ -146,4 +146,10 @@ members:
 organization: PayPal
 roles: committer
 time_zone: "+2"
+  - name: Jingsong Lee
+apache_id: lzljs3620320
+email: lzljs3620320 [at] apache [dot] org
+organization: Alibaba
+roles: committer
+time_zone: "+8"
 ---

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 02/03: Restore index.html and fix order

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 153a09528571216c6325ae66c8a3872159ede8cc
Author: JingsongLi 
AuthorDate: Sat Jul 8 13:00:07 2017 +0800

Restore index.html and fix order
---
 content/contribute/team/index.html |  9 -
 src/_beam_team/team.md | 12 ++--
 2 files changed, 6 insertions(+), 15 deletions(-)

diff --git a/content/contribute/team/index.html 
b/content/contribute/team/index.html
index 7794d38..ade8a0c 100644
--- a/content/contribute/team/index.html
+++ b/content/contribute/team/index.html
@@ -373,15 +373,6 @@
   committer
   +2
 
-
-
-  Jingsong Lee
-  lzljs3620320
-  lzljs3620320 [at] apache [dot] org
-  Alibaba
-  committer
-  +8
-
   
 
   
diff --git a/src/_beam_team/team.md b/src/_beam_team/team.md
index 9a1f7e2..fd3feb2 100644
--- a/src/_beam_team/team.md
+++ b/src/_beam_team/team.md
@@ -92,6 +92,12 @@ members:
 organization: data Artisans
 roles: committer, PMC
 time_zone: "+1"
+  - name: Jingsong Lee
+apache_id: lzljs3620320
+email: lzljs3620320 [at] apache [dot] org
+organization: Alibaba
+roles: committer
+time_zone: "+8"
   - name: Stas Levin
 apache_id: staslevin
 email: staslevin [at] apache [dot] org
@@ -146,10 +152,4 @@ members:
 organization: PayPal
 roles: committer
 time_zone: "+2"
-  - name: Jingsong Lee
-apache_id: lzljs3620320
-email: lzljs3620320 [at] apache [dot] org
-organization: Alibaba
-roles: committer
-time_zone: "+8"
 ---

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #1946

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2590

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3356

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-934) Findbugs doesn't pass in Java8 Examples

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-934:
-

Assignee: Jean-Baptiste Onofré

> Findbugs doesn't pass in Java8 Examples
> ---
>
> Key: BEAM-934
> URL: https://issues.apache.org/jira/browse/BEAM-934
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>  Labels: newbie, starter
>
> {code}
> [INFO] --- findbugs-maven-plugin:3.0.1:check (default) @ beam-examples-java8 
> ---
> [INFO] BugInstance size is 2
> [INFO] Error size is 0
> [INFO] Total bugs: 2
> [INFO] Result of integer multiplication cast to long in 
> org.apache.beam.examples.complete.game.injector.Injector$TeamInfo.getEndTimeInMillis()
>  [org.apache.beam.examples.complete.game.injector.Injector$TeamInfo] At 
> Injector.java:[line 170]
> [INFO] Format string should use %n rather than \n in 
> org.apache.beam.examples.complete.game.injector.InjectorUtils.createTopic(Pubsub,
>  String) [org.apache.beam.examples.complete.game.injector.InjectorUtils] At 
> InjectorUtils.java:[line 96]
> [INFO]  
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2203) Arithmetic operators: support DATETIME & DATETIME_INTERVAL

[Kai Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079517#comment-16079517
 ] 

Kai Jiang commented on BEAM-2203:
-

[~xumingming] I would take a try on this one. 
Should we implement these two first?
TIMESTAMPADD(timeUnit, integer, datetime)
TIMESTAMPDIFF(timeUnit, datetime, datetime2)

> Arithmetic operators: support DATETIME & DATETIME_INTERVAL
> --
>
> Key: BEAM-2203
> URL: https://issues.apache.org/jira/browse/BEAM-2203
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: James Xu
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2204) support TUMBLE/HOP/SESSION_START

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079499#comment-16079499
 ] 

ASF GitHub Bot commented on BEAM-2204:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3527

[BEAM-2204] support TUMBLE/HOP/SESSION_START

*_START function is supported for TUMBLE/HOP/SESSION window, however *_END 
is not available due to some issue with Calcite relational expressions, need a 
follow up with Calcite team.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2204

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3527.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3527


commit 1d33e7ba1a0efe27d5b38ded15f7e3f4b85b0869
Author: mingmxu 
Date:   2017-07-09T07:52:23Z

support TUMBLE/HOP/SESSION _START function




> support TUMBLE/HOP/SESSION_START
> 
>
> Key: BEAM-2204
> URL: https://issues.apache.org/jira/browse/BEAM-2204
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>
> As described in {{CALCITE-1761}}, TUMBLE/HOP/SESSION_START/END do not resolve 
> time field correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3527: [BEAM-2204] support TUMBLE/HOP/SESSION_START

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3527

[BEAM-2204] support TUMBLE/HOP/SESSION_START

*_START function is supported for TUMBLE/HOP/SESSION window, however *_END 
is not available due to some issue with Calcite relational expressions, need a 
follow up with Calcite team.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2204

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3527.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3527


commit 1d33e7ba1a0efe27d5b38ded15f7e3f4b85b0869
Author: mingmxu 
Date:   2017-07-09T07:52:23Z

support TUMBLE/HOP/SESSION _START function




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-2204) support TUMBLE/HOP/SESSION_START

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2204:
-
Summary: support TUMBLE/HOP/SESSION_START  (was: fix error in CALCITE-1761)

> support TUMBLE/HOP/SESSION_START
> 
>
> Key: BEAM-2204
> URL: https://issues.apache.org/jira/browse/BEAM-2204
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>
> As described in {{CALCITE-1761}}, TUMBLE/HOP/SESSION_START/END do not resolve 
> time field correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2589

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3526: BEAM-934 Findbugs doesn't pass in Java8 Examples

[eralmas7]

GitHub user eralmas7 opened a pull request:

https://github.com/apache/beam/pull/3526

BEAM-934 Findbugs doesn't pass in Java8 Examples

Maven Build result:

...
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ 
beam-examples-java8 ---
[INFO] Installing 
C:\workspace-apache\beam\examples\java8\target\beam-examples-java8-2.2.0-SNAPSHOT.jar
 to 
C:\Users\almass\.m2\repository\org\apache\beam\beam-examples-java8\2.2.0-SNAPSHOT\beam-examples-java8-2.2.0-SNAPSHOT.jar
[INFO] Installing 
C:\workspace-apache\beam\examples\java8\dependency-reduced-pom.xml to 
C:\Users\almass\.m2\repository\org\apache\beam\beam-examples-java8\2.2.0-SNAPSHOT\beam-examples-java8-2.2.0-SNAPSHOT.pom
[INFO] Installing 
C:\workspace-apache\beam\examples\java8\target\beam-examples-java8-2.2.0-SNAPSHOT-tests.jar
 to 
C:\Users\almass\.m2\repository\org\apache\beam\beam-examples-java8\2.2.0-SNAPSHOT\beam-examples-java8-2.2.0-SNAPSHOT-tests.jar
[INFO] Installing 
C:\workspace-apache\beam\examples\java8\target\beam-examples-java8-2.2.0-SNAPSHOT-tests.jar
 to 
C:\Users\almass\.m2\repository\org\apache\beam\beam-examples-java8\2.2.0-SNAPSHOT\beam-examples-java8-2.2.0-SNAPSHOT-tests.jar
[INFO] 

[INFO] BUILD SUCCESS
[INFO] 

[INFO] Total time: 02:20 min
[INFO] Finished at: 2017-07-09T11:49:43+05:30
[INFO] Final Memory: 30M/109M
[INFO] 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eralmas7/beam bugFix-934

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3526.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3526


commit c909c9efe97632a9a9ab19ca6a6d99f3c8c5ffe0
Author: eralmas7 
Date:   2017-07-09T06:20:52Z

BEAM-934 Fixed build by fixing firebug error.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-934) Findbugs doesn't pass in Java8 Examples

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079482#comment-16079482
 ] 

ASF GitHub Bot commented on BEAM-934:
-

GitHub user eralmas7 opened a pull request:

https://github.com/apache/beam/pull/3526

BEAM-934 Findbugs doesn't pass in Java8 Examples

Maven Build result:

...
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ 
beam-examples-java8 ---
[INFO] Installing 
C:\workspace-apache\beam\examples\java8\target\beam-examples-java8-2.2.0-SNAPSHOT.jar
 to 
C:\Users\almass\.m2\repository\org\apache\beam\beam-examples-java8\2.2.0-SNAPSHOT\beam-examples-java8-2.2.0-SNAPSHOT.jar
[INFO] Installing 
C:\workspace-apache\beam\examples\java8\dependency-reduced-pom.xml to 
C:\Users\almass\.m2\repository\org\apache\beam\beam-examples-java8\2.2.0-SNAPSHOT\beam-examples-java8-2.2.0-SNAPSHOT.pom
[INFO] Installing 
C:\workspace-apache\beam\examples\java8\target\beam-examples-java8-2.2.0-SNAPSHOT-tests.jar
 to 
C:\Users\almass\.m2\repository\org\apache\beam\beam-examples-java8\2.2.0-SNAPSHOT\beam-examples-java8-2.2.0-SNAPSHOT-tests.jar
[INFO] Installing 
C:\workspace-apache\beam\examples\java8\target\beam-examples-java8-2.2.0-SNAPSHOT-tests.jar
 to 
C:\Users\almass\.m2\repository\org\apache\beam\beam-examples-java8\2.2.0-SNAPSHOT\beam-examples-java8-2.2.0-SNAPSHOT-tests.jar
[INFO] 

[INFO] BUILD SUCCESS
[INFO] 

[INFO] Total time: 02:20 min
[INFO] Finished at: 2017-07-09T11:49:43+05:30
[INFO] Final Memory: 30M/109M
[INFO] 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eralmas7/beam bugFix-934

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3526.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3526


commit c909c9efe97632a9a9ab19ca6a6d99f3c8c5ffe0
Author: eralmas7 
Date:   2017-07-09T06:20:52Z

BEAM-934 Fixed build by fixing firebug error.




> Findbugs doesn't pass in Java8 Examples
> ---
>
> Key: BEAM-934
> URL: https://issues.apache.org/jira/browse/BEAM-934
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Daniel Halperin
>  Labels: newbie, starter
>
> {code}
> [INFO] --- findbugs-maven-plugin:3.0.1:check (default) @ beam-examples-java8 
> ---
> [INFO] BugInstance size is 2
> [INFO] Error size is 0
> [INFO] Total bugs: 2
> [INFO] Result of integer multiplication cast to long in 
> org.apache.beam.examples.complete.game.injector.Injector$TeamInfo.getEndTimeInMillis()
>  [org.apache.beam.examples.complete.game.injector.Injector$TeamInfo] At 
> Injector.java:[line 170]
> [INFO] Format string should use %n rather than \n in 
> org.apache.beam.examples.complete.game.injector.InjectorUtils.createTopic(Pubsub,
>  String) [org.apache.beam.examples.complete.game.injector.InjectorUtils] At 
> InjectorUtils.java:[line 96]
> [INFO]  
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3355

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #1945

[Apache Jenkins Server]

See 


--
[...truncated 471.02 KB...]
2017-07-09T06:05:42.881 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.pom
 (2 KB at 62.1 KB/sec)
2017-07-09T06:05:42.882 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.6.1/slf4j-parent-1.6.1.pom
2017-07-09T06:05:42.915 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.6.1/slf4j-parent-1.6.1.pom
 (10 KB at 275.9 KB/sec)
2017-07-09T06:05:42.917 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/log4j/log4j/1.2.16/log4j-1.2.16.pom
2017-07-09T06:05:42.943 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/log4j/log4j/1.2.16/log4j-1.2.16.pom (20 KB 
at 764.0 KB/sec)
2017-07-09T06:05:42.946 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/jline/jline/0.9.94/jline-0.9.94.pom
2017-07-09T06:05:42.972 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/jline/jline/0.9.94/jline-0.9.94.pom (7 KB 
at 238.8 KB/sec)
2017-07-09T06:05:42.974 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.pom
2017-07-09T06:05:43.002 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.pom
 (26 KB at 913.9 KB/sec)
2017-07-09T06:05:43.007 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-framework/2.7.1/curator-framework-2.7.1.pom
2017-07-09T06:05:43.033 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-framework/2.7.1/curator-framework-2.7.1.pom
 (3 KB at 78.3 KB/sec)
2017-07-09T06:05:43.035 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.7.1/apache-curator-2.7.1.pom
2017-07-09T06:05:43.081 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/apache-curator/2.7.1/apache-curator-2.7.1.pom
 (32 KB at 675.7 KB/sec)
2017-07-09T06:05:43.084 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.pom
2017-07-09T06:05:43.124 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.pom
 (3 KB at 56.6 KB/sec)
2017-07-09T06:05:43.126 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.42/jsch-0.1.42.pom
2017-07-09T06:05:43.153 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.42/jsch-0.1.42.pom 
(967 B at 35.0 KB/sec)
2017-07-09T06:05:43.154 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-recipes/2.7.1/curator-recipes-2.7.1.pom
2017-07-09T06:05:43.180 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-recipes/2.7.1/curator-recipes-2.7.1.pom
 (3 KB at 88.8 KB/sec)
2017-07-09T06:05:43.183 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.pom
2017-07-09T06:05:43.209 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.pom
 (4 KB at 153.6 KB/sec)
2017-07-09T06:05:43.211 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/htrace/htrace/3.1.0-incubating/htrace-3.1.0-incubating.pom
2017-07-09T06:05:43.237 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/htrace/htrace/3.1.0-incubating/htrace-3.1.0-incubating.pom
 (12 KB at 436.4 KB/sec)
2017-07-09T06:05:43.239 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/12/apache-12.pom
2017-07-09T06:05:43.267 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/12/apache-12.pom (16 KB 
at 561.3 KB/sec)
2017-07-09T06:05:43.269 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/datatorrent/netlet/1.2.1/netlet-1.2.1.pom
2017-07-09T06:05:43.298 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/datatorrent/netlet/1.2.1/netlet-1.2.1.pom
 (19 KB at 629.8 KB/sec)
2017-07-09T06:05:43.301 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.pom
2017-07-09T06:05:43.327 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.pom
 (2 KB at 60.1 KB/sec)
2017-07-09T06:05:43.331 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.pom
2017-07-09T06:05:43.358 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.pom
 (7 KB at 225.0 KB/sec)
2017-07-09T06:05:43.361 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/objenesis/objenesis/2.1/objenesis-2.1.pom
2017-07-09T06:05:43.386 [INFO] Downloaded: