Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2090

[Apache Jenkins Server]

See 

[GitHub] beam pull request #1829: Add checkstyle:check to verify

[ravwojdyla]

GitHub user ravwojdyla opened a pull request:

https://github.com/apache/beam/pull/1829

Add checkstyle:check to verify

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ravwojdyla/incubator-beam check_verify

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1829.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1829


commit 1f3f38e26e024868a85783c4991eec494856e9bb
Author: Rafal Wojdyla 
Date:   2017-01-24T05:20:50Z

Add checkstyle:check to verify




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1828: [BEAM-79] Fix PostCommit test confs for Gearpump ru...

[manuzhang]

GitHub user manuzhang opened a pull request:

https://github.com/apache/beam/pull/1828

[BEAM-79] Fix PostCommit test confs for Gearpump runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam 
gearpump-runner-post-commit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1828.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1828


commit 4fd216b6b9f7427a7df118a60202752ebeb1836d
Author: manuzhang 
Date:   2017-01-24T05:16:07Z

[BEAM-79] Fix PostCommit test confs for Gearpump runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2089

[Apache Jenkins Server]

See 

[GitHub] beam pull request #1827: [BEAM-27] Allow absolute timers

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/1827

[BEAM-27] Allow absolute timers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam absolute-timers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1827.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1827


commit 352e8411612fa1915878428746aa21b618e39218
Author: Kenneth Knowles 
Date:   2017-01-24T04:50:50Z

Allow absolute timers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1424

[Apache Jenkins Server]

See 

[2/2] beam git commit: This closes #1822: Revert "Simplified API surface verifications"

[kenn]

This closes #1822: Revert "Simplified API surface verifications"


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cb6e0a80
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cb6e0a80
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cb6e0a80

Branch: refs/heads/master
Commit: cb6e0a80c57b056489d447cde092cffdd041eed5
Parents: 6ecbfb9 9248bef
Author: Kenneth Knowles 
Authored: Mon Jan 23 19:47:47 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Jan 23 19:47:47 2017 -0800

--
 .../org/apache/beam/sdk/util/ApiSurface.java| 420 +--
 .../org/apache/beam/SdkCoreApiSurfaceTest.java  |  61 ---
 .../apache/beam/sdk/util/ApiSurfaceTest.java| 152 +--
 .../apache/beam/sdk/io/gcp/ApiSurfaceTest.java  | 134 ++
 .../beam/sdk/io/gcp/GcpApiSurfaceTest.java  |  76 
 5 files changed, 359 insertions(+), 484 deletions(-)
--

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #2088

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Gearpump #158

[Apache Jenkins Server]

See 


Changes:

[owenzhang1990] [BEAM-1180] Implement GearpumpPipelineResult

[owenzhang1990] update ROS configurations

[owenzhang1990] activate ROS on Gearpump by default

[owenzhang1990] fix group by window

[owenzhang1990] update to latest gearpump dsl function interface

[owenzhang1990] support OutputTimeFn

[owenzhang1990] return encoded key for GroupByKey translation

[owenzhang1990] fix ParDo.BoundMulti translation

[owenzhang1990] reduce timeout to wait for result

[owenzhang1990] Remove cache for Gearpump on travis

[owenzhang1990] note thread is interrupted on InterruptedException

--
[...truncated 17787 lines...]
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:155)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:81)
at 
org.apache.maven.plugin.surefire.InPluginVMSurefireStarter.runSuitesInProcess(InPluginVMSurefireStarter.java:82)
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1010)
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:862)
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:755)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at 
org.jvnet.hudson.maven3.launcher.Maven32Launcher.main(Maven32Launcher.java:132)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
at jenkins.maven3.agent.Maven32Main.launch(Maven32Main.java:186)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:136)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:71)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 80 more
Caused by: java.lang.VerifyError: Inconsistent stackmap frames at branch target 
152
Exception Details:
  Location:
akka/dispatch/Mailbox.processAllSystemMessages()V @152: getstatic
  Reason:
Type top (current frame, locals[9]) is not assignable to 
'akka/dispatch/sysmsg/SystemMessage' (stack map, locals[9])
  Current Frame:
bci: @131
flags: { }

[11/12] beam git commit: note thread is interrupted on InterruptedException

[kenn]

note thread is interrupted on InterruptedException


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d814857a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d814857a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d814857a

Branch: refs/heads/gearpump-runner
Commit: d814857a6c372ba3f87106d49d3ce1ef7c3c7766
Parents: 85dcfbd
Author: manuzhang 
Authored: Fri Jan 20 13:21:24 2017 +0800
Committer: manuzhang 
Committed: Fri Jan 20 13:21:24 2017 +0800

--
 .../apache/beam/runners/gearpump/GearpumpPipelineResult.java | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d814857a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
index 3dd78de..9e53517 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
@@ -67,7 +67,13 @@ public class GearpumpPipelineResult implements 
PipelineResult {
 do {
   try {
 Thread.sleep(defaultWaitInterval.getMillis());
-  } catch (InterruptedException e) {
+  } catch (Exception e) {
+if (e instanceof InterruptedException) {
+  Thread.currentThread().interrupt();
+}
+if (e instanceof RuntimeException) {
+  throw (RuntimeException) e;
+}
 throw new RuntimeException(e);
   }
 } while (State.RUNNING == getGearpumpState()

[08/12] beam git commit: fix ParDo.BoundMulti translation

[kenn]

fix ParDo.BoundMulti translation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b2d326ff
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b2d326ff
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b2d326ff

Branch: refs/heads/gearpump-runner
Commit: b2d326ff73afca5c8e941c8006e9d74261a6b9df
Parents: 364a3f0
Author: manuzhang 
Authored: Mon Jan 16 12:31:26 2017 +0800
Committer: manuzhang 
Committed: Mon Jan 16 12:31:26 2017 +0800

--
 .../gearpump/translators/ParDoBoundMultiTranslator.java| 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b2d326ff/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoBoundMultiTranslator.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoBoundMultiTranslator.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoBoundMultiTranslator.java
index 0d5b8bc..bf7073b 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoBoundMultiTranslator.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoBoundMultiTranslator.java
@@ -91,8 +91,7 @@ public class ParDoBoundMultiTranslator 
implements
 private final DoFnRunnerFactory doFnRunnerFactory;
 private DoFnRunner doFnRunner;
 private final DoFn doFn;
-private final List>> outputs 
= Lists
-.newArrayList();
+private List>> outputs;
 
 public DoFnMultiFunction(
 GearpumpPipelineOptions pipelineOptions,
@@ -127,6 +126,8 @@ public class ParDoBoundMultiTranslator 
implements
 
 @Override
 public Iterator>> 
apply(WindowedValue wv) {
+  outputs = Lists.newArrayList();
+
   if (null == doFnRunner) {
 doFnRunner = doFnRunnerFactory.createRunner();
   }
@@ -166,6 +167,7 @@ public class ParDoBoundMultiTranslator 
implements
 
 @Override
 public WindowedValue apply(WindowedValue> wv) {
+  // System.out.println(wv.getValue().getKey() + ":" + 
wv.getValue().getValue());
   return WindowedValue.of(wv.getValue().getValue(), wv.getTimestamp(),
   wv.getWindows(), wv.getPane());
 }

[09/12] beam git commit: reduce timeout to wait for result

[kenn]

reduce timeout to wait for result


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7613ec44
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7613ec44
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7613ec44

Branch: refs/heads/gearpump-runner
Commit: 7613ec44cedf12d1e7bf80e8bb6a505f09653c4f
Parents: b2d326f
Author: manuzhang 
Authored: Mon Jan 16 13:25:12 2017 +0800
Committer: manuzhang 
Committed: Mon Jan 16 13:25:12 2017 +0800

--
 .../org/apache/beam/runners/gearpump/GearpumpPipelineResult.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7613ec44/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
index 9c8f7b3..3dd78de 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
@@ -42,8 +42,8 @@ public class GearpumpPipelineResult implements PipelineResult 
{
 
   private final ClientContext client;
   private final int appId;
-  private final Duration defaultWaitDuration = Duration.standardSeconds(60);
-  private final Duration defaultWaitInterval = Duration.standardSeconds(10);
+  private final Duration defaultWaitDuration = Duration.standardSeconds(30);
+  private final Duration defaultWaitInterval = Duration.standardSeconds(5);
 
   public GearpumpPipelineResult(ClientContext client, int appId) {
 this.client = client;

[04/12] beam git commit: fix group by window

[kenn]

fix group by window


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e63d42d1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e63d42d1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e63d42d1

Branch: refs/heads/gearpump-runner
Commit: e63d42d1113728badc66285e7ce7a8ce204a82d9
Parents: ea633d2
Author: manuzhang 
Authored: Sat Jan 7 23:07:23 2017 +0800
Committer: manuzhang 
Committed: Sat Jan 14 13:35:31 2017 +0800

--
 .../beam/runners/gearpump/GearpumpRunner.java   |  3 ++-
 .../translators/GroupByKeyTranslator.java   |  4 +--
 .../translators/TranslationContext.java |  1 -
 .../translators/WindowBoundTranslator.java  | 27 ++--
 .../gearpump/translators/io/GearpumpSource.java |  4 +--
 5 files changed, 30 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e63d42d1/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunner.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunner.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunner.java
index 9c44da3..01fdb3b 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunner.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunner.java
@@ -102,8 +102,9 @@ public class GearpumpRunner extends 
PipelineRunner {
 options.getSerializers());
 ClientContext clientContext = getClientContext(options, config);
 options.setClientContext(clientContext);
+UserConfig userConfig = UserConfig.empty();
 JavaStreamApp streamApp = new JavaStreamApp(
-appName, clientContext, UserConfig.empty());
+appName, clientContext, userConfig);
 TranslationContext translationContext = new TranslationContext(streamApp, 
options);
 GearpumpPipelineTranslator translator = new 
GearpumpPipelineTranslator(translationContext);
 translator.translate(pipeline);

http://git-wip-us.apache.org/repos/asf/beam/blob/e63d42d1/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
index 989957f..8e3ffe3 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
@@ -37,7 +37,7 @@ import org.apache.beam.sdk.values.KV;
 
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.gearpump.streaming.dsl.javaapi.JavaStream;
-import org.apache.gearpump.streaming.dsl.window.api.Accumulating$;
+import org.apache.gearpump.streaming.dsl.window.api.Discarding$;
 import org.apache.gearpump.streaming.dsl.window.api.EventTimeTrigger$;
 import org.apache.gearpump.streaming.dsl.window.api.Window;
 import org.apache.gearpump.streaming.dsl.window.api.WindowFn;
@@ -60,7 +60,7 @@ public class GroupByKeyTranslator implements 
TransformTranslator>>> outputStream = inputStream
 .window(Window.apply(new 
GearpumpWindowFn(input.getWindowingStrategy().getWindowFn()),
-EventTimeTrigger$.MODULE$, Accumulating$.MODULE$), "assign_window")
+EventTimeTrigger$.MODULE$, Discarding$.MODULE$), "assign_window")
 .groupBy(new GroupByFn(), parallelism, "group_by_Key_and_Window")
 .map(new ValueToIterable(), "map_value_to_iterable")
 .reduce(new MergeValue(), "merge_value");

http://git-wip-us.apache.org/repos/asf/beam/blob/e63d42d1/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
index 63fb619..b2cff8a 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
@@ -50,7 +50,6 @@ public class TranslationContext {
   public TranslationContext(JavaStreamApp streamApp, GearpumpPipelineOptions 
pipelineOptions) {
 this.streamApp = streamApp;
 this.pipelineOptions = 

[01/12] beam git commit: [BEAM-1180] Implement GearpumpPipelineResult

[kenn]

Repository: beam
Updated Branches:
  refs/heads/gearpump-runner 4c445dd0b -> 1ed16f11a


[BEAM-1180] Implement GearpumpPipelineResult


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/21554764
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/21554764
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/21554764

Branch: refs/heads/gearpump-runner
Commit: 21554764056c45ea18be1e844b4ca1bfb71e544a
Parents: 4c445dd
Author: manuzhang 
Authored: Tue Dec 20 10:39:56 2016 +0800
Committer: manuzhang 
Committed: Wed Jan 4 12:59:08 2017 +0800

--
 runners/gearpump/pom.xml|  1 +
 .../gearpump/GearpumpPipelineResult.java| 59 ++--
 .../beam/runners/gearpump/GearpumpRunner.java   |  4 +-
 .../runners/gearpump/TestGearpumpRunner.java|  4 ++
 .../translators/GroupByKeyTranslator.java   |  1 -
 5 files changed, 62 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/21554764/runners/gearpump/pom.xml
--
diff --git a/runners/gearpump/pom.xml b/runners/gearpump/pom.xml
index bb35ad7..777ad34 100644
--- a/runners/gearpump/pom.xml
+++ b/runners/gearpump/pom.xml
@@ -69,6 +69,7 @@
   
 
org.apache.beam:beam-sdks-java-core
   
+  -noverify
   
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/21554764/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
index ed1201d..9c8f7b3 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
@@ -18,6 +18,7 @@
 package org.apache.beam.runners.gearpump;
 
 import java.io.IOException;
+import java.util.List;
 
 import org.apache.beam.sdk.AggregatorRetrievalException;
 import org.apache.beam.sdk.AggregatorValues;
@@ -26,31 +27,62 @@ import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.metrics.MetricResults;
 import org.apache.beam.sdk.transforms.Aggregator;
 
+import org.apache.gearpump.cluster.MasterToAppMaster;
+import org.apache.gearpump.cluster.MasterToAppMaster.AppMasterData;
+import org.apache.gearpump.cluster.client.ClientContext;
 import org.joda.time.Duration;
 
+import scala.collection.JavaConverters;
+import scala.collection.Seq;
 
 /**
  * Result of executing a {@link Pipeline} with Gearpump.
  */
 public class GearpumpPipelineResult implements PipelineResult {
+
+  private final ClientContext client;
+  private final int appId;
+  private final Duration defaultWaitDuration = Duration.standardSeconds(60);
+  private final Duration defaultWaitInterval = Duration.standardSeconds(10);
+
+  public GearpumpPipelineResult(ClientContext client, int appId) {
+this.client = client;
+this.appId = appId;
+  }
+
   @Override
   public State getState() {
-return null;
+return getGearpumpState();
   }
 
   @Override
   public State cancel() throws IOException {
-return null;
+client.shutdown(appId);
+return State.CANCELLED;
   }
 
   @Override
   public State waitUntilFinish(Duration duration) {
-return null;
+long start = System.currentTimeMillis();
+do {
+  try {
+Thread.sleep(defaultWaitInterval.getMillis());
+  } catch (InterruptedException e) {
+throw new RuntimeException(e);
+  }
+} while (State.RUNNING == getGearpumpState()
+&& (System.currentTimeMillis() - start) < duration.getMillis());
+
+if (State.RUNNING == getGearpumpState()) {
+  return State.DONE;
+} else {
+  return State.FAILED;
+}
   }
 
   @Override
   public State waitUntilFinish() {
-return null;
+return waitUntilFinish(defaultWaitDuration);
   }
 
   @Override
@@ -66,4 +98,23 @@ public class GearpumpPipelineResult implements 
PipelineResult {
 return null;
   }
 
+  private State getGearpumpState() {
+String status = null;
+List apps =
+JavaConverters.seqAsJavaListConverter(
+(Seq) client.listApps().appMasters()).asJava();
+for (AppMasterData app: apps) {
+  if (app.appId() == appId) {
+status = app.status();
+  }
+}
+if (null == status || 
status.equals(MasterToAppMaster.AppMasterNonExist())) {
+  return State.UNKNOWN;
+} else if 

[12/12] beam git commit: This closes #1661: Implement GearpumpPipelineResult

[kenn]

This closes #1661: Implement GearpumpPipelineResult

  note thread is interrupted on InterruptedException
  Remove cache for Gearpump on travis
  reduce timeout to wait for result
  fix ParDo.BoundMulti translation
  return encoded key for GroupByKey translation
  support OutputTimeFn
  update to latest gearpump dsl function interface
  fix group by window
  activate ROS on Gearpump by default
  update ROS configurations
  [BEAM-1180] Implement GearpumpPipelineResult


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1ed16f11
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1ed16f11
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1ed16f11

Branch: refs/heads/gearpump-runner
Commit: 1ed16f11a3fb24c3cc6773235651c4a9255d6fbc
Parents: 4c445dd d814857
Author: Kenneth Knowles 
Authored: Mon Jan 23 19:22:31 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Jan 23 19:22:31 2017 -0800

--
 .travis.yml |  1 +
 runners/gearpump/pom.xml|  9 +-
 .../gearpump/GearpumpPipelineResult.java| 65 -
 .../beam/runners/gearpump/GearpumpRunner.java   |  7 +-
 .../runners/gearpump/TestGearpumpRunner.java|  4 +
 .../translators/GroupByKeyTranslator.java   | 96 
 .../translators/ParDoBoundMultiTranslator.java  | 35 +--
 .../translators/TranslationContext.java |  1 -
 .../translators/WindowBoundTranslator.java  | 49 --
 .../translators/functions/DoFnFunction.java | 21 -
 .../gearpump/translators/io/GearpumpSource.java |  4 +-
 11 files changed, 238 insertions(+), 54 deletions(-)
--

[07/12] beam git commit: return encoded key for GroupByKey translation

[kenn]

return encoded key for GroupByKey translation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/364a3f08
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/364a3f08
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/364a3f08

Branch: refs/heads/gearpump-runner
Commit: 364a3f089747ff4761cb5b54c963c8a8013574a0
Parents: f6aaf0d
Author: manuzhang 
Authored: Mon Jan 16 11:16:05 2017 +0800
Committer: manuzhang 
Committed: Mon Jan 16 11:16:05 2017 +0800

--
 .../translators/GroupByKeyTranslator.java   | 24 
 1 file changed, 20 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/364a3f08/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
index e16a178..ac8e218 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
@@ -22,17 +22,22 @@ import com.google.common.collect.Iterables;
 import com.google.common.collect.Lists;
 
 import java.io.Serializable;
+import java.nio.ByteBuffer;
 import java.time.Instant;
 import java.util.Collection;
 import java.util.LinkedList;
 import java.util.List;
 
 import org.apache.beam.runners.gearpump.translators.utils.TranslatorUtils;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.transforms.windowing.OutputTimeFn;
+import org.apache.beam.sdk.util.CoderUtils;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.KV;
 
@@ -56,6 +61,7 @@ public class GroupByKeyTranslator implements 
TransformTranslator transform, TranslationContext 
context) {
 PCollection> input = context.getInput(transform);
+Coder inputKeyCoder = ((KvCoder) input.getCoder()).getKeyCoder();
 JavaStream>> inputStream =
 context.getInputStream(input);
 int parallelism = context.getPipelineOptions().getParallelism();
@@ -64,7 +70,7 @@ public class GroupByKeyTranslator implements 
TransformTranslator>>> outputStream = inputStream
 .window(Window.apply(new 
GearpumpWindowFn(input.getWindowingStrategy().getWindowFn()),
 EventTimeTrigger$.MODULE$, Discarding$.MODULE$), "assign_window")
-.groupBy(new GroupByFn(), parallelism, "group_by_Key_and_Window")
+.groupBy(new GroupByFn(inputKeyCoder), parallelism, 
"group_by_Key_and_Window")
 .map(new ValueToIterable(), "map_value_to_iterable")
 .map(new KeyedByTimestamp(), "keyed_by_timestamp")
 .reduce(new Merge(outputTimeFn), "merge")
@@ -128,11 +134,21 @@ public class GroupByKeyTranslator implements 
TransformTranslator extends
-  GroupByFunction>, K> {
+  GroupByFunction>, ByteBuffer> {
+
+private final Coder keyCoder;
+
+GroupByFn(Coder keyCoder) {
+  this.keyCoder = keyCoder;
+}
 
 @Override
-public K apply(WindowedValue> wv) {
-  return wv.getValue().getKey();
+public ByteBuffer apply(WindowedValue> wv) {
+  try {
+return ByteBuffer.wrap(CoderUtils.encodeToByteArray(keyCoder, 
wv.getValue().getKey()));
+  } catch (CoderException e) {
+throw new RuntimeException(e);
+  }
 }
   }
 

[05/12] beam git commit: update to latest gearpump dsl function interface

[kenn]

update to latest gearpump dsl function interface


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3bf82638
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3bf82638
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3bf82638

Branch: refs/heads/gearpump-runner
Commit: 3bf82638096ae7aa91c7d3c862c2994772bee51b
Parents: e63d42d
Author: manuzhang 
Authored: Sat Jan 14 13:36:07 2017 +0800
Committer: manuzhang 
Committed: Sat Jan 14 21:40:18 2017 +0800

--
 .../translators/GroupByKeyTranslator.java   | 12 
 .../translators/ParDoBoundMultiTranslator.java  | 29 ++--
 .../translators/WindowBoundTranslator.java  |  4 +--
 .../translators/functions/DoFnFunction.java | 21 +++---
 4 files changed, 46 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3bf82638/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
index 8e3ffe3..4eaf755 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
@@ -36,15 +36,15 @@ import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.KV;
 
 import org.apache.beam.sdk.values.PCollection;
+import org.apache.gearpump.streaming.dsl.api.functions.MapFunction;
+import org.apache.gearpump.streaming.dsl.api.functions.ReduceFunction;
 import org.apache.gearpump.streaming.dsl.javaapi.JavaStream;
+import org.apache.gearpump.streaming.dsl.javaapi.functions.GroupByFunction;
 import org.apache.gearpump.streaming.dsl.window.api.Discarding$;
 import org.apache.gearpump.streaming.dsl.window.api.EventTimeTrigger$;
 import org.apache.gearpump.streaming.dsl.window.api.Window;
 import org.apache.gearpump.streaming.dsl.window.api.WindowFn;
 import org.apache.gearpump.streaming.dsl.window.impl.Bucket;
-import org.apache.gearpump.streaming.javaapi.dsl.functions.GroupByFunction;
-import org.apache.gearpump.streaming.javaapi.dsl.functions.MapFunction;
-import org.apache.gearpump.streaming.javaapi.dsl.functions.ReduceFunction;
 import scala.collection.JavaConversions;
 
 
@@ -122,7 +122,7 @@ public class GroupByKeyTranslator implements 
TransformTranslator implements
+  private static class GroupByFn extends
   GroupByFunction>, K> {
 
 @Override
@@ -132,7 +132,7 @@ public class GroupByKeyTranslator implements 
TransformTranslator
-  implements MapFunction>, WindowedValue>> {
+  extends MapFunction>, WindowedValue>> {
 
 @Override
 public WindowedValue> apply(WindowedValue> wv) 
{
@@ -141,7 +141,7 @@ public class GroupByKeyTranslator implements 
TransformTranslator implements
+  private static class MergeValue extends
   ReduceFunction>> {
 
 @Override

http://git-wip-us.apache.org/repos/asf/beam/blob/3bf82638/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoBoundMultiTranslator.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoBoundMultiTranslator.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoBoundMultiTranslator.java
index 24f9734..0d5b8bc 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoBoundMultiTranslator.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoBoundMultiTranslator.java
@@ -33,6 +33,7 @@ import 
org.apache.beam.runners.gearpump.translators.utils.NoOpSideInputReader;
 import org.apache.beam.runners.gearpump.translators.utils.NoOpStepContext;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.reflect.DoFnInvokers;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;
@@ -41,10 +42,10 @@ import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TupleTag;
 import org.apache.beam.sdk.values.TupleTagList;
 
+import 

[02/12] beam git commit: update ROS configurations

[kenn]

update ROS configurations


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cfdc971f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cfdc971f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cfdc971f

Branch: refs/heads/gearpump-runner
Commit: cfdc971f45ff716b7bd88b3e054ca7077454ab07
Parents: 2155476
Author: manuzhang 
Authored: Thu Jan 5 13:47:42 2017 +0800
Committer: manuzhang 
Committed: Thu Jan 5 13:47:42 2017 +0800

--
 runners/gearpump/pom.xml | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cfdc971f/runners/gearpump/pom.xml
--
diff --git a/runners/gearpump/pom.xml b/runners/gearpump/pom.xml
index 777ad34..4e3722c 100644
--- a/runners/gearpump/pom.xml
+++ b/runners/gearpump/pom.xml
@@ -64,6 +64,12 @@
 
 
   
org.apache.beam.sdk.testing.RunnableOnService
+  
+org.apache.beam.sdk.testing.UsesStatefulParDo,
+org.apache.beam.sdk.testing.UsesTimersInParDo,
+org.apache.beam.sdk.testing.UsesSplittableParDo,
+org.apache.beam.sdk.testing.UsesMetrics
+  
   none
   true
   

Build failed in Jenkins: beam_PostCommit_Python_Verify #1076

[Apache Jenkins Server]

See 

--
[...truncated 8345 lines...]
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Match.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s16"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Match"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2017-01-23_19_18_55-6733368171544728562]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2017-01-23_19_18_55-6733368171544728562
root: INFO: Job 2017-01-23_19_18_55-6733368171544728562 is in state 
JOB_STATE_RUNNING
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bceef: 
2017-01-24T03:18:57.519Z: JOB_MESSAGE_DETAILED: (e11bab53777fc3a3): Checking 
required Cloud APIs are enabled.
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd4b2: 
2017-01-24T03:18:58.994Z: JOB_MESSAGE_DEBUG: (e11bab53777fc1b2): Combiner 
lifting skipped for step assert_that/Group: GroupByKey not followed by a 
combiner.
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd4b4: 
2017-01-24T03:18:58.996Z: JOB_MESSAGE_DEBUG: (e11bab53777fc860): Combiner 
lifting skipped for step assert:even/Group: GroupByKey not followed by a 
combiner.
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd4b6: 
2017-01-24T03:18:58.998Z: JOB_MESSAGE_DEBUG: (e11bab53777fcf0e): Combiner 
lifting skipped for step assert:odd/Group: GroupByKey not followed by a 
combiner.
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd4c0: 
2017-01-24T03:18:59.008Z: JOB_MESSAGE_DETAILED: (e11bab53777fc5bc): Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd4d0: 
2017-01-24T03:18:59.024Z: JOB_MESSAGE_DETAILED: (e11bab53777fcc6a): Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd4eb: 
2017-01-24T03:18:59.051Z: JOB_MESSAGE_DETAILED: (e11bab53777fc722): Annotating 
graph with Autotuner information.
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd51f: 
2017-01-24T03:18:59.103Z: JOB_MESSAGE_DETAILED: (e11bab53777fc7d5): Fusing 
adjacent ParDo, Read, Write, and Flatten operations
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd524: 
2017-01-24T03:18:59.108Z: JOB_MESSAGE_DETAILED: (e11bab53777fc531): Fusing 
consumer assert:odd/ToVoidKey into assert:odd/WindowInto
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd527: 
2017-01-24T03:18:59.111Z: JOB_MESSAGE_DETAILED: (e11bab53777fcbdf): Fusing 
consumer assert:odd/UnKey into assert:odd/Group/GroupByWindow
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd529: 
2017-01-24T03:18:59.113Z: JOB_MESSAGE_DETAILED: (e11bab53777fc28d): Fusing 
consumer assert:even/UnKey into assert:even/Group/GroupByWindow
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd52b: 
2017-01-24T03:18:59.115Z: JOB_MESSAGE_DETAILED: (e11bab53777fc93b): Fusing 
consumer assert:even/Group/GroupByWindow into assert:even/Group/Read
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd52e: 
2017-01-24T03:18:59.118Z: JOB_MESSAGE_DETAILED: (e11bab53777fcfe9): Fusing 
consumer assert_that/Match into assert_that/UnKey
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd530: 
2017-01-24T03:18:59.120Z: JOB_MESSAGE_DETAILED: (e11bab53777fc697): Fusing 
consumer assert_that/UnKey into assert_that/Group/GroupByWindow
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd533: 
2017-01-24T03:18:59.123Z: JOB_MESSAGE_DETAILED: (e11bab53777fcd45): Fusing 
consumer assert_that/Group/GroupByWindow into assert_that/Group/Read
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd535: 
2017-01-24T03:18:59.125Z: JOB_MESSAGE_DETAILED: (e11bab53777fc3f3): Fusing 
consumer assert_that/Group/Write into assert_that/Group/Reify
root: INFO: 2017-01-23_19_18_55-6733368171544728562_0159ce7bd538: 
2017-01-24T03:18:59.128Z: JOB_MESSAGE_DETAILED: (e11bab53777fcaa1): Fusing 
consumer assert_that/Group/Reify into assert_that/ToVoidKey
root: INFO: 

[10/12] beam git commit: Remove cache for Gearpump on travis

[kenn]

Remove cache for Gearpump on travis


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/85dcfbd1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/85dcfbd1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/85dcfbd1

Branch: refs/heads/gearpump-runner
Commit: 85dcfbd153acb4e450a4f0f94fc54b19b76507d3
Parents: 7613ec4
Author: manuzhang 
Authored: Fri Jan 20 08:33:04 2017 +0800
Committer: manuzhang 
Committed: Fri Jan 20 10:52:11 2017 +0800

--
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/85dcfbd1/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index a806477..7dcd5d1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -59,6 +59,7 @@ before_install:
 install:
   # Removing this here protects from inadvertent caching
   - rm -rf "$HOME/.m2/repository/org/apache/beam"
+  - rm -rf "$HOME/.m2/repository/org/apache/gearpump"
 
 script:
   - travis_retry mvn --batch-mode --update-snapshots --no-snapshot-updates 
$MAVEN_OVERRIDE install && travis_retry bash -ex .travis/test_wordcount.sh

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2087

[Apache Jenkins Server]

See 

[GitHub] beam pull request #1826: [BEAM-59] Beam FileSystem.setDefaultConfig: remove ...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/1826

[BEAM-59] Beam FileSystem.setDefaultConfig: remove scheme from the 
signature.


Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam file-system-set-default

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1826.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1826


commit a5193aecd37082171acddcb267d3d920567a38fc
Author: Pei He 
Date:   2017-01-24T02:08:44Z

[BEAM-59] Beam FileSystem.setDefaultConfig: remove scheme from the 
signature.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1825: [BEAM-1258] Improve logging in BigQueryIO.verifyTab...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/1825

[BEAM-1258] Improve logging in BigQueryIO.verifyTableEmpty().

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam bigquery-warning

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1825.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1825


commit 92ae19a54d83f9eab155b4fc22bf717c19cc1acf
Author: Pei He 
Date:   2017-01-23T22:52:30Z

[BEAM-1258] Improve logging in BigQueryIO.verifyTableEmpty().

commit 9c2bf49c66f789c854493dc8fef46879f1c19e11
Author: Pei He 
Date:   2017-01-24T01:39:20Z

[BEAM-1302] BigQueryServicesImpl: skip logging warning if exceptions do not 
need retry.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #2416

[Apache Jenkins Server]

See 

[GitHub] beam pull request #1820: [BEAM-1299] Removes Dataflow native text source and...

[chamikaramj]

GitHub user chamikaramj reopened a pull request:

https://github.com/apache/beam/pull/1820

[BEAM-1299] Removes Dataflow native text source and sink from Beam Python 
SDK.

Users should be using Beam text source and sink available in module 
'textio.py' instead of this.

Also removes Dataflow native file source/sink that is only used by native 
text source/sink.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
remove_native_text_source_sink

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1820.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1820


commit ab6b7026da410a3084da58de873c6b8b809dd1fb
Author: Chamikara Jayalath 
Date:   2017-01-23T21:23:45Z

Removes Dataflow native text source and sink from Beam SDK.

Users should be using Beam text source and sink available in module 
'textio.py' instead of this.

Also removes Dataflow native file source/sink that is only used by native 
text source/sink.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1820: [BEAM-1299] Removes Dataflow native text source and...

[chamikaramj]

Github user chamikaramj closed the pull request at:

https://github.com/apache/beam/pull/1820


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2086

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_RunnableOnService_Flink #1423

[Apache Jenkins Server]

See 

[GitHub] beam pull request #1824: Splittable DoFn: Gives the runner access to Restric...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/1824

Splittable DoFn: Gives the runner access to RestrictionTracker

Changes the SplittableParDo transform so that ProcessFn uses a
runner-supplied hook to run the @ProcessElement method, giving
it, among other things, the RestrictionTracker, so the runner
can initiate checkpointing/splitting with it at will.

Introduces a default implementation of said hook, which limits
the number of outputs and duration of the call. This implementation
is used in tests and in Direct runner. Dataflow Streaming runner
will also use this implementation, while Dataflow Batch runner
will be more sophisticated.

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam runner-hooks

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1824


commit fe8be1a66cef83e92ed0358682eb150ea04455fe
Author: Eugene Kirpichov 
Date:   2016-11-18T19:21:19Z

Gives the runner access to RestrictionTracker

Changes the SplittableParDo transform so that ProcessFn uses a
runner-supplied hook to run the @ProcessElement method, giving
it, among other things, the RestrictionTracker, so the runner
can initiate checkpointing/splitting with it at will.

Introduces a default implementation of said hook, which limits
the number of outputs and duration of the call. This implementation
is used in tests and in Direct runner. Dataflow Streaming runner
will also use this implementation, while Dataflow Batch runner
will be more sophisticated.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1823: [BEAM-1235] BigQueryIO.Write: log failed load/copy ...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/1823

[BEAM-1235] BigQueryIO.Write: log failed load/copy jobs.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam bq-job-error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1823.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1823


commit c186bc4e39e96e8946b7cb70f7d3918ae616bda4
Author: Pei He 
Date:   2017-01-24T00:25:43Z

[BEAM-1235] BigQueryIO.Write: log failed load/copy jobs.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1788: [BEAM-1279] Ensure Composite Nodes produce no outpu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1788


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1788

[tgroh]

This closes #1788


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6ecbfb9e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6ecbfb9e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6ecbfb9e

Branch: refs/heads/master
Commit: 6ecbfb9e155755b5fa39df6a5212cd38a00ea45e
Parents: 338012d 0e1893a
Author: Thomas Groh 
Authored: Mon Jan 23 16:24:24 2017 -0800
Committer: Thomas Groh 
Committed: Mon Jan 23 16:24:24 2017 -0800

--
 .../beam/sdk/runners/TransformHierarchy.java| 33 +++-
 .../sdk/runners/TransformHierarchyTest.java | 30 --
 2 files changed, 46 insertions(+), 17 deletions(-)
--

[1/2] beam git commit: PubSubIO: fix and improve testing for DisplayData

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master f15b52fa3 -> 338012d14


PubSubIO: fix and improve testing for DisplayData

Also adds better type/nullability checking in the code.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4406414a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4406414a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4406414a

Branch: refs/heads/master
Commit: 4406414a52e45213de5521bff4a7599b8aa53a71
Parents: f15b52f
Author: Dan Halperin 
Authored: Mon Jan 23 10:01:22 2017 -0800
Committer: Dan Halperin 
Committed: Mon Jan 23 16:21:23 2017 -0800

--
 .../java/org/apache/beam/sdk/io/PubsubIO.java   | 28 ++
 .../beam/sdk/io/PubsubUnboundedSource.java  | 32 +++-
 .../org/apache/beam/sdk/io/PubsubIOTest.java| 39 
 3 files changed, 75 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4406414a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
index 1471953..806b7da 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
@@ -556,9 +556,10 @@ public class PubsubIO {
 // Validate.
 PubsubSubscription.fromPath(subscription.get());
   }
-  return new Read<>(name,
-  NestedValueProvider.of(subscription, new SubscriptionTranslator()),
-  topic, timestampLabel, coder, idLabel, maxNumRecords, maxReadTime, 
parseFn);
+  return new Read<>(
+  name, NestedValueProvider.of(subscription, new 
SubscriptionTranslator()),
+  null /* reset topic to null */, timestampLabel, coder, idLabel, 
maxNumRecords,
+  maxReadTime, parseFn);
 }
 
 /**
@@ -584,7 +585,7 @@ public class PubsubIO {
 // Validate.
 PubsubTopic.fromPath(topic.get());
   }
-  return new Read<>(name, subscription,
+  return new Read<>(name, null /* reset subscription to null */,
   NestedValueProvider.of(topic, new TopicTranslator()),
   timestampLabel, coder, idLabel, maxNumRecords, maxReadTime, parseFn);
 }
@@ -757,6 +758,7 @@ public class PubsubIO {
 /**
  * Get the topic being read from.
  */
+@Nullable
 public PubsubTopic getTopic() {
   return topic == null ? null : topic.get();
 }
@@ -771,6 +773,7 @@ public class PubsubIO {
 /**
  * Get the subscription being read from.
  */
+@Nullable
 public PubsubSubscription getSubscription() {
   return subscription == null ? null : subscription.get();
 }
@@ -785,6 +788,7 @@ public class PubsubIO {
 /**
  * Get the timestamp label.
  */
+@Nullable
 public String getTimestampLabel() {
   return timestampLabel;
 }
@@ -792,6 +796,7 @@ public class PubsubIO {
 /**
  * Get the id label.
  */
+@Nullable
 public String getIdLabel() {
   return idLabel;
 }
@@ -800,6 +805,7 @@ public class PubsubIO {
 /**
  * Get the {@link Coder} used for the transform's output.
  */
+@Nullable
 public Coder getCoder() {
   return coder;
 }
@@ -814,6 +820,7 @@ public class PubsubIO {
 /**
  * Get the maximum read time.
  */
+@Nullable
 public Duration getMaxReadTime() {
   return maxReadTime;
 }
@@ -821,6 +828,7 @@ public class PubsubIO {
 /**
  * Get the parse function used for PubSub attributes.
  */
+@Nullable
 public SimpleFunction getPubSubMessageParseFn() {
   return parseFn;
 }
@@ -1074,15 +1082,17 @@ public class PubsubIO {
 }
 
 /**
- * Returns the PubSub topic being read from.
+ * Returns the PubSub topic being written to.
  */
+@Nullable
 public PubsubTopic getTopic() {
-  return topic.get();
+  return (topic == null) ? null : topic.get();
 }
 
 /**
- * Returns the {@link ValueProvider} for the topic being read from.
+ * Returns the {@link ValueProvider} for the topic being written to.
  */
+@Nullable
 public ValueProvider getTopicProvider() {
   return topic;
 }
@@ -1090,6 +1100,7 @@ public class PubsubIO {
 /**
  * Returns the timestamp label.
  */
+@Nullable
 public String getTimestampLabel() {
   return timestampLabel;
 }
@@ -1097,6 +1108,7 @@ public class PubsubIO {
 /**
  * Returns the id label.
  */
+@Nullable
 public String getIdLabel() {
   return idLabel;
 }
@@ -1104,6 

[GitHub] beam pull request #1817: [BEAM-1292] PubSubIO: fix and improve testing for D...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1817


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1735: [BEAM-646] Always expand getInputs, getOutputs in A...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1735


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2085

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #1075

[Apache Jenkins Server]

See 

Changes:

[chamikara] Increments major version used by Dataflow runner to 5

[robertwb] Remove dataflow_test.py

[robertwb] Code cleanup now that all runners support windowed side inputs.

--
[...truncated 8425 lines...]
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/UnKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s15"
}, 
"serialized_fn": "", 
"user_name": "assert:even/UnKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s17", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Match.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s16"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Match"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2017-01-23_15_30_54-12011182360156094633]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2017-01-23_15_30_54-12011182360156094633
root: INFO: Job 2017-01-23_15_30_54-12011182360156094633 is in state 
JOB_STATE_RUNNING
root: INFO: 2017-01-23_15_30_54-12011182360156094633_0159cdab092b: 
2017-01-23T23:30:55.403Z: JOB_MESSAGE_DETAILED: (3fadbfc3db312046): Checking 
required Cloud APIs are enabled.
root: INFO: 2017-01-23_15_30_54-12011182360156094633_0159cdab0b0b: 
2017-01-23T23:30:55.883Z: JOB_MESSAGE_DEBUG: (3fadbfc3db312195): Combiner 
lifting skipped for step assert_that/Group: GroupByKey not followed by a 
combiner.
root: INFO: 2017-01-23_15_30_54-12011182360156094633_0159cdab0b0d: 
2017-01-23T23:30:55.885Z: JOB_MESSAGE_DEBUG: (3fadbfc3db312ac3): Combiner 
lifting skipped for step assert:even/Group: GroupByKey not followed by a 
combiner.
root: INFO: 2017-01-23_15_30_54-12011182360156094633_0159cdab0b0f: 
2017-01-23T23:30:55.887Z: JOB_MESSAGE_DEBUG: (3fadbfc3db3123f1): Combiner 
lifting skipped for step assert:odd/Group: GroupByKey not followed by a 
combiner.
root: INFO: 2017-01-23_15_30_54-12011182360156094633_0159cdab0b12: 
2017-01-23T23:30:55.890Z: JOB_MESSAGE_DETAILED: (3fadbfc3db312d1f): Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2017-01-23_15_30_54-12011182360156094633_0159cdab0b14: 
2017-01-23T23:30:55.892Z: JOB_MESSAGE_DETAILED: (3fadbfc3db31264d): Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2017-01-23_15_30_54-12011182360156094633_0159cdab0b1e: 
2017-01-23T23:30:55.902Z: JOB_MESSAGE_DETAILED: (3fadbfc3db312b05): Annotating 
graph with Autotuner information.
root: INFO: 2017-01-23_15_30_54-12011182360156094633_0159cdab0b29: 
2017-01-23T23:30:55.913Z: JOB_MESSAGE_DETAILED: (3fadbfc3db3129f8): Fusing 
adjacent ParDo, 

[GitHub] beam pull request #1821: [BEAM-1301] Support for BigQuery table description

[ravwojdyla]

GitHub user ravwojdyla opened a pull request:

https://github.com/apache/beam/pull/1821

[BEAM-1301] Support for BigQuery table description

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ravwojdyla/incubator-beam tbl_desc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1821.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1821


commit 268cf7a0e10ed2b8bdc758e5dccf0524ec97a1cf
Author: Rafal Wojdyla 
Date:   2017-01-23T23:14:15Z

Fix BEAM-1301. Support table description in BigQuery IO.

commit 6950238a55cb5dab1dd03225b52b239bd147e3ee
Author: Rafal Wojdyla 
Date:   2017-01-23T23:14:47Z

Add tests for BigQueryIO table description.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1812: Code cleanup now that all runners support windowed ...

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/beam/pull/1812


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Code cleanup now that all runners support windowed side inputs.

[robertwb]

Code cleanup now that all runners support windowed side inputs.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6cb2f37e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6cb2f37e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6cb2f37e

Branch: refs/heads/python-sdk
Commit: 6cb2f37efadfb52138b125fcaf51e703c2c5fd5a
Parents: deb2aea
Author: Robert Bradshaw 
Authored: Sat Jan 21 21:13:36 2017 -0800
Committer: Robert Bradshaw 
Committed: Mon Jan 23 14:37:45 2017 -0800

--
 sdks/python/apache_beam/runners/common.py | 34 ++
 1 file changed, 19 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6cb2f37e/sdks/python/apache_beam/runners/common.py
--
diff --git a/sdks/python/apache_beam/runners/common.py 
b/sdks/python/apache_beam/runners/common.py
index 0f63cbc..9c8fdfc 100644
--- a/sdks/python/apache_beam/runners/common.py
+++ b/sdks/python/apache_beam/runners/common.py
@@ -24,7 +24,6 @@ import sys
 from apache_beam.internal import util
 from apache_beam.pvalue import SideOutputValue
 from apache_beam.transforms import core
-from apache_beam.transforms import sideinputs
 from apache_beam.transforms import window
 from apache_beam.transforms.window import TimestampedValue
 from apache_beam.transforms.window import WindowFn
@@ -71,6 +70,21 @@ class DoFnRunner(Receiver):
# Preferred alternative to context
# TODO(robertwb): Remove once all runners are updated.
state=None):
+"""Initializes a DoFnRunner.
+
+Args:
+  fn: user DoFn to invoke
+  args: positional side input arguments (static and placeholder), if any
+  kwargs: keyword side input arguments (static and placeholder), if any
+  side_inputs: list of sideinput.SideInputMaps for deferred side inputs
+  windowing: windowing properties of the output PCollection(s)
+  context: a DoFnContext to use (deprecated)
+  tagged_receivers: a dict of tag name to Receiver objects
+  logger: a logging module (deprecated)
+  step_name: the name of this step
+  logging_context: a LoggingContext object
+  state: handle for accessing DoFn state
+"""
 self.step_name = step_name
 self.window_fn = windowing.windowfn
 self.tagged_receivers = tagged_receivers
@@ -97,14 +111,10 @@ class DoFnRunner(Receiver):
 if isinstance(fn, core.NewDoFn):
   self.is_new_dofn = True
 
-  # SideInputs
-  self.side_inputs = [side_input
-  if isinstance(side_input, sideinputs.SideInputMap)
-  else {global_window: side_input}
-  for side_input in side_inputs]
+  # Stash values for use in new_dofn_process.
+  self.side_inputs = side_inputs
   self.has_windowed_side_inputs = not all(
-  isinstance(si, dict) or si.is_globally_windowed()
-  for si in self.side_inputs)
+  si.is_globally_windowed() for si in self.side_inputs)
 
   self.args = args if args else []
   self.kwargs = kwargs if kwargs else {}
@@ -117,14 +127,8 @@ class DoFnRunner(Receiver):
 self.dofn = fn
 self.dofn_process = fn.process
   else:
-# TODO(robertwb): Remove when all runners pass side input maps.
-side_inputs = [side_input
-   if isinstance(side_input, sideinputs.SideInputMap)
-   else {global_window: side_input}
-   for side_input in side_inputs]
 if side_inputs and all(
-isinstance(side_input, dict) or side_input.is_globally_windowed()
-for side_input in side_inputs):
+side_input.is_globally_windowed() for side_input in side_inputs):
   args, kwargs = util.insert_values_in_args(
   args, kwargs, [side_input[global_window]
  for side_input in side_inputs])

[GitHub] beam pull request #1813: [BEAM-1218] Remove dataflow_test.py

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/beam/pull/1813


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Closes #1813

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/python-sdk 01b362807 -> deb2aea4e


Closes #1813


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/deb2aea4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/deb2aea4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/deb2aea4

Branch: refs/heads/python-sdk
Commit: deb2aea4e96e7e8dc90489f8ad1cf17f1fd1f0ec
Parents: 01b3628 d5b90d8
Author: Robert Bradshaw 
Authored: Mon Jan 23 14:36:55 2017 -0800
Committer: Robert Bradshaw 
Committed: Mon Jan 23 14:36:55 2017 -0800

--
 sdks/python/apache_beam/dataflow_test.py| 418 ---
 .../apache_beam/transforms/ptransform_test.py   |  67 +++
 .../apache_beam/transforms/sideinputs_test.py   | 208 -
 3 files changed, 274 insertions(+), 419 deletions(-)
--

[GitHub] beam pull request #1818: [BEAM-1298] Increments major version used by Datafl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1818


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Increments major version used by Dataflow runner to 5

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/python-sdk 894461e64 -> 01b362807


Increments major version used by Dataflow runner to 5


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1811458b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1811458b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1811458b

Branch: refs/heads/python-sdk
Commit: 1811458b0c33fba0dde909fc655452ad8a37c9f9
Parents: 894461e
Author: Chamikara Jayalath 
Authored: Mon Jan 23 10:25:28 2017 -0800
Committer: Chamikara Jayalath 
Committed: Mon Jan 23 10:25:28 2017 -0800

--
 sdks/python/apache_beam/runners/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1811458b/sdks/python/apache_beam/runners/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow_runner.py
index 31d3386..abcc764 100644
--- a/sdks/python/apache_beam/runners/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow_runner.py
@@ -62,7 +62,7 @@ class DataflowRunner(PipelineRunner):
   # Environment version information. It is passed to the service during a
   # a job submission and is used by the service to establish what features
   # are expected by the workers.
-  BATCH_ENVIRONMENT_MAJOR_VERSION = '4'
+  BATCH_ENVIRONMENT_MAJOR_VERSION = '5'
   STREAMING_ENVIRONMENT_MAJOR_VERSION = '0'
 
   def __init__(self, cache=None, blocking=False):

[GitHub] beam pull request #1780: [BEAM-882,BEAM-883,BEAM-878] Simplified the creatio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1780


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1780: Simplified API surface verifications

[kenn]

This closes #1780: Simplified API surface verifications


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/26a2c47f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/26a2c47f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/26a2c47f

Branch: refs/heads/master
Commit: 26a2c47f48ddc1ea1389023a0fb72e449fb8f592
Parents: a1a022d 29ffaf3
Author: Kenneth Knowles 
Authored: Mon Jan 23 13:56:55 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Jan 23 13:56:55 2017 -0800

--
 .../org/apache/beam/sdk/util/ApiSurface.java| 420 ++-
 .../org/apache/beam/SdkCoreApiSurfaceTest.java  |  61 +++
 .../apache/beam/sdk/util/ApiSurfaceTest.java| 152 ++-
 .../apache/beam/sdk/io/gcp/ApiSurfaceTest.java  | 134 --
 .../beam/sdk/io/gcp/GcpApiSurfaceTest.java  |  76 
 5 files changed, 484 insertions(+), 359 deletions(-)
--

[1/2] beam git commit: [BEAM-882, BEAM-883, BEAM-878] Simplified API surface verifications.

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master a1a022d6b -> 26a2c47f4


[BEAM-882,BEAM-883,BEAM-878] Simplified API surface verifications.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/29ffaf38
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/29ffaf38
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/29ffaf38

Branch: refs/heads/master
Commit: 29ffaf3859ba9b4d8ba8529efc96fd5e105e21a3
Parents: a1a022d
Author: Stas Levin 
Authored: Mon Jan 16 16:20:25 2017 +0200
Committer: Kenneth Knowles 
Committed: Mon Jan 23 13:56:45 2017 -0800

--
 .../org/apache/beam/sdk/util/ApiSurface.java| 420 ++-
 .../org/apache/beam/SdkCoreApiSurfaceTest.java  |  61 +++
 .../apache/beam/sdk/util/ApiSurfaceTest.java| 152 ++-
 .../apache/beam/sdk/io/gcp/ApiSurfaceTest.java  | 134 --
 .../beam/sdk/io/gcp/GcpApiSurfaceTest.java  |  76 
 5 files changed, 484 insertions(+), 359 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/29ffaf38/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
index 2040161..b6b0b32 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
@@ -17,12 +17,21 @@
  */
 package org.apache.beam.sdk.util;
 
+import static org.hamcrest.Matchers.anyOf;
+
+import com.google.common.base.Function;
 import com.google.common.base.Joiner;
+import com.google.common.base.Predicate;
 import com.google.common.base.Supplier;
+import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Lists;
 import com.google.common.collect.Maps;
 import com.google.common.collect.Multimap;
 import com.google.common.collect.Multimaps;
+import com.google.common.collect.Ordering;
 import com.google.common.collect.Sets;
 import com.google.common.reflect.ClassPath;
 import com.google.common.reflect.ClassPath.ClassInfo;
@@ -45,15 +54,20 @@ import java.util.Collections;
 import java.util.List;
 import java.util.Set;
 import java.util.regex.Pattern;
+import javax.annotation.Nonnull;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.hamcrest.StringDescription;
+import org.hamcrest.TypeSafeDiagnosingMatcher;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
- * Represents the API surface of a package prefix. Used for accessing public 
classes,
- * methods, and the types they reference, to control what dependencies are 
re-exported.
+ * Represents the API surface of a package prefix. Used for accessing public 
classes, methods, and
+ * the types they reference, to control what dependencies are re-exported.
  *
- * For the purposes of calculating the public API surface, exposure 
includes any public
- * or protected occurrence of:
+ * For the purposes of calculating the public API surface, exposure 
includes any public or
+ * protected occurrence of:
  *
  * 
  * superclasses
@@ -66,42 +80,272 @@ import org.slf4j.LoggerFactory;
  * wildcard bounds
  * 
  *
- * Exposure is a transitive property. The resulting map excludes primitives
- * and array classes themselves.
+ * Exposure is a transitive property. The resulting map excludes primitives 
and array classes
+ * themselves.
  *
- * It is prudent (though not required) to prune prefixes like "java" via 
the builder
- * method {@link #pruningPrefix} to halt the traversal so it does not 
uselessly catalog references
- * that are not interesting.
+ * It is prudent (though not required) to prune prefixes like "java" via 
the builder method
+ * {@link #pruningPrefix} to halt the traversal so it does not uselessly 
catalog references that are
+ * not interesting.
  */
 @SuppressWarnings("rawtypes")
 public class ApiSurface {
   private static final Logger LOG = LoggerFactory.getLogger(ApiSurface.class);
 
+  /** A factory method to create a {@link Class} matcher for classes residing 
in a given package. */
+  public static Matcher classesInPackage(final String packageName) {
+return new Matchers.ClassInPackage(packageName);
+  }
+
+  /**
+   * A factory method to create an {@link ApiSurface} matcher, producing a 
positive match if the
+   * queried api surface contains ONLY classes described by the provided 
matchers.
+   */
+  public static Matcher containsOnlyClassesMatching(
+  final Set> classMatchers) {
+return 

[GitHub] beam pull request #1820: [BEAM-1299] Removes Dataflow native text source and...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/1820

[BEAM-1299] Removes Dataflow native text source and sink from Beam Python 
SDK.

Users should be using Beam text source and sink available in module 
'textio.py' instead of this.

Also removes Dataflow native file source/sink that is only used by native 
text source/sink.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
remove_native_text_source_sink

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1820.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1820


commit ab6b7026da410a3084da58de873c6b8b809dd1fb
Author: Chamikara Jayalath 
Date:   2017-01-23T21:23:45Z

Removes Dataflow native text source and sink from Beam SDK.

Users should be using Beam text source and sink available in module 
'textio.py' instead of this.

Also removes Dataflow native file source/sink that is only used by native 
text source/sink.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1819

[dhalperi]

This closes #1819


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a1a022d6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a1a022d6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a1a022d6

Branch: refs/heads/master
Commit: a1a022d6bc08f50c1815b1c129858840d0deae5d
Parents: daed01a 0ee34d9
Author: Dan Halperin 
Authored: Mon Jan 23 13:05:03 2017 -0800
Committer: Dan Halperin 
Committed: Mon Jan 23 13:05:03 2017 -0800

--
 .../runners/direct/UnboundedReadEvaluatorFactory.java  |  9 +
 .../direct/UnboundedReadEvaluatorFactoryTest.java  | 13 -
 2 files changed, 17 insertions(+), 5 deletions(-)
--

[GitHub] beam pull request #1819: [BEAM-1300] Finalize Checkpoints before resuming fr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1819


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Finalize Checkpoints before resuming from them

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master daed01a69 -> a1a022d6b


Finalize Checkpoints before resuming from them

This moves checkpoint finalization in the DirectRunner to occur before
the call to createReader instead of between that call and the call to
reader.start().


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0ee34d94
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0ee34d94
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0ee34d94

Branch: refs/heads/master
Commit: 0ee34d9436574adecd276759cdfec62e60ba7b66
Parents: daed01a
Author: Thomas Groh 
Authored: Mon Jan 23 10:25:04 2017 -0800
Committer: Dan Halperin 
Committed: Mon Jan 23 13:05:00 2017 -0800

--
 .../runners/direct/UnboundedReadEvaluatorFactory.java  |  9 +
 .../direct/UnboundedReadEvaluatorFactoryTest.java  | 13 -
 2 files changed, 17 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0ee34d94/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
index a4aebc9..013e929 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
@@ -164,9 +164,13 @@ class UnboundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 throws IOException {
   UnboundedReader existing = shard.getExistingReader();
   if (existing == null) {
+CheckpointMarkT checkpoint = shard.getCheckpoint();
+if (checkpoint != null) {
+  checkpoint.finalizeCheckpoint();
+}
 return shard
 .getSource()
-.createReader(evaluationContext.getPipelineOptions(), 
shard.getCheckpoint());
+.createReader(evaluationContext.getPipelineOptions(), checkpoint);
   } else {
 return existing;
   }
@@ -176,9 +180,6 @@ class UnboundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 UnboundedReader reader, UnboundedSourceShard shard)
 throws IOException {
   if (shard.getExistingReader() == null) {
-if (shard.getCheckpoint() != null) {
-  shard.getCheckpoint().finalizeCheckpoint();
-}
 return reader.start();
   } else {
 return shard.getExistingReader().advance();

http://git-wip-us.apache.org/repos/asf/beam/blob/0ee34d94/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
index 92d668e..987f927 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
@@ -21,6 +21,7 @@ import static 
org.apache.beam.runners.direct.DirectGraphs.getProducer;
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.is;
 import static org.hamcrest.Matchers.not;
 import static org.hamcrest.Matchers.nullValue;
 import static org.junit.Assert.assertThat;
@@ -414,6 +415,9 @@ public class UnboundedReadEvaluatorFactoryTest {
 @Override
 public UnboundedSource.UnboundedReader createReader(
 PipelineOptions options, @Nullable TestCheckpointMark checkpointMark) {
+  if (checkpointMark != null) {
+assertThat(checkpointMark.isFinalized(), is(true));
+  }
   return new TestUnboundedReader(elems, checkpointMark == null ? -1 : 
checkpointMark.index);
 }
 
@@ -505,13 +509,20 @@ public class UnboundedReadEvaluatorFactoryTest {
 
   private static class TestCheckpointMark implements CheckpointMark {
 final int index;
+private boolean finalized = false;
 
 private TestCheckpointMark(int index) {
   this.index = index;
 }
 
 @Override
-public void finalizeCheckpoint() throws IOException {}
+public void finalizeCheckpoint() throws IOException {
+  finalized = true;
+}
+
+boolean isFinalized() 

[GitHub] beam pull request #1795: [BEAM-560] Use ThreadLocal to cache Marshaller/Unma...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1795


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam git commit: Refactor EmptyOnDeserializationThreadLocal to util

[tgroh]

Refactor EmptyOnDeserializationThreadLocal to util

This is a serialization-capable ThreadLocal used in AvroCoder.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e1ee05e8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e1ee05e8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e1ee05e8

Branch: refs/heads/master
Commit: e1ee05e80e54e61b511a71a98a66868b81745533
Parents: 9db5f74
Author: Kai Jiang 
Authored: Mon Jan 23 11:45:36 2017 -0800
Committer: Thomas Groh 
Committed: Mon Jan 23 11:46:48 2017 -0800

--
 .../org/apache/beam/sdk/coders/AvroCoder.java   | 19 +-
 .../util/EmptyOnDeserializationThreadLocal.java | 39 
 2 files changed, 40 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e1ee05e8/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
index 2c88c9a..9316224 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
@@ -24,7 +24,6 @@ import com.fasterxml.jackson.annotation.JsonProperty;
 import com.google.common.base.Supplier;
 import java.io.IOException;
 import java.io.InputStream;
-import java.io.ObjectStreamException;
 import java.io.OutputStream;
 import java.io.Serializable;
 import java.lang.reflect.Field;
@@ -59,6 +58,7 @@ import org.apache.avro.specific.SpecificData;
 import org.apache.avro.util.ClassUtils;
 import org.apache.avro.util.Utf8;
 import org.apache.beam.sdk.util.CloudObject;
+import org.apache.beam.sdk.util.EmptyOnDeserializationThreadLocal;
 import org.apache.beam.sdk.values.TypeDescriptor;
 
 /**
@@ -176,23 +176,6 @@ public class AvroCoder extends StandardCoder {
   private static final DecoderFactory DECODER_FACTORY = DecoderFactory.get();
 
   /**
-   * A {@link Serializable} {@link ThreadLocal} which discards any "stored" 
objects. This allows
-   * for Kryo to serialize an {@link AvroCoder} as a final field.
-   */
-  private static class EmptyOnDeserializationThreadLocal
-  extends ThreadLocal implements Serializable {
-private void writeObject(java.io.ObjectOutputStream out) throws 
IOException {
-}
-
-private void readObject(java.io.ObjectInputStream in)
-throws IOException, ClassNotFoundException {
-}
-
-private void readObjectNoData() throws ObjectStreamException {
-}
-  }
-
-  /**
* A {@link Serializable} object that holds the {@link String} version of a 
{@link Schema}.
* This is paired with the {@link SerializableSchemaSupplier} via {@link 
Serializable}'s usage
* of the {@link #readResolve} method.

http://git-wip-us.apache.org/repos/asf/beam/blob/e1ee05e8/sdks/java/core/src/main/java/org/apache/beam/sdk/util/EmptyOnDeserializationThreadLocal.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/EmptyOnDeserializationThreadLocal.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/EmptyOnDeserializationThreadLocal.java
new file mode 100644
index 000..890728a
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/EmptyOnDeserializationThreadLocal.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import java.io.IOException;
+import java.io.ObjectStreamException;
+import java.io.Serializable;
+import org.apache.beam.sdk.coders.Coder;
+
+/**
+ * A {@link Serializable} {@link ThreadLocal} which discards any "stored" 
objects. This allows
+ * for Kryo to serialize a {@link Coder} as a final field.
+ */
+public class EmptyOnDeserializationThreadLocal extends ThreadLocal 
implements Serializable {
+private void 

[1/3] beam git commit: Use a ThreadLocal to for Marshaller/Unmarshaller in JAXBCoder

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 9db5f746a -> daed01a69


Use a ThreadLocal to for Marshaller/Unmarshaller in JAXBCoder

This allows reuse of thread-unsafe marshallers and unmarshallers while
encoding elements, while the coder remains thread-safe.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cf0b990b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cf0b990b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cf0b990b

Branch: refs/heads/master
Commit: cf0b990b0336f46b4d4775c93e86bd2310d622b5
Parents: e1ee05e
Author: Kai Jiang 
Authored: Thu Jan 19 05:06:13 2017 -0800
Committer: Thomas Groh 
Committed: Mon Jan 23 11:46:48 2017 -0800

--
 .../org/apache/beam/sdk/coders/JAXBCoder.java   | 36 +++-
 1 file changed, 27 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cf0b990b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/JAXBCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/JAXBCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/JAXBCoder.java
index 0a4f9cc..ea636fc 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/JAXBCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/JAXBCoder.java
@@ -30,6 +30,7 @@ import javax.xml.bind.JAXBException;
 import javax.xml.bind.Marshaller;
 import javax.xml.bind.Unmarshaller;
 import org.apache.beam.sdk.util.CloudObject;
+import org.apache.beam.sdk.util.EmptyOnDeserializationThreadLocal;
 import org.apache.beam.sdk.util.Structs;
 import org.apache.beam.sdk.util.VarInt;
 import org.apache.beam.sdk.values.TypeDescriptor;
@@ -45,6 +46,8 @@ public class JAXBCoder extends AtomicCoder {
   private final Class jaxbClass;
   private final TypeDescriptor typeDescriptor;
   private transient volatile JAXBContext jaxbContext;
+  private final EmptyOnDeserializationThreadLocal jaxbMarshaller;
+  private final EmptyOnDeserializationThreadLocal 
jaxbUnmarshaller;
 
   public Class getJAXBClass() {
 return jaxbClass;
@@ -53,6 +56,28 @@ public class JAXBCoder extends AtomicCoder {
   private JAXBCoder(Class jaxbClass) {
 this.jaxbClass = jaxbClass;
 this.typeDescriptor = TypeDescriptor.of(jaxbClass);
+this.jaxbMarshaller = new EmptyOnDeserializationThreadLocal() {
+  @Override
+  protected Marshaller initialValue() {
+try {
+  JAXBContext jaxbContext = getContext();
+  return jaxbContext.createMarshaller();
+} catch (JAXBException e) {
+  throw new RuntimeException("Error when creating marshaller from JAXB 
Context.", e);
+}
+  }
+};
+this.jaxbUnmarshaller = new 
EmptyOnDeserializationThreadLocal() {
+  @Override
+  protected Unmarshaller initialValue() {
+try {
+  JAXBContext jaxbContext = getContext();
+  return jaxbContext.createUnmarshaller();
+} catch (Exception e) {
+  throw new RuntimeException("Error when creating unmarshaller from 
JAXB Context.", e);
+}
+  }
+};
   }
 
   /**
@@ -68,9 +93,6 @@ public class JAXBCoder extends AtomicCoder {
   public void encode(T value, OutputStream outStream, Context context)
   throws CoderException, IOException {
 try {
-  JAXBContext jaxbContext = getContext();
-  // TODO: Consider caching in a ThreadLocal if this impacts performance
-  Marshaller jaxbMarshaller = jaxbContext.createMarshaller();
   if (!context.isWholeStream) {
 try {
   long size = getEncodedElementByteSize(value, Context.OUTER);
@@ -83,7 +105,7 @@ public class JAXBCoder extends AtomicCoder {
 }
   }
 
-  jaxbMarshaller.marshal(value, new CloseIgnoringOutputStream(outStream));
+  jaxbMarshaller.get().marshal(value, new 
CloseIgnoringOutputStream(outStream));
 } catch (JAXBException e) {
   throw new CoderException(e);
 }
@@ -92,17 +114,13 @@ public class JAXBCoder extends AtomicCoder {
   @Override
   public T decode(InputStream inStream, Context context) throws 
CoderException, IOException {
 try {
-  JAXBContext jaxbContext = getContext();
-  // TODO: Consider caching in a ThreadLocal if this impacts performance
-  Unmarshaller jaxbUnmarshaller = jaxbContext.createUnmarshaller();
-
   InputStream stream = inStream;
   if (!context.isWholeStream) {
 long limit = VarInt.decodeLong(inStream);
 stream = ByteStreams.limit(inStream, limit);
   }
   @SuppressWarnings("unchecked")
-  T obj = (T) jaxbUnmarshaller.unmarshal(new 
CloseIgnoringInputStream(stream));
+  T obj = (T) jaxbUnmarshaller.get().unmarshal(new 

[3/3] beam git commit: This closes #1795

[tgroh]

This closes #1795


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/daed01a6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/daed01a6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/daed01a6

Branch: refs/heads/master
Commit: daed01a69af45cee3d3f250e1b60afde82d71e84
Parents: 9db5f74 cf0b990
Author: Thomas Groh 
Authored: Mon Jan 23 13:00:42 2017 -0800
Committer: Thomas Groh 
Committed: Mon Jan 23 13:00:45 2017 -0800

--
 .../org/apache/beam/sdk/coders/AvroCoder.java   | 19 +-
 .../org/apache/beam/sdk/coders/JAXBCoder.java   | 36 +-
 .../util/EmptyOnDeserializationThreadLocal.java | 39 
 3 files changed, 67 insertions(+), 27 deletions(-)
--

[GitHub] beam pull request #1818: [BEAM-1298] Increments major used by Dataflow runne...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/1818

[BEAM-1298] Increments major used by Dataflow runner to 5

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
increment_major_version_5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1818.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1818


commit 1811458b0c33fba0dde909fc655452ad8a37c9f9
Author: Chamikara Jayalath 
Date:   2017-01-23T18:25:28Z

Increments major version used by Dataflow runner to 5




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1817: [BEAM-1292] PubSubIO: fix and improve testing for D...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1817

[BEAM-1292] PubSubIO: fix and improve testing for DisplayData

Also adds better type/nullability checking in the code.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam pubsub-display-data

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1817.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1817


commit da745375a9045a7b2e8294a7d993498f4bcdb265
Author: Dan Halperin 
Date:   2017-01-23T18:01:22Z

PubSubIO: fix and improve testing for DisplayData

Also adds better type/nullability checking in the code.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1810: Add some typing to prevent speed regression for old...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1810


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1816: [BEAM-1297] Update maven plugin versions and fix so...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/1816

[BEAM-1297] Update maven plugin versions and fix some maven related issues

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-1297-update-plugins

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1816.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1816


commit a94f18e04c1d5b0e152166fcd5479e523d9afca1
Author: Ismaël Mejía 
Date:   2017-01-23T14:40:35Z

[BEAM-1297] Move release profile plugins to parent

commit 1844715c8acb518c03ef6b7aeeaad74e41366cbd
Author: Ismaël Mejía 
Date:   2017-01-23T14:59:03Z

[BEAM-1297] Remove explicit plugin versions for spark runner

commit 93bdc6766b0dcc0c9af0d3358c3e3d4f30586831
Author: Ismaël Mejía 
Date:   2017-01-23T16:07:11Z

[BEAM-1297] Fix error message on minimum maven version for some plugins

commit d601a126b10eddb392b4cc4912a84b23b6581026
Author: Ismaël Mejía 
Date:   2017-01-23T17:30:53Z

[BEAM-1297] Extract non-parent plugin execution from pluginManagement

commit 22c3daf3422140889ad158a0c37d8f3131707305
Author: Ismaël Mejía 
Date:   2017-01-23T17:32:09Z

[BEAM-1297] Update all maven plugin versions to the latest release




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Closes #1810

[robertwb]

Closes #1810


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/894461e6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/894461e6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/894461e6

Branch: refs/heads/python-sdk
Commit: 894461e64b09e6d719ba3eef282cd36cea550f7b
Parents: d0474ab f68c9dc
Author: Robert Bradshaw 
Authored: Mon Jan 23 09:48:55 2017 -0800
Committer: Robert Bradshaw 
Committed: Mon Jan 23 09:48:55 2017 -0800

--
 sdks/python/apache_beam/runners/common.pxd | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--

[1/2] beam git commit: Add some typing to prevent speed regression for old_dofn.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/python-sdk d0474ab5b -> 894461e64


Add some typing to prevent speed regression for old_dofn.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f68c9dc8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f68c9dc8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f68c9dc8

Branch: refs/heads/python-sdk
Commit: f68c9dc8d17881c43c31922375fec9593265cc5d
Parents: d0474ab
Author: Robert Bradshaw 
Authored: Sat Jan 21 20:52:02 2017 -0800
Committer: Robert Bradshaw 
Committed: Sat Jan 21 20:52:02 2017 -0800

--
 sdks/python/apache_beam/runners/common.pxd | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f68c9dc8/sdks/python/apache_beam/runners/common.pxd
--
diff --git a/sdks/python/apache_beam/runners/common.pxd 
b/sdks/python/apache_beam/runners/common.pxd
index 06fe434..10d1f96 100644
--- a/sdks/python/apache_beam/runners/common.pxd
+++ b/sdks/python/apache_beam/runners/common.pxd
@@ -36,15 +36,17 @@ cdef class DoFnRunner(Receiver):
   cdef object tagged_receivers
   cdef LoggingContext logging_context
   cdef object step_name
-  cdef object is_new_dofn
+  cdef bint is_new_dofn
   cdef object args
-  cdef object kwargs
+  cdef dict kwargs
   cdef object side_inputs
   cdef bint has_windowed_side_inputs
 
   cdef Receiver main_receivers
 
   cpdef process(self, WindowedValue element)
+  cdef old_dofn_process(self, WindowedValue element)
+  cdef new_dofn_process(self, WindowedValue element)
 
   @cython.locals(windowed_value=WindowedValue)
   cpdef _process_outputs(self, WindowedValue element, results)

[1/2] beam git commit: [BEAM-1273] Error with FlinkPipelineOptions serialization after setStateBackend

[aljoscha]

Repository: beam
Updated Branches:
  refs/heads/master ddde35327 -> 9db5f746a


[BEAM-1273] Error with FlinkPipelineOptions serialization after setStateBackend


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b5bbadf5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b5bbadf5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b5bbadf5

Branch: refs/heads/master
Commit: b5bbadf59625563d1755f66a24d27c96c5fd3492
Parents: ddde353
Author: Alexey Diomin 
Authored: Mon Jan 16 14:46:08 2017 +0400
Committer: Aljoscha Krettek 
Committed: Mon Jan 23 17:22:58 2017 +0100

--
 .../beam/runners/flink/FlinkPipelineOptions.java   |  6 +++---
 .../apache/beam/runners/flink/PipelineOptionsTest.java | 13 +
 2 files changed, 16 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b5bbadf5/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
index 3bb358e..ef9afea 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
@@ -89,13 +89,13 @@ public interface FlinkPipelineOptions
   void setObjectReuse(Boolean reuse);
 
   /**
-   * Sets a state backend to store Beam's state during computation.
+   * State backend to store Beam's state during computation.
* Note: Only applicable when executing in streaming mode.
-   * @param stateBackend The state backend to use
*/
   @Description("Sets the state backend to use in streaming mode. "
   + "Otherwise the default is read from the Flink config.")
-  void setStateBackend(AbstractStateBackend stateBackend);
+  @JsonIgnore
   AbstractStateBackend getStateBackend();
+  void setStateBackend(AbstractStateBackend stateBackend);
 
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/b5bbadf5/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/PipelineOptionsTest.java
--
diff --git 
a/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/PipelineOptionsTest.java
 
b/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/PipelineOptionsTest.java
index 4c97cc7..23bc6a2 100644
--- 
a/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/PipelineOptionsTest.java
+++ 
b/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/PipelineOptionsTest.java
@@ -19,6 +19,7 @@ package org.apache.beam.runners.flink;
 
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
 import static org.junit.Assert.assertTrue;
 
 import java.util.Collections;
@@ -40,6 +41,7 @@ import org.apache.commons.lang.SerializationUtils;
 import org.apache.flink.api.common.ExecutionConfig;
 import org.apache.flink.api.common.typeinfo.TypeHint;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.runtime.state.memory.MemoryStateBackend;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
 import org.joda.time.Instant;
@@ -80,6 +82,17 @@ public class PipelineOptionsTest {
   }
 
   @Test
+  public void testIgnoredFieldSerialization() {
+FlinkPipelineOptions options = 
PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+options.setStateBackend(new MemoryStateBackend());
+
+FlinkPipelineOptions deserialized =
+new 
SerializedPipelineOptions(options).getPipelineOptions().as(FlinkPipelineOptions.class);
+
+assertNull(deserialized.getStateBackend());
+  }
+
+  @Test
   public void testCaching() {
 PipelineOptions deserializedOptions =
 serializedOptions.getPipelineOptions().as(PipelineOptions.class);

[GitHub] beam pull request #1779: [BEAM-1273] Error with FlinkPipelineOptions seriali...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1779


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1779

[aljoscha]

This closes #1779


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9db5f746
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9db5f746
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9db5f746

Branch: refs/heads/master
Commit: 9db5f746ae81c4fa755a9a8f2c4888759c8e7042
Parents: ddde353 b5bbadf
Author: Aljoscha Krettek 
Authored: Mon Jan 23 17:38:37 2017 +0100
Committer: Aljoscha Krettek 
Committed: Mon Jan 23 17:38:37 2017 +0100

--
 .../beam/runners/flink/FlinkPipelineOptions.java   |  6 +++---
 .../apache/beam/runners/flink/PipelineOptionsTest.java | 13 +
 2 files changed, 16 insertions(+), 3 deletions(-)
--

[jira] [Closed] (BEAM-1281) GlobalWindow needs non-empty encoding in StateNamespaces

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-1281.
-
   Resolution: Won't Fix
Fix Version/s: Not applicable

> GlobalWindow needs non-empty encoding in StateNamespaces
> 
>
> Key: BEAM-1281
> URL: https://issues.apache.org/jira/browse/BEAM-1281
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: Not applicable
>
>
> Because the GlobalWindow is encoded to zero bytes, a StateNamespace built 
> from the window has a stringKey "//" while the global namespace's stringKey 
> is "/". As paths, these are identical, though we don't currently treat them 
> as paths, quite. It isn't clear whether this is desirable. Maybe it is 
> harmless but it complicates parsing and interpretation.
> For a system that actually builds hierarchical paths out of, say, some 
> prefix, the StateNamespace, and a subsequent ID, the canonicalized path is 
> the same so it is not possible to deserialize to the original.
> There are other gotchas associated with zero-length encodings, such as APIs 
> that return zero bytes when there is no more data ready, versus returning 
> zero bytes because the data is representable in zero bytes, given the context 
> of knowing what type of data is expected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)