Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1186

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-668) Reinstate runner direct translation for TextIO and AvroIO once Beam SDK supports hdfs.

[Amit Sela (JIRA)]

Amit Sela created BEAM-668:
--

 Summary: Reinstate runner direct translation for TextIO and AvroIO 
once Beam SDK supports hdfs.
 Key: BEAM-668
 URL: https://issues.apache.org/jira/browse/BEAM-668
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Amit Sela
Assignee: Amit Sela


The runner has an implementation of a direct translation (not with 
Read.Bounded) for TextIO and AvroIO.
Those cannot be used properly until BEAM-59 is resolved and hdfs is properly 
supported.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-644) Primitive to shift the watermark while assigning timestamps

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512585#comment-15512585
 ] 

Aljoscha Krettek commented on BEAM-644:
---

Doesn't a shift by {{D}} just shift (hehe, yes) the problem by a bit but in the 
end you still don't know what timestamps future data might have?

> Primitive to shift the watermark while assigning timestamps
> ---
>
> Key: BEAM-644
> URL: https://issues.apache.org/jira/browse/BEAM-644
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> There is a general need, especially important in the presence of 
> SplittableDoFn, to be able to assign new timestamps to elements without 
> making them late or droppable.
>  - DoFn.withAllowedTimestampSkew is inadequate, because it simply allows one 
> to produce late data, but does not allow one to shift the watermark so the 
> new data is on-time.
>  - For a SplittableDoFn, one may receive an element such as the name of a log 
> file that contains elements for the day preceding the log file. The timestamp 
> on the filename must currently be the beginning of the log. If such elements 
> are constantly flowing, it may be OK, but since we don't know that element is 
> coming, in that absence of data, the watermark may advance. We need a way to 
> keep it far enough back even in the absence of data holding it back.
> One idea is a new primitive ShiftWatermark / AdjustTimestamps with the 
> following pieces:
>  - A constant duration (positive or negative) D by which to shift the 
> watermark.
>  - A function from TimestampedElement to new timestamp that always falls 
> within D of the original timestamp.
> With this primitive added, outputWithTimestamp and withAllowedTimestampSkew 
> could be removed, simplifying DoFn.
> Alternatively, all of this functionality could be bolted on to DoFn.
> This ticket is not a proposal, but a record of the issue and ideas that were 
> mentioned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-642] Support Flink Detached Mode for JOB execution

[mxm]

Repository: incubator-beam
Updated Branches:
  refs/heads/master f62d04e22 -> 843275210


[BEAM-642] Support Flink Detached Mode for JOB execution


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/dc69bc48
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/dc69bc48
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/dc69bc48

Branch: refs/heads/master
Commit: dc69bc48b0057f45d849d3cfec848fa066ee0854
Parents: f62d04e
Author: Sumit Chawla 
Authored: Mon Sep 19 15:10:53 2016 -0700
Committer: Maximilian Michels 
Committed: Thu Sep 22 11:30:09 2016 +0200

--
 .../apache/beam/runners/flink/FlinkRunner.java  | 25 +---
 1 file changed, 16 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/dc69bc48/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
index d3c65c0..137fdeb 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
@@ -25,6 +25,7 @@ import java.net.URL;
 import java.net.URLClassLoader;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
@@ -56,6 +57,7 @@ import org.apache.beam.sdk.values.POutput;
 import org.apache.beam.sdk.values.PValue;
 
 import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.client.program.DetachedEnvironment;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -151,18 +153,23 @@ public class FlinkRunner extends 
PipelineRunner {
   throw new RuntimeException("Pipeline execution failed", e);
 }
 
-LOG.info("Execution finished in {} msecs", result.getNetRuntime());
-
-Map accumulators = result.getAllAccumulatorResults();
-if (accumulators != null && !accumulators.isEmpty()) {
-  LOG.info("Final aggregator values:");
+if (result instanceof DetachedEnvironment.DetachedJobExecutionResult) {
+  LOG.info("Pipeline submitted in Detached mode");
+  Map accumulators = Collections.emptyMap();
+  return new FlinkRunnerResult(accumulators, -1L);
+} else {
+  LOG.info("Execution finished in {} msecs", result.getNetRuntime());
+  Map accumulators = result.getAllAccumulatorResults();
+  if (accumulators != null && !accumulators.isEmpty()) {
+LOG.info("Final aggregator values:");
 
-  for (Map.Entry entry : 
result.getAllAccumulatorResults().entrySet()) {
-LOG.info("{} : {}", entry.getKey(), entry.getValue());
+for (Map.Entry entry : 
result.getAllAccumulatorResults().entrySet()) {
+  LOG.info("{} : {}", entry.getKey(), entry.getValue());
+}
   }
-}
 
-return new FlinkRunnerResult(accumulators, result.getNetRuntime());
+  return new FlinkRunnerResult(accumulators, result.getNetRuntime());
+}
   }
 
   /**

[2/2] incubator-beam git commit: This closes #974

[mxm]

This closes #974


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/84327521
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/84327521
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/84327521

Branch: refs/heads/master
Commit: 843275210b12ebf35b45d5f4d98c689dfb0820b1
Parents: f62d04e dc69bc4
Author: Maximilian Michels 
Authored: Thu Sep 22 11:30:16 2016 +0200
Committer: Maximilian Michels 
Committed: Thu Sep 22 11:30:16 2016 +0200

--
 .../apache/beam/runners/flink/FlinkRunner.java  | 25 +---
 1 file changed, 16 insertions(+), 9 deletions(-)
--

[GitHub] incubator-beam pull request #974: BEAM-642 - Support Flink Detached Mode for...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/974


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-642) FlinkRunner does not support Detached Mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512753#comment-15512753
 ] 

ASF GitHub Bot commented on BEAM-642:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/974


> FlinkRunner does not support Detached Mode
> --
>
> Key: BEAM-642
> URL: https://issues.apache.org/jira/browse/BEAM-642
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Sumit Chawla
>Assignee: Amit Sela
>
> FlinkRunner gives exception when job is submitted in detached mode.
> Following code will throw exception: 
> {code}
>  LOG.info("Execution finished in {} msecs", result.getNetRuntime()); 
> {code}
> {code}
> org.apache.flink.api.common.InvalidProgramException: Job was submitted in 
> detached mode. Results of job execution, such as accumulators, runtime, job 
> id etc. are not available. 
>   at 
> org.apache.flink.client.program.DetachedEnvironment$DetachedJobExecutionResult.getNetRuntime(DetachedEnvironment.java:98)
>  ~[flink-clients_2.10-1.1.2.jar:1.1.2]
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:116) 
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:48) 
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:183) 
> ~[beam-sdks-java-core-0.2.0-incubating.jar:0.2.0-incubating]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_92]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_92]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_92]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
>  [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>  [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:320) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-642) FlinkRunner does not support Detached Mode

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels resolved BEAM-642.
-
   Resolution: Fixed
 Assignee: Sumit Chawla  (was: Amit Sela)
Fix Version/s: 0.3.0-incubating

Resolved with 843275210b12ebf35b45d5f4d98c689dfb0820b1

[~sumitkchawla] was so kind to fix this. I hope you don't mind assigning him, 
[~amitsela].



> FlinkRunner does not support Detached Mode
> --
>
> Key: BEAM-642
> URL: https://issues.apache.org/jira/browse/BEAM-642
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Sumit Chawla
>Assignee: Sumit Chawla
> Fix For: 0.3.0-incubating
>
>
> FlinkRunner gives exception when job is submitted in detached mode.
> Following code will throw exception: 
> {code}
>  LOG.info("Execution finished in {} msecs", result.getNetRuntime()); 
> {code}
> {code}
> org.apache.flink.api.common.InvalidProgramException: Job was submitted in 
> detached mode. Results of job execution, such as accumulators, runtime, job 
> id etc. are not available. 
>   at 
> org.apache.flink.client.program.DetachedEnvironment$DetachedJobExecutionResult.getNetRuntime(DetachedEnvironment.java:98)
>  ~[flink-clients_2.10-1.1.2.jar:1.1.2]
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:116) 
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:48) 
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:183) 
> ~[beam-sdks-java-core-0.2.0-incubating.jar:0.2.0-incubating]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_92]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_92]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_92]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
>  [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>  [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:320) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-642) FlinkRunner does not support Detached Mode

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512839#comment-15512839
 ] 

Amit Sela commented on BEAM-642:


[~mxm] I really have no idea what happened here - my guess is this ticket was 
built on top of a clone of BEAM-637. I definitely wasn't assigned to any 
FlinkRunner tickets AFAIK.

> FlinkRunner does not support Detached Mode
> --
>
> Key: BEAM-642
> URL: https://issues.apache.org/jira/browse/BEAM-642
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Sumit Chawla
>Assignee: Sumit Chawla
> Fix For: 0.3.0-incubating
>
>
> FlinkRunner gives exception when job is submitted in detached mode.
> Following code will throw exception: 
> {code}
>  LOG.info("Execution finished in {} msecs", result.getNetRuntime()); 
> {code}
> {code}
> org.apache.flink.api.common.InvalidProgramException: Job was submitted in 
> detached mode. Results of job execution, such as accumulators, runtime, job 
> id etc. are not available. 
>   at 
> org.apache.flink.client.program.DetachedEnvironment$DetachedJobExecutionResult.getNetRuntime(DetachedEnvironment.java:98)
>  ~[flink-clients_2.10-1.1.2.jar:1.1.2]
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:116) 
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:48) 
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:183) 
> ~[beam-sdks-java-core-0.2.0-incubating.jar:0.2.0-incubating]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_92]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_92]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_92]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
>  [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>  [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:320) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1187

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-613) Add windwoding use cases to SimpleStreamingWordCountTest

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin updated BEAM-613:

Summary: Add windwoding use cases to SimpleStreamingWordCountTest  (was: 
SimpleStreamingWordCountTest tests only a single batch)

> Add windwoding use cases to SimpleStreamingWordCountTest
> 
>
> Key: BEAM-613
> URL: https://issues.apache.org/jira/browse/BEAM-613
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
>
> {{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
>  aims to test a simple Spark streaming job, but only tests a single batch, 
> which is uncharacteristic of an actual (even simple) streaming job, usually 
> consisting of multiple batches.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-613) Add windwoding use cases to SimpleStreamingWordCountTest

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin updated BEAM-613:

Description: 
{{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
 should have tests for both {{FixedWindows}} and {{SlidingWindows}}.  (was: 
{{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
 aims to test a simple Spark streaming job, but only tests a single batch, 
which is uncharacteristic of an actual (even simple) streaming job, usually 
consisting of multiple batches.)

> Add windwoding use cases to SimpleStreamingWordCountTest
> 
>
> Key: BEAM-613
> URL: https://issues.apache.org/jira/browse/BEAM-613
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
>
> {{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
>  should have tests for both {{FixedWindows}} and {{SlidingWindows}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-645) Running Wordcount in Spark Checks Locally and Outputs in HDFS

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513074#comment-15513074
 ] 

Amit Sela commented on BEAM-645:


This https://github.com/apache/incubator-beam/pull/983 should fix Read.Bounded 
support for the SparkRunner.

> Running Wordcount in Spark Checks Locally and Outputs in HDFS
> -
>
> Key: BEAM-645
> URL: https://issues.apache.org/jira/browse/BEAM-645
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Jesse Anderson
>Assignee: Amit Sela
>
> When running the Wordcount example with the Spark runner, the Spark runner 
> uses the input file in HDFS. When the program performs its startup checks, it 
> looks for the file in the local filesystem.
> To workaround this issue, you have to create a file in the local filesystem 
> and put the actual file in HDFS.
> Here is the stack trace when the file doesn't exist in the local filesystem:
> {quote}Exception in thread "main" java.lang.IllegalStateException: Unable to 
> find any files matching Macbeth.txt
>   at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:199)
>   at org.apache.beam.sdk.io.TextIO$Read$Bound.apply(TextIO.java:279)
>   at org.apache.beam.sdk.io.TextIO$Read$Bound.apply(TextIO.java:192)
>   at 
> org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:76)
>   at org.apache.beam.runners.spark.SparkRunner.apply(SparkRunner.java:128)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:400)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:323)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:58)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:173)
>   at org.apache.beam.examples.WordCount.main(WordCount.java:195)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>   at 
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>   at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-642) FlinkRunner does not support Detached Mode

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513164#comment-15513164
 ] 

Maximilian Michels commented on BEAM-642:
-

I was also wondering why you were assigned. Not that I mind :)

Actually, you got assigned because the issue was originally falsely tagged with 
the "runner-spark" component. This component has you as the default assignee.

> FlinkRunner does not support Detached Mode
> --
>
> Key: BEAM-642
> URL: https://issues.apache.org/jira/browse/BEAM-642
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Sumit Chawla
>Assignee: Sumit Chawla
> Fix For: 0.3.0-incubating
>
>
> FlinkRunner gives exception when job is submitted in detached mode.
> Following code will throw exception: 
> {code}
>  LOG.info("Execution finished in {} msecs", result.getNetRuntime()); 
> {code}
> {code}
> org.apache.flink.api.common.InvalidProgramException: Job was submitted in 
> detached mode. Results of job execution, such as accumulators, runtime, job 
> id etc. are not available. 
>   at 
> org.apache.flink.client.program.DetachedEnvironment$DetachedJobExecutionResult.getNetRuntime(DetachedEnvironment.java:98)
>  ~[flink-clients_2.10-1.1.2.jar:1.1.2]
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:116) 
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:48) 
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:183) 
> ~[beam-sdks-java-core-0.2.0-incubating.jar:0.2.0-incubating]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_92]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_92]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_92]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
>  [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>  [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:320) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1188

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-613) SimpleStreamingWordCountTest does not properly test fixed windows

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin updated BEAM-613:

Summary: SimpleStreamingWordCountTest does not properly test fixed windows  
(was: Add windwoding use cases to SimpleStreamingWordCountTest)

> SimpleStreamingWordCountTest does not properly test fixed windows
> -
>
> Key: BEAM-613
> URL: https://issues.apache.org/jira/browse/BEAM-613
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
>
> {{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
>  should have tests for both {{FixedWindows}} and {{SlidingWindows}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-613) SimpleStreamingWordCountTest does not properly test fixed windows

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin updated BEAM-613:

Description: 
{{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
 does not properly test {{FixedWindows}}.  (was: 
{{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
 should have tests for both {{FixedWindows}} and {{SlidingWindows}}.)

> SimpleStreamingWordCountTest does not properly test fixed windows
> -
>
> Key: BEAM-613
> URL: https://issues.apache.org/jira/browse/BEAM-613
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
>
> {{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
>  does not properly test {{FixedWindows}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-669) SimpleStreamingWordCountTest does not have a test for sliding windows

[Stas Levin (JIRA)]

Stas Levin created BEAM-669:
---

 Summary: SimpleStreamingWordCountTest does not have a test for 
sliding windows
 Key: BEAM-669
 URL: https://issues.apache.org/jira/browse/BEAM-669
 Project: Beam
  Issue Type: Improvement
  Components: runner-spark
Reporter: Stas Levin
Assignee: Amit Sela


{{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
 does not have a test for {{SlidingWindows}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-613) SimpleStreamingWordCountTest does not properly test fixed windows

[Stas Levin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513318#comment-15513318
 ] 

Stas Levin commented on BEAM-613:
-

I'll open a separate issue for adding a test for sliding windows.

> SimpleStreamingWordCountTest does not properly test fixed windows
> -
>
> Key: BEAM-613
> URL: https://issues.apache.org/jira/browse/BEAM-613
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
>
> {{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
>  does not properly test {{FixedWindows}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #982: [BEAM-613] Revised SimpleStreamingWordCoun...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/982


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-613] Revised SimpleStreamingWordCountTest to better test fixed windows.

[amitsela]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 843275210 -> 6082ebcce


[BEAM-613] Revised SimpleStreamingWordCountTest to better test fixed windows.

Revised the test to test multiple batches

Set the timeout to 1 ms since it essentially plays no role here.
Removed blank lines between imports.

Refactored the timeout related stuff to make it more natural from Beam model's 
perspective.

Fix windowing bug.

Expected result if for the entire window.

Renamed the test's name to better reflect the use case it's testing.

Fixed a typo.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b1474a18
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b1474a18
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b1474a18

Branch: refs/heads/master
Commit: b1474a18c4fe3b3aefdb6cd364fce9dfc227b6df
Parents: 8432752
Author: Stas Levin 
Authored: Mon Sep 5 18:22:59 2016 +0300
Committer: Sela 
Committed: Thu Sep 22 18:18:19 2016 +0300

--
 .../streaming/StreamingTransformTranslator.java | 28 +--
 .../streaming/SimpleStreamingWordCountTest.java | 49 +---
 2 files changed, 46 insertions(+), 31 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b1474a18/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
index 64ddc57..9cb377d 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
@@ -191,27 +191,29 @@ public final class StreamingTransformTranslator {
 @SuppressWarnings("unchecked")
 JavaDStream> dStream =
 (JavaDStream>) sec.getStream(transform);
+// get the right window durations.
+Duration windowDuration;
+Duration slideDuration;
 if (windowFn instanceof FixedWindows) {
-  Duration windowDuration = Durations.milliseconds(((FixedWindows) 
windowFn).getSize()
-  .getMillis());
-  sec.setStream(transform, dStream.window(windowDuration));
+  windowDuration = Durations.milliseconds(((FixedWindows) 
windowFn).getSize().getMillis());
+  slideDuration = windowDuration;
 } else if (windowFn instanceof SlidingWindows) {
-  Duration windowDuration = Durations.milliseconds(((SlidingWindows) 
windowFn).getSize()
-  .getMillis());
-  Duration slideDuration = Durations.milliseconds(((SlidingWindows) 
windowFn).getPeriod()
-  .getMillis());
-  sec.setStream(transform, dStream.window(windowDuration, 
slideDuration));
+  SlidingWindows slidingWindows = (SlidingWindows) windowFn;
+  windowDuration = 
Durations.milliseconds(slidingWindows.getSize().getMillis());
+  slideDuration = 
Durations.milliseconds(slidingWindows.getPeriod().getMillis());
+} else {
+  throw new UnsupportedOperationException(String.format("WindowFn %s 
is not supported.",
+  windowFn.getClass().getCanonicalName()));
 }
+JavaDStream> windowedDStream =
+dStream.window(windowDuration, slideDuration);
 //--- then we apply windowing to the elements
-@SuppressWarnings("unchecked")
-JavaDStream> dStream2 =
-(JavaDStream>) sec.getStream(transform);
 if (TranslationUtils.skipAssignWindows(transform, context)) {
-  sec.setStream(transform, dStream2);
+  sec.setStream(transform, windowedDStream);
 } else {
   final OldDoFn addWindowsDoFn = new 
AssignWindowsDoFn<>(windowFn);
   final SparkRuntimeContext runtimeContext = sec.getRuntimeContext();
-  JavaDStream> outStream = dStream2.transform(
+  JavaDStream> outStream = windowedDStream.transform(
   new Function>, 
JavaRDD>>() {
 @Override
 public JavaRDD> call(JavaRDD> 
rdd) throws Exception {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b1474a18/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SimpleStreamingWordCountTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SimpleStreamingWordCountTest.java
 
b/runners/spark/src

[GitHub] incubator-beam pull request #986: Update grpc to 1.0.1 and protobuf to 3.0.0

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/986

Update grpc to 1.0.1 and protobuf to 3.0.0

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam update_grpc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/986.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #986






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #982

[amitsela]

This closes #982


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6082ebcc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6082ebcc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6082ebcc

Branch: refs/heads/master
Commit: 6082ebccedec076140720aefdb8f35e263847082
Parents: 8432752 b1474a1
Author: Sela 
Authored: Thu Sep 22 18:19:43 2016 +0300
Committer: Sela 
Committed: Thu Sep 22 18:19:43 2016 +0300

--
 .../streaming/StreamingTransformTranslator.java | 28 +--
 .../streaming/SimpleStreamingWordCountTest.java | 49 +---
 2 files changed, 46 insertions(+), 31 deletions(-)
--

[jira] [Commented] (BEAM-613) SimpleStreamingWordCountTest does not properly test fixed windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513611#comment-15513611
 ] 

ASF GitHub Bot commented on BEAM-613:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/982


> SimpleStreamingWordCountTest does not properly test fixed windows
> -
>
> Key: BEAM-613
> URL: https://issues.apache.org/jira/browse/BEAM-613
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
>
> {{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
>  does not properly test {{FixedWindows}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-669) SimpleStreamingWordCountTest does not have a test for sliding windows

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-669:
---
Assignee: Stas Levin  (was: Amit Sela)

> SimpleStreamingWordCountTest does not have a test for sliding windows
> -
>
> Key: BEAM-669
> URL: https://issues.apache.org/jira/browse/BEAM-669
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
>
> {{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
>  does not have a test for {{SlidingWindows}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-613) SimpleStreamingWordCountTest does not properly test fixed windows

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-613.

   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> SimpleStreamingWordCountTest does not properly test fixed windows
> -
>
> Key: BEAM-613
> URL: https://issues.apache.org/jira/browse/BEAM-613
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
> Fix For: 0.3.0-incubating
>
>
> {{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
>  does not properly test {{FixedWindows}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-15) Applying windowing to cached RDDs fails

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-15?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-15.
---
   Resolution: Duplicate
Fix Version/s: 0.3.0-incubating

Resolved by BEAM-610

> Applying windowing to cached RDDs fails
> ---
>
> Key: BEAM-15
> URL: https://issues.apache.org/jira/browse/BEAM-15
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.3.0-incubating
>
>
> The Spark runner caches RDDs that are accessed more than once. If applying 
> window operations to a cached RDD, it will fail because windowed RDDs will 
> try to cache with a different cache level - windowing cache level is 
> StorageLevel.MEMORY_ONLY_SER and RDD cache level is StorageLevel.MEMORY_ONLY.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-549) SparkRunner should support Beam's KafkaIO instead of providing it's own.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513629#comment-15513629
 ] 

ASF GitHub Bot commented on BEAM-549:
-

Github user amitsela closed the pull request at:

https://github.com/apache/incubator-beam/pull/822


> SparkRunner should support Beam's KafkaIO instead of providing it's own.
> 
>
> Key: BEAM-549
> URL: https://issues.apache.org/jira/browse/BEAM-549
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> For portability, and in the spirit of Apache Beam, the SparkRunner should use 
> the Beam implementation of KafkaIO instead of it's own.
> Having said that, the runner will translate the KafkaIO as defined in the 
> pipeline into it's own internal implementation, but should still map the 
> properties the user defined in the pipeline in a way that the IO behaves the 
> same - i.e., brokers, topic, etc.
> Eventually, the SparkRunner will implement reading from Kafka using Spark's 
> KafakUtils.createDirectStream() as described here: 
> http://spark.apache.org/docs/1.6.2/streaming-kafka-integration.html#approach-2-direct-approach-no-receivers
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-549) SparkRunner should support Beam's KafkaIO instead of providing it's own.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela closed BEAM-549.
--
   Resolution: Duplicate
Fix Version/s: 0.3.0-incubating

> SparkRunner should support Beam's KafkaIO instead of providing it's own.
> 
>
> Key: BEAM-549
> URL: https://issues.apache.org/jira/browse/BEAM-549
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.3.0-incubating
>
>
> For portability, and in the spirit of Apache Beam, the SparkRunner should use 
> the Beam implementation of KafkaIO instead of it's own.
> Having said that, the runner will translate the KafkaIO as defined in the 
> pipeline into it's own internal implementation, but should still map the 
> properties the user defined in the pipeline in a way that the IO behaves the 
> same - i.e., brokers, topic, etc.
> Eventually, the SparkRunner will implement reading from Kafka using Spark's 
> KafakUtils.createDirectStream() as described here: 
> http://spark.apache.org/docs/1.6.2/streaming-kafka-integration.html#approach-2-direct-approach-no-receivers
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-669) SimpleStreamingWordCountTest does not have a test for sliding windows

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-669:
---
Assignee: (was: Stas Levin)

> SimpleStreamingWordCountTest does not have a test for sliding windows
> -
>
> Key: BEAM-669
> URL: https://issues.apache.org/jira/browse/BEAM-669
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Stas Levin
>
> {{org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest}}
>  does not have a test for {{SlidingWindows}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-645) Running Wordcount in Spark Checks Locally and Outputs in HDFS

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela closed BEAM-645.
--
   Resolution: Duplicate
Fix Version/s: 0.3.0-incubating

> Running Wordcount in Spark Checks Locally and Outputs in HDFS
> -
>
> Key: BEAM-645
> URL: https://issues.apache.org/jira/browse/BEAM-645
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Jesse Anderson
>Assignee: Amit Sela
> Fix For: 0.3.0-incubating
>
>
> When running the Wordcount example with the Spark runner, the Spark runner 
> uses the input file in HDFS. When the program performs its startup checks, it 
> looks for the file in the local filesystem.
> To workaround this issue, you have to create a file in the local filesystem 
> and put the actual file in HDFS.
> Here is the stack trace when the file doesn't exist in the local filesystem:
> {quote}Exception in thread "main" java.lang.IllegalStateException: Unable to 
> find any files matching Macbeth.txt
>   at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:199)
>   at org.apache.beam.sdk.io.TextIO$Read$Bound.apply(TextIO.java:279)
>   at org.apache.beam.sdk.io.TextIO$Read$Bound.apply(TextIO.java:192)
>   at 
> org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:76)
>   at org.apache.beam.runners.spark.SparkRunner.apply(SparkRunner.java:128)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:400)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:323)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:58)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:173)
>   at org.apache.beam.examples.WordCount.main(WordCount.java:195)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>   at 
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>   at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-637) WindowedValue$ValueInGlobalWindow is not serializable when using JavaSerializer instead of Kryo

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513640#comment-15513640
 ] 

Amit Sela commented on BEAM-637:


[~staslev] do you think this ticket could be about "Spark runner to support 
JavaSerializer" instead ? this is only the symptom...

> WindowedValue$ValueInGlobalWindow is not serializable when using 
> JavaSerializer instead of Kryo 
> 
>
> Key: BEAM-637
> URL: https://issues.apache.org/jira/browse/BEAM-637
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Amit Sela
>
> When using {{JavaSerializer}} instead of {{KryoSerializer}} by configuring 
> {{JavaSparkContext}} with  {{"spark.serializer"}} set to 
> {{JavaSerializer.class.getCanonicalName()}}, an exception is thrown:
> {noformat}
> object not serializable (class: 
> org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow, value: 
> ValueInGlobalWindow{value=hi there, pane=PaneInfo.NO_FIRING})
> {noformat}
> {noformat}
> Serialization stack:
>   - object not serializable (class: 
> org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow, value: 
> ValueInGlobalWindow{value=hi there, pane=PaneInfo.NO_FIRING})
>   at 
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   at 
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
>   at 
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
>   at scala.Option.foreach(Option.scala:236)
>   at 
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
>   at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
>   at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>   at 
> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
>   at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
>   at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
>   at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
>   at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
>   at org.apache.spark.rdd.RDD.count(RDD.scala:1157)
>   at 
> org.apache.spark.api.java.JavaRDDLike$class.count(JavaRDDLike.scala:440)
>   at 
> org.apache.spark.api.java.AbstractJavaRDDLike.count(JavaRDDLike.scala:46)
>   at 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext$1.call(StreamingEvaluationContext.java:175)
>   at 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext$1.call(StreamingEvaluationContext.java:172)
>   at 
> org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$3.apply(JavaDStreamLike.scala:335)
>   at 
> org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$3.apply(JavaDStreamLike.scala:335)
>   at 
> org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
>   at 
> org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
>   at 
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
>   at 
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
>   at 
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
>   at 
> org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
>   at 
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
>   at 
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
>   at 
> org.apache.spark.streaming.dstream.

[GitHub] incubator-beam pull request #822: [BEAM-549] SparkRunner should support Beam...

[amitsela]

Github user amitsela closed the pull request at:

https://github.com/apache/incubator-beam/pull/822


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #984

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 6082ebcce -> 495d8b358


Closes #984


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/495d8b35
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/495d8b35
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/495d8b35

Branch: refs/heads/master
Commit: 495d8b3580e87db3929f0e70f1e97efa57d34c48
Parents: 6082ebc d73f8e7
Author: Dan Halperin 
Authored: Thu Sep 22 08:59:23 2016 -0700
Committer: Dan Halperin 
Committed: Thu Sep 22 08:59:23 2016 -0700

--
 .../org/apache/beam/sdk/DataflowMatchers.java   | 64 
 1 file changed, 64 deletions(-)
--

[GitHub] incubator-beam pull request #984: Remove DataflowMatchers

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/984


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Remove DataflowMatchers

[dhalperi]

Remove DataflowMatchers

This is a Dataflow-specific requirement and should not be present within
the core Beam SDK.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d73f8e77
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d73f8e77
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d73f8e77

Branch: refs/heads/master
Commit: d73f8e7792f396e6c3fe444f24672d53795bb864
Parents: 6082ebc
Author: Thomas Groh 
Authored: Wed Sep 21 18:16:45 2016 -0700
Committer: Dan Halperin 
Committed: Thu Sep 22 08:59:23 2016 -0700

--
 .../org/apache/beam/sdk/DataflowMatchers.java   | 64 
 1 file changed, 64 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d73f8e77/sdks/java/core/src/test/java/org/apache/beam/sdk/DataflowMatchers.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/DataflowMatchers.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/DataflowMatchers.java
deleted file mode 100644
index c76dcf8..000
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/DataflowMatchers.java
+++ /dev/null
@@ -1,64 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk;
-
-import com.google.protobuf.ByteString;
-import java.io.Serializable;
-import org.hamcrest.Description;
-import org.hamcrest.TypeSafeMatcher;
-
-/**
- * Matchers that are useful when writing Dataflow tests.
- */
-public class DataflowMatchers {
-  /**
-   * Matcher for {@link ByteString} that prints the strings in UTF8.
-   */
-  public static class ByteStringMatcher extends TypeSafeMatcher
-  implements Serializable {
-private ByteString expected;
-private ByteStringMatcher(ByteString expected) {
-  this.expected = expected;
-}
-
-public static ByteStringMatcher byteStringEq(ByteString expected) {
-  return new ByteStringMatcher(expected);
-}
-
-@Override
-public void describeTo(Description description) {
-  description
-  .appendText("ByteString(")
-  .appendText(expected.toStringUtf8())
-  .appendText(")");
-}
-
-@Override
-public void describeMismatchSafely(ByteString actual, Description 
description) {
-  description
-  .appendText("was ByteString(")
-  .appendText(actual.toStringUtf8())
-  .appendText(")");
-}
-
-@Override
-protected boolean matchesSafely(ByteString actual) {
-  return actual.equals(expected);
-}
-  }
-}

[GitHub] incubator-beam pull request #987: Adds __all__ tags to source modules.

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/987

Adds __all__ tags to source modules.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
add_all_tags_to_source_modules

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/987.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #987


commit fbda5eeff89b65d8022fc32babb328386cae0bca
Author: Chamikara Jayalath 
Date:   2016-09-22T16:08:11Z

Adds __all__ tags to source modules.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1189

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #988: Add initial scaffolding for default bucket

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/988

Add initial scaffolding for default bucket

@tgroh, could you please take a look?

Before reviewing further, I was hoping to get clarity on whether the travis 
failures I'm seeing are related to this PR:

[ERROR] 
/Users/travis/build/sammcveety/incubator-beam/sdks/java/core/target/generated-sources/annotations/org/apache/beam/sdk/testing/AutoValue_TestStream_ElementEvent.java:[8,7]
 duplicate class: org.apache.beam.sdk.testing.AutoValue_TestStream_ElementEvent
[ERROR] 
/Users/travis/build/sammcveety/incubator-beam/sdks/java/core/target/generated-sources/annotations/org/apache/beam/sdk/testing/AutoValue_TestStream_WatermarkEvent.java:[8,7]
 duplicate class: 
org.apache.beam.sdk.testing.AutoValue_TestStream_WatermarkEvent
[ERROR] 
/Users/travis/build/sammcveety/incubator-beam/sdks/java/core/target/generated-sources/annotations/org/apache/beam/sdk/testing/AutoValue_TestStream_ProcessingTimeEvent.java:[8,7]
 duplicate class: 
org.apache.beam.sdk.testing.AutoValue_TestStream_ProcessingTimeEvent
[ERROR] 
/Users/travis/build/sammcveety/incubator-beam/sdks/java/core/target/generated-sources/annotations/org/apache/beam/sdk/transforms/display/AutoValue_DisplayData_Item.java:[7,7]
 duplicate class: 
org.apache.beam.sdk.transforms.display.AutoValue_DisplayData_Item
[ERROR] 
/Users/travis/build/sammcveety/incubator-beam/sdks/java/core/target/generated-sources/annotations/org/apache/beam/sdk/options/AutoValue_PipelineOptionSpec.java:[8,7]
 duplicate class: org.apache.beam.sdk.options.AutoValue_PipelineOptionSpec
[ERROR] 
/Users/travis/build/sammcveety/incubator-beam/sdks/java/core/target/generated-sources/annotations/org/apache/beam/sdk/options/AutoValue_ProxyInvocationHandler_BoundValue.java:[7,7]
 duplicate class: 
org.apache.beam.sdk.options.AutoValue_ProxyInvocationHandler_BoundValue
[ERROR] 
/Users/travis/build/sammcveety/incubator-beam/sdks/java/core/target/generated-sources/annotations/org/apache/beam/sdk/transforms/reflect/AutoValue_DoFnSignature.java:[8,7]
 duplicate class: org.apache.beam.sdk.transforms.reflect.AutoValue_DoFnSignature
[ERROR] 
/Users/travis/build/sammcveety/incubator-beam/sdks/java/core/target/generated-sources/annotations/org/apache/beam/sdk/transforms/reflect/AutoValue_DoFnSignature_ProcessElementMethod.java:[9,7]
 duplicate class: 
org.apache.beam.sdk.transforms.reflect.AutoValue_DoFnSignature_ProcessElementMethod
[ERROR] 
/Users/travis/build/sammcveety/incubator-beam/sdks/java/core/target/generated-sources/annotations/org/apache/beam/sdk/transforms/reflect/AutoValue_DoFnSignature_BundleMethod.java:[8,7]
 duplicate class: 
org.apache.beam.sdk.transforms.reflect.AutoValue_DoFnSignature_BundleMethod
[ERROR] 
/Users/travis/build/sammcveety/incubator-beam/sdks/java/core/target/generated-sources/annotations/org/apache/beam/sdk/transforms/reflect/AutoValue_DoFnSignature_LifecycleMethod.java:[8,7]
 duplicate class: 
org.apache.beam.sdk.transforms.reflect.AutoValue_DoFnSignature_LifecycleMethod

- Provide a hook for determining the project number associated with a bucket
- Provide an API for looking up the project number from a project ID

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/default_bucket

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/988.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #988


commit bb87b3a735e87eac3105f15294c4e2e782ae1870
Author: sammcveety 
Date:   2016-09-18T01:19:53Z

Add initial bucket stuff.

commit f6b4616d9f19b5f484bf573a36c90d075a155cb7
Author: sammcveety 
Date:   2016-09-18T01:45:39Z

Basic CRM client.

commit b7e93be0501cf26092456c757fdf7affa71cdd2f
Author: sammcveety 
Date:   2016-09-18T02:08:05Z

Fix some compilation errors and update pom.

commit 4077ef3d7b9d906dbfbb4c4189fde1ac81a0e8f9
Author: sammcveety 
Date:   2016-09-18T02:22:48Z

Minor updates.

commit dab7ec2f6aaf19dc5cf382b926bd1766861d8127
Author: sammcveety 
Date:   2016-09-18T02:36:38Z

Add options.

commit 0ab2a1f11364b0db707eccaeaa7342479b24a594
Author: sammcveety 
Date:   2016-09-18T02:39:35Z

Fix more things.

commit b005d0f80c73c1dca78d64f057731e4bd91420cf
Author: sammcveety 
Date:   2016-09-18T03:04:40Z

Actually include file.

commit a5a616a74b33074ce22d26efc12d3b19867f8d7b
Author: sammcveety 
Date:   2016-09-18T03:17:26Z

Checkstyle fixes.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.or

[jira] [Commented] (BEAM-642) FlinkRunner does not support Detached Mode

[Sumit Chawla (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513831#comment-15513831
 ] 

Sumit Chawla commented on BEAM-642:
---

[~maxmil23] [~amitsela] Sorry i did not have permission to create a new issue.  
So i cloned an existing one and updated that one. I could not change the 
assignee as well.

> FlinkRunner does not support Detached Mode
> --
>
> Key: BEAM-642
> URL: https://issues.apache.org/jira/browse/BEAM-642
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Sumit Chawla
>Assignee: Sumit Chawla
> Fix For: 0.3.0-incubating
>
>
> FlinkRunner gives exception when job is submitted in detached mode.
> Following code will throw exception: 
> {code}
>  LOG.info("Execution finished in {} msecs", result.getNetRuntime()); 
> {code}
> {code}
> org.apache.flink.api.common.InvalidProgramException: Job was submitted in 
> detached mode. Results of job execution, such as accumulators, runtime, job 
> id etc. are not available. 
>   at 
> org.apache.flink.client.program.DetachedEnvironment$DetachedJobExecutionResult.getNetRuntime(DetachedEnvironment.java:98)
>  ~[flink-clients_2.10-1.1.2.jar:1.1.2]
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:116) 
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:48) 
> ~[beam-runners-flink_2.10-0.2.0-incubating.jar:0.2.0-incubating]
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:183) 
> ~[beam-sdks-java-core-0.2.0-incubating.jar:0.2.0-incubating]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_92]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_92]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_92]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
>  [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>  [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:320) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253) 
> [flink-clients_2.10-1.1.2.jar:1.1.2]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-644) Primitive to shift the watermark while assigning timestamps

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-644:
-
Description: 
There is a general need, especially important in the presence of 
SplittableDoFn, to be able to assign new timestamps to elements without making 
them late or droppable.

 - DoFn.withAllowedTimestampSkew is inadequate, because it simply allows one to 
produce late data, but does not allow one to shift the watermark so the new 
data is on-time.
 - For a SplittableDoFn, one may receive an element such as the name of a log 
file that contains elements for the day preceding the log file. The timestamp 
on the filename must currently be the beginning of the log. If such elements 
are constantly flowing, it may be OK, but since we don't know that element is 
coming, in that absence of data, the watermark may advance. We need a way to 
keep it far enough back even in the absence of data holding it back.

One idea is a new primitive ShiftWatermark / AdjustTimestamps with the 
following pieces:

 - A constant duration (positive or negative) D by which to shift the watermark.
 - A function from TimestampedElement to new timestamp that is >= t + D

So, for example, AdjustTimestamps(<-60 minutes>, f) would allow f to make 
timestamps up to 60 minutes earlier.

With this primitive added, outputWithTimestamp and withAllowedTimestampSkew 
could be removed, simplifying DoFn.

Alternatively, all of this functionality could be bolted on to DoFn.

This ticket is not a proposal, but a record of the issue and ideas that were 
mentioned.

  was:
There is a general need, especially important in the presence of 
SplittableDoFn, to be able to assign new timestamps to elements without making 
them late or droppable.

 - DoFn.withAllowedTimestampSkew is inadequate, because it simply allows one to 
produce late data, but does not allow one to shift the watermark so the new 
data is on-time.
 - For a SplittableDoFn, one may receive an element such as the name of a log 
file that contains elements for the day preceding the log file. The timestamp 
on the filename must currently be the beginning of the log. If such elements 
are constantly flowing, it may be OK, but since we don't know that element is 
coming, in that absence of data, the watermark may advance. We need a way to 
keep it far enough back even in the absence of data holding it back.

One idea is a new primitive ShiftWatermark / AdjustTimestamps with the 
following pieces:

 - A constant duration (positive or negative) D by which to shift the watermark.
 - A function from TimestampedElement to new timestamp that always falls 
within D of the original timestamp.

With this primitive added, outputWithTimestamp and withAllowedTimestampSkew 
could be removed, simplifying DoFn.

Alternatively, all of this functionality could be bolted on to DoFn.

This ticket is not a proposal, but a record of the issue and ideas that were 
mentioned.


> Primitive to shift the watermark while assigning timestamps
> ---
>
> Key: BEAM-644
> URL: https://issues.apache.org/jira/browse/BEAM-644
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> There is a general need, especially important in the presence of 
> SplittableDoFn, to be able to assign new timestamps to elements without 
> making them late or droppable.
>  - DoFn.withAllowedTimestampSkew is inadequate, because it simply allows one 
> to produce late data, but does not allow one to shift the watermark so the 
> new data is on-time.
>  - For a SplittableDoFn, one may receive an element such as the name of a log 
> file that contains elements for the day preceding the log file. The timestamp 
> on the filename must currently be the beginning of the log. If such elements 
> are constantly flowing, it may be OK, but since we don't know that element is 
> coming, in that absence of data, the watermark may advance. We need a way to 
> keep it far enough back even in the absence of data holding it back.
> One idea is a new primitive ShiftWatermark / AdjustTimestamps with the 
> following pieces:
>  - A constant duration (positive or negative) D by which to shift the 
> watermark.
>  - A function from TimestampedElement to new timestamp that is >= t + D
> So, for example, AdjustTimestamps(<-60 minutes>, f) would allow f to make 
> timestamps up to 60 minutes earlier.
> With this primitive added, outputWithTimestamp and withAllowedTimestampSkew 
> could be removed, simplifying DoFn.
> Alternatively, all of this functionality could be bolted on to DoFn.
> This ticket is not a proposal, but a record of the issue and ideas that were 
> mentioned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-644) Primitive to shift the watermark while assigning timestamps

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513859#comment-15513859
 ] 

Kenneth Knowles commented on BEAM-644:
--

I don't quite understand the comment [~aljoscha]. Can you say more? I have 
incorporated the correction by [~bchambers] and also added one example.

> Primitive to shift the watermark while assigning timestamps
> ---
>
> Key: BEAM-644
> URL: https://issues.apache.org/jira/browse/BEAM-644
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> There is a general need, especially important in the presence of 
> SplittableDoFn, to be able to assign new timestamps to elements without 
> making them late or droppable.
>  - DoFn.withAllowedTimestampSkew is inadequate, because it simply allows one 
> to produce late data, but does not allow one to shift the watermark so the 
> new data is on-time.
>  - For a SplittableDoFn, one may receive an element such as the name of a log 
> file that contains elements for the day preceding the log file. The timestamp 
> on the filename must currently be the beginning of the log. If such elements 
> are constantly flowing, it may be OK, but since we don't know that element is 
> coming, in that absence of data, the watermark may advance. We need a way to 
> keep it far enough back even in the absence of data holding it back.
> One idea is a new primitive ShiftWatermark / AdjustTimestamps with the 
> following pieces:
>  - A constant duration (positive or negative) D by which to shift the 
> watermark.
>  - A function from TimestampedElement to new timestamp that is >= t + D
> So, for example, AdjustTimestamps(<-60 minutes>, f) would allow f to make 
> timestamps up to 60 minutes earlier.
> With this primitive added, outputWithTimestamp and withAllowedTimestampSkew 
> could be removed, simplifying DoFn.
> Alternatively, all of this functionality could be bolted on to DoFn.
> This ticket is not a proposal, but a record of the issue and ideas that were 
> mentioned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #989: [BEAM-625] Better documentation for Compre...

[katsiapis]

GitHub user katsiapis opened a pull request:

https://github.com/apache/incubator-beam/pull/989

[BEAM-625] Better documentation for CompressionTypes.





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/katsiapis/incubator-beam compression

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/989.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #989


commit cb6d891aed0faf92f2e720fca98a658cfbddc959
Author: Gus Katsiapis 
Date:   2016-09-22T17:25:45Z

Better documentation for CompressionTypes.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-625) Make Dataflow Python Materialized PCollection representation more efficient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513900#comment-15513900
 ] 

ASF GitHub Bot commented on BEAM-625:
-

GitHub user katsiapis opened a pull request:

https://github.com/apache/incubator-beam/pull/989

[BEAM-625] Better documentation for CompressionTypes.





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/katsiapis/incubator-beam compression

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/989.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #989


commit cb6d891aed0faf92f2e720fca98a658cfbddc959
Author: Gus Katsiapis 
Date:   2016-09-22T17:25:45Z

Better documentation for CompressionTypes.




> Make Dataflow Python Materialized PCollection representation more efficient
> ---
>
> Key: BEAM-625
> URL: https://issues.apache.org/jira/browse/BEAM-625
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Konstantinos Katsiapis
>Assignee: Frances Perry
>
> This will be a several step process which will involve adding better support 
> for compression as well as Avro.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-644) Primitive to shift the watermark while assigning timestamps

[Ben Chambers (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513902#comment-15513902
 ] 

Ben Chambers commented on BEAM-644:
---

I'm not sure if this will clarify, but there are two parts:
1. A function which for a given input timestamp (or input element) computes an 
output timestamp.
2. A duration which declares the maximum distance the output timestamp could be 
behind the input timestamp.

Consider a case where elements are read from a stream with timestamps based on 
when they were published.
But each element also has a timestamp that indicates when it actually happened.
And we know that there is at most a 60 minute delay between when it happened 
and when it was published.

We want to adjust the timestamps of elements to be when they actually happened. 
But this may shift timestamps up to 60 minutes into the past. Naively, this 
will make data later, since the watermark logic doesn't know about the shift. 
But, with this proposal we'd write something that couples the shift up to 60 
minutes into the past with a declaration that we'll shift at most up to 60 
minutes into the past. The first applies to each element while the second 
relates the output watermark to the input watermark.

Similar reasoning could apply the other direction as well (shifting the output 
watermark forward) if we knew that we'd always adjust timestamps forward by at 
least a certain amount.

> Primitive to shift the watermark while assigning timestamps
> ---
>
> Key: BEAM-644
> URL: https://issues.apache.org/jira/browse/BEAM-644
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> There is a general need, especially important in the presence of 
> SplittableDoFn, to be able to assign new timestamps to elements without 
> making them late or droppable.
>  - DoFn.withAllowedTimestampSkew is inadequate, because it simply allows one 
> to produce late data, but does not allow one to shift the watermark so the 
> new data is on-time.
>  - For a SplittableDoFn, one may receive an element such as the name of a log 
> file that contains elements for the day preceding the log file. The timestamp 
> on the filename must currently be the beginning of the log. If such elements 
> are constantly flowing, it may be OK, but since we don't know that element is 
> coming, in that absence of data, the watermark may advance. We need a way to 
> keep it far enough back even in the absence of data holding it back.
> One idea is a new primitive ShiftWatermark / AdjustTimestamps with the 
> following pieces:
>  - A constant duration (positive or negative) D by which to shift the 
> watermark.
>  - A function from TimestampedElement to new timestamp that is >= t + D
> So, for example, AdjustTimestamps(<-60 minutes>, f) would allow f to make 
> timestamps up to 60 minutes earlier.
> With this primitive added, outputWithTimestamp and withAllowedTimestampSkew 
> could be removed, simplifying DoFn.
> Alternatively, all of this functionality could be bolted on to DoFn.
> This ticket is not a proposal, but a record of the issue and ideas that were 
> mentioned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1190

[Apache Jenkins Server]

See 

[1/2] incubator-beam git commit: Adds __all__ tags to source modules.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 345fc6985 -> 9565b2c27


Adds __all__ tags to source modules.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fbda5eef
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fbda5eef
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fbda5eef

Branch: refs/heads/python-sdk
Commit: fbda5eeff89b65d8022fc32babb328386cae0bca
Parents: 345fc69
Author: Chamikara Jayalath 
Authored: Thu Sep 22 09:08:11 2016 -0700
Committer: Chamikara Jayalath 
Committed: Thu Sep 22 09:08:11 2016 -0700

--
 sdks/python/apache_beam/io/avroio.py | 2 ++
 sdks/python/apache_beam/io/textio.py | 2 ++
 2 files changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fbda5eef/sdks/python/apache_beam/io/avroio.py
--
diff --git a/sdks/python/apache_beam/io/avroio.py 
b/sdks/python/apache_beam/io/avroio.py
index 196e760..3415d22 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -28,6 +28,8 @@ from apache_beam.io import filebasedsource
 from apache_beam.io.iobase import Read
 from apache_beam.transforms import PTransform
 
+__all__ = ['ReadFromAvro']
+
 
 class ReadFromAvro(PTransform):
   """A ``PTransform`` for reading avro files."""

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fbda5eef/sdks/python/apache_beam/io/textio.py
--
diff --git a/sdks/python/apache_beam/io/textio.py 
b/sdks/python/apache_beam/io/textio.py
index 28fd949..6042576 100644
--- a/sdks/python/apache_beam/io/textio.py
+++ b/sdks/python/apache_beam/io/textio.py
@@ -26,6 +26,8 @@ from apache_beam.io.iobase import Read
 from apache_beam.io.iobase import Write
 from apache_beam.transforms import PTransform
 
+__all__ = ['ReadFromText', 'WriteToText']
+
 
 class _TextSource(filebasedsource.FileBasedSource):
   """A source for reading text files.

[2/2] incubator-beam git commit: Closes #987

[robertwb]

Closes #987


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9565b2c2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9565b2c2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9565b2c2

Branch: refs/heads/python-sdk
Commit: 9565b2c27765a72c27d58038248204398a739933
Parents: 345fc69 fbda5ee
Author: Robert Bradshaw 
Authored: Thu Sep 22 11:26:05 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Sep 22 11:26:05 2016 -0700

--
 sdks/python/apache_beam/io/avroio.py | 2 ++
 sdks/python/apache_beam/io/textio.py | 2 ++
 2 files changed, 4 insertions(+)
--

[GitHub] incubator-beam pull request #987: Adds __all__ tags to source modules.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/987


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1191

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1192

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-670) FluentBackoff incorrectly handles nextBackOff millis

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-670:
--

 Summary: FluentBackoff incorrectly handles nextBackOff millis
 Key: BEAM-670
 URL: https://issues.apache.org/jira/browse/BEAM-670
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Luke Cwik
Assignee: Daniel Halperin
Priority: Minor


From:
https://github.com/GoogleCloudPlatform/DataflowJavaSDK/commit/94d57207924ed8650cf3c97fccb2a45f27bcc6a3#commitcomment-19135952

Also present in:
https://github.com/apache/incubator-beam/pull/888/files#diff-f6d45f28c12083c9556bb410bde8b109R614

The check is inverted, it should be nextBackoffMillis != BackOff.STOP
Otherwise it causes Thread.sleep() to be called with value -1 which causes an 
IllegalArgumentException exception to be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-670) BigQuery TableRow inserter incorrectly handles nextBackOff millis

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-670:
---
Summary: BigQuery TableRow inserter incorrectly handles nextBackOff millis  
(was: FluentBackoff incorrectly handles nextBackOff millis)

> BigQuery TableRow inserter incorrectly handles nextBackOff millis
> -
>
> Key: BEAM-670
> URL: https://issues.apache.org/jira/browse/BEAM-670
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Daniel Halperin
>Priority: Minor
>
> From:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/commit/94d57207924ed8650cf3c97fccb2a45f27bcc6a3#commitcomment-19135952
> Also present in:
> https://github.com/apache/incubator-beam/pull/888/files#diff-f6d45f28c12083c9556bb410bde8b109R614
> The check is inverted, it should be nextBackoffMillis != BackOff.STOP
> Otherwise it causes Thread.sleep() to be called with value -1 which causes an 
> IllegalArgumentException exception to be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-670) BigQuery TableRow inserter incorrectly handles nextBackOff millis

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514375#comment-15514375
 ] 

Daniel Halperin commented on BEAM-670:
--

The bug is actually at L618. We're calling nextBackoffMillis twice in a row 
instead of using the cached value.

https://github.com/apache/incubator-beam/pull/888/files#diff-f6d45f28c12083c9556bb410bde8b109R618

Good catch. How did tests not get this?

> BigQuery TableRow inserter incorrectly handles nextBackOff millis
> -
>
> Key: BEAM-670
> URL: https://issues.apache.org/jira/browse/BEAM-670
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Daniel Halperin
>Priority: Minor
>
> From:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/commit/94d57207924ed8650cf3c97fccb2a45f27bcc6a3#commitcomment-19135952
> Also present in:
> https://github.com/apache/incubator-beam/pull/888/files#diff-f6d45f28c12083c9556bb410bde8b109R614
> The check is inverted, it should be nextBackoffMillis != BackOff.STOP
> Otherwise it causes Thread.sleep() to be called with value -1 which causes an 
> IllegalArgumentException exception to be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-670) BigQuery TableRow inserter incorrectly handles nextBackOff millis

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514377#comment-15514377
 ] 

Daniel Halperin commented on BEAM-670:
--

In other words, I believe the check is good, but I might be wrong.

> BigQuery TableRow inserter incorrectly handles nextBackOff millis
> -
>
> Key: BEAM-670
> URL: https://issues.apache.org/jira/browse/BEAM-670
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Daniel Halperin
>Priority: Minor
>
> From:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/commit/94d57207924ed8650cf3c97fccb2a45f27bcc6a3#commitcomment-19135952
> Also present in:
> https://github.com/apache/incubator-beam/pull/888/files#diff-f6d45f28c12083c9556bb410bde8b109R614
> The check is inverted, it should be nextBackoffMillis != BackOff.STOP
> Otherwise it causes Thread.sleep() to be called with value -1 which causes an 
> IllegalArgumentException exception to be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #962: [BEAM-] Use sys.executable a...

[chrish42]

Github user chrish42 closed the pull request at:

https://github.com/apache/incubator-beam/pull/962


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Support BigQuery DATE type

[lcwik]

Support BigQuery DATE type

This closes #985


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8245f9b4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8245f9b4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8245f9b4

Branch: refs/heads/master
Commit: 8245f9b4b2ec28ad76c2cf0813b3a2de28aaebcd
Parents: 495d8b3 287061b
Author: Luke Cwik 
Authored: Thu Sep 22 14:27:43 2016 -0700
Committer: Luke Cwik 
Committed: Thu Sep 22 14:27:43 2016 -0700

--
 .../beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java  |  4 
 .../sdk/io/gcp/bigquery/BigQueryTableRowIterator.java|  2 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java  | 11 ---
 .../io/gcp/bigquery/BigQueryTableRowIteratorTest.java|  9 ++---
 4 files changed, 19 insertions(+), 7 deletions(-)
--

[GitHub] incubator-beam pull request #990: [BEAM-605] Create Bigquery Verifier

[markflyhigh]

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/990

[BEAM-605] Create Bigquery Verifier

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - Create BigqueryMatcher that is used to verify Bigquery data in 
integration test.
 - Add BigqueryMatcher to BigQueryTornadoesIT for output verification. 

Example command is used to running BigQueryTornadoesIT:
```
mvn clean verify -pl examples/java -am -DskipITs=false 
-Dit.test=BigQueryTornadoesIT -DintegrationTestPipelineOptions='[ 
"--tempRoot=gs://clouddfe-testing-temp-storage", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]'
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
bigquery-data-IT-verifier

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/990.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #990


commit 26f69fd9a3efb38f49a17f4474af3c004e1b897d
Author: Mark Liu 
Date:   2016-09-22T21:34:20Z

[BEAM-605] Create Bigquery Verifier




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #985: Support BigQuery DATE type

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/985


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-605) Create BigQuery Verifier

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514548#comment-15514548
 ] 

ASF GitHub Bot commented on BEAM-605:
-

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/990

[BEAM-605] Create Bigquery Verifier

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - Create BigqueryMatcher that is used to verify Bigquery data in 
integration test.
 - Add BigqueryMatcher to BigQueryTornadoesIT for output verification. 

Example command is used to running BigQueryTornadoesIT:
```
mvn clean verify -pl examples/java -am -DskipITs=false 
-Dit.test=BigQueryTornadoesIT -DintegrationTestPipelineOptions='[ 
"--tempRoot=gs://clouddfe-testing-temp-storage", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]'
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
bigquery-data-IT-verifier

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/990.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #990


commit 26f69fd9a3efb38f49a17f4474af3c004e1b897d
Author: Mark Liu 
Date:   2016-09-22T21:34:20Z

[BEAM-605] Create Bigquery Verifier




> Create BigQuery Verifier
> 
>
> Key: BEAM-605
> URL: https://issues.apache.org/jira/browse/BEAM-605
> Project: Beam
>  Issue Type: Task
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> Create BigQuery verifier that is used to verify output of integration test 
> which is using BigQuery as output source. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/5] incubator-beam-site git commit: This closes #37

[frances]

This closes #37


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/1bbf0868
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/1bbf0868
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/1bbf0868

Branch: refs/heads/asf-site
Commit: 1bbf0868993d0f33d6d89e38b3ea08eacc2a3658
Parents: dcdd8b7 14879db
Author: Frances Perry 
Authored: Thu Sep 22 14:34:26 2016 -0700
Committer: Frances Perry 
Committed: Thu Sep 22 14:34:26 2016 -0700

--
 _includes/header.html |   1 +
 contribute/testing.md | 388 +
 2 files changed, 389 insertions(+)
--

[2/5] incubator-beam-site git commit: Updated per PR comments

[frances]

Updated per PR comments

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/ec1ef53b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/ec1ef53b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/ec1ef53b

Branch: refs/heads/asf-site
Commit: ec1ef53b8bbc20bcf5f61b6985468b15f376a3a7
Parents: 6ad7ff0
Author: Jason Kuster 
Authored: Tue Sep 20 17:11:37 2016 -0700
Committer: Jason Kuster 
Committed: Tue Sep 20 17:11:37 2016 -0700

--
 contribute/testing.md | 228 +
 1 file changed, 190 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ec1ef53b/contribute/testing.md
--
diff --git a/contribute/testing.md b/contribute/testing.md
index 21d1e87..d8891bd 100644
--- a/contribute/testing.md
+++ b/contribute/testing.md
@@ -1,11 +1,27 @@
+---
+layout: default
+title: 'Beam Testing Guide'
+redirect_from: '/docs/testing'
+---
+
 # Beam Testing Documentation
 
-# Overview
+## Overview
 
-Apache Beam (incubating) is a rapidly-maturing software project with a strong 
commitment to testing. Consequently, it has many testing-related needs. It 
requires precommit tests to ensure code going into the repository meets a 
certain quality bar and it requires ongoing postcommit tests to make sure that 
more subtle changes which escape precommit are nonetheless caught. This 
document outlines how to write tests, which tests are appropriate where, and 
when tests are run, with some additional information about the testing systems 
at the bottom.
+Apache Beam (incubating) is a rapidly-maturing software project with a strong
+commitment to testing. Consequently, it has many testing-related needs. It
+requires precommit tests to ensure code going into the repository meets a
+certain quality bar and it requires ongoing postcommit tests to make sure that
+more subtle changes which escape precommit are nonetheless caught. This 
document
+outlines how to write tests, which tests are appropriate where, and when tests
+are run, with some additional information about the testing systems at the
+bottom.
 
-## Beam Java Testing Matrix
+If you’re writing tests, take a look at the testing matrix first, find what 
you
+want to test, then look into the “Scenarios” and “Types” sections 
below for more
+details on those testing types.
 
+## Beam Java Testing Matrix
 
 
   
@@ -17,6 +33,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co

Link to Example

+   Type
+   
+   Runs In
+   
   
   
DoFn
@@ -27,6 +47,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co

https://github.com/apache/incubator-beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java#L1325";>BigQueryIOTest

+   Unit
+   
+   Precommit, Postcommit
+   
   
   
BoundedSource
@@ -37,6 +61,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co

https://github.com/apache/incubator-beam/blob/84a0dd1714028370befa80dea16f720edce05252/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java#L972";>TextIOTest

+   Unit
+   
+   Precommit, Postcommit
+   
   
   

@@ -47,6 +75,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co

https://github.com/apache/incubator-beam/blob/8b1e64a668489297e11926124c4eee6c8f69a3a7/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java#L339";>BigtableTest

+   Unit
+   
+   Precommit, Postcommit
+   
   
   

@@ -57,6 +89,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co

https://github.com/apache/incubator-beam/blob/84a0dd1714028370befa80dea16f720edce05252/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java#L1021";>TextIOTest

+   Unit
+   
+   Precommit, Postcommit
+   
   
   
Transform
@@ -67,6 +103,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co

https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java#L1199";>ParDoTest

+   @NeedsRunner
+   
+   
+   
   
   
Example Pipeline
@@ -77,6 +117,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co

https://github.com/apache/incubator-beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java#L76";>WordCountIT

+   E2E
+   
+   Postcommit (Except WordCountIT

[5/5] incubator-beam-site git commit: Regenerated html.

[frances]

Regenerated html.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/a02bf8d3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/a02bf8d3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/a02bf8d3

Branch: refs/heads/asf-site
Commit: a02bf8d3a730c46bcb92f88f9d0722a7ac8728a9
Parents: 1bbf086
Author: Frances Perry 
Authored: Thu Sep 22 14:36:20 2016 -0700
Committer: Frances Perry 
Committed: Thu Sep 22 14:36:20 2016 -0700

--
 .../2016/03/17/capability-matrix.html   |   1 +
 .../2016/04/03/presentation-materials.html  |   1 +
 .../sdk/2016/02/25/python-sdk-now-public.html   |   1 +
 .../beam/release/2016/06/15/first-release.html  |   1 +
 .../website/2016/02/22/beam-has-a-logo.html |   1 +
 .../blog/2016/05/18/splitAtFraction-method.html |   1 +
 .../05/27/where-is-my-pcollection-dot-map.html  |   1 +
 .../06/13/flink-batch-runner-milestone.html |   1 +
 content/blog/2016/08/03/six-months.html |   1 +
 content/blog/index.html |   1 +
 content/coming-soon.html|   1 +
 .../contribute/contribution-guide/index.html|   1 +
 content/contribute/index.html   |   1 +
 content/contribute/source-repository/index.html |   1 +
 content/contribute/testing/index.html   | 573 +++
 content/feed.xml|   4 +-
 content/index.html  |   1 +
 content/learn/index.html|   1 +
 content/learn/presentation-materials/index.html |   1 +
 content/learn/programming-guide/index.html  |   1 +
 content/learn/resources/index.html  |   1 +
 .../learn/runners/capability-matrix/index.html  |   3 +-
 content/learn/runners/dataflow/index.html   |   1 +
 content/learn/runners/direct/index.html |   1 +
 content/learn/runners/flink/index.html  |   1 +
 content/learn/runners/index.html|   1 +
 content/learn/runners/spark/index.html  |   1 +
 content/learn/sdks/index.html   |   1 +
 content/learn/sdks/java/index.html  |   1 +
 content/learn/sdks/javadoc/index.html   |   1 +
 content/privacy_policy/index.html   |   1 +
 content/project/logos/index.html|   1 +
 content/project/public-meetings/index.html  |   1 +
 content/project/team/index.html |   1 +
 content/use/beam-overview/index.html|   1 +
 content/use/index.html  |   1 +
 content/use/issue-tracking/index.html   |   1 +
 content/use/mailing-lists/index.html|   1 +
 content/use/quickstart/index.html   |   1 +
 content/use/releases/index.html |   1 +
 content/use/walkthroughs/index.html |   1 +
 41 files changed, 615 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/a02bf8d3/content/beam/capability/2016/03/17/capability-matrix.html
--
diff --git a/content/beam/capability/2016/03/17/capability-matrix.html 
b/content/beam/capability/2016/03/17/capability-matrix.html
index b23323c..cbf07e3 100644
--- a/content/beam/capability/2016/03/17/capability-matrix.html
+++ b/content/beam/capability/2016/03/17/capability-matrix.html
@@ -96,6 +96,7 @@
  
  Basics
  Contribution Guide
+ Testing
  Mailing 
Lists
   Source 
Repository
   Issue Tracking

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/a02bf8d3/content/beam/capability/2016/04/03/presentation-materials.html
--
diff --git a/content/beam/capability/2016/04/03/presentation-materials.html 
b/content/beam/capability/2016/04/03/presentation-materials.html
index 58033db..6cc61f9 100644
--- a/content/beam/capability/2016/04/03/presentation-materials.html
+++ b/content/beam/capability/2016/04/03/presentation-materials.html
@@ -96,6 +96,7 @@
  
  Basics
  Contribution Guide
+ Testing
  Mailing 
Lists
   Source 
Repository
   Issue Tracking

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/a02bf8d3/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html
--
diff --git a/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html 
b/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html
i

[3/5] incubator-beam-site git commit: Added toc

[frances]

Added toc


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/14879db2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/14879db2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/14879db2

Branch: refs/heads/asf-site
Commit: 14879db2a8ed1eedc5fc8b9dd71a784fbba8af43
Parents: ec1ef53
Author: Frances Perry 
Authored: Thu Sep 22 14:32:04 2016 -0700
Committer: Frances Perry 
Committed: Thu Sep 22 14:32:04 2016 -0700

--
 contribute/testing.md | 16 +++-
 1 file changed, 11 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/14879db2/contribute/testing.md
--
diff --git a/contribute/testing.md b/contribute/testing.md
index d8891bd..b613afa 100644
--- a/contribute/testing.md
+++ b/contribute/testing.md
@@ -1,11 +1,14 @@
 ---
 layout: default
 title: 'Beam Testing Guide'
-redirect_from: '/docs/testing'
+permalink: /contribute/testing/
 ---
 
 # Beam Testing Documentation
 
+* TOC
+{:toc}
+
 ## Overview
 
 Apache Beam (incubating) is a rapidly-maturing software project with a strong
@@ -21,7 +24,9 @@ If you’re writing tests, take a look at the testing matrix 
first, find what yo
 want to test, then look into the “Scenarios” and “Types” sections 
below for more
 details on those testing types.
 
-## Beam Java Testing Matrix
+## Testing Matrix
+
+### Java SDK
 
 
   
@@ -250,10 +255,11 @@ details on those testing types.
   
 
 
-## Beam Python Testing Matrix
+### Python SDK
 
-Beam Python Testing Matrix coming soon. Python is currently tested in 
postcommit
-by a Jenkins build; precommit testing will be coming soon.
+The Python SDK is currently under development on a feature branch. We have 
initial
+postcommit tests by a Jenkins build; precommit testing and a full testing 
+matrix will be coming soon.
 
 ## Testing Scenarios
 

[1/5] incubator-beam-site git commit: Add testing guide to website.

[frances]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site dcdd8b742 -> a02bf8d3a


Add testing guide to website.

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/6ad7ff02
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/6ad7ff02
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/6ad7ff02

Branch: refs/heads/asf-site
Commit: 6ad7ff02b7adfe3d870b0beffe30e8c625d136d5
Parents: e2430eb
Author: Jason Kuster 
Authored: Fri Aug 12 11:18:46 2016 -0700
Committer: Jason Kuster 
Committed: Fri Aug 12 11:36:46 2016 -0700

--
 _includes/header.html |   1 +
 contribute/testing.md | 230 +
 2 files changed, 231 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/6ad7ff02/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index 653bda5..182b30a 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -63,6 +63,7 @@
  
  Basics
  Contribution Guide
+ Testing
  Mailing Lists
   Source Repository
   Issue 
Tracking

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/6ad7ff02/contribute/testing.md
--
diff --git a/contribute/testing.md b/contribute/testing.md
new file mode 100644
index 000..21d1e87
--- /dev/null
+++ b/contribute/testing.md
@@ -0,0 +1,230 @@
+# Beam Testing Documentation
+
+# Overview
+
+Apache Beam (incubating) is a rapidly-maturing software project with a strong 
commitment to testing. Consequently, it has many testing-related needs. It 
requires precommit tests to ensure code going into the repository meets a 
certain quality bar and it requires ongoing postcommit tests to make sure that 
more subtle changes which escape precommit are nonetheless caught. This 
document outlines how to write tests, which tests are appropriate where, and 
when tests are run, with some additional information about the testing systems 
at the bottom.
+
+## Beam Java Testing Matrix
+
+
+
+  
+   Component to Test
+   
+   Test Scenario
+   
+   Tool to Use
+   
+   Link to Example
+   
+  
+  
+   DoFn
+   
+   Correctness on one/few bundles
+   
+   DoFnTester
+   
+   https://github.com/apache/incubator-beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java#L1325";>BigQueryIOTest
+   
+  
+  
+   BoundedSource
+   
+   Correctly Reads Input
+   
+   https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SourceTestUtils.java#L128";>SourceTestUtils.readFromSource
+   
+   https://github.com/apache/incubator-beam/blob/84a0dd1714028370befa80dea16f720edce05252/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java#L972";>TextIOTest
+   
+  
+  
+   
+   
+   Correct Initial Splitting
+   
+   https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SourceTestUtils.java#L201";>SourceTestUtils.assertSourcesEqualReferenceSource
+   
+   https://github.com/apache/incubator-beam/blob/8b1e64a668489297e11926124c4eee6c8f69a3a7/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java#L339";>BigtableTest
+   
+  
+  
+   
+   
+   Correct Dynamic Splitting
+   
+   https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SourceTestUtils.java#L541";>SourceTestUtils.
 assertSplitAtFractionExhaustive
+   
+   https://github.com/apache/incubator-beam/blob/84a0dd1714028370befa80dea16f720edce05252/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java#L1021";>TextIOTest
+   
+  
+  
+   Transform
+   
+   Correctness
+   
+   @NeedsRunner Test
+   
+   https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java#L1199";>ParDoTest
+   
+  
+  
+   Example Pipeline
+   
+   Verify Behavior on Each Runner
+   
+   E2E Test
+   
+   https://github.com/apache/incubator-beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java#L76";>WordCountIT
+   
+  
+  
+   Source/Sink with external resource
+   
+   External Resource Faked
+   
+   Unit / @NeedsRunner Test
+   
+   https://github.com/apache/incubator-beam/blob/84a0dd1714028370befa80dea16f720edce05252/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/Bigtab

[GitHub] incubator-beam-site pull request #37: Add testing guide to website.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/37


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Support BigQuery DATE type

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 495d8b358 -> 8245f9b4b


Support BigQuery DATE type


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/287061b7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/287061b7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/287061b7

Branch: refs/heads/master
Commit: 287061b75ddaf6908dc14726876059443083f62c
Parents: 495d8b3
Author: Pei He 
Authored: Wed Sep 21 19:56:01 2016 -0700
Committer: Luke Cwik 
Committed: Thu Sep 22 14:25:51 2016 -0700

--
 .../beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java  |  4 
 .../sdk/io/gcp/bigquery/BigQueryTableRowIterator.java|  2 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java  | 11 ---
 .../io/gcp/bigquery/BigQueryTableRowIteratorTest.java|  9 ++---
 4 files changed, 19 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/287061b7/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
index d9b5423..2cc2df7 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
@@ -20,6 +20,7 @@ package org.apache.beam.sdk.io.gcp.bigquery;
 import static com.google.common.base.MoreObjects.firstNonNull;
 import static com.google.common.base.Preconditions.checkNotNull;
 import static com.google.common.base.Verify.verify;
+import static com.google.common.base.Verify.verifyNotNull;
 
 import com.google.api.services.bigquery.model.TableFieldSchema;
 import com.google.api.services.bigquery.model.TableRow;
@@ -162,11 +163,13 @@ class BigQueryAvroUtils {
 .put("BOOLEAN", Type.BOOLEAN)
 .put("TIMESTAMP", Type.LONG)
 .put("RECORD", Type.RECORD)
+.put("DATE", Type.STRING)
 .build();
 // Per https://cloud.google.com/bigquery/docs/reference/v2/tables#schema, 
the type field
 // is required, so it may not be null.
 String bqType = fieldSchema.getType();
 Type expectedAvroType = fieldMap.get(bqType);
+verifyNotNull(expectedAvroType, "Unsupported BigQuery type: %s", bqType);
 verify(
 avroType == expectedAvroType,
 "Expected Avro schema type %s, not %s, for BigQuery %s field %s",
@@ -176,6 +179,7 @@ class BigQueryAvroUtils {
 fieldSchema.getName());
 switch (fieldSchema.getType()) {
   case "STRING":
+  case "DATE":
 // Avro will use a CharSequence to represent String objects, but it 
may not always use
 // java.lang.String; for example, it may prefer 
org.apache.avro.util.Utf8.
 verify(v instanceof CharSequence, "Expected CharSequence (String), got 
%s", v.getClass());

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/287061b7/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableRowIterator.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableRowIterator.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableRowIterator.java
index 420f30c..d7423a1 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableRowIterator.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableRowIterator.java
@@ -253,7 +253,7 @@ class BigQueryTableRowIterator implements AutoCloseable {
   return BigQueryAvroUtils.formatTimestamp((String) v);
 }
 
-// Returns the original value for String and base64 encoded BYTES
+// Returns the original value for String, base64 encoded BYTES, and DATE 
string.
 return v;
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/287061b7/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java
index 

[3/3] incubator-beam git commit: Update grpc to 1.0.1 and protobuf to 3.0.0

[lcwik]

Update grpc to 1.0.1 and protobuf to 3.0.0

This closes #986


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4872bde8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4872bde8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4872bde8

Branch: refs/heads/master
Commit: 4872bde8f4ad2655074c262b30373c9cd53ae103
Parents: 8245f9b a447d13
Author: Luke Cwik 
Authored: Thu Sep 22 14:37:32 2016 -0700
Committer: Luke Cwik 
Committed: Thu Sep 22 14:37:32 2016 -0700

--
 pom.xml |  54 +-
 runners/google-cloud-dataflow-java/pom.xml  |   6 ++
 sdks/java/core/pom.xml  |  34 +++---
 .../apache/beam/sdk/util/PubsubGrpcClient.java  |  34 ++
 .../beam/sdk/util/PubsubGrpcClientTest.java | 108 ---
 sdks/java/io/google-cloud-platform/pom.xml  |   7 +-
 6 files changed, 159 insertions(+), 84 deletions(-)
--

[GitHub] incubator-beam pull request #986: Update grpc to 1.0.1 and protobuf to 3.0.0

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/986


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: Update grpc to 1.0.1 and protobuf to 3.0.0

[lcwik]

Update grpc to 1.0.1 and protobuf to 3.0.0


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f93ca9ce
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f93ca9ce
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f93ca9ce

Branch: refs/heads/master
Commit: f93ca9ce803a8847a7178ff0d7c5e1631bed8f2d
Parents: 8245f9b
Author: Luke Cwik 
Authored: Wed Sep 21 17:11:22 2016 -0700
Committer: Luke Cwik 
Committed: Thu Sep 22 14:37:00 2016 -0700

--
 pom.xml| 30 -
 sdks/java/core/pom.xml | 17 +-
 sdks/java/io/google-cloud-platform/pom.xml |  2 +-
 3 files changed, 27 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f93ca9ce/pom.xml
--
diff --git a/pom.xml b/pom.xml
index fb3a8a4..70373ec 100644
--- a/pom.xml
+++ b/pom.xml
@@ -103,19 +103,19 @@
 
 1.8.1
 v2-rev295-1.22.0
-0.0.2
+0.1.0
 v2-rev8-1.22.0
 v1b3-rev36-1.22.0
 0.5.160222
-1.1.0
-1.0.1
+1.2.0
+1.2.0
 1.0-rc2
 1.1
 1.22.0
 1.4.5
 
0.5.160304
 19.0
-0.14.1
+1.0.1
 1.3
 2.7.2
 3.0.1
@@ -124,7 +124,7 @@
 1.9.5
 4.1.1.Final
 1.4.0.Final
-3.0.0-beta-1
+3.0.0
 v1-rev10-1.22.0
 1.7.14
 3.1.4
@@ -470,6 +470,26 @@
   
 
   
+com.google.api.grpc
+grpc-google-pubsub-v1
+${pubsubgrpc.version}
+
+  
+  
+com.google.guava
+guava-jdk5
+  
+  
+  
+io.grpc
+grpc-all
+  
+
+  
+
+  
 com.google.apis
 google-api-services-storage
 ${storage.version}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f93ca9ce/sdks/java/core/pom.xml
--
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index 78aec85..1cf7ba5 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -337,22 +337,7 @@
 
 
   com.google.api.grpc
-  grpc-pubsub-v1
-  ${pubsubgrpc.version}
-  
-
-
-  com.google.guava
-  guava-jdk5
-
-
-
-  io.grpc
-  grpc-all
-
-  
+  grpc-google-pubsub-v1
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f93ca9ce/sdks/java/io/google-cloud-platform/pom.xml
--
diff --git a/sdks/java/io/google-cloud-platform/pom.xml 
b/sdks/java/io/google-cloud-platform/pom.xml
index 8075335..a0f3fef 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -32,7 +32,7 @@
   jar
 
   
-0.9.1
+0.9.2
   
 
   

[1/3] incubator-beam git commit: Update PubsubGrpcClient to not take in mocks as they are not needed anymore Update PubsubGrpcClientTest to use an inprocess server to handle requests/responses for tes

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8245f9b4b -> 4872bde8f


Update PubsubGrpcClient to not take in mocks as they are not needed anymore
Update PubsubGrpcClientTest to use an inprocess server to handle 
requests/responses for testing.
Add/remove required dependencies found missing by dep-analyzer plugin during 
upgrade to protobuf 3.0.0 and grpc 1.0.1


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a447d130
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a447d130
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a447d130

Branch: refs/heads/master
Commit: a447d130172ec7c270fc9f1d16a2f7404898461e
Parents: f93ca9c
Author: Luke Cwik 
Authored: Wed Sep 21 19:49:10 2016 -0700
Committer: Luke Cwik 
Committed: Thu Sep 22 14:37:00 2016 -0700

--
 pom.xml |  24 +
 runners/google-cloud-dataflow-java/pom.xml  |   6 ++
 sdks/java/core/pom.xml  |  17 +++
 .../apache/beam/sdk/util/PubsubGrpcClient.java  |  34 ++
 .../beam/sdk/util/PubsubGrpcClientTest.java | 108 ---
 sdks/java/io/google-cloud-platform/pom.xml  |   5 +
 6 files changed, 132 insertions(+), 62 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a447d130/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 70373ec..cc93bb9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -321,11 +321,35 @@
 
   
 io.grpc
+grpc-protobuf
+${grpc.version}
+  
+
+  
+io.grpc
+grpc-protobuf-lite
+${grpc.version}
+  
+
+  
+com.google.protobuf
+protobuf-lite
+3.0.1
+  
+
+  
+io.grpc
 grpc-netty
 ${grpc.version}
   
 
   
+io.grpc
+grpc-stub
+${grpc.version}
+  
+
+  
 com.google.api-client
 google-api-client
 ${google-clients.version}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a447d130/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index bf66f38..acf6cce 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -321,6 +321,12 @@
 
 
 
+  com.google.protobuf
+  protobuf-lite
+  runtime
+
+
+
   com.google.code.findbugs
   annotations
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a447d130/sdks/java/core/pom.xml
--
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index 1cf7ba5..c4d3e64 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -308,6 +308,11 @@
   grpc-netty
 
 
+
+  io.grpc
+  grpc-stub
+
+
 
 
@@ -317,6 +322,18 @@
 
 
 
+  io.grpc
+  grpc-protobuf-lite
+  runtime
+
+
+
+  com.google.protobuf
+  protobuf-lite
+  runtime
+
+
+
   com.google.auth
   google-auth-library-oauth2-http
   0.4.0

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a447d130/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PubsubGrpcClient.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PubsubGrpcClient.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PubsubGrpcClient.java
index 988b90f..02152ba 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PubsubGrpcClient.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PubsubGrpcClient.java
@@ -100,9 +100,7 @@ public class PubsubGrpcClient extends PubsubClient {
   idLabel,
   DEFAULT_TIMEOUT_S,
   channel,
-  credentials,
-  null /* publisher stub */,
-  null /* subscriber stub */);
+  credentials);
 }
 
 @Override
@@ -159,16 +157,12 @@ public class PubsubGrpcClient extends PubsubClient {
   @Nullable String idLabel,
   int timeoutSec,
   ManagedChannel publisherChannel,
-  GoogleCredentials credentials,
-  PublisherGrpc.PublisherBlockingStub cachedPublisherStub,
-  SubscriberGrpc.SubscriberBlockingStub cachedSubscriberStub) {
+  GoogleCredentials credentials) {
 this.timestampLabel = timestampLabel;
 this.idLabel = idLabel;

[jira] [Assigned] (BEAM-259) Execute selected RunnableOnService tests with Spark runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-259:


Assignee: Kenneth Knowles

> Execute selected RunnableOnService tests with Spark runner
> --
>
> Key: BEAM-259
> URL: https://issues.apache.org/jira/browse/BEAM-259
> Project: Beam
>  Issue Type: Test
>  Components: runner-spark
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_MavenVerify #1378

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1378

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-259) Execute selected RunnableOnService tests with Spark runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514657#comment-15514657
 ] 

Kenneth Knowles commented on BEAM-259:
--

[~jasonkuster] any chance you can take a look at why this build isn't running 
in postcommit?

> Execute selected RunnableOnService tests with Spark runner
> --
>
> Key: BEAM-259
> URL: https://issues.apache.org/jira/browse/BEAM-259
> Project: Beam
>  Issue Type: Test
>  Components: runner-spark
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-259) Execute selected RunnableOnService tests with Spark runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-259:
-
Assignee: Jason Kuster  (was: Kenneth Knowles)

> Execute selected RunnableOnService tests with Spark runner
> --
>
> Key: BEAM-259
> URL: https://issues.apache.org/jira/browse/BEAM-259
> Project: Beam
>  Issue Type: Test
>  Components: runner-spark
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #983

[amitsela]

This closes #983


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a00d2f81
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a00d2f81
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a00d2f81

Branch: refs/heads/master
Commit: a00d2f810992c450c14eab2bb5e3aa3ad3f80f74
Parents: 4872bde 584
Author: Sela 
Authored: Fri Sep 23 01:32:26 2016 +0300
Committer: Sela 
Committed: Fri Sep 23 01:32:26 2016 +0300

--
 examples/java/pom.xml   |   2 +-
 .../org/apache/beam/examples/WordCountIT.java   |   6 -
 runners/spark/pom.xml   |   6 +
 .../apache/beam/runners/spark/SparkRunner.java  |   5 -
 .../apache/beam/runners/spark/io/SourceRDD.java | 198 +++
 .../spark/translation/TransformTranslator.java  |  22 ++-
 6 files changed, 223 insertions(+), 16 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1193

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #983: [BEAM-657] Support Read.Bounded primitive

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/983


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-657] Support Read.Bounded primitive.

[amitsela]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4872bde8f -> a00d2f810


[BEAM-657] Support Read.Bounded primitive.

Support Read.Bounded primitive.

Avro requires this for snappy.

Create is supported by Read.Bounded now.

Read.Bounded support should solve gs issues now.

remove unused direct translations. Addressed by BEAM-668.

Assert deault parallelism, close reader on exception, and other improvements.

Adressed more comments.

Extra line.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5848
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5848
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5848

Branch: refs/heads/master
Commit: 58489bc8cbe4d702ff9ae07f932fb96141a1
Parents: 4872bde
Author: Sela 
Authored: Wed Sep 21 15:13:02 2016 +0300
Committer: Sela 
Committed: Fri Sep 23 01:31:16 2016 +0300

--
 examples/java/pom.xml   |   2 +-
 .../org/apache/beam/examples/WordCountIT.java   |   6 -
 runners/spark/pom.xml   |   6 +
 .../apache/beam/runners/spark/SparkRunner.java  |   5 -
 .../apache/beam/runners/spark/io/SourceRDD.java | 198 +++
 .../spark/translation/TransformTranslator.java  |  22 ++-
 6 files changed, 223 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5848/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 6a39f64..9a48ec6 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -184,7 +184,7 @@
 
   [
   "--project=apache-beam-testing",
-  "--tempRoot=/tmp",
+  "--tempRoot=gs://temp-storage-for-end-to-end-tests",
   "--runner=org.apache.beam.runners.spark.TestSparkRunner"
   ]
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5848/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java 
b/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
index b0e0fe0..2f2ea46 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
@@ -18,7 +18,6 @@
 
 package org.apache.beam.examples;
 
-import com.google.common.io.Resources;
 import java.util.Date;
 import org.apache.beam.examples.WordCount.WordCountOptions;
 import org.apache.beam.sdk.options.Default;
@@ -63,11 +62,6 @@ public class WordCountIT {
 new FileChecksumMatcher(options.getOutputChecksum(), 
options.getOutput() + "*"));
 
 String e2eTestInputPath = "gs://apache-beam-samples/apache/LICENSE";
-// Spark runner currently doesn't support GCS I/O, change default input to:
-// .../src/test/resources/LICENSE
-if (options.getRunner().getName().contains("SparkRunner")) {
-  e2eTestInputPath = Resources.getResource("LICENSE").getPath();
-}
 options.setInputFile(e2eTestInputPath);
 
 WordCount.main(TestPipeline.convertToArgs(options));

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5848/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index 228a90b..60b2de9 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -180,6 +180,12 @@
   joda-time
 
 
+  org.apache.commons
+  commons-compress
+  1.9
+  provided
+
+
   commons-io
   commons-io
   2.4

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5848/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
index 63dfe0d..3888ec2 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
@@ -28,7 +28,6 @@ import 
org.apache.beam.runners.spark.translation.TransformEvaluator;
 import org.apache.beam.runners.spark.translation.TransformTranslator;
 import 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory;
 import 
org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext;
-import org.apache.beam.r

[jira] [Commented] (BEAM-657) Support Read.Bounded primitive

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514689#comment-15514689
 ] 

ASF GitHub Bot commented on BEAM-657:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/983


> Support Read.Bounded primitive
> --
>
> Key: BEAM-657
> URL: https://issues.apache.org/jira/browse/BEAM-657
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> Spark runner support for Beam's primitive Read.Bounded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-657) Support Read.Bounded primitive

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-657.

   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> Support Read.Bounded primitive
> --
>
> Key: BEAM-657
> URL: https://issues.apache.org/jira/browse/BEAM-657
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.3.0-incubating
>
>
> Spark runner support for Beam's primitive Read.Bounded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1379

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1379

[Apache Jenkins Server]

See 

[1/3] incubator-beam git commit: Using strings instead of integers for identifying CompressionTypes.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 9565b2c27 -> 0fa9c4be6


Using strings instead of integers for identifying CompressionTypes.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8403ec1c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8403ec1c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8403ec1c

Branch: refs/heads/python-sdk
Commit: 8403ec1c446cc825eb81a46588a1222fcd42f523
Parents: 04d84c9
Author: Gus Katsiapis 
Authored: Thu Sep 22 13:15:51 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Sep 22 16:18:28 2016 -0700

--
 sdks/python/apache_beam/io/fileio.py | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8403ec1c/sdks/python/apache_beam/io/fileio.py
--
diff --git a/sdks/python/apache_beam/io/fileio.py 
b/sdks/python/apache_beam/io/fileio.py
index 6c254c7..54ca891 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -67,20 +67,20 @@ class CompressionTypes(object):
   #   .gz (implies GZIP as described below)
   #   .z  (implies ZLIB as described below).
   # Any non-recognized extension implies UNCOMPRESSED as described below.
-  AUTO = _CompressionType(1)
+  AUTO = _CompressionType('auto')
 
   # GZIP compression (deflate with GZIP headers).
-  GZIP = _CompressionType(2)
+  GZIP = _CompressionType('gzip')
 
   # ZLIB compression (deflate with ZLIB headers).
-  ZLIB = _CompressionType(3)
+  ZLIB = _CompressionType('zlib')
 
   # Uncompressed (i.e., may be split).
-  UNCOMPRESSED = _CompressionType(4)
+  UNCOMPRESSED = _CompressionType('uncompressed')
 
   # TODO: Remove this backwards-compatibility soon.
-  NO_COMPRESSION = _CompressionType(4)  # Deprecated. Use UNCOMPRESSED instead.
-  assert NO_COMPRESSION == UNCOMPRESSED
+  # Deprecated. Use UNCOMPRESSED instead.
+  NO_COMPRESSION = UNCOMPRESSED
 
   @classmethod
   def is_valid_compression_type(cls, compression_type):

[3/3] incubator-beam git commit: Closes #989

[robertwb]

Closes #989


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0fa9c4be
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0fa9c4be
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0fa9c4be

Branch: refs/heads/python-sdk
Commit: 0fa9c4be6185e15129447983173ae036f9d4fa89
Parents: 9565b2c 8403ec1
Author: Robert Bradshaw 
Authored: Thu Sep 22 16:18:29 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Sep 22 16:18:29 2016 -0700

--
 sdks/python/apache_beam/io/fileio.py | 25 +++--
 1 file changed, 19 insertions(+), 6 deletions(-)
--

[2/3] incubator-beam git commit: Better documentation for CompressionTypes.

[robertwb]

Better documentation for CompressionTypes.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/04d84c99
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/04d84c99
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/04d84c99

Branch: refs/heads/python-sdk
Commit: 04d84c996a14b557647a4c85703a29fbeaaa0db6
Parents: 9565b2c
Author: Gus Katsiapis 
Authored: Thu Sep 22 10:25:45 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Sep 22 16:18:28 2016 -0700

--
 sdks/python/apache_beam/io/fileio.py | 21 +
 1 file changed, 17 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/04d84c99/sdks/python/apache_beam/io/fileio.py
--
diff --git a/sdks/python/apache_beam/io/fileio.py 
b/sdks/python/apache_beam/io/fileio.py
index d640d50..6c254c7 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -60,10 +60,23 @@ class _CompressionType(object):
 
 class CompressionTypes(object):
   """Enum-like class representing known compression types."""
-  AUTO = _CompressionType(1)  # Detect compression based on filename extension.
-  GZIP = _CompressionType(2)  # gzip compression (deflate with gzip headers).
-  ZLIB = _CompressionType(3)  # zlib compression (deflate with zlib headers).
-  UNCOMPRESSED = _CompressionType(4)  # Uncompressed (i.e., may be split).
+
+  # Detect compression based on filename extension.
+  #
+  # The following extensions are currently recognized by auto-detection:
+  #   .gz (implies GZIP as described below)
+  #   .z  (implies ZLIB as described below).
+  # Any non-recognized extension implies UNCOMPRESSED as described below.
+  AUTO = _CompressionType(1)
+
+  # GZIP compression (deflate with GZIP headers).
+  GZIP = _CompressionType(2)
+
+  # ZLIB compression (deflate with ZLIB headers).
+  ZLIB = _CompressionType(3)
+
+  # Uncompressed (i.e., may be split).
+  UNCOMPRESSED = _CompressionType(4)
 
   # TODO: Remove this backwards-compatibility soon.
   NO_COMPRESSION = _CompressionType(4)  # Deprecated. Use UNCOMPRESSED instead.

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1380

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1380

[Apache Jenkins Server]

See 

[incubator-beam] Git Push Summary

[dhalperi]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.2.0-incubating [created] 71aaffb7f

[GitHub] incubator-beam pull request #989: [BEAM-989] Better documentation for Compre...

[katsiapis]

Github user katsiapis closed the pull request at:

https://github.com/apache/incubator-beam/pull/989


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-147) Rename Aggregator to [P]Metric

[Ben Chambers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Chambers reassigned BEAM-147:
-

Assignee: Ben Chambers  (was: Frances Perry)

> Rename Aggregator to [P]Metric
> --
>
> Key: BEAM-147
> URL: https://issues.apache.org/jira/browse/BEAM-147
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Robert Bradshaw
>Assignee: Ben Chambers
>
> The name "Aggregator" is confusing.
> See discussion at 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
>  and 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser
>  . Exact name still being bikeshedded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #991: Allow .whl files to be staged with --extra...

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/incubator-beam/pull/991

Allow .whl files to be staged with --extra_package



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/incubator-beam allow-whl

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/991.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #991


commit b92058bed5125fc25fb0ca3da1d820f90b5ed25b
Author: Charles Chen 
Date:   2016-09-23T03:19:05Z

Allow .whl files to be staged with --extra_package




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1381

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1381

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1381

[Apache Jenkins Server]

See