[jira] [Resolved] (BEAM-1133) Add maxNumRecords per micro-batch for Spark runner options.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1133.
-
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> Add maxNumRecords per micro-batch for Spark runner options. 
> 
>
> Key: BEAM-1133
> URL: https://issues.apache.org/jira/browse/BEAM-1133
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> Allow users to set a max records bound (per micro-batch) when reading from 
> {{UnboundedSource}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1130) SparkRunner ResumeFromCheckpointStreamingTest Failing

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1130.
-
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> SparkRunner ResumeFromCheckpointStreamingTest Failing
> -
>
> Key: BEAM-1130
> URL: https://issues.apache.org/jira/browse/BEAM-1130
> Project: Beam
>  Issue Type: Bug
>Reporter: Jason Kuster
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Failing in (at least) suites:
> https://builds.apache.org/job/beam_PostCommit_Java_RunnableOnService_Spark/
> 381 - 385
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/
> 2063 - 2066
> Culprit change seems to be:
> https://github.com/apache/incubator-beam/commit/aba40e2de9ba058f33086eb6a913fa583a82b058
> Error:
> {code}
> Success aggregator should be greater than zero.
> Expected: not <0>
>  but: was <0>
> {code}
> Sample output:
> https://builds.apache.org/job/beam_PostCommit_Java_RunnableOnService_Spark/lastCompletedBuild/org.apache.beam$beam-runners-spark/testReport/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testRun/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1126) Expose UnboundedSource split backlog in number of events

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744487#comment-15744487
 ] 

Aviem Zur commented on BEAM-1126:
-

The intent is indeed to report number of events via a metric/aggregator, but 
the context for this number is inside the {{UnboundedSource}} implementation, 
which is why exposing this number via a method is required.

> Expose UnboundedSource split backlog in number of events
> 
>
> Key: BEAM-1126
> URL: https://issues.apache.org/jira/browse/BEAM-1126
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Daniel Halperin
>Priority: Minor
>
> Today {{UnboundedSource}} exposes split backlog in bytes via 
> {{getSplitBacklogBytes()}}
> There is value in exposing backlog in number of events as well, since this 
> number can be more human comprehensible than bytes. something like 
> {{getSplitBacklogEvents()}} or {{getSplitBacklogCount()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #2080

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2079

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1589: FileBasedSinkTest: fix tests in Windows O...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1589


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1589

[davor]

This closes #1589


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e776d1d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e776d1d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e776d1d7

Branch: refs/heads/master
Commit: e776d1d705857309107d78abf4eab5b5bc0c27da
Parents: f898ff7 592c4d9
Author: Davor Bonaci 
Authored: Mon Dec 12 21:23:11 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Dec 12 21:23:11 2016 -0800

--
 .../test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java   | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)
--

[1/2] incubator-beam git commit: FileBasedSinkTest: fix tests in Windows OS by using IOChannelUtils.resolve().

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master f898ff785 -> e776d1d70


FileBasedSinkTest: fix tests in Windows OS by using IOChannelUtils.resolve().


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/592c4d9f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/592c4d9f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/592c4d9f

Branch: refs/heads/master
Commit: 592c4d9fb9cbca22035e75fdcc77b887d40e86ec
Parents: f898ff7
Author: Pei He 
Authored: Mon Dec 12 17:52:07 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Dec 12 21:22:47 2016 -0800

--
 .../test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java   | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/592c4d9f/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
index 930ca29..9f0c424 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
@@ -51,6 +51,7 @@ import org.apache.beam.sdk.io.FileBasedSink.FileResult;
 import org.apache.beam.sdk.io.FileBasedSink.WritableByteChannelFactory;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.util.IOChannelUtils;
 import 
org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream;
 import org.junit.Rule;
 import org.junit.Test;
@@ -88,8 +89,7 @@ public class FileBasedSinkTest {
   @Test
   public void testWriter() throws Exception {
 String testUid = "testId";
-String expectedFilename =
-getBaseTempDirectory() + "/" + testUid;
+String expectedFilename = IOChannelUtils.resolve(getBaseTempDirectory(), 
testUid);
 SimpleSink.SimpleWriter writer = buildWriter();
 
 List values = Arrays.asList("sympathetic vulture", "boresome 
hummingbird");
@@ -488,8 +488,7 @@ public class FileBasedSinkTest {
 .createWriteOperation(null);
 final FileBasedWriter writer =
 writeOp.createWriter(null);
-final String expectedFilename =
-writeOp.tempDirectory.get() + "/" + testUid;
+final String expectedFilename = 
IOChannelUtils.resolve(writeOp.tempDirectory.get(), testUid);
 
 final List expected = new ArrayList<>();
 expected.add("header");

[jira] [Resolved] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-1020.

Resolution: Duplicate

Confirmed duplicate, which is fixed in the short-term. A new JIRA issue to be 
filed for the long-term fix.

[~tenghuanhe], thanks for your contribution! I'd love to see you continue 
contributing to the project!

> FileBasedSink fails to remove temporary directory on Windows operating system
> -
>
> Key: BEAM-1020
> URL: https://issues.apache.org/jira/browse/BEAM-1020
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Tenghuan He
>Assignee: Tenghuan He
> Fix For: 0.4.0-incubating
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
>  Beam fails to remove temporary directory after completing pipelines and 
> sinking output results on Windows operating system and throws the following 
> exception
> {noformat} 
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
>   at 
> org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
>   at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
>   at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
>   at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
>   at 
> org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
>   at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
> {noformat}
> This is due to the use of wildcard * when matching all files in the temporary 
> directory, while Windows does not support wildcard * in the file path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-863) WordCount on Windows: java.nio.file.InvalidPathException

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-863.
---
Resolution: Fixed

This is now fixed, by catching and ignoring the exception.

[~pei...@gmail.com], would you mind filing a separate issue for the long term 
fix?

> WordCount on Windows: java.nio.file.InvalidPathException
> 
>
> Key: BEAM-863
> URL: https://issues.apache.org/jira/browse/BEAM-863
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Davor Bonaci
>Assignee: Pei He
> Fix For: 0.4.0-incubating
>
>
> Running basic, unchanged WordCount on Windows with local files results in an 
> exception during cleanup:
> Caused by: java.nio.file.InvalidPathException: Illegal char <\*> at index 46: 
> C:\\dir\\beam_material\\result-temp-\* at 
> sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.jav‌​a:18) at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation‌​.removeTemporaryFile‌​s(FileBasedSink.java‌​:392)
> More details: 
> http://stackoverflow.com/questions/40285433/apache-beam-exception-while-running-wordcount-example-in-eclipse
> We need to address this for the next release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-863) WordCount on Windows: java.nio.file.InvalidPathException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744196#comment-15744196
 ] 

ASF GitHub Bot commented on BEAM-863:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1587


> WordCount on Windows: java.nio.file.InvalidPathException
> 
>
> Key: BEAM-863
> URL: https://issues.apache.org/jira/browse/BEAM-863
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Davor Bonaci
>Assignee: Pei He
> Fix For: 0.4.0-incubating
>
>
> Running basic, unchanged WordCount on Windows with local files results in an 
> exception during cleanup:
> Caused by: java.nio.file.InvalidPathException: Illegal char <\*> at index 46: 
> C:\\dir\\beam_material\\result-temp-\* at 
> sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.jav‌​a:18) at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation‌​.removeTemporaryFile‌​s(FileBasedSink.java‌​:392)
> More details: 
> http://stackoverflow.com/questions/40285433/apache-beam-exception-while-running-wordcount-example-in-eclipse
> We need to address this for the next release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1587: [BEAM-863] FileBasedSink: ignore exceptio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1587


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1587

[davor]

This closes #1587


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f898ff78
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f898ff78
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f898ff78

Branch: refs/heads/master
Commit: f898ff785e613394d07bfb3ca266d000524e4adb
Parents: 5ef34f7 c62f0a8
Author: Davor Bonaci 
Authored: Mon Dec 12 21:14:15 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Dec 12 21:14:15 2016 -0800

--
 .../org/apache/beam/sdk/io/FileBasedSink.java| 19 +++
 1 file changed, 15 insertions(+), 4 deletions(-)
--

[jira] [Commented] (BEAM-1142) maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488]

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744185#comment-15744185
 ] 

ASF GitHub Bot commented on BEAM-1142:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1588


> maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488]
> ---
>
> Key: BEAM-1142
> URL: https://issues.apache.org/jira/browse/BEAM-1142
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Pei He
>Assignee: Pei He
> Fix For: 0.4.0-incubating
>
>
> mvn clean install -DskipTests failed in maven-archetype-starter
> (Windows OS, maven version 3.3.x)
> Caused by: https://issues.apache.org/jira/browse/ARCHETYPE-488
> Logs:
> [INFO] Invoking post-archetype-generation goals: verify
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 32.082 s
> [INFO] Finished at: 2016-12-13T00:21:12+00:00
> [INFO] Final Memory: 32M/78M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-archetype-plugin:2
> .4:integration-test (default-integration-test) on project 
> beam-sdks-java-maven-a
> rchetypes-starter:
> [ERROR] Archetype IT 'basic' failed: Cannot run additions goals.
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal o
> rg.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test 
> (default-int
> egration-test) on project beam-sdks-java-maven-archetypes-starter:
> Archetype IT 'basic' failed: Cannot run additions goals.
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:212)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:153)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:145)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
> ct(LifecycleModuleBuilder.java:116)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
> ct(LifecycleModuleBuilder.java:80)
> at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThre
> adedBuilder.build(SingleThreadedBuilder.java:51)
> at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(Lifecycl
> eStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun
> cher.java:289)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
> a:229)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
> uncher.java:415)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
> 356)
> Caused by: org.apache.maven.plugin.MojoExecutionException:
> Archetype IT 'basic' failed: Cannot run additions goals.
> at 
> org.apache.maven.archetype.mojos.IntegrationTestMojo.execute(Integrat
> ionTestMojo.java:269)
> at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(Default
> BuildPluginManager.java:134)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:207)
> ... 20 more
> [ERROR]
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> rea
> d the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE
> xception



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1588: [BEAM-1142] Upgrade maven-invoker to addr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1588


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1142) maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488]

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-1142.

   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488]
> ---
>
> Key: BEAM-1142
> URL: https://issues.apache.org/jira/browse/BEAM-1142
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Pei He
>Assignee: Pei He
> Fix For: 0.4.0-incubating
>
>
> mvn clean install -DskipTests failed in maven-archetype-starter
> (Windows OS, maven version 3.3.x)
> Caused by: https://issues.apache.org/jira/browse/ARCHETYPE-488
> Logs:
> [INFO] Invoking post-archetype-generation goals: verify
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 32.082 s
> [INFO] Finished at: 2016-12-13T00:21:12+00:00
> [INFO] Final Memory: 32M/78M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-archetype-plugin:2
> .4:integration-test (default-integration-test) on project 
> beam-sdks-java-maven-a
> rchetypes-starter:
> [ERROR] Archetype IT 'basic' failed: Cannot run additions goals.
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal o
> rg.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test 
> (default-int
> egration-test) on project beam-sdks-java-maven-archetypes-starter:
> Archetype IT 'basic' failed: Cannot run additions goals.
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:212)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:153)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:145)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
> ct(LifecycleModuleBuilder.java:116)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
> ct(LifecycleModuleBuilder.java:80)
> at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThre
> adedBuilder.build(SingleThreadedBuilder.java:51)
> at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(Lifecycl
> eStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun
> cher.java:289)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
> a:229)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
> uncher.java:415)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
> 356)
> Caused by: org.apache.maven.plugin.MojoExecutionException:
> Archetype IT 'basic' failed: Cannot run additions goals.
> at 
> org.apache.maven.archetype.mojos.IntegrationTestMojo.execute(Integrat
> ionTestMojo.java:269)
> at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(Default
> BuildPluginManager.java:134)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:207)
> ... 20 more
> [ERROR]
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> rea
> d the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE
> xception



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-1142] Upgrade maven-invoker to address maven bug ARCHETYPE-488.

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 91cc606b4 -> 5ef34f7ee


[BEAM-1142] Upgrade maven-invoker to address maven bug ARCHETYPE-488.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2b84b7bc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2b84b7bc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2b84b7bc

Branch: refs/heads/master
Commit: 2b84b7bc08e4e351074f5cbd9f20e908295bd7d8
Parents: 91cc606
Author: Pei He 
Authored: Mon Dec 12 17:41:59 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Dec 12 21:08:45 2016 -0800

--
 sdks/java/maven-archetypes/examples-java8/pom.xml | 7 +++
 sdks/java/maven-archetypes/examples/pom.xml   | 7 +++
 sdks/java/maven-archetypes/starter/pom.xml| 7 +++
 3 files changed, 21 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2b84b7bc/sdks/java/maven-archetypes/examples-java8/pom.xml
--
diff --git a/sdks/java/maven-archetypes/examples-java8/pom.xml 
b/sdks/java/maven-archetypes/examples-java8/pom.xml
index 5cf094e..4aeb627 100644
--- a/sdks/java/maven-archetypes/examples-java8/pom.xml
+++ b/sdks/java/maven-archetypes/examples-java8/pom.xml
@@ -50,6 +50,13 @@
 
   maven-archetype-plugin
   2.4
+  
+
+  org.apache.maven.shared
+  maven-invoker
+  2.2
+
+  
   
 

[2/2] incubator-beam git commit: This closes #1588

[davor]

This closes #1588


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5ef34f7e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5ef34f7e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5ef34f7e

Branch: refs/heads/master
Commit: 5ef34f7ee8ecb89e4eae3b06fcd19e2102cd367b
Parents: 91cc606 2b84b7b
Author: Davor Bonaci 
Authored: Mon Dec 12 21:08:56 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Dec 12 21:08:56 2016 -0800

--
 sdks/java/maven-archetypes/examples-java8/pom.xml | 7 +++
 sdks/java/maven-archetypes/examples/pom.xml   | 7 +++
 sdks/java/maven-archetypes/starter/pom.xml| 7 +++
 3 files changed, 21 insertions(+)
--

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744106#comment-15744106
 ] 

ASF GitHub Bot commented on BEAM-362:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1592

[BEAM-362] Move ActiveWindowSet and implementations to runners-core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam ActiveWindowSet

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1592.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1592


commit 9a71e6e1558721dc5f80868eeb599ffedd025773
Author: Kenneth Knowles 
Date:   2016-12-13T04:18:02Z

Move ActiveWindowSet and implementations to runners-core




> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1592: [BEAM-362] Move ActiveWindowSet and imple...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1592

[BEAM-362] Move ActiveWindowSet and implementations to runners-core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam ActiveWindowSet

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1592.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1592


commit 9a71e6e1558721dc5f80868eeb599ffedd025773
Author: Kenneth Knowles 
Date:   2016-12-13T04:18:02Z

Move ActiveWindowSet and implementations to runners-core




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1089) Jenkins comments on PRs are too many & too large

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744080#comment-15744080
 ] 

Kenneth Knowles commented on BEAM-1089:
---

[~jasonkuster] I see what you mean [here in our 
configs|https://github.com/apache/incubator-beam/blob/master/.jenkins/common_job_properties.groovy#L124].
 Can the whole block be deleted or some such?

> Jenkins comments on PRs are too many & too large
> 
>
> Key: BEAM-1089
> URL: https://issues.apache.org/jira/browse/BEAM-1089
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>
> Lately, I've been finding review comments somewhat drowned out by asfbot 
> copying build results onto a PR. It also generates a lot of needless email. I 
> have not yet tried to devise just the right filter, hoping we can just return 
> to the normal practice of leaving just a commit status.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1089) Jenkins comments on PRs are too many & too large

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744077#comment-15744077
 ] 

Kenneth Knowles commented on BEAM-1089:
---

[~neelesh77] that sounds really great as well!

> Jenkins comments on PRs are too many & too large
> 
>
> Key: BEAM-1089
> URL: https://issues.apache.org/jira/browse/BEAM-1089
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>
> Lately, I've been finding review comments somewhat drowned out by asfbot 
> copying build results onto a PR. It also generates a lot of needless email. I 
> have not yet tried to devise just the right filter, hoping we can just return 
> to the normal practice of leaving just a commit status.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1143) Timestamps on Jenkins log lines

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1143:
-

 Summary: Timestamps on Jenkins log lines
 Key: BEAM-1143
 URL: https://issues.apache.org/jira/browse/BEAM-1143
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Kenneth Knowles
Assignee: Jason Kuster
Priority: Minor


I suspect this might be doable more universally in the groovy DSL scripts, but 
we would gain some value by a port of 
https://github.com/apache/incubator-beam/commit/7f82a573d00a5a30331b7bbb8757e55f4a2d93ae
 to the most appropriate analog for Jenkins. (in the worst case, just exactly 
porting the env var)

We are currently regularly bottlenecked on build duration/backlog and the time 
seems to exist outside of the durations accounted for by Maven's usual output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-1139) Investigate failures in precommit - Apex & Kryo

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743997#comment-15743997
 ] 

Kenneth Knowles edited comment on BEAM-1139 at 12/13/16 3:18 AM:
-

Unfortunately not. Further bummer: the postcommit is not a superset of the 
precommit so it never broke. That would provide a much easier trail.


was (Author: kenn):
Unfortunately not. Further bummer: the postcommit does not mirror the precommit 
so it never broke. That would provide a much easier trail.

> Investigate failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Thomas Weise
>Priority: Minor
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1139) Investigate failures in precommit - Apex & Kryo

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1139:
--
Summary: Investigate failures in precommit - Apex & Kryo  (was: Failures in 
precommit - Apex & Kryo)

> Investigate failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Thomas Weise
>Priority: Minor
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1139) Failures in precommit - Apex & Kryo

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1139:
--
Priority: Minor  (was: Blocker)

> Failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Thomas Weise
>Priority: Minor
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1139) Failures in precommit - Apex & Kryo

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743997#comment-15743997
 ] 

Kenneth Knowles commented on BEAM-1139:
---

Unfortunately not. Further bummer: the postcommit does not mirror the precommit 
so it never broke. That would provide a much easier trail.

> Failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Thomas Weise
>Priority: Blocker
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1591: Do not test pickling native sink objects

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1591

Do not test pickling native sink objects




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam dire

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1591.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1591


commit e36ee8d797965e2e2101cce046a14399f2008fc6
Author: Ahmet Altay 
Date:   2016-12-13T03:09:37Z

Do not test pickling native sink objects




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1800

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1800

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-1139) Failures in precommit - Apex & Kryo

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743947#comment-15743947
 ] 

Daniel Halperin edited comment on BEAM-1139 at 12/13/16 2:51 AM:
-

Do we know the PR or other root cause of this precommit break?


was (Author: dhalp...@google.com):
Do we know the PR root cause of this precommit break?

> Failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Thomas Weise
>Priority: Blocker
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1139) Failures in precommit - Apex & Kryo

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743947#comment-15743947
 ] 

Daniel Halperin commented on BEAM-1139:
---

Do we know the PR root cause of this precommit break?

> Failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Thomas Weise
>Priority: Blocker
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1071) Support pre-existing tables with streaming BigQueryIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743938#comment-15743938
 ] 

ASF GitHub Bot commented on BEAM-1071:
--

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1590

[BEAM-1071] Allow for streaming tables with CREATE_NEVER disposition



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam 
sgmc/never_disposition

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1590.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1590


commit 76240403cc3a14d413d597d24b8871832a3b7192
Author: Sam McVeety 
Date:   2016-12-13T02:47:20Z

Allow for BQ tables never




> Support pre-existing tables with streaming BigQueryIO
> -
>
> Key: BEAM-1071
> URL: https://issues.apache.org/jira/browse/BEAM-1071
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Sam McVeety
>Priority: Minor
>
> Specifically, with a tableRef function, CREATE_NEVER should be allowed for 
> pre-existing tables.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #866

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1590: [BEAM-1071] Allow for streaming tables wi...

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1590

[BEAM-1071] Allow for streaming tables with CREATE_NEVER disposition



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam 
sgmc/never_disposition

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1590.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1590


commit 76240403cc3a14d413d597d24b8871832a3b7192
Author: Sam McVeety 
Date:   2016-12-13T02:47:20Z

Allow for BQ tables never




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-967) Add Jenkins postcommit for Apex runner RunnableOnService tests

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743877#comment-15743877
 ] 

Kenneth Knowles commented on BEAM-967:
--

This would likely have caught BEAM-1139 sooner. Any progress?

> Add Jenkins postcommit for Apex runner RunnableOnService tests
> --
>
> Key: BEAM-967
> URL: https://issues.apache.org/jira/browse/BEAM-967
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex, testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>
> During development of the Apex runner, all RunnableOnService tests were run 
> in precommit. Now that it has been merged, they are not. It needs the usual 
> treatment that we give runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1139) Failures in precommit - Apex & Kryo

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743872#comment-15743872
 ] 

ASF GitHub Bot commented on BEAM-1139:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1586


> Failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Thomas Weise
>Priority: Blocker
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1586: [BEAM-1139] Add no-arg constructor for Un...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1586


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: Add Tests for Kryo Serialization of URFBS

[tgroh]

Add Tests for Kryo Serialization of URFBS


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/47cc2dca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/47cc2dca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/47cc2dca

Branch: refs/heads/master
Commit: 47cc2dca05daa4075093c414e13bf0cacaa77744
Parents: 3c2e550
Author: Thomas Groh 
Authored: Mon Dec 12 16:33:53 2016 -0800
Committer: Thomas Groh 
Committed: Mon Dec 12 18:17:08 2016 -0800

--
 runners/core-java/pom.xml   |  7 ++
 .../UnboundedReadFromBoundedSourceTest.java | 97 ++--
 2 files changed, 94 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/47cc2dca/runners/core-java/pom.xml
--
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index 0e2b4b0..bab9d57 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -152,6 +152,13 @@
 
 
 
+
+  com.esotericsoftware.kryo
+  kryo
+  2.21
+  test
+
+
 
 
   org.apache.beam

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/47cc2dca/runners/core-java/src/test/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSourceTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSourceTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSourceTest.java
index 7fd8807..8a1b70b 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSourceTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSourceTest.java
@@ -17,19 +17,28 @@
  */
 package org.apache.beam.runners.core;
 
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.hasSize;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
 
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Lists;
 import com.google.common.collect.Sets;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
 import java.io.File;
 import java.io.FileOutputStream;
 import java.io.IOException;
 import java.io.OutputStream;
 import java.nio.ByteBuffer;
 import java.nio.channels.ReadableByteChannel;
+import java.util.ArrayList;
 import java.util.List;
 import java.util.NoSuchElementException;
 import java.util.Random;
@@ -44,11 +53,13 @@ import org.apache.beam.sdk.io.BoundedSource;
 import org.apache.beam.sdk.io.CountingSource;
 import org.apache.beam.sdk.io.FileBasedSource;
 import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.Source;
 import org.apache.beam.sdk.io.UnboundedSource;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.testing.NeedsRunner;
 import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.SourceTestUtils;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.Count;
 import org.apache.beam.sdk.transforms.Distinct;
@@ -65,6 +76,7 @@ import org.junit.rules.ExpectedException;
 import org.junit.rules.TemporaryFolder;
 import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
+import org.objenesis.strategy.StdInstantiatorStrategy;
 
 /**
  * Unit tests for {@link UnboundedReadFromBoundedSource}.
@@ -101,28 +113,93 @@ public class UnboundedReadFromBoundedSourceTest {
 
 PCollection output =
 p.apply(Read.from(unboundedSource).withMaxNumRecords(numElements));
-
 // Count == numElements
 PAssert
-  .thatSingleton(output.apply("Count", Count.globally()))
-  .isEqualTo(numElements);
+.thatSingleton(output.apply("Count", Count.globally()))
+.isEqualTo(numElements);
 // Unique count == numElements
 PAssert
-  .thatSingleton(output.apply(Distinct.create())
-  .apply("UniqueCount", Count.globally()))
-  .isEqualTo(numElements);
+.thatSingleton(output.apply(Distinct.create())
+.apply("UniqueCount", Count.globally()))
+.isEqualTo(numElements);
 // Min == 0
 PAssert
-  .thatSingleton(output.apply("Min", Min.globally()))
-  .isEqualTo(0L);
+

[3/3] incubator-beam git commit: Add no-arg constructor for UnboundedReadFromBoundedSource

[tgroh]

Add no-arg constructor for UnboundedReadFromBoundedSource

This allows Kryo to work with the type, currently required by
the Apex runner.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3c2e550a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3c2e550a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3c2e550a

Branch: refs/heads/master
Commit: 3c2e550aa7a56fe16f278cb353314a1ee51dbfe3
Parents: d9657ff
Author: Kenneth Knowles 
Authored: Mon Dec 12 14:59:36 2016 -0800
Committer: Thomas Groh 
Committed: Mon Dec 12 18:17:08 2016 -0800

--
 .../beam/runners/core/UnboundedReadFromBoundedSource.java | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3c2e550a/runners/core-java/src/main/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSource.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSource.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSource.java
index f3f93e1..c804725 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSource.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/UnboundedReadFromBoundedSource.java
@@ -118,7 +118,12 @@ public class UnboundedReadFromBoundedSource extends 
PTransform {
 
-private BoundedSource boundedSource;
+@SuppressWarnings("unused") // for Kryo
+private BoundedToUnboundedSourceAdapter() {
+  this.boundedSource = null;
+}
+
+private final BoundedSource boundedSource;
 
 public BoundedToUnboundedSourceAdapter(BoundedSource boundedSource) {
   this.boundedSource = boundedSource;

[1/3] incubator-beam git commit: This closes #1586

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master d9657ffc3 -> 91cc606b4


This closes #1586


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/91cc606b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/91cc606b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/91cc606b

Branch: refs/heads/master
Commit: 91cc606b469b1166b2a2d41425eab63a4795b1e0
Parents: d9657ff 47cc2dc
Author: Thomas Groh 
Authored: Mon Dec 12 18:17:08 2016 -0800
Committer: Thomas Groh 
Committed: Mon Dec 12 18:17:08 2016 -0800

--
 runners/core-java/pom.xml   |  7 ++
 .../core/UnboundedReadFromBoundedSource.java|  7 +-
 .../UnboundedReadFromBoundedSourceTest.java | 97 ++--
 3 files changed, 100 insertions(+), 11 deletions(-)
--

[GitHub] incubator-beam pull request #1584: DO NOT REVIEW. Debugging MqttIO failures.

[jkff]

Github user jkff closed the pull request at:

https://github.com/apache/incubator-beam/pull/1584


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1589: FileBasedSinkTest: fix tests in Windows O...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1589

FileBasedSinkTest: fix tests in Windows OS by using 
IOChannelUtils.resolve().

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam files-resolve

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1589.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1589


commit c91425065c9a2dee1beab077dc5450858e44c47a
Author: Pei He 
Date:   2016-12-13T01:52:07Z

FileBasedSinkTest: fix tests in Windows OS by using 
IOChannelUtils.resolve().




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1142) maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488]

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743800#comment-15743800
 ] 

ASF GitHub Bot commented on BEAM-1142:
--

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1588

[BEAM-1142] Upgrade maven-invoker to address maven bug ARCHETYPE-488.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam fix-archetypes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1588.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1588


commit b461617aaf3d727ff3319296929cc4e3e2f4c325
Author: Pei He 
Date:   2016-12-13T01:41:59Z

[BEAM-1142] Upgrade maven-invoker to address maven bug ARCHETYPE-488.




> maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488]
> ---
>
> Key: BEAM-1142
> URL: https://issues.apache.org/jira/browse/BEAM-1142
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Pei He
>Assignee: Pei He
>
> mvn clean install -DskipTests failed in maven-archetype-starter
> (Windows OS, maven version 3.3.x)
> Caused by: https://issues.apache.org/jira/browse/ARCHETYPE-488
> Logs:
> [INFO] Invoking post-archetype-generation goals: verify
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 32.082 s
> [INFO] Finished at: 2016-12-13T00:21:12+00:00
> [INFO] Final Memory: 32M/78M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-archetype-plugin:2
> .4:integration-test (default-integration-test) on project 
> beam-sdks-java-maven-a
> rchetypes-starter:
> [ERROR] Archetype IT 'basic' failed: Cannot run additions goals.
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal o
> rg.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test 
> (default-int
> egration-test) on project beam-sdks-java-maven-archetypes-starter:
> Archetype IT 'basic' failed: Cannot run additions goals.
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:212)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:153)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:145)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
> ct(LifecycleModuleBuilder.java:116)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
> ct(LifecycleModuleBuilder.java:80)
> at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThre
> adedBuilder.build(SingleThreadedBuilder.java:51)
> at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(Lifecycl
> eStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun
> cher.java:289)
> at 
> 

[jira] [Updated] (BEAM-1142) maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488]

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-1142:
-
Component/s: examples-java

> maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488]
> ---
>
> Key: BEAM-1142
> URL: https://issues.apache.org/jira/browse/BEAM-1142
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Pei He
>Assignee: Pei He
>
> mvn clean install -DskipTests failed in maven-archetype-starter
> (Windows OS, maven version 3.3.x)
> Caused by: https://issues.apache.org/jira/browse/ARCHETYPE-488
> Logs:
> [INFO] Invoking post-archetype-generation goals: verify
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 32.082 s
> [INFO] Finished at: 2016-12-13T00:21:12+00:00
> [INFO] Final Memory: 32M/78M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-archetype-plugin:2
> .4:integration-test (default-integration-test) on project 
> beam-sdks-java-maven-a
> rchetypes-starter:
> [ERROR] Archetype IT 'basic' failed: Cannot run additions goals.
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal o
> rg.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test 
> (default-int
> egration-test) on project beam-sdks-java-maven-archetypes-starter:
> Archetype IT 'basic' failed: Cannot run additions goals.
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:212)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:153)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:145)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
> ct(LifecycleModuleBuilder.java:116)
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
> ct(LifecycleModuleBuilder.java:80)
> at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThre
> adedBuilder.build(SingleThreadedBuilder.java:51)
> at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(Lifecycl
> eStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun
> cher.java:289)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
> a:229)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
> uncher.java:415)
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
> 356)
> Caused by: org.apache.maven.plugin.MojoExecutionException:
> Archetype IT 'basic' failed: Cannot run additions goals.
> at 
> org.apache.maven.archetype.mojos.IntegrationTestMojo.execute(Integrat
> ionTestMojo.java:269)
> at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(Default
> BuildPluginManager.java:134)
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
> .java:207)
> ... 20 more
> [ERROR]
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> rea
> d the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE
> xception



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1142) maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488]

[Pei He (JIRA)]

Pei He created BEAM-1142:


 Summary: maven-archetype-starter failed in Windows OS due to 
[ARCHETYPE-488]
 Key: BEAM-1142
 URL: https://issues.apache.org/jira/browse/BEAM-1142
 Project: Beam
  Issue Type: Bug
Reporter: Pei He
Assignee: Pei He


mvn clean install -DskipTests failed in maven-archetype-starter
(Windows OS, maven version 3.3.x)
Caused by: https://issues.apache.org/jira/browse/ARCHETYPE-488

Logs:

[INFO] Invoking post-archetype-generation goals: verify
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 32.082 s
[INFO] Finished at: 2016-12-13T00:21:12+00:00
[INFO] Final Memory: 32M/78M
[INFO] 
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:2
.4:integration-test (default-integration-test) on project beam-sdks-java-maven-a
rchetypes-starter:
[ERROR] Archetype IT 'basic' failed: Cannot run additions goals.
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal o
rg.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test (default-int
egration-test) on project beam-sdks-java-maven-archetypes-starter:
Archetype IT 'basic' failed: Cannot run additions goals.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
ct(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje
ct(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThre
adedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(Lifecycl
eStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun
cher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
a:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
uncher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
356)
Caused by: org.apache.maven.plugin.MojoExecutionException:
Archetype IT 'basic' failed: Cannot run additions goals.
at org.apache.maven.archetype.mojos.IntegrationTestMojo.execute(Integrat
ionTestMojo.java:269)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(Default
BuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:207)
... 20 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please rea
d the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE
xception




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #107: Quickstart fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/107


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam-site git commit: This closes #107

[davor]

This closes #107


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/b46a41fa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/b46a41fa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/b46a41fa

Branch: refs/heads/asf-site
Commit: b46a41fab2375ced039706dcfeaa6f122cdfe5d7
Parents: a4eb4de d1b19ee
Author: Davor Bonaci 
Authored: Mon Dec 12 17:33:41 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Dec 12 17:33:41 2016 -0800

--
 content/get-started/quickstart/index.html | 3 ++-
 src/get-started/quickstart.md | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)
--

[2/3] incubator-beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/d1b19ee3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/d1b19ee3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/d1b19ee3

Branch: refs/heads/asf-site
Commit: d1b19ee3997c5fb1518074d715f74153c9f0d2f7
Parents: 5eaa92b
Author: Davor Bonaci 
Authored: Mon Dec 12 17:33:41 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Dec 12 17:33:41 2016 -0800

--
 content/get-started/quickstart/index.html | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/d1b19ee3/content/get-started/quickstart/index.html
--
diff --git a/content/get-started/quickstart/index.html 
b/content/get-started/quickstart/index.html
index cc5e1ed..2e2fc6a 100644
--- a/content/get-started/quickstart/index.html
+++ b/content/get-started/quickstart/index.html
@@ -247,7 +247,8 @@ You can monitor the running job by visiting the Flink 
dashboard at http://fl
 
 $ 
mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
 -Dexec.args="--runner=DataflowRunner 
--gcpTempLocation=gs://your-gcs-bucket/tmp \
- --inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs://your-gcs-bucket/counts"
+ --inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs://your-gcs-bucket/counts" \
+ -Pdataflow-runner
 
 
 

[GitHub] incubator-beam pull request #1348: Update subnetwork documentation

[sammcveety]

Github user sammcveety closed the pull request at:

https://github.com/apache/incubator-beam/pull/1348


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] incubator-beam-site git commit: Dataflow quickstart fix

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site a4eb4de53 -> b46a41fab


Dataflow quickstart fix

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/5eaa92b2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/5eaa92b2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/5eaa92b2

Branch: refs/heads/asf-site
Commit: 5eaa92b20f322a4f54214e9976632e5a30d05421
Parents: a4eb4de
Author: Jason Kuster 
Authored: Mon Dec 12 17:29:34 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Dec 12 17:33:13 2016 -0800

--
 src/get-started/quickstart.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5eaa92b2/src/get-started/quickstart.md
--
diff --git a/src/get-started/quickstart.md b/src/get-started/quickstart.md
index 76777b0..e10f9f3 100644
--- a/src/get-started/quickstart.md
+++ b/src/get-started/quickstart.md
@@ -105,7 +105,8 @@ $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.WordCount \
 ```
 $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
 -Dexec.args="--runner=DataflowRunner 
--gcpTempLocation=gs:///tmp \
- --inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs:///counts"
+ --inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs:///counts" \
+ -Pdataflow-runner
 ```
 
 

[GitHub] incubator-beam-site pull request #107: Quickstart fix

[jasonkuster]

GitHub user jasonkuster opened a pull request:

https://github.com/apache/incubator-beam-site/pull/107

Quickstart fix

R: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/incubator-beam-site quickstart-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/107.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #107


commit 8d58a4b3eb692db20b41ac8fe1d377c67c02734a
Author: Jason Kuster 
Date:   2016-12-13T01:29:34Z

Dataflow quickstart fix

Signed-off-by: Jason Kuster 

commit 45c9a751fc6e4413351c292a3d56c7b215d4457e
Author: Jason Kuster 
Date:   2016-12-13T01:31:14Z

fixup! Dataflow quickstart fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-975) Issue with MongoDBIO

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-975:
--
Fix Version/s: (was: 0.4.0-incubating)

> Issue with MongoDBIO
> 
>
> Key: BEAM-975
> URL: https://issues.apache.org/jira/browse/BEAM-975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Reza Nouri
>Assignee: Jean-Baptiste Onofré
>
> It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
> REST service that reads data from Mongo. After a number of requests, mongoIO 
> throws the following exception:
> com.mongodb.MongoSocketReadException: Prematurely reached end of stream
>   at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
>   at 
> com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
>   at 
> com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
>   at 
> com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
>   at 
> com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
>   at 
> com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
>   at 
> com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
>   at 
> com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
>   at 
> com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
>   at 
> com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
>   at 
> com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
>   at com.mongodb.Mongo.execute(Mongo.java:772)
>   at com.mongodb.Mongo$2.execute(Mongo.java:759)
>   at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
>   at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I suppose there must be a problem with Mongo connection which causes this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1141) Beam Quickstart doesn't work on DataflowRunner

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743694#comment-15743694
 ] 

Davor Bonaci commented on BEAM-1141:


Need to add a profile for the Dataflow runner, as all other runners are doing.

> Beam Quickstart doesn't work on DataflowRunner
> --
>
> Key: BEAM-1141
> URL: https://issues.apache.org/jira/browse/BEAM-1141
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Jason Kuster
>Assignee: Davor Bonaci
>
> Repro steps:
> In new dir, run current archetype:generate command:
> {code}
> mvn archetype:generate \
>   
> -DarchetypeRepository=https://repository.apache.org/content/groups/snapshots \
>   -DarchetypeGroupId=org.apache.beam \
>   -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
>   -DarchetypeVersion=LATEST \
>   -DgroupId=org.example \
>   -DartifactId=word-count-beam \
>   -Dversion="0.1" \
>   -Dpackage=org.apache.beam.examples \
>   -DinteractiveMode=false
> {code}
> Then run the sample command for DataflowRunner:
> {code}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>-Dexec.args="--runner=DataflowRunner 
> --gcpTempLocation=gs:///tmp \
> --inputFile=gs://apache-beam-samples/shakespeare/* 
> --output=gs:///counts"
> {code}
> Expected outcome: WordCount runs on Dataflow
> Actual outcome:
> {code}
> [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ word-count-beam ---
> [WARNING]
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Unknown 'runner' specified 
> 'DataflowRunner', supported pipeline runners [DirectRunner]
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1610)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:105)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:290)
>   at org.apache.beam.examples.WordCount.main(WordCount.java:174)
>   ... 6 more
> Caused by: java.lang.ClassNotFoundException: DataflowRunner
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:264)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1596)
>   ... 9 more
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 17.916 s
> [INFO] Finished at: 2016-12-12T16:38:55-08:00
> [INFO] Final Memory: 29M/266M
> [INFO] 
> 
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java 
> (default-cli) on project word-count-beam: An exception occured while 
> executing the Java class. null: InvocationTargetException: Unknown 'runner' 
> specified 'DataflowRunner', supported pipeline runners [DirectRunner] -> 
> [Help 1]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1141) Beam Quickstart doesn't work on DataflowRunner

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1141:
---
Component/s: website

> Beam Quickstart doesn't work on DataflowRunner
> --
>
> Key: BEAM-1141
> URL: https://issues.apache.org/jira/browse/BEAM-1141
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Jason Kuster
>Assignee: Davor Bonaci
>
> Repro steps:
> In new dir, run current archetype:generate command:
> {code}
> mvn archetype:generate \
>   
> -DarchetypeRepository=https://repository.apache.org/content/groups/snapshots \
>   -DarchetypeGroupId=org.apache.beam \
>   -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
>   -DarchetypeVersion=LATEST \
>   -DgroupId=org.example \
>   -DartifactId=word-count-beam \
>   -Dversion="0.1" \
>   -Dpackage=org.apache.beam.examples \
>   -DinteractiveMode=false
> {code}
> Then run the sample command for DataflowRunner:
> {code}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>-Dexec.args="--runner=DataflowRunner 
> --gcpTempLocation=gs:///tmp \
> --inputFile=gs://apache-beam-samples/shakespeare/* 
> --output=gs:///counts"
> {code}
> Expected outcome: WordCount runs on Dataflow
> Actual outcome:
> {code}
> [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ word-count-beam ---
> [WARNING]
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Unknown 'runner' specified 
> 'DataflowRunner', supported pipeline runners [DirectRunner]
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1610)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:105)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:290)
>   at org.apache.beam.examples.WordCount.main(WordCount.java:174)
>   ... 6 more
> Caused by: java.lang.ClassNotFoundException: DataflowRunner
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:264)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1596)
>   ... 9 more
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 17.916 s
> [INFO] Finished at: 2016-12-12T16:38:55-08:00
> [INFO] Final Memory: 29M/266M
> [INFO] 
> 
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java 
> (default-cli) on project word-count-beam: An exception occured while 
> executing the Java class. null: InvocationTargetException: Unknown 'runner' 
> specified 'DataflowRunner', supported pipeline runners [DirectRunner] -> 
> [Help 1]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1141) Beam Quickstart doesn't work on DataflowRunner

[Jason Kuster (JIRA)]

Jason Kuster created BEAM-1141:
--

 Summary: Beam Quickstart doesn't work on DataflowRunner
 Key: BEAM-1141
 URL: https://issues.apache.org/jira/browse/BEAM-1141
 Project: Beam
  Issue Type: Bug
Reporter: Jason Kuster
Assignee: Davor Bonaci


Repro steps:

In new dir, run current archetype:generate command:

{code}
mvn archetype:generate \
  
-DarchetypeRepository=https://repository.apache.org/content/groups/snapshots \
  -DarchetypeGroupId=org.apache.beam \
  -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
  -DarchetypeVersion=LATEST \
  -DgroupId=org.example \
  -DartifactId=word-count-beam \
  -Dversion="0.1" \
  -Dpackage=org.apache.beam.examples \
  -DinteractiveMode=false
{code}

Then run the sample command for DataflowRunner:

{code}
mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
 -Dexec.args="--runner=DataflowRunner 
--gcpTempLocation=gs:///tmp \
  --inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs:///counts"
{code}

Expected outcome: WordCount runs on Dataflow

Actual outcome:

{code}
[INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ word-count-beam ---
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Unknown 'runner' specified 
'DataflowRunner', supported pipeline runners [DirectRunner]
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1610)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:105)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:290)
at org.apache.beam.examples.WordCount.main(WordCount.java:174)
... 6 more
Caused by: java.lang.ClassNotFoundException: DataflowRunner
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1596)
... 9 more
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 17.916 s
[INFO] Finished at: 2016-12-12T16:38:55-08:00
[INFO] Final Memory: 29M/266M
[INFO] 
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java 
(default-cli) on project word-count-beam: An exception occured while executing 
the Java class. null: InvocationTargetException: Unknown 'runner' specified 
'DataflowRunner', supported pipeline runners [DirectRunner] -> [Help 1]
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1091) Python E2E WordCount test in Precommit

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-1091:
---
Summary: Python E2E WordCount test in Precommit  (was: E2E WordCount test 
in Precommit)

> Python E2E WordCount test in Precommit
> --
>
> Key: BEAM-1091
> URL: https://issues.apache.org/jira/browse/BEAM-1091
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> We want to include some e2e test in precommit in order to catch bugs earlier, 
> instead of breaking postcommit very often.
> As what we have in postcommit, we want the same wordcount test in precommit 
> executed by DirectPipelineRunner and DataflowRunner. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1521: Add reference to the >> and | operators f...

[mariapython]

Github user mariapython closed the pull request at:

https://github.com/apache/incubator-beam/pull/1521


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-863) WordCount on Windows: java.nio.file.InvalidPathException

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-863:
--
Description: 
Running basic, unchanged WordCount on Windows with local files results in an 
exception during cleanup:

Caused by: java.nio.file.InvalidPathException: Illegal char <\*> at index 46: 
C:\\dir\\beam_material\\result-temp-\* at 
sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.jav‌​a:18) at 
org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation‌​.removeTemporaryFile‌​s(FileBasedSink.java‌​:392)

More details: 
http://stackoverflow.com/questions/40285433/apache-beam-exception-while-running-wordcount-example-in-eclipse

We need to address this for the next release.

  was:
Running basic, unchanged WordCount on Windows with local files results in an 
exception during cleanup:

Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 46: 
C:\\dir\\beam_material\\result-temp-* at 
sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.jav‌​a:18) at 
org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation‌​.removeTemporaryFile‌​s(FileBasedSink.java‌​:392)

More details: 
http://stackoverflow.com/questions/40285433/apache-beam-exception-while-running-wordcount-example-in-eclipse

We need to address this for the next release.


> WordCount on Windows: java.nio.file.InvalidPathException
> 
>
> Key: BEAM-863
> URL: https://issues.apache.org/jira/browse/BEAM-863
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Davor Bonaci
>Assignee: Pei He
> Fix For: 0.4.0-incubating
>
>
> Running basic, unchanged WordCount on Windows with local files results in an 
> exception during cleanup:
> Caused by: java.nio.file.InvalidPathException: Illegal char <\*> at index 46: 
> C:\\dir\\beam_material\\result-temp-\* at 
> sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.jav‌​a:18) at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation‌​.removeTemporaryFile‌​s(FileBasedSink.java‌​:392)
> More details: 
> http://stackoverflow.com/questions/40285433/apache-beam-exception-while-running-wordcount-example-in-eclipse
> We need to address this for the next release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-843) Use New DoFn Directly in Flink Runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743596#comment-15743596
 ] 

Kenneth Knowles commented on BEAM-843:
--

Removed the version tag because I think it was just added when it was filed, 
not deliberately to block release. Aljoscha, feel free to indicate that you 
really mean to ship this now. We don't mean to remove any intentional tag.

> Use New DoFn Directly in Flink Runner
> -
>
> Key: BEAM-843
> URL: https://issues.apache.org/jira/browse/BEAM-843
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-843) Use New DoFn Directly in Flink Runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-843:
-
Fix Version/s: (was: 0.4.0-incubating)

> Use New DoFn Directly in Flink Runner
> -
>
> Key: BEAM-843
> URL: https://issues.apache.org/jira/browse/BEAM-843
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-843) Use New DoFn Directly in Flink Runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743588#comment-15743588
 ] 

Kenneth Knowles commented on BEAM-843:
--

I think the only new capabilities it would allow are not yet implemented anyhow.

> Use New DoFn Directly in Flink Runner
> -
>
> Key: BEAM-843
> URL: https://issues.apache.org/jira/browse/BEAM-843
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
> Fix For: 0.4.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-863) WordCount on Windows: java.nio.file.InvalidPathException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743585#comment-15743585
 ] 

ASF GitHub Bot commented on BEAM-863:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1587

[BEAM-863] FileBasedSink: ignore exceptions when removing temp output files 
for issues in Windows OS.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam windows-os-path

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1587.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1587


commit 1ef4eb68f5e9a8550bd45630190ac39d29e3e405
Author: Pei He 
Date:   2016-12-12T23:43:14Z

FileBasedSink: ignore exceptions when removing temp output files for issues 
in Windows OS.




> WordCount on Windows: java.nio.file.InvalidPathException
> 
>
> Key: BEAM-863
> URL: https://issues.apache.org/jira/browse/BEAM-863
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Davor Bonaci
>Assignee: Pei He
> Fix For: 0.4.0-incubating
>
>
> Running basic, unchanged WordCount on Windows with local files results in an 
> exception during cleanup:
> Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 46: 
> C:\\dir\\beam_material\\result-temp-* at 
> sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.jav‌​a:18) at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation‌​.removeTemporaryFile‌​s(FileBasedSink.java‌​:392)
> More details: 
> http://stackoverflow.com/questions/40285433/apache-beam-exception-while-running-wordcount-example-in-eclipse
> We need to address this for the next release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1587: [BEAM-863] FileBasedSink: ignore exceptio...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1587

[BEAM-863] FileBasedSink: ignore exceptions when removing temp output files 
for issues in Windows OS.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam windows-os-path

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1587.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1587


commit 1ef4eb68f5e9a8550bd45630190ac39d29e3e405
Author: Pei He 
Date:   2016-12-12T23:43:14Z

FileBasedSink: ignore exceptions when removing temp output files for issues 
in Windows OS.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1140) Apex ParDoBoundTranslatorTest.testMultiOutputParDoWithSideInputs is flaky

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1140:
--

 Summary: Apex 
ParDoBoundTranslatorTest.testMultiOutputParDoWithSideInputs is flaky
 Key: BEAM-1140
 URL: https://issues.apache.org/jira/browse/BEAM-1140
 Project: Beam
  Issue Type: Bug
  Components: runner-apex
Reporter: Eugene Kirpichov
Assignee: Thomas Weise


The test fails about one in 10-20 times or so when run in "Run until failure" 
mode in IntelliJ (I did it 3 times; once it failed on the 3rd run, once on the 
15th, once on the 40th).

It's making precommits unstable (e.g. currently 
https://github.com/apache/incubator-beam/pull/1565 failed with this test error).

Unfortunately I don't understand enough about the Apex runner to understand 
what's going on and fix this myself...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1586: [BEAM-1139] Add no-arg constructor for Un...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1586

[BEAM-1139] Add no-arg constructor for UnboundedReadFromBoundedSource

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This allows Kryo to work with the type, currently required by
the Apex runner.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam take_over_1585

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1586.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1586


commit e88e9edfcca65066d3f40d0fa49ba7386e09652f
Author: Kenneth Knowles 
Date:   2016-12-12T22:59:36Z

Add no-arg constructor for UnboundedReadFromBoundedSource

This allows Kryo to work with the type, currently required by
the Apex runner.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1139) Failures in precommit - Apex & Kryo

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743569#comment-15743569
 ] 

ASF GitHub Bot commented on BEAM-1139:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1586

[BEAM-1139] Add no-arg constructor for UnboundedReadFromBoundedSource

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This allows Kryo to work with the type, currently required by
the Apex runner.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam take_over_1585

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1586.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1586


commit e88e9edfcca65066d3f40d0fa49ba7386e09652f
Author: Kenneth Knowles 
Date:   2016-12-12T22:59:36Z

Add no-arg constructor for UnboundedReadFromBoundedSource

This allows Kryo to work with the type, currently required by
the Apex runner.




> Failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Thomas Weise
>Priority: Blocker
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1109) Python ValidatesRunner Tests on Dataflow Service Timeout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743558#comment-15743558
 ] 

ASF GitHub Bot commented on BEAM-1109:
--

Github user markflyhigh closed the pull request at:

https://github.com/apache/incubator-beam/pull/1551


> Python ValidatesRunner Tests on Dataflow Service Timeout
> 
>
> Key: BEAM-1109
> URL: https://issues.apache.org/jira/browse/BEAM-1109
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> ValidatesRunner tests timeout with following logs:
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/839/console
> Need to increase "--process-timeout" in execution command 
> (https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/run_postcommit.sh#L77).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1551: [BEAM-1109] Fix Python Postcommit Test Ti...

[markflyhigh]

Github user markflyhigh closed the pull request at:

https://github.com/apache/incubator-beam/pull/1551


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1132) Jenkins JaCoCo plugin for Beam

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743538#comment-15743538
 ] 

Davor Bonaci commented on BEAM-1132:


Yup -- something to consider. Not urgent, however.

> Jenkins JaCoCo plugin for Beam
> --
>
> Key: BEAM-1132
> URL: https://issues.apache.org/jira/browse/BEAM-1132
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>Priority: Minor
>
> Jenkins has a JaCoCo plugin and other Apache projects use it. We should try 
> it too (and might as well disable coveralls, as I don't know anyone who looks 
> at it).
> If this takes more than 10 minutes to set up, then another option is to just 
> archive JaCoCo's HTML reports so we can browse them.
> Either of these should take just minutes and yield huge benefits in 
> visibility of where we have really bad coverage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1132) Jenkins JaCoCo plugin for Beam

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1132:
---
Priority: Minor  (was: Major)

> Jenkins JaCoCo plugin for Beam
> --
>
> Key: BEAM-1132
> URL: https://issues.apache.org/jira/browse/BEAM-1132
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>Priority: Minor
>
> Jenkins has a JaCoCo plugin and other Apache projects use it. We should try 
> it too (and might as well disable coveralls, as I don't know anyone who looks 
> at it).
> If this takes more than 10 minutes to set up, then another option is to just 
> archive JaCoCo's HTML reports so we can browse them.
> Either of these should take just minutes and yield huge benefits in 
> visibility of where we have really bad coverage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1132) Jenkins JaCoCo plugin for Beam

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1132:
---
Assignee: Jason Kuster  (was: Davor Bonaci)

> Jenkins JaCoCo plugin for Beam
> --
>
> Key: BEAM-1132
> URL: https://issues.apache.org/jira/browse/BEAM-1132
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>
> Jenkins has a JaCoCo plugin and other Apache projects use it. We should try 
> it too (and might as well disable coveralls, as I don't know anyone who looks 
> at it).
> If this takes more than 10 minutes to set up, then another option is to just 
> archive JaCoCo's HTML reports so we can browse them.
> Either of these should take just minutes and yield huge benefits in 
> visibility of where we have really bad coverage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1135) Make WindowedWordCount example more independent of runner and execution style

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-1135.

Resolution: Fixed

[~kenn] just merged a PR that fixes this.

> Make WindowedWordCount example more independent of runner and execution style
> -
>
> Key: BEAM-1135
> URL: https://issues.apache.org/jira/browse/BEAM-1135
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 0.4.0-incubating
>
>
> Right now, WindowedWordCount has not received the same nice revision as our 
> other examples. The PR has been open for a bit I am breaking it out into its 
> own ticket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1135) Make WindowedWordCount example more independent of runner and execution style

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743522#comment-15743522
 ] 

ASF GitHub Bot commented on BEAM-1135:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1536


> Make WindowedWordCount example more independent of runner and execution style
> -
>
> Key: BEAM-1135
> URL: https://issues.apache.org/jira/browse/BEAM-1135
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 0.4.0-incubating
>
>
> Right now, WindowedWordCount has not received the same nice revision as our 
> other examples. The PR has been open for a bit I am breaking it out into its 
> own ticket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1139) Failures in precommit - Apex & Kryo

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743521#comment-15743521
 ] 

ASF GitHub Bot commented on BEAM-1139:
--

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1585


> Failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Thomas Weise
>Priority: Blocker
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/4] incubator-beam git commit: Add IntervalWindow coder to the standard registry

[kenn]

Add IntervalWindow coder to the standard registry


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9678b1cc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9678b1cc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9678b1cc

Branch: refs/heads/master
Commit: 9678b1cc6c799767e48ebc4d9071db099b4d135d
Parents: 59f1fb2
Author: Kenneth Knowles 
Authored: Thu Nov 3 20:44:45 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Dec 12 15:12:05 2016 -0800

--
 .../src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java| 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9678b1cc/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
index 9110de0..65f4209 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
@@ -40,6 +40,7 @@ import javax.annotation.Nullable;
 import org.apache.beam.sdk.coders.CannotProvideCoderException.ReasonCode;
 import org.apache.beam.sdk.coders.protobuf.ProtoCoder;
 import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.util.CoderUtils;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.TimestampedValue;
@@ -107,6 +108,7 @@ public class CoderRegistry implements CoderProvider {
 registerCoder(TimestampedValue.class, 
TimestampedValue.TimestampedValueCoder.class);
 registerCoder(Void.class, VoidCoder.class);
 registerCoder(byte[].class, ByteArrayCoder.class);
+registerCoder(IntervalWindow.class, IntervalWindow.getCoder());
   }
 
   /**

[1/4] incubator-beam git commit: Factor out ShardedFile from FileChecksumMatcher

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 59f1fb26a -> d9657ffc3


Factor out ShardedFile from FileChecksumMatcher


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/db41940f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/db41940f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/db41940f

Branch: refs/heads/master
Commit: db41940f977bf3315ea7e5460d188d8f9b4fa119
Parents: 9678b1c
Author: Kenneth Knowles 
Authored: Mon Dec 5 14:32:12 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Dec 12 15:12:05 2016 -0800

--
 .../beam/sdk/testing/FileChecksumMatcher.java   | 114 ++
 .../beam/sdk/util/ExplicitShardedFile.java  | 120 ++
 .../beam/sdk/util/NumberedShardedFile.java  | 220 +++
 .../org/apache/beam/sdk/util/ShardedFile.java   |  42 
 .../sdk/testing/FileChecksumMatcherTest.java|  77 ---
 .../beam/sdk/util/NumberedShardedFileTest.java  | 181 +++
 6 files changed, 581 insertions(+), 173 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/db41940f/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
index 4b249fe..82a6b71 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
@@ -21,31 +21,19 @@ package org.apache.beam.sdk.testing;
 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
 
-import com.google.api.client.util.BackOff;
-import com.google.api.client.util.BackOffUtils;
 import com.google.api.client.util.Sleeper;
-import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Strings;
-import com.google.common.collect.Lists;
 import com.google.common.hash.HashCode;
 import com.google.common.hash.Hashing;
-import com.google.common.io.CharStreams;
-import java.io.IOException;
-import java.io.Reader;
-import java.nio.channels.Channels;
 import java.nio.charset.StandardCharsets;
-import java.nio.file.Path;
-import java.nio.file.Paths;
 import java.util.ArrayList;
-import java.util.Collection;
 import java.util.List;
-import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 import javax.annotation.Nonnull;
 import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.util.FluentBackoff;
-import org.apache.beam.sdk.util.IOChannelFactory;
-import org.apache.beam.sdk.util.IOChannelUtils;
+import org.apache.beam.sdk.util.NumberedShardedFile;
+import org.apache.beam.sdk.util.ShardedFile;
 import org.hamcrest.Description;
 import org.hamcrest.TypeSafeMatcher;
 import org.joda.time.Duration;
@@ -83,9 +71,8 @@ public class FileChecksumMatcher extends 
TypeSafeMatcher
   Pattern.compile("(?x) \\S* (? \\d+) -of- (? \\d+)");
 
   private final String expectedChecksum;
-  private final String filePath;
-  private final Pattern shardTemplate;
   private String actualChecksum;
+  private final ShardedFile shardedFile;
 
   /**
* Constructor that uses default shard template.
@@ -98,7 +85,7 @@ public class FileChecksumMatcher extends 
TypeSafeMatcher
   }
 
   /**
-   * Constructor.
+   * Constructor using a custom shard template.
*
* @param checksum expected checksum string used to verify file content.
* @param filePath path of files that's to be verified.
@@ -121,8 +108,17 @@ public class FileChecksumMatcher extends 
TypeSafeMatcher
 DEFAULT_SHARD_TEMPLATE);
 
 this.expectedChecksum = checksum;
-this.filePath = filePath;
-this.shardTemplate = shardTemplate;
+this.shardedFile = new NumberedShardedFile(filePath, shardTemplate);
+  }
+
+  /**
+   * Constructor using an entirely custom {@link ShardedFile} implementation.
+   *
+   * For internal use only.
+   */
+  public FileChecksumMatcher(String expectedChecksum, ShardedFile shardedFile) 
{
+this.expectedChecksum = expectedChecksum;
+this.shardedFile = shardedFile;
   }
 
   @Override
@@ -130,9 +126,10 @@ public class FileChecksumMatcher extends 
TypeSafeMatcher
 // Load output data
 List outputs;
 try {
-  outputs = readFilesWithRetries(Sleeper.DEFAULT, 
BACK_OFF_FACTORY.backoff());
+  outputs = shardedFile.readFilesWithRetries(Sleeper.DEFAULT, 
BACK_OFF_FACTORY.backoff());
 } catch (Exception e) {
-  throw new RuntimeException(String.format("Failed to read from: %s", 

[GitHub] incubator-beam pull request #1536: [BEAM-1135] Revise WindowedWordCount exam...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1536


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/4] incubator-beam git commit: Revise WindowedWordCount for runner and execution mode portability

[kenn]

Revise WindowedWordCount for runner and execution mode portability


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/42595dcd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/42595dcd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/42595dcd

Branch: refs/heads/master
Commit: 42595dcd29c248bd3572596c9bb8464d18acd19b
Parents: db41940
Author: Kenneth Knowles 
Authored: Thu Nov 3 14:37:26 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Dec 12 15:23:38 2016 -0800

--
 .../apache/beam/examples/WindowedWordCount.java | 177 +-
 .../examples/common/WriteWindowedFilesDoFn.java |  77 
 .../beam/examples/WindowedWordCountIT.java  | 182 ---
 3 files changed, 326 insertions(+), 110 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/42595dcd/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java
index 4e254bd..5c19454 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java
@@ -17,26 +17,25 @@
  */
 package org.apache.beam.examples;
 
-import com.google.api.services.bigquery.model.TableFieldSchema;
-import com.google.api.services.bigquery.model.TableReference;
-import com.google.api.services.bigquery.model.TableRow;
-import com.google.api.services.bigquery.model.TableSchema;
 import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
+import java.util.concurrent.ThreadLocalRandom;
 import org.apache.beam.examples.common.ExampleBigQueryTableOptions;
 import org.apache.beam.examples.common.ExampleOptions;
-import org.apache.beam.examples.common.ExampleUtils;
+import org.apache.beam.examples.common.WriteWindowedFilesDoFn;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
 import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.DefaultValueFactory;
 import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
@@ -63,7 +62,8 @@ import org.joda.time.Instant;
  *   2. Adding timestamps to data
  *   3. Windowing
  *   4. Re-using PTransforms over windowed PCollections
- *   5. Writing to BigQuery
+ *   5. Accessing the window of an element
+ *   6. Writing data to per-window text files
  * 
  *
  * By default, the examples will run with the {@code DirectRunner}.
@@ -74,25 +74,23 @@ import org.joda.time.Instant;
  * 
  * See examples/java/README.md for instructions about how to configure 
different runners.
  *
- * Optionally specify the input file path via:
- * {@code --inputFile=gs://INPUT_PATH},
- * which defaults to {@code gs://apache-beam-samples/shakespeare/kinglear.txt}.
+ * To execute this pipeline locally, specify a local output file (if using 
the
+ * {@code DirectRunner}) or output prefix on a supported distributed file 
system.
+ * {@code
+ *   --output=[YOUR_LOCAL_FILE | YOUR_OUTPUT_PREFIX]
+ * }
  *
- * Specify an output BigQuery dataset and optionally, a table for the 
output. If you don't
- * specify the table, one will be created for you using the job name. If you 
don't specify the
- * dataset, a dataset called {@code beam_examples} must already exist in your 
project.
- * {@code --bigQueryDataset=YOUR-DATASET --bigQueryTable=YOUR-NEW-TABLE-NAME}.
+ * The input file defaults to a public data set containing the text of of 
King Lear,
+ * by William Shakespeare. You can override it and choose your own input with 
{@code --inputFile}.
  *
  * By default, the pipeline will do fixed windowing, on 1-minute windows.  
You can
  * change this interval by setting the {@code --windowSize} parameter, e.g. 
{@code --windowSize=10}
  * for 10-minute windows.
  *
- * The example will try to cancel the pipelines on the signal to terminate 
the process 

[4/4] incubator-beam git commit: This closes #1536

[kenn]

This closes #1536


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d9657ffc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d9657ffc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d9657ffc

Branch: refs/heads/master
Commit: d9657ffc37490b063835672e0b5287b4d18aba96
Parents: 59f1fb2 42595dc
Author: Kenneth Knowles 
Authored: Mon Dec 12 15:33:29 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Dec 12 15:33:29 2016 -0800

--
 .../apache/beam/examples/WindowedWordCount.java | 177 ---
 .../examples/common/WriteWindowedFilesDoFn.java |  77 +++
 .../beam/examples/WindowedWordCountIT.java  | 182 ---
 .../apache/beam/sdk/coders/CoderRegistry.java   |   2 +
 .../beam/sdk/testing/FileChecksumMatcher.java   | 114 ++
 .../beam/sdk/util/ExplicitShardedFile.java  | 120 ++
 .../beam/sdk/util/NumberedShardedFile.java  | 220 +++
 .../org/apache/beam/sdk/util/ShardedFile.java   |  42 
 .../sdk/testing/FileChecksumMatcherTest.java|  77 ---
 .../beam/sdk/util/NumberedShardedFileTest.java  | 181 +++
 10 files changed, 909 insertions(+), 283 deletions(-)
--

[GitHub] incubator-beam pull request #1585: [BEAM-1139] Add no-arg constructor for Un...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1585


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1139) Failures in precommit - Apex & Kryo

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1139:
--
Assignee: Thomas Weise  (was: Kenneth Knowles)

> Failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Thomas Weise
>Priority: Blocker
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1799

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1799

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #865

[Apache Jenkins Server]

See 

--
[...truncated 2748 lines...]
  "component_encodings": [
{
  "@type": 
"PickleCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwBmcnZOanOIA5XIYNmYyFjbSFTkh4A9DYR/g==",
 
  "component_encodings": []
}, 
{
  "@type": 
"PickleCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwBmcnZOanOIA5XIYNmYyFjbSFTkh4A9DYR/g==",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": "ViewAsSingleton(side.None)/CreatePCollectionView.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"user_name": "ViewAsSingleton(side.None)/CreatePCollectionView"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s4", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {
  "s3": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "s3"
  }
}, 
"output_info": [
  {
"encoding": {
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
}, 
{
  "@type": "SingletonCoder$", 
  "component_encodings": []
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "compute.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s1"
}, 
"serialized_fn": "", 
"user_name": "compute"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2016-12-12_14_37_00-14980560058396649741]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2016-12-12_14_37_00-14980560058396649741
root: INFO: Job 2016-12-12_14_37_00-14980560058396649741 is in state 
JOB_STATE_RUNNING
root: INFO: 2016-12-12_14_37_00-14980560058396649741_0158f52e97f6: 
2016-12-12T22:37:01.302Z: JOB_MESSAGE_DETAILED: (d532e8081463d3f0): Checking 
required Cloud APIs are enabled.
root: INFO: 2016-12-12_14_37_00-14980560058396649741_0158f52e9978: 
2016-12-12T22:37:01.688Z: JOB_MESSAGE_DETAILED: (d532e8081463d065): Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2016-12-12_14_37_00-14980560058396649741_0158f52e997b: 
2016-12-12T22:37:01.691Z: JOB_MESSAGE_DETAILED: (d532e8081463d87f): Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2016-12-12_14_37_00-14980560058396649741_0158f52e997d: 
2016-12-12T22:37:01.693Z: JOB_MESSAGE_DETAILED: (d532e8081463d099): Annotating 
graph with Autotuner information.
root: INFO: 

[jira] [Commented] (BEAM-1139) Failures in precommit - Apex & Kryo

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743450#comment-15743450
 ] 

ASF GitHub Bot commented on BEAM-1139:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1585

[BEAM-1139] Add no-arg constructor for UnboundedReadFromBoundedSource

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This allows Kryo to work with the type, currently required by the Apex 
runner.

This seems to be failing in precommit reliably.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam build-break

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1585.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1585


commit 4124f9b5cd44c78b6e6a892ed5ecd6b13bc1a23a
Author: Kenneth Knowles 
Date:   2016-12-12T22:59:36Z

Add no-arg constructor for UnboundedReadFromBoundedSource

This allows Kryo to work with the type, currently required by
the Apex runner.




> Failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1585: [BEAM-1139] Add no-arg constructor for Un...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1585

[BEAM-1139] Add no-arg constructor for UnboundedReadFromBoundedSource

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This allows Kryo to work with the type, currently required by the Apex 
runner.

This seems to be failing in precommit reliably.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam build-break

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1585.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1585


commit 4124f9b5cd44c78b6e6a892ed5ecd6b13bc1a23a
Author: Kenneth Knowles 
Date:   2016-12-12T22:59:36Z

Add no-arg constructor for UnboundedReadFromBoundedSource

This allows Kryo to work with the type, currently required by
the Apex runner.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1124) Python ValidateRunner Test test_multi_valued_singleton_side_input Break Postcommit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743425#comment-15743425
 ] 

ASF GitHub Bot commented on BEAM-1124:
--

Github user markflyhigh closed the pull request at:

https://github.com/apache/incubator-beam/pull/1571


> Python ValidateRunner Test test_multi_valued_singleton_side_input Break 
> Postcommit
> --
>
> Key: BEAM-1124
> URL: https://issues.apache.org/jira/browse/BEAM-1124
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-py, testing
>Reporter: Mark Liu
>Assignee: Ahmet Altay
>
> Python test_multi_valued_singleton_side_input test, a ValidatesRunner test 
> that running on dataflow service, failed and broke 
> postcommit(https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/853/).
> Here is the stack trace:
> {code}
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/dataflow_test.py",
>  line 186, in test_multi_valued_singleton_side_input
> pipeline.run()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 159, in run
> return self.runner.run(self)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow_runner.py",
>  line 195, in run
> % getattr(self, 'last_error_msg', None), self.result)
> DataflowRuntimeException: Dataflow pipeline failed:
> (99aeafa7a8dffcc7): Traceback (most recent call last):
>   File 
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 
> 514, in do_work
> work_executor.execute()
>   File "dataflow_worker/executor.py", line 892, in 
> dataflow_worker.executor.MapTaskExecutor.execute 
> (dataflow_worker/executor.c:24008)
> op.start()
>   File "dataflow_worker/executor.py", line 456, in 
> dataflow_worker.executor.DoOperation.start (dataflow_worker/executor.c:13870)
> def start(self):
>   File "dataflow_worker/executor.py", line 483, in 
> dataflow_worker.executor.DoOperation.start (dataflow_worker/executor.c:13685)
> self.dofn_runner = common.DoFnRunner(
>   File "apache_beam/runners/common.py", line 89, in 
> apache_beam.runners.common.DoFnRunner.__init__ 
> (apache_beam/runners/common.c:3469)
> args, kwargs, [side_input[global_window]
>   File 
> "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py",
>  line 192, in __getitem__
> _FilteringIterable(self._iterable, target_window), self._view_options)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/pvalue.py", line 
> 279, in _from_runtime_iterable
> 'PCollection with more than one element accessed as '
> ValueError: PCollection with more than one element accessed as a singleton 
> view.
> {code}
> Worker logs in here:
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/853/console
> In order to temporarily ignore this test in postcommit, we can comment out 
> annotation "@attr('ValidatesRunner')" of this test. Then it will only run as 
> a unit test (execute by DirectRunner), but not run as a ValidatesRunner test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1571: [BEAM-1124] Temporarily Ignore a Validate...

[markflyhigh]

Github user markflyhigh closed the pull request at:

https://github.com/apache/incubator-beam/pull/1571


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1139) Failures in precommit - Apex & Kryo

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1139:
-

 Summary: Failures in precommit - Apex & Kryo
 Key: BEAM-1139
 URL: https://issues.apache.org/jira/browse/BEAM-1139
 Project: Beam
  Issue Type: Improvement
  Components: runner-apex
Reporter: Kenneth Knowles
Assignee: Thomas Weise
Priority: Blocker


https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/

This is not necessarily a bug in the Apex runner, but it looks like this class 
cannot be serialized via Kryo while the Apex runner needs it to be. Probably 
the fix is to roll-forwards a simple change to make it Kryo serializable.

It is not clear to me the difference between this test run and others. Clearly 
there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-1124] Temporarily Ignore a ValidatesRunnerTest That Broke Postcommit

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk b265dceaa -> f086afe12


[BEAM-1124] Temporarily Ignore a ValidatesRunnerTest That Broke Postcommit


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0a70e581
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0a70e581
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0a70e581

Branch: refs/heads/python-sdk
Commit: 0a70e581bef4258dfb18f3c5db8f9a9369ab13e8
Parents: b265dce
Author: Mark Liu 
Authored: Fri Dec 9 16:55:45 2016 -0800
Committer: Robert Bradshaw 
Committed: Mon Dec 12 14:19:57 2016 -0800

--
 sdks/python/apache_beam/dataflow_test.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0a70e581/sdks/python/apache_beam/dataflow_test.py
--
diff --git a/sdks/python/apache_beam/dataflow_test.py 
b/sdks/python/apache_beam/dataflow_test.py
index ba3553a..f410230 100644
--- a/sdks/python/apache_beam/dataflow_test.py
+++ b/sdks/python/apache_beam/dataflow_test.py
@@ -176,7 +176,9 @@ class DataflowTest(unittest.TestCase):
 assert_that(result, equal_to([(1, 'empty'), (2, 'empty')]))
 pipeline.run()
 
-  @attr('ValidatesRunner')
+  # @attr('ValidatesRunner')
+  # TODO(BEAM-1124): Temporarily disable it due to test failed running on
+  # Dataflow service.
   def test_multi_valued_singleton_side_input(self):
 pipeline = TestPipeline()
 pcol = pipeline | 'start' >> Create([1, 2])

[2/2] incubator-beam git commit: Closes #1571

[robertwb]

Closes #1571


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f086afe1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f086afe1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f086afe1

Branch: refs/heads/python-sdk
Commit: f086afe125cc413e7626ab186268d5bb3c067a89
Parents: b265dce 0a70e58
Author: Robert Bradshaw 
Authored: Mon Dec 12 14:19:58 2016 -0800
Committer: Robert Bradshaw 
Committed: Mon Dec 12 14:19:58 2016 -0800

--
 sdks/python/apache_beam/dataflow_test.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--

[GitHub] incubator-beam pull request #1584: DO NOT REVIEW. Debugging MqttIO failures.

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1584

DO NOT REVIEW. Debugging MqttIO failures.

(can't reproduce the error locally, and don't know how to run something on 
Travis - so creating a PR just for the sake of getting a Travis run)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam mqttio-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1584.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1584


commit 234916835fcd83a3a43409e81c94880d0988dde9
Author: Jean-Baptiste Onofré 
Date:   2016-09-12T16:49:36Z

[BEAM-606] Create MqttIO

commit 6ed144825fcfc978103c5cc8c96546ae283f86b2
Author: Jean-Baptiste Onofré 
Date:   2016-09-23T21:27:40Z

[BEAM-606] Use AutoValue, remove checkpoint, introduce BlockingQueue and 
improve tests.

commit 9fbee3fc87f92d2bd43d5dc79cc277d8e648ebb4
Author: Jean-Baptiste Onofré 
Date:   2016-10-05T10:28:44Z

[BEAM-606] Set topic required. Move some subscriber specific MQTT code on 
the reader. Improve tests.

commit 6f4713006f9062f42bad9643ce9dbf115c7af119
Author: Jean-Baptiste Onofré 
Date:   2016-10-05T14:46:24Z

[BEAM-606] Replace KV internal use with a regular POJO

commit 6bb48572c8cb2099d3782de5465eddade0ff3326
Author: Jean-Baptiste Onofré 
Date:   2016-10-05T14:55:43Z

[BEAM-606] Use a better value for watermark

commit e02b2a25d4698eafda492192c339246db9f670af
Author: Jean-Baptiste Onofré 
Date:   2016-10-10T19:12:37Z

[BEAM-606] Validation is now performed only in create(), maxNumRecords and 
maxReadTime are now exclusive, use ByteArrayCoder instead of SerializableCoder, 
use poll instead of take on the blocking queue, add javadoc on the with* 
methods, code cleanup

commit 775a066625f5f250404ad2571aabf6e278aed662
Author: Jean-Baptiste Onofré 
Date:   2016-10-13T07:44:00Z

[BEAM-606] Rename connection configuration, improve javadoc

commit 352b5fb279fe7dfd2d5a80bc023cbe7c2106eb1c
Author: Jean-Baptiste Onofré 
Date:   2016-10-13T08:41:52Z

[BEAM-606] Refactore read test to be more reliable

commit 42ff71deaf0bdedf9ee62f25d21aeb5efff2ec9b
Author: Jean-Baptiste Onofré 
Date:   2016-10-21T13:50:44Z

[BEAM-606] Update documentation, change the QoS in the test

commit 532704604658ac081eb16611958e4ad7419b6855
Author: Jean-Baptiste Onofré 
Date:   2016-11-02T06:01:39Z

[BEAM-606] Start ActiveMQ broker on free network port in the tests

commit dda9092d1e2fb17c80641d3490de5a7d4ae242a5
Author: Jean-Baptiste Onofré 
Date:   2016-11-02T09:38:55Z

[BEAM-606] Fix parent pom version

commit b92eb296a8755f2f92bbfa3f1ad9acc12d6ea080
Author: Jean-Baptiste Onofré 
Date:   2016-11-03T08:50:00Z

[BEAM-606] Exclude support of QoS 2, cleanup in javadoc

commit 75d24726ec9ffb71844c760a3f321ebab0e7a228
Author: Jean-Baptiste Onofré 
Date:   2016-11-14T12:07:01Z

[BEAM-606] Add core MQTT test

commit fc1195ad917144ed2ac829ea8f8c3df620e0cbb4
Author: Jean-Baptiste Onofré 
Date:   2016-11-22T16:58:14Z

[BEAM-606] Improved checkpoint with manual ack, update comments about QoS 
(relationship between publisher and subsribers), auto generate clientId 
(reliability)

commit f7063fbc9343101a921715082b7cc06da0184483
Author: Jean-Baptiste Onofré 
Date:   2016-12-05T10:14:41Z

[BEAM-606] Fix checkpoint mark

commit 7d9b9cdcd155cdd35082095887728afa9372e500
Author: Jean-Baptiste Onofré 
Date:   2016-12-09T17:14:07Z

[BEAM-606] Only support QoS 1 in Write, updated to last SDK changes

commit aed01b9696f2bb87eb01d8477b2d322b24fb7119
Author: Jean-Baptiste Onofré 
Date:   2016-12-10T15:56:35Z

[BEAM-606] Use ActiveMQ KahaDB in test for persistence adapter

commit e77fea21d065cf926506c1a49f05f3c9d7101d21
Author: Eugene Kirpichov 
Date:   2016-12-12T22:04:26Z

For debugging: dont drop stack in UserCodeException




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1583: Implement GetDefaultOutputCoder in Direct...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1583

Implement GetDefaultOutputCoder in DirectGroupByKey

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This uses the standard Coder Inference path to set coders, rather than
explicitly setting the output coders for intermediate PCollections.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
coder_inference_direct_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1583.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1583


commit 06796b2396ea5d56cd4c75e82a7f13a2d5ad52aa
Author: Thomas Groh 
Date:   2016-12-12T21:55:49Z

Implement GetDefaultOutputCoder in DirectGroupByKey

This uses the standard Coder Inference path to set coders, rather than
explicitly setting the output coders for intermediate PCollections.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Stop expanding PValues in DirectRunner visitors

[tgroh]

Stop expanding PValues in DirectRunner visitors

A PValue always expands to itself, and these calls are unneccessary.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8ef74a74
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8ef74a74
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8ef74a74

Branch: refs/heads/master
Commit: 8ef74a744327c40fbb05030fd7657db8a865cb94
Parents: 5d619e8
Author: Thomas Groh 
Authored: Fri Dec 9 15:52:15 2016 -0800
Committer: Thomas Groh 
Committed: Mon Dec 12 13:51:29 2016 -0800

--
 .../beam/runners/direct/DirectGraphVisitor.java   | 14 ++
 .../runners/direct/KeyedPValueTrackingVisitor.java|  2 +-
 2 files changed, 7 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8ef74a74/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraphVisitor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraphVisitor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraphVisitor.java
index 4f38bce..0283d03 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraphVisitor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraphVisitor.java
@@ -99,14 +99,12 @@ class DirectGraphVisitor extends PipelineVisitor.Defaults {
 if (!producers.containsKey(value)) {
   producers.put(value, appliedTransform);
 }
-for (PValue expandedValue : value.expand()) {
-  if (expandedValue instanceof PCollectionView) {
-views.add((PCollectionView) expandedValue);
-  }
-  if (!producers.containsKey(expandedValue)) {
-producers.put(value, appliedTransform);
-  }
-}
+   if (value instanceof PCollectionView) {
+ views.add((PCollectionView) value);
+   }
+   if (!producers.containsKey(value)) {
+ producers.put(value, appliedTransform);
+   }
   }
 
   private AppliedPTransform 
getAppliedTransform(TransformHierarchy.Node node) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8ef74a74/runners/direct-java/src/main/java/org/apache/beam/runners/direct/KeyedPValueTrackingVisitor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/KeyedPValueTrackingVisitor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/KeyedPValueTrackingVisitor.java
index 4161f9e..7f85169 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/KeyedPValueTrackingVisitor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/KeyedPValueTrackingVisitor.java
@@ -84,7 +84,7 @@ class KeyedPValueTrackingVisitor implements PipelineVisitor {
   @Override
   public void visitValue(PValue value, TransformHierarchy.Node producer) {
 if (producesKeyedOutputs.contains(producer.getTransform().getClass())) {
-  keyedValues.addAll(value.expand());
+  keyedValues.add(value);
 }
   }
 

[GitHub] incubator-beam pull request #1570: Stop expanding PValues in DirectRunner vi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1570


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: This closes #1570

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5d619e8e3 -> 59f1fb26a


This closes #1570


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/59f1fb26
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/59f1fb26
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/59f1fb26

Branch: refs/heads/master
Commit: 59f1fb26a900b4e4cbdb4d16b94cfe646b1c5f7e
Parents: 5d619e8 8ef74a7
Author: Thomas Groh 
Authored: Mon Dec 12 13:51:29 2016 -0800
Committer: Thomas Groh 
Committed: Mon Dec 12 13:51:29 2016 -0800

--
 .../beam/runners/direct/DirectGraphVisitor.java   | 14 ++
 .../runners/direct/KeyedPValueTrackingVisitor.java|  2 +-
 2 files changed, 7 insertions(+), 9 deletions(-)
--

[jira] [Commented] (BEAM-646) Get runners out of the apply()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743239#comment-15743239
 ] 

ASF GitHub Bot commented on BEAM-646:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1582

[BEAM-646] Add Parameters to finishSpecifying

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
This removes use of getProducingTransformInternal() in TypedPValue.

Ensure that all nodes are finished specifying before a call to
`Pipeline#traverseTopologically` or `PipelineRunner#run`. This ensures
that all nodes are fully specified without requiring the `PipelineRunner`
to do so explicitly.

Use Coder Inference rather than explicitly setting Coders within
DirectRunner overrides.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
parametered_finish_specifying

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1582.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1582


commit 05d3151de529e1db8ffe0e16ea03e4d2674b4696
Author: Thomas Groh 
Date:   2016-12-08T22:33:36Z

Improve Coder Inference within DirectRunner Overrides

This cleans up how coders are provided for DirectRunner overrides.

Ensure that coders are provided within tests that use an empty Flatten.

commit 97d81855b7dd54b0362121fe2b7d92aafb7d6594
Author: Thomas Groh 
Date:   2016-12-08T17:09:33Z

Add Parameters to finishSpecifying

Remove the need to use getProducingTransformInternal in TypedPValue.




> Get runners out of the apply()
> --
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now, the runner intercepts calls to apply() and replaces transforms as 
> we go. This means that there is no "original" user graph. For portability and 
> misc architectural benefits, we would like to build the original graph first, 
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more 
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline 
> object.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1582: [BEAM-646] Add Parameters to finishSpecif...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1582

[BEAM-646] Add Parameters to finishSpecifying

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
This removes use of getProducingTransformInternal() in TypedPValue.

Ensure that all nodes are finished specifying before a call to
`Pipeline#traverseTopologically` or `PipelineRunner#run`. This ensures
that all nodes are fully specified without requiring the `PipelineRunner`
to do so explicitly.

Use Coder Inference rather than explicitly setting Coders within
DirectRunner overrides.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
parametered_finish_specifying

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1582.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1582


commit 05d3151de529e1db8ffe0e16ea03e4d2674b4696
Author: Thomas Groh 
Date:   2016-12-08T22:33:36Z

Improve Coder Inference within DirectRunner Overrides

This cleans up how coders are provided for DirectRunner overrides.

Ensure that coders are provided within tests that use an empty Flatten.

commit 97d81855b7dd54b0362121fe2b7d92aafb7d6594
Author: Thomas Groh 
Date:   2016-12-08T17:09:33Z

Add Parameters to finishSpecifying

Remove the need to use getProducingTransformInternal in TypedPValue.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1556: [BEAM-23] Refactors to support key-preser...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1556


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---