[jira] [Updated] (BEAM-1187) GCP Transport not performing timed backoff after connection failure
[ https://issues.apache.org/jira/browse/BEAM-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-1187: --- Assignee: Pei He (was: Davor Bonaci) > GCP Transport not performing timed backoff after connection failure > --- > > Key: BEAM-1187 > URL: https://issues.apache.org/jira/browse/BEAM-1187 > Project: Beam > Issue Type: Bug > Components: runner-dataflow, sdk-java-core, sdk-java-gcp >Reporter: Luke Cwik >Assignee: Pei He >Priority: Minor > > The http request retries are failing and seemingly being immediately retried > if there is a connection exception. Note that below all the times are the > same, and also that we are logging too much. This seems to be related to the > interaction by the chaining http request initializer combining the Credential > initializer followed by the RetryHttpRequestInitializer. Also, note that we > never log "Request failed with IOException, will NOT retry" which implies > that the retry logic never made it to the RetryHttpRequestInitializer. > Action items are: > 1) Ensure that the RetryHttpRequestInitializer is used > 2) Ensure that calls do backoff > 3) Reduce the logging to one terminal statement saying that we retried X > times and final failure was YYY. > Dump of console output: > Dec 20, 2016 9:12:20 AM > com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner fromOptions > INFO: PipelineOptions.filesToStage was not specified. Defaulting to files > from the classpath: will stage 1 files. Enable logging at DEBUG level to see > which files will be staged. > Dec 20, 2016 9:12:21 AM > com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner run > INFO: Executing pipeline on the Dataflow Service, which will have billing > implications related to Google Compute Engine usage and other Google Cloud > Services. > Dec 20, 2016 9:12:21 AM com.google.cloud.dataflow.sdk.util.PackageUtil > stageClasspathElements > INFO: Uploading 1 files from PipelineOptions.filesToStage to staging location > to prepare for execution. > Dec 20, 2016 9:12:21 AM com.google.cloud.dataflow.sdk.util.PackageUtil > stageClasspathElements > INFO: Uploading PipelineOptions.filesToStage complete: 1 files newly > uploaded, 0 files cached > Dec 20, 2016 9:12:22 AM com.google.api.client.http.HttpRequest execute > WARNING: exception thrown while executing request > java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:589) > at sun.net.NetworkClient.doConnect(NetworkClient.java:175) > at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) > at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) > at sun.net.www.http.HttpClient.(HttpClient.java:211) > at sun.net.www.http.HttpClient.New(HttpClient.java:308) > at sun.net.www.http.HttpClient.New(HttpClient.java:326) > at > sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169) > at > sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) > at > sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) > at > sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933) > at > sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283) > at > sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258) > at > com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77) > at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981) > at > com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) > at > com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) > at > com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) > at > com.google.cloud.dataflow.sdk.r
[jira] [Updated] (BEAM-1184) Add integration tests for ElasticsearchIO
[ https://issues.apache.org/jira/browse/BEAM-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-1184: --- Assignee: Stephen Sisk (was: Davor Bonaci) > Add integration tests for ElasticsearchIO > - > > Key: BEAM-1184 > URL: https://issues.apache.org/jira/browse/BEAM-1184 > Project: Beam > Issue Type: Test > Components: sdk-java-core >Reporter: Stephen Sisk >Assignee: Stephen Sisk > > In https://github.com/apache/incubator-beam/pull/1439#pullrequestreview, the > original PR included a dockerfile and an integration test that ran against an > ES instance in that dockerfile. > We are working on infrastructure to stand up docker containers for this kind > of testing, but aren't quite ready for it, so we pulled the scripts, > dockerfile+IT out of that PR. This issue tracks checking that work in once we > can take advantage of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
svn commit: r17413 - /dev/incubator/beam/0.4.0-incubating/
Author: davor Date: Fri Dec 16 01:14:10 2016 New Revision: 17413 Log: Update Apache Beam, version 0.4.0-incubating, to RC#3. Modified: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 Modified: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip == Binary files - no diff available. Modified: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc == --- dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc (original) +++ dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc Fri Dec 16 01:14:10 2016 @@ -1,11 +1,11 @@ -BEGIN PGP SIGNATURE- Version: GnuPG v1 -iQEcBAABAgAGBQJYUwWjAAoJEMkEN+GPDTNPfrkH/j1yLtwxDIKd5R199iKDYzoU -o7imDtwFFvLbex7npObvwZeW5LRJ/Fkt+KOmZMCb03imFYZCuUeUZTvEJnuk5jM5 -scjcm8qBds0ih+7e/nQ31CSXfeT8opK9r/9UGPalAb/2cIhz2M37zDSTBvtnq9/B -KT/cCrWc58HfiZ7sxNmal0c1ziKZN7XJnZ3KXixAlN+n9cO+FrsRiExtXXW5u3Qo -QGlxY5qms/uPYJ7FfM03qytLUQ8xltVrRfPIZzC2lQ9BQ/9YMo3V8q7qAYrFCn/z -WZmyp8Ofb/FGjra/OPCDHYGRhmrIhQdqYTdiGmIQmL8+zj+U5n2YjSxXEO25Zh8= -=MQyA +iQEcBAABAgAGBQJYUzpvAAoJEMkEN+GPDTNPSh0H/irdPLgrZ+9Heixo4UhB9m1p +74bljmblc2XHPryT46fvrbT81aYgOj9LyIuvKzM0O1JLocaVhiHdiMzQDZVEFXMJ +6WnQ1SbGaK0lfMCLF+9N28cb7MtZaY4wxPxFOI5Oxwvh11uJzmSpnGVTnqd26sBA +Tr26rRVPLudZZPUQ6caSjh+9u437kXNBcPAU+htEPmeSidJlmSWqrxSFkb79iyTA +RDXmm2r+2m5cNjJPLdqMHdpiFPs6BGBRlS2oWNvEX5aO7xEbVq/qv6SRUWAqCrZE +PhlRtMeeSrKuDLjnmpRX/DCzfi1bz4qIiXK+lNdiOmXLhyxRY1Z8yd/5dTAbJpU= +=OrGH -END PGP SIGNATURE- Modified: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 == --- dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 (original) +++ dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 Fri Dec 16 01:14:10 2016 @@ -1 +1 @@ -53fde7af8554145e413935b6ef0c29ce apache-beam-0.4.0-incubating-source-release.zip +2b39170726302eb85e16ac87f71909dd apache-beam-0.4.0-incubating-source-release.zip Modified: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 == --- dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 (original) +++ dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 Fri Dec 16 01:14:10 2016 @@ -1 +1 @@ -8863b82a3947ed718aea8ff9367f50f6dd5f1c30 apache-beam-0.4.0-incubating-source-release.zip +861a6f87370c872afabc104bdcbb4f584b891c96 apache-beam-0.4.0-incubating-source-release.zip
svn commit: r17411 - /dev/incubator/beam/0.4.0-incubating/
Author: davor Date: Thu Dec 15 21:58:20 2016 New Revision: 17411 Log: Update Apache Beam version 0.4.0-incubating to RC#2. Modified: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 Modified: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip == Binary files - no diff available. Modified: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc == --- dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc (original) +++ dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc Thu Dec 15 21:58:20 2016 @@ -1,11 +1,11 @@ -BEGIN PGP SIGNATURE- Version: GnuPG v1 -iQEcBAABAgAGBQJYUgRnAAoJEMkEN+GPDTNPQSAH/1RntayJB6+YMCHV0s5ABRL0 -/FSYTdRb6k24D14olLkNJ+LAjEyou6hn5ZPpVR8AZ+wM/QE+TmGwDYTbZslHcG1l -Ncudyts8D4fDyy1KK1lpnDMDMYajg9MbgQMvVr5Bqv0ORLZf7eAcNAjtBa9+aNGY -pEVaehqHKcs8ubjbrVFACaT/BlmhkgExzwmnYr2orDRT8YR3FjMuXFWlq+lNI1TQ -BsJeRfz/SisdSV6IdZpIyx0+TXW/vbmyU5iIH8KK7v28QpMXQGZ+28TkXREvCuk3 -xhyIF5PK4dR2JKNsvm2w0XyJigSuIO2kJGb2pgKs8lA95UC50uy/ZGkvqSg2zbM= -=6Aq9 +iQEcBAABAgAGBQJYUwWjAAoJEMkEN+GPDTNPfrkH/j1yLtwxDIKd5R199iKDYzoU +o7imDtwFFvLbex7npObvwZeW5LRJ/Fkt+KOmZMCb03imFYZCuUeUZTvEJnuk5jM5 +scjcm8qBds0ih+7e/nQ31CSXfeT8opK9r/9UGPalAb/2cIhz2M37zDSTBvtnq9/B +KT/cCrWc58HfiZ7sxNmal0c1ziKZN7XJnZ3KXixAlN+n9cO+FrsRiExtXXW5u3Qo +QGlxY5qms/uPYJ7FfM03qytLUQ8xltVrRfPIZzC2lQ9BQ/9YMo3V8q7qAYrFCn/z +WZmyp8Ofb/FGjra/OPCDHYGRhmrIhQdqYTdiGmIQmL8+zj+U5n2YjSxXEO25Zh8= +=MQyA -END PGP SIGNATURE- Modified: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 == --- dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 (original) +++ dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 Thu Dec 15 21:58:20 2016 @@ -1 +1 @@ -120fb9d0d04dd598188d78cc720e1ee4 apache-beam-0.4.0-incubating-source-release.zip +53fde7af8554145e413935b6ef0c29ce apache-beam-0.4.0-incubating-source-release.zip Modified: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 == --- dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 (original) +++ dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 Thu Dec 15 21:58:20 2016 @@ -1 +1 @@ -10df1dcc63cc891946a55456608ebb1e188336c1 apache-beam-0.4.0-incubating-source-release.zip +8863b82a3947ed718aea8ff9367f50f6dd5f1c30 apache-beam-0.4.0-incubating-source-release.zip
svn commit: r17390 - /dev/incubator/beam/0.4.0-incubating/
Author: davor Date: Thu Dec 15 03:31:02 2016 New Revision: 17390 Log: Add RC#1 for Apache Beam, version 0.4.0-incubating. Added: dev/incubator/beam/0.4.0-incubating/ dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip (with props) dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 Added: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip == Binary file - no diff available. Propchange: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc == --- dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc (added) +++ dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.asc Thu Dec 15 03:31:02 2016 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- +Version: GnuPG v1 + +iQEcBAABAgAGBQJYUgRnAAoJEMkEN+GPDTNPQSAH/1RntayJB6+YMCHV0s5ABRL0 +/FSYTdRb6k24D14olLkNJ+LAjEyou6hn5ZPpVR8AZ+wM/QE+TmGwDYTbZslHcG1l +Ncudyts8D4fDyy1KK1lpnDMDMYajg9MbgQMvVr5Bqv0ORLZf7eAcNAjtBa9+aNGY +pEVaehqHKcs8ubjbrVFACaT/BlmhkgExzwmnYr2orDRT8YR3FjMuXFWlq+lNI1TQ +BsJeRfz/SisdSV6IdZpIyx0+TXW/vbmyU5iIH8KK7v28QpMXQGZ+28TkXREvCuk3 +xhyIF5PK4dR2JKNsvm2w0XyJigSuIO2kJGb2pgKs8lA95UC50uy/ZGkvqSg2zbM= +=6Aq9 +-END PGP SIGNATURE- Added: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 == --- dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 (added) +++ dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.md5 Thu Dec 15 03:31:02 2016 @@ -0,0 +1 @@ +120fb9d0d04dd598188d78cc720e1ee4 apache-beam-0.4.0-incubating-source-release.zip Added: dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 == --- dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 (added) +++ dev/incubator/beam/0.4.0-incubating/apache-beam-0.4.0-incubating-source-release.zip.sha1 Thu Dec 15 03:31:02 2016 @@ -0,0 +1 @@ +10df1dcc63cc891946a55456608ebb1e188336c1 apache-beam-0.4.0-incubating-source-release.zip
incubator-beam git commit: [maven-release-plugin] rollback changes from release preparation of v0.4.0-incubating-RC3
Repository: incubator-beam Updated Branches: refs/heads/release-0.4.0-incubating 3d67e7a1f -> 93045e9c1 [maven-release-plugin] rollback changes from release preparation of v0.4.0-incubating-RC3 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/93045e9c Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/93045e9c Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/93045e9c Branch: refs/heads/release-0.4.0-incubating Commit: 93045e9c1b99b540bed2b3d93d568fd18ec9423e Parents: 3d67e7a Author: Davor Bonaci Authored: Thu Dec 15 17:01:44 2016 -0800 Committer: Davor Bonaci Committed: Thu Dec 15 17:01:44 2016 -0800 -- examples/java/pom.xml | 2 +- examples/java8/pom.xml| 2 +- examples/pom.xml | 2 +- pom.xml | 4 ++-- runners/apex/pom.xml | 2 +- runners/core-java/pom.xml | 2 +- runners/direct-java/pom.xml | 2 +- runners/flink/examples/pom.xml| 2 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml | 2 +- runners/google-cloud-dataflow-java/pom.xml| 2 +- runners/pom.xml | 2 +- runners/spark/pom.xml | 2 +- sdks/java/build-tools/pom.xml | 2 +- sdks/java/core/pom.xml| 2 +- sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml | 2 +- sdks/java/extensions/sorter/pom.xml | 2 +- sdks/java/io/google-cloud-platform/pom.xml| 2 +- sdks/java/io/hdfs/pom.xml | 2 +- sdks/java/io/jdbc/pom.xml | 2 +- sdks/java/io/jms/pom.xml | 2 +- sdks/java/io/kafka/pom.xml| 2 +- sdks/java/io/kinesis/pom.xml | 2 +- sdks/java/io/mongodb/pom.xml | 2 +- sdks/java/io/pom.xml | 2 +- sdks/java/java8tests/pom.xml | 2 +- sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +- sdks/java/maven-archetypes/examples/pom.xml | 2 +- sdks/java/maven-archetypes/pom.xml| 2 +- sdks/java/maven-archetypes/starter/pom.xml| 2 +- sdks/java/pom.xml | 2 +- sdks/pom.xml | 2 +- 33 files changed, 34 insertions(+), 34 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93045e9c/examples/java/pom.xml -- diff --git a/examples/java/pom.xml b/examples/java/pom.xml index 7333c10..e541edb 100644 --- a/examples/java/pom.xml +++ b/examples/java/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating +0.4.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93045e9c/examples/java8/pom.xml -- diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml index 59d8f4d..8591955 100644 --- a/examples/java8/pom.xml +++ b/examples/java8/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating +0.4.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93045e9c/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 65c6f1b..eb42861 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-parent -0.4.0-incubating +0.4.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93045e9c/pom.xml -- diff --git a/pom.xml b/pom.xml index 6d7da5d..e860c7a 100644 --- a/pom.xml +++ b/pom.xml @@ -34,7 +34,7 @@ http://beam.incubator.apache.org 2016 - 0.4.0-incubating + 0.4.0-incubating-SNAPSHOT @@ -48,7 +48,7 @@ scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary -v0.4.0-incubating-RC3 +release-0.4.0-incubating http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93045e9c/runners/apex/pom.
incubator-beam git commit: [maven-release-plugin] prepare release v0.4.0-incubating-RC3
Repository: incubator-beam Updated Branches: refs/heads/release-0.4.0-incubating bce0a0d50 -> 3d67e7a1f [maven-release-plugin] prepare release v0.4.0-incubating-RC3 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3d67e7a1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3d67e7a1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3d67e7a1 Branch: refs/heads/release-0.4.0-incubating Commit: 3d67e7a1fd2f988ca286aa6b8a8d927b986848de Parents: bce0a0d Author: Davor Bonaci Authored: Thu Dec 15 17:01:29 2016 -0800 Committer: Davor Bonaci Committed: Thu Dec 15 17:01:29 2016 -0800 -- examples/java/pom.xml | 2 +- examples/java8/pom.xml| 2 +- examples/pom.xml | 2 +- pom.xml | 4 ++-- runners/apex/pom.xml | 2 +- runners/core-java/pom.xml | 2 +- runners/direct-java/pom.xml | 2 +- runners/flink/examples/pom.xml| 2 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml | 2 +- runners/google-cloud-dataflow-java/pom.xml| 2 +- runners/pom.xml | 2 +- runners/spark/pom.xml | 2 +- sdks/java/build-tools/pom.xml | 2 +- sdks/java/core/pom.xml| 2 +- sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml | 2 +- sdks/java/extensions/sorter/pom.xml | 2 +- sdks/java/io/google-cloud-platform/pom.xml| 2 +- sdks/java/io/hdfs/pom.xml | 2 +- sdks/java/io/jdbc/pom.xml | 2 +- sdks/java/io/jms/pom.xml | 2 +- sdks/java/io/kafka/pom.xml| 2 +- sdks/java/io/kinesis/pom.xml | 2 +- sdks/java/io/mongodb/pom.xml | 2 +- sdks/java/io/pom.xml | 2 +- sdks/java/java8tests/pom.xml | 2 +- sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +- sdks/java/maven-archetypes/examples/pom.xml | 2 +- sdks/java/maven-archetypes/pom.xml| 2 +- sdks/java/maven-archetypes/starter/pom.xml| 2 +- sdks/java/pom.xml | 2 +- sdks/pom.xml | 2 +- 33 files changed, 34 insertions(+), 34 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d67e7a1/examples/java/pom.xml -- diff --git a/examples/java/pom.xml b/examples/java/pom.xml index e541edb..7333c10 100644 --- a/examples/java/pom.xml +++ b/examples/java/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating-SNAPSHOT +0.4.0-incubating ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d67e7a1/examples/java8/pom.xml -- diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml index 8591955..59d8f4d 100644 --- a/examples/java8/pom.xml +++ b/examples/java8/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating-SNAPSHOT +0.4.0-incubating ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d67e7a1/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index eb42861..65c6f1b 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-parent -0.4.0-incubating-SNAPSHOT +0.4.0-incubating ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d67e7a1/pom.xml -- diff --git a/pom.xml b/pom.xml index e860c7a..6d7da5d 100644 --- a/pom.xml +++ b/pom.xml @@ -34,7 +34,7 @@ http://beam.incubator.apache.org 2016 - 0.4.0-incubating-SNAPSHOT + 0.4.0-incubating @@ -48,7 +48,7 @@ scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary -release-0.4.0-incubating +v0.4.0-incubating-RC3 http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d67e7a1/runners/apex/pom.xml -- diff --git a/runn
[incubator-beam] Git Push Summary
Repository: incubator-beam Updated Tags: refs/tags/v0.4.0-incubating-RC3 [created] 112e38e4a
[jira] [Resolved] (BEAM-1153) GcsUtil needs to set timeout and retry explicitly in BatchRequest.
[ https://issues.apache.org/jira/browse/BEAM-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci resolved BEAM-1153. Resolution: Fixed Fix Version/s: 0.4.0-incubating > GcsUtil needs to set timeout and retry explicitly in BatchRequest. > -- > > Key: BEAM-1153 > URL: https://issues.apache.org/jira/browse/BEAM-1153 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Pei He >Assignee: Pei He >Priority: Blocker > Fix For: 0.4.0-incubating > > > Non-batch requests uses RetryHttpRequestInitializer, which set read timeout > as 80 seconds, and does more retries. > Google Cloud auto generated Json library doesn't set HttpRequestInitializer > for batch requests. > GcsUtil uses storageClient.batch(), and it is defined in here: > https://github.com/vparfonov/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/services/AbstractGoogleClient.java#L256 > Without the HttpRequestInitializer, the default read timeout is 20 seconds. > Possible fix is: https://github.com/apache/incubator-beam/pull/1608 > In additional, we can partially rollback > https://github.com/apache/incubator-beam/pull/1359 to keep using non-batch > API for fileSize() for single files. This will make sure existing code will > keep work as the same way. > PR: https://github.com/apache/incubator-beam/pull/1611 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/2] incubator-beam git commit: [BEAM-1153] GcsUtil: use non-batch API for single file size requests.
Repository: incubator-beam
Updated Branches:
refs/heads/release-0.4.0-incubating f753422a3 -> bce0a0d50
[BEAM-1153] GcsUtil: use non-batch API for single file size requests.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/58601f8c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/58601f8c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/58601f8c
Branch: refs/heads/release-0.4.0-incubating
Commit: 58601f8c69b97dbdd9087b27c356c50bca7a1c8b
Parents: f753422
Author: Pei He
Authored: Tue Dec 13 18:29:17 2016 -0800
Committer: Pei He
Committed: Thu Dec 15 16:30:54 2016 -0800
--
.../java/org/apache/beam/sdk/util/GcsUtil.java | 29 -
.../org/apache/beam/sdk/util/GcsUtilTest.java | 65 +++-
2 files changed, 92 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/58601f8c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
index 2edb1d6..dcdba46 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
@@ -267,7 +267,34 @@ public class GcsUtil {
* if the resource does not exist.
*/
public long fileSize(GcsPath path) throws IOException {
-return fileSizes(ImmutableList.of(path)).get(0);
+return fileSize(
+path,
+BACKOFF_FACTORY.backoff(),
+Sleeper.DEFAULT);
+ }
+
+ /**
+ * Returns the file size from GCS or throws {@link FileNotFoundException}
+ * if the resource does not exist.
+ */
+ @VisibleForTesting
+ long fileSize(GcsPath path, BackOff backoff, Sleeper sleeper) throws
IOException {
+Storage.Objects.Get getObject =
+storageClient.objects().get(path.getBucket(), path.getObject());
+try {
+ StorageObject object = ResilientOperation.retry(
+ ResilientOperation.getGoogleRequestCallable(getObject),
+ backoff,
+ RetryDeterminer.SOCKET_ERRORS,
+ IOException.class,
+ sleeper);
+ return object.getSize().longValue();
+} catch (Exception e) {
+ if (e instanceof IOException &&
errorExtractor.itemNotFound((IOException) e)) {
+throw new FileNotFoundException(path.toString());
+ }
+ throw new IOException("Unable to get file size", e);
+}
}
/**
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/58601f8c/sdks/java/core/src/test/java/org/apache/beam/sdk/util/GcsUtilTest.java
--
diff --git
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/GcsUtilTest.java
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/GcsUtilTest.java
index c8ed402..6ca87f9 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/GcsUtilTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/GcsUtilTest.java
@@ -57,6 +57,7 @@ import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
import java.io.FileNotFoundException;
import java.io.IOException;
+import java.math.BigInteger;
import java.net.SocketTimeoutException;
import java.nio.channels.SeekableByteChannel;
import java.nio.file.AccessDeniedException;
@@ -320,7 +321,69 @@ public class GcsUtilTest {
}
@Test
- public void testGetSizeBytesWhenFileNotFound() throws Exception {
+ public void testFileSizeNonBatch() throws Exception {
+GcsOptions pipelineOptions = gcsOptionsWithTestCredential();
+GcsUtil gcsUtil = pipelineOptions.getGcsUtil();
+
+Storage mockStorage = Mockito.mock(Storage.class);
+gcsUtil.setStorageClient(mockStorage);
+
+Storage.Objects mockStorageObjects = Mockito.mock(Storage.Objects.class);
+Storage.Objects.Get mockStorageGet =
Mockito.mock(Storage.Objects.Get.class);
+
+when(mockStorage.objects()).thenReturn(mockStorageObjects);
+when(mockStorageObjects.get("testbucket",
"testobject")).thenReturn(mockStorageGet);
+when(mockStorageGet.execute()).thenReturn(
+new StorageObject().setSize(BigInteger.valueOf(1000)));
+
+assertEquals(1000, gcsUtil.fileSize(GcsPath.fromComponents("testbucket",
"testobject")));
+ }
+
+ @Test
+ public void testFileSizeWhenFileNotFoundNonBatch() throws Exception {
+MockLowLevelHttpResponse notFoundResponse = new MockLowLevelHttpResponse();
+notFoundResponse.setContent("");
+notFoundResponse.setStatusCode(HttpStatusCodes.STATUS_CODE_NOT_FOUND);
+
+MockHttpTransport mockTransport =
+new
MockHttpTransport.Builder(
[2/2] incubator-beam git commit: This closes #1637
This closes #1637 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bce0a0d5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bce0a0d5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bce0a0d5 Branch: refs/heads/release-0.4.0-incubating Commit: bce0a0d50bdb4e560790c8292f031f5757a9bb8c Parents: f753422 58601f8 Author: Davor Bonaci Authored: Thu Dec 15 16:49:07 2016 -0800 Committer: Davor Bonaci Committed: Thu Dec 15 16:49:07 2016 -0800 -- .../java/org/apache/beam/sdk/util/GcsUtil.java | 29 - .../org/apache/beam/sdk/util/GcsUtilTest.java | 65 +++- 2 files changed, 92 insertions(+), 2 deletions(-) --
[incubator-beam] Git Push Summary
Repository: incubator-beam Updated Tags: refs/tags/v0.4.0-incubating-RC2 [created] 76246ac4f
incubator-beam git commit: [maven-release-plugin] rollback changes from release preparation of v0.4.0-incubating-RC2
Repository: incubator-beam Updated Branches: refs/heads/release-0.4.0-incubating a29affd05 -> f753422a3 [maven-release-plugin] rollback changes from release preparation of v0.4.0-incubating-RC2 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f753422a Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f753422a Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f753422a Branch: refs/heads/release-0.4.0-incubating Commit: f753422a36ec6cbcf56dd6f69e1c7ba29e53a350 Parents: a29affd Author: Davor Bonaci Authored: Thu Dec 15 13:22:04 2016 -0800 Committer: Davor Bonaci Committed: Thu Dec 15 13:22:04 2016 -0800 -- examples/java/pom.xml | 2 +- examples/java8/pom.xml| 2 +- examples/pom.xml | 2 +- pom.xml | 4 ++-- runners/apex/pom.xml | 2 +- runners/core-java/pom.xml | 2 +- runners/direct-java/pom.xml | 2 +- runners/flink/examples/pom.xml| 2 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml | 2 +- runners/google-cloud-dataflow-java/pom.xml| 2 +- runners/pom.xml | 2 +- runners/spark/pom.xml | 2 +- sdks/java/build-tools/pom.xml | 2 +- sdks/java/core/pom.xml| 2 +- sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml | 2 +- sdks/java/extensions/sorter/pom.xml | 2 +- sdks/java/io/google-cloud-platform/pom.xml| 2 +- sdks/java/io/hdfs/pom.xml | 2 +- sdks/java/io/jdbc/pom.xml | 2 +- sdks/java/io/jms/pom.xml | 2 +- sdks/java/io/kafka/pom.xml| 2 +- sdks/java/io/kinesis/pom.xml | 2 +- sdks/java/io/mongodb/pom.xml | 2 +- sdks/java/io/pom.xml | 2 +- sdks/java/java8tests/pom.xml | 2 +- sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +- sdks/java/maven-archetypes/examples/pom.xml | 2 +- sdks/java/maven-archetypes/pom.xml| 2 +- sdks/java/maven-archetypes/starter/pom.xml| 2 +- sdks/java/pom.xml | 2 +- sdks/pom.xml | 2 +- 33 files changed, 34 insertions(+), 34 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/examples/java/pom.xml -- diff --git a/examples/java/pom.xml b/examples/java/pom.xml index 7333c10..e541edb 100644 --- a/examples/java/pom.xml +++ b/examples/java/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating +0.4.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/examples/java8/pom.xml -- diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml index 59d8f4d..8591955 100644 --- a/examples/java8/pom.xml +++ b/examples/java8/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating +0.4.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 65c6f1b..eb42861 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-parent -0.4.0-incubating +0.4.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/pom.xml -- diff --git a/pom.xml b/pom.xml index 0d3db9d..e860c7a 100644 --- a/pom.xml +++ b/pom.xml @@ -34,7 +34,7 @@ http://beam.incubator.apache.org 2016 - 0.4.0-incubating + 0.4.0-incubating-SNAPSHOT @@ -48,7 +48,7 @@ scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary -v0.4.0-incubating-RC2 +release-0.4.0-incubating http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/runners/apex/pom.
incubator-beam git commit: [maven-release-plugin] prepare release v0.4.0-incubating-RC2
Repository: incubator-beam Updated Branches: refs/heads/release-0.4.0-incubating 3ca8d2bfc -> a29affd05 [maven-release-plugin] prepare release v0.4.0-incubating-RC2 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a29affd0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a29affd0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a29affd0 Branch: refs/heads/release-0.4.0-incubating Commit: a29affd05cfa3f387fd12296250d61a4bbe78946 Parents: 3ca8d2b Author: Davor Bonaci Authored: Thu Dec 15 13:16:49 2016 -0800 Committer: Davor Bonaci Committed: Thu Dec 15 13:16:49 2016 -0800 -- examples/java/pom.xml | 2 +- examples/java8/pom.xml| 2 +- examples/pom.xml | 2 +- pom.xml | 4 ++-- runners/apex/pom.xml | 2 +- runners/core-java/pom.xml | 2 +- runners/direct-java/pom.xml | 2 +- runners/flink/examples/pom.xml| 2 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml | 2 +- runners/google-cloud-dataflow-java/pom.xml| 2 +- runners/pom.xml | 2 +- runners/spark/pom.xml | 2 +- sdks/java/build-tools/pom.xml | 2 +- sdks/java/core/pom.xml| 2 +- sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml | 2 +- sdks/java/extensions/sorter/pom.xml | 2 +- sdks/java/io/google-cloud-platform/pom.xml| 2 +- sdks/java/io/hdfs/pom.xml | 2 +- sdks/java/io/jdbc/pom.xml | 2 +- sdks/java/io/jms/pom.xml | 2 +- sdks/java/io/kafka/pom.xml| 2 +- sdks/java/io/kinesis/pom.xml | 2 +- sdks/java/io/mongodb/pom.xml | 2 +- sdks/java/io/pom.xml | 2 +- sdks/java/java8tests/pom.xml | 2 +- sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +- sdks/java/maven-archetypes/examples/pom.xml | 2 +- sdks/java/maven-archetypes/pom.xml| 2 +- sdks/java/maven-archetypes/starter/pom.xml| 2 +- sdks/java/pom.xml | 2 +- sdks/pom.xml | 2 +- 33 files changed, 34 insertions(+), 34 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a29affd0/examples/java/pom.xml -- diff --git a/examples/java/pom.xml b/examples/java/pom.xml index e541edb..7333c10 100644 --- a/examples/java/pom.xml +++ b/examples/java/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating-SNAPSHOT +0.4.0-incubating ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a29affd0/examples/java8/pom.xml -- diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml index 8591955..59d8f4d 100644 --- a/examples/java8/pom.xml +++ b/examples/java8/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating-SNAPSHOT +0.4.0-incubating ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a29affd0/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index eb42861..65c6f1b 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-parent -0.4.0-incubating-SNAPSHOT +0.4.0-incubating ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a29affd0/pom.xml -- diff --git a/pom.xml b/pom.xml index e860c7a..0d3db9d 100644 --- a/pom.xml +++ b/pom.xml @@ -34,7 +34,7 @@ http://beam.incubator.apache.org 2016 - 0.4.0-incubating-SNAPSHOT + 0.4.0-incubating @@ -48,7 +48,7 @@ scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary -release-0.4.0-incubating +v0.4.0-incubating-RC2 http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a29affd0/runners/apex/pom.xml -- diff --git a/runn
[2/2] incubator-beam git commit: This closes #1629
This closes #1629 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5dcffe06 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5dcffe06 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5dcffe06 Branch: refs/heads/release-0.4.0-incubating Commit: 5dcffe066944e8cc302e716337770551efa48aed Parents: 7e234aa 21507b8 Author: Davor Bonaci Authored: Thu Dec 15 10:50:08 2016 -0800 Committer: Davor Bonaci Committed: Thu Dec 15 10:50:08 2016 -0800 -- .../org/apache/beam/runners/dataflow/dataflow.properties | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) --
[1/2] incubator-beam git commit: Set Dataflow worker version to 0.4.0-incubating
Repository: incubator-beam Updated Branches: refs/heads/release-0.4.0-incubating 7e234aa79 -> 5dcffe066 Set Dataflow worker version to 0.4.0-incubating Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/21507b8d Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/21507b8d Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/21507b8d Branch: refs/heads/release-0.4.0-incubating Commit: 21507b8da0bb74cc8d844ede985648ac4508df2b Parents: 7e234aa Author: Kenneth Knowles Authored: Thu Dec 15 10:17:49 2016 -0800 Committer: Kenneth Knowles Committed: Thu Dec 15 10:17:49 2016 -0800 -- .../org/apache/beam/runners/dataflow/dataflow.properties | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/21507b8d/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties -- diff --git a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties index 1eae8cb..abfea0c 100644 --- a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties +++ b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties @@ -18,6 +18,6 @@ environment.major.version=6 -worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20161205 +worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-0.4.0-incubating -worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20161205 +worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-0.4.0-incubating
incubator-beam git commit: [maven-release-plugin] rollback changes from release preparation of v0.4.0-incubating-RC1
Repository: incubator-beam Updated Branches: refs/heads/release-0.4.0-incubating af3989b08 -> 7e234aa79 [maven-release-plugin] rollback changes from release preparation of v0.4.0-incubating-RC1 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7e234aa7 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7e234aa7 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7e234aa7 Branch: refs/heads/release-0.4.0-incubating Commit: 7e234aa7938606d1a01064a6fdccf01bb90d4f2d Parents: af3989b Author: Davor Bonaci Authored: Wed Dec 14 18:59:29 2016 -0800 Committer: Davor Bonaci Committed: Wed Dec 14 18:59:29 2016 -0800 -- examples/java/pom.xml | 2 +- examples/java8/pom.xml| 2 +- examples/pom.xml | 2 +- pom.xml | 4 ++-- runners/apex/pom.xml | 2 +- runners/core-java/pom.xml | 2 +- runners/direct-java/pom.xml | 2 +- runners/flink/examples/pom.xml| 2 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml | 2 +- runners/google-cloud-dataflow-java/pom.xml| 2 +- runners/pom.xml | 2 +- runners/spark/pom.xml | 2 +- sdks/java/build-tools/pom.xml | 2 +- sdks/java/core/pom.xml| 2 +- sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml | 2 +- sdks/java/extensions/sorter/pom.xml | 2 +- sdks/java/io/google-cloud-platform/pom.xml| 2 +- sdks/java/io/hdfs/pom.xml | 2 +- sdks/java/io/jdbc/pom.xml | 2 +- sdks/java/io/jms/pom.xml | 2 +- sdks/java/io/kafka/pom.xml| 2 +- sdks/java/io/kinesis/pom.xml | 2 +- sdks/java/io/mongodb/pom.xml | 2 +- sdks/java/io/pom.xml | 2 +- sdks/java/java8tests/pom.xml | 2 +- sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +- sdks/java/maven-archetypes/examples/pom.xml | 2 +- sdks/java/maven-archetypes/pom.xml| 2 +- sdks/java/maven-archetypes/starter/pom.xml| 2 +- sdks/java/pom.xml | 2 +- sdks/pom.xml | 2 +- 33 files changed, 34 insertions(+), 34 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7e234aa7/examples/java/pom.xml -- diff --git a/examples/java/pom.xml b/examples/java/pom.xml index 7333c10..e541edb 100644 --- a/examples/java/pom.xml +++ b/examples/java/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating +0.4.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7e234aa7/examples/java8/pom.xml -- diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml index 59d8f4d..8591955 100644 --- a/examples/java8/pom.xml +++ b/examples/java8/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating +0.4.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7e234aa7/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 65c6f1b..eb42861 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-parent -0.4.0-incubating +0.4.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7e234aa7/pom.xml -- diff --git a/pom.xml b/pom.xml index 37f698c..e860c7a 100644 --- a/pom.xml +++ b/pom.xml @@ -34,7 +34,7 @@ http://beam.incubator.apache.org 2016 - 0.4.0-incubating + 0.4.0-incubating-SNAPSHOT @@ -48,7 +48,7 @@ scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary -v0.4.0-incubating-RC1 +release-0.4.0-incubating http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7e234aa7/runners/apex/pom.
[incubator-beam] Git Push Summary
Repository: incubator-beam Updated Tags: refs/tags/v0.4.0-incubating-RC1 [created] 85d1c8a2f
incubator-beam git commit: [maven-release-plugin] prepare release v0.4.0-incubating-RC1
Repository: incubator-beam Updated Branches: refs/heads/release-0.4.0-incubating ea0d7f5c2 -> af3989b08 [maven-release-plugin] prepare release v0.4.0-incubating-RC1 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/af3989b0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/af3989b0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/af3989b0 Branch: refs/heads/release-0.4.0-incubating Commit: af3989b086ed2e1bf0a08358ce6b7620c334cf49 Parents: ea0d7f5 Author: Davor Bonaci Authored: Wed Dec 14 18:58:57 2016 -0800 Committer: Davor Bonaci Committed: Wed Dec 14 18:58:57 2016 -0800 -- examples/java/pom.xml | 2 +- examples/java8/pom.xml| 2 +- examples/pom.xml | 2 +- pom.xml | 4 ++-- runners/apex/pom.xml | 2 +- runners/core-java/pom.xml | 2 +- runners/direct-java/pom.xml | 2 +- runners/flink/examples/pom.xml| 2 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml | 2 +- runners/google-cloud-dataflow-java/pom.xml| 2 +- runners/pom.xml | 2 +- runners/spark/pom.xml | 2 +- sdks/java/build-tools/pom.xml | 2 +- sdks/java/core/pom.xml| 2 +- sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml | 2 +- sdks/java/extensions/sorter/pom.xml | 2 +- sdks/java/io/google-cloud-platform/pom.xml| 2 +- sdks/java/io/hdfs/pom.xml | 2 +- sdks/java/io/jdbc/pom.xml | 2 +- sdks/java/io/jms/pom.xml | 2 +- sdks/java/io/kafka/pom.xml| 2 +- sdks/java/io/kinesis/pom.xml | 2 +- sdks/java/io/mongodb/pom.xml | 2 +- sdks/java/io/pom.xml | 2 +- sdks/java/java8tests/pom.xml | 2 +- sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +- sdks/java/maven-archetypes/examples/pom.xml | 2 +- sdks/java/maven-archetypes/pom.xml| 2 +- sdks/java/maven-archetypes/starter/pom.xml| 2 +- sdks/java/pom.xml | 2 +- sdks/pom.xml | 2 +- 33 files changed, 34 insertions(+), 34 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/af3989b0/examples/java/pom.xml -- diff --git a/examples/java/pom.xml b/examples/java/pom.xml index e541edb..7333c10 100644 --- a/examples/java/pom.xml +++ b/examples/java/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating-SNAPSHOT +0.4.0-incubating ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/af3989b0/examples/java8/pom.xml -- diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml index 8591955..59d8f4d 100644 --- a/examples/java8/pom.xml +++ b/examples/java8/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating-SNAPSHOT +0.4.0-incubating ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/af3989b0/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index eb42861..65c6f1b 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-parent -0.4.0-incubating-SNAPSHOT +0.4.0-incubating ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/af3989b0/pom.xml -- diff --git a/pom.xml b/pom.xml index e860c7a..37f698c 100644 --- a/pom.xml +++ b/pom.xml @@ -34,7 +34,7 @@ http://beam.incubator.apache.org 2016 - 0.4.0-incubating-SNAPSHOT + 0.4.0-incubating @@ -48,7 +48,7 @@ scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary -release-0.4.0-incubating +v0.4.0-incubating-RC1 http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/af3989b0/runners/apex/pom.xml -- diff --git a/runn
[1/2] incubator-beam git commit: Update version from 0.5.0-SNAPSHOT to 0.5.0-incubating-SNAPSHOT
Repository: incubator-beam Updated Branches: refs/heads/master cf2c1d685 -> dad5ba5a6 Update version from 0.5.0-SNAPSHOT to 0.5.0-incubating-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/25215889 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/25215889 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/25215889 Branch: refs/heads/master Commit: 25215889381f7da61766054af68c84ffed4c0c71 Parents: cf2c1d6 Author: Davor Bonaci Authored: Tue Dec 13 14:27:55 2016 -0800 Committer: Davor Bonaci Committed: Tue Dec 13 14:27:55 2016 -0800 -- examples/java/pom.xml | 2 +- examples/java8/pom.xml | 2 +- examples/pom.xml| 2 +- pom.xml | 2 +- runners/apex/pom.xml| 2 +- runners/core-java/pom.xml | 2 +- runners/direct-java/pom.xml | 2 +- runners/flink/examples/pom.xml | 2 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml| 2 +- runners/google-cloud-dataflow-java/pom.xml | 2 +- runners/pom.xml | 2 +- runners/spark/pom.xml | 2 +- sdks/java/build-tools/pom.xml | 2 +- sdks/java/core/pom.xml | 2 +- sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml| 2 +- sdks/java/extensions/sorter/pom.xml | 2 +- sdks/java/io/google-cloud-platform/pom.xml | 2 +- sdks/java/io/hdfs/pom.xml | 2 +- sdks/java/io/jdbc/pom.xml | 2 +- sdks/java/io/jms/pom.xml| 2 +- sdks/java/io/kafka/pom.xml | 2 +- sdks/java/io/kinesis/pom.xml| 2 +- sdks/java/io/mongodb/pom.xml| 2 +- sdks/java/io/pom.xml| 2 +- sdks/java/java8tests/pom.xml| 2 +- sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +- .../src/main/resources/archetype-resources/pom.xml | 2 +- sdks/java/maven-archetypes/examples/pom.xml | 2 +- .../examples/src/main/resources/archetype-resources/pom.xml | 2 +- sdks/java/maven-archetypes/pom.xml | 2 +- sdks/java/maven-archetypes/starter/pom.xml | 2 +- .../starter/src/main/resources/archetype-resources/pom.xml | 5 ++--- .../starter/src/test/resources/projects/basic/reference/pom.xml | 2 +- sdks/java/pom.xml | 2 +- sdks/pom.xml| 2 +- 37 files changed, 38 insertions(+), 39 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/25215889/examples/java/pom.xml -- diff --git a/examples/java/pom.xml b/examples/java/pom.xml index dc86532..f1e0fe1 100644 --- a/examples/java/pom.xml +++ b/examples/java/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.5.0-SNAPSHOT +0.5.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/25215889/examples/java8/pom.xml -- diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml index 370d79f..1abf6fd 100644 --- a/examples/java8/pom.xml +++ b/examples/java8/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.5.0-SNAPSHOT +0.5.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/25215889/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index adfbaa9..3e0fbdd 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-parent -0.5.0-SNAPSHOT +0.5.0-incubating-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/25215889/pom.
[2/2] incubator-beam git commit: This closes #1600
This closes #1600 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/dad5ba5a Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/dad5ba5a Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/dad5ba5a Branch: refs/heads/master Commit: dad5ba5a66b971c272b25b8da368df8593196edb Parents: cf2c1d6 2521588 Author: Davor Bonaci Authored: Tue Dec 13 14:44:59 2016 -0800 Committer: Davor Bonaci Committed: Tue Dec 13 14:44:59 2016 -0800 -- examples/java/pom.xml | 2 +- examples/java8/pom.xml | 2 +- examples/pom.xml| 2 +- pom.xml | 2 +- runners/apex/pom.xml| 2 +- runners/core-java/pom.xml | 2 +- runners/direct-java/pom.xml | 2 +- runners/flink/examples/pom.xml | 2 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml| 2 +- runners/google-cloud-dataflow-java/pom.xml | 2 +- runners/pom.xml | 2 +- runners/spark/pom.xml | 2 +- sdks/java/build-tools/pom.xml | 2 +- sdks/java/core/pom.xml | 2 +- sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml| 2 +- sdks/java/extensions/sorter/pom.xml | 2 +- sdks/java/io/google-cloud-platform/pom.xml | 2 +- sdks/java/io/hdfs/pom.xml | 2 +- sdks/java/io/jdbc/pom.xml | 2 +- sdks/java/io/jms/pom.xml| 2 +- sdks/java/io/kafka/pom.xml | 2 +- sdks/java/io/kinesis/pom.xml| 2 +- sdks/java/io/mongodb/pom.xml| 2 +- sdks/java/io/pom.xml| 2 +- sdks/java/java8tests/pom.xml| 2 +- sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +- .../src/main/resources/archetype-resources/pom.xml | 2 +- sdks/java/maven-archetypes/examples/pom.xml | 2 +- .../examples/src/main/resources/archetype-resources/pom.xml | 2 +- sdks/java/maven-archetypes/pom.xml | 2 +- sdks/java/maven-archetypes/starter/pom.xml | 2 +- .../starter/src/main/resources/archetype-resources/pom.xml | 5 ++--- .../starter/src/test/resources/projects/basic/reference/pom.xml | 2 +- sdks/java/pom.xml | 2 +- sdks/pom.xml| 2 +- 37 files changed, 38 insertions(+), 39 deletions(-) --
[1/2] incubator-beam git commit: Update Maven Archetype versions after cutting the release branch
Repository: incubator-beam
Updated Branches:
refs/heads/master a52600b68 -> cf2c1d685
Update Maven Archetype versions after cutting the release branch
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5d6a50c6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5d6a50c6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5d6a50c6
Branch: refs/heads/master
Commit: 5d6a50c63360d8e706c095b73b55734390615135
Parents: a52600b
Author: Davor Bonaci
Authored: Tue Dec 13 13:43:58 2016 -0800
Committer: Davor Bonaci
Committed: Tue Dec 13 14:20:51 2016 -0800
--
.../src/main/resources/archetype-resources/pom.xml | 6 +++---
.../examples/src/main/resources/archetype-resources/pom.xml| 2 +-
.../starter/src/main/resources/archetype-resources/pom.xml | 2 +-
.../src/test/resources/projects/basic/reference/pom.xml| 2 +-
4 files changed, 6 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5d6a50c6/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
--
diff --git
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
index 693850a..a440424 100644
---
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
+++
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
@@ -27,7 +27,7 @@
jar
-0.4.0-incubating-SNAPSHOT
+0.5.0-SNAPSHOT
@@ -177,14 +177,14 @@
org.apache.beam
beam-sdks-java-core
- 0.4.0-incubating-SNAPSHOT
+ ${beam.version}
org.apache.beam
beam-sdks-java-io-google-cloud-platform
- 0.4.0-incubating-SNAPSHOT
+ ${beam.version}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5d6a50c6/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
--
diff --git
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 9bbc646..b6eab15 100644
---
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -27,7 +27,7 @@
jar
-0.4.0-incubating-SNAPSHOT
+0.5.0-SNAPSHOT
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5d6a50c6/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
--
diff --git
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
index efafeca..4a6e89d 100644
---
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
+++
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
@@ -25,7 +25,7 @@
${version}
-0.4.0-incubating-SNAPSHOT
+0.5.0-SNAPSHOT
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5d6a50c6/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
--
diff --git
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
index a86bd11..97f9e72 100644
---
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
+++
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
@@ -25,7 +25,7 @@
0.1
-0.4.0-incubating-SNAPSHOT
+0.5.0-SNAPSHOT
[2/2] incubator-beam git commit: This closes #1599
This closes #1599 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cf2c1d68 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cf2c1d68 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cf2c1d68 Branch: refs/heads/master Commit: cf2c1d685035d7da00efb1a72cd06caa006cccd7 Parents: a52600b 5d6a50c Author: Davor Bonaci Authored: Tue Dec 13 14:20:55 2016 -0800 Committer: Davor Bonaci Committed: Tue Dec 13 14:20:55 2016 -0800 -- .../src/main/resources/archetype-resources/pom.xml | 6 +++--- .../examples/src/main/resources/archetype-resources/pom.xml| 2 +- .../starter/src/main/resources/archetype-resources/pom.xml | 2 +- .../src/test/resources/projects/basic/reference/pom.xml| 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) --
incubator-beam git commit: Update Maven Archetype version; fixup for Java8.
Repository: incubator-beam
Updated Branches:
refs/heads/release-0.4.0-incubating 2c1d4eb4d -> 9f3b06359
Update Maven Archetype version; fixup for Java8.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9f3b0635
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9f3b0635
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9f3b0635
Branch: refs/heads/release-0.4.0-incubating
Commit: 9f3b0635971a45f8baf8fa68ba5ec2f8c47cec6b
Parents: 2c1d4eb
Author: Davor Bonaci
Authored: Tue Dec 13 14:18:32 2016 -0800
Committer: Davor Bonaci
Committed: Tue Dec 13 14:18:32 2016 -0800
--
.../src/main/resources/archetype-resources/pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9f3b0635/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
--
diff --git
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
index 2771d5d..c9645d3 100644
---
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
+++
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
@@ -177,14 +177,14 @@
org.apache.beam
beam-sdks-java-core
- 0.4.0-incubating-SNAPSHOT
+ ${beam.version}
org.apache.beam
beam-sdks-java-io-google-cloud-platform
- 0.4.0-incubating-SNAPSHOT
+ ${beam.version}
incubator-beam git commit: Update Maven Archetype versions to pin them to 0.4.0-incubating to be released
Repository: incubator-beam
Updated Branches:
refs/heads/release-0.4.0-incubating c8dd9989c -> 2c1d4eb4d
Update Maven Archetype versions to pin them to 0.4.0-incubating to be released
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2c1d4eb4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2c1d4eb4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2c1d4eb4
Branch: refs/heads/release-0.4.0-incubating
Commit: 2c1d4eb4d878d1a8b8adf68b10eb8744cf524b5c
Parents: c8dd998
Author: Davor Bonaci
Authored: Tue Dec 13 13:48:26 2016 -0800
Committer: Davor Bonaci
Committed: Tue Dec 13 13:48:26 2016 -0800
--
.../examples-java8/src/main/resources/archetype-resources/pom.xml | 2 +-
.../examples/src/main/resources/archetype-resources/pom.xml| 2 +-
.../starter/src/main/resources/archetype-resources/pom.xml | 2 +-
.../starter/src/test/resources/projects/basic/reference/pom.xml| 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2c1d4eb4/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
--
diff --git
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
index 693850a..2771d5d 100644
---
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
+++
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
@@ -27,7 +27,7 @@
jar
-0.4.0-incubating-SNAPSHOT
+0.4.0-incubating
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2c1d4eb4/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
--
diff --git
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 9bbc646..62a2de5 100644
---
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -27,7 +27,7 @@
jar
-0.4.0-incubating-SNAPSHOT
+0.4.0-incubating
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2c1d4eb4/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
--
diff --git
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
index efafeca..08139b2 100644
---
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
+++
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
@@ -25,7 +25,7 @@
${version}
-0.4.0-incubating-SNAPSHOT
+0.4.0-incubating
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2c1d4eb4/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
--
diff --git
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
index a86bd11..2a668e8 100644
---
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
+++
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
@@ -25,7 +25,7 @@
0.1
-0.4.0-incubating-SNAPSHOT
+0.4.0-incubating
incubator-beam git commit: [maven-release-plugin] prepare branch release-0.4.0-incubating
Repository: incubator-beam Updated Branches: refs/heads/master 6a89fffe3 -> c8dd9989c [maven-release-plugin] prepare branch release-0.4.0-incubating Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c8dd9989 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c8dd9989 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c8dd9989 Branch: refs/heads/master Commit: c8dd9989c49aab1cede743a45cad321b9b9b5a2d Parents: 6a89fff Author: Davor Bonaci Authored: Tue Dec 13 13:34:22 2016 -0800 Committer: Davor Bonaci Committed: Tue Dec 13 13:34:22 2016 -0800 -- pom.xml | 2 +- runners/apex/pom.xml | 6 ++ sdks/java/maven-archetypes/examples-java8/pom.xml | 4 +--- sdks/java/maven-archetypes/examples/pom.xml | 4 +--- 4 files changed, 5 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c8dd9989/pom.xml -- diff --git a/pom.xml b/pom.xml index 32bcab0..e860c7a 100644 --- a/pom.xml +++ b/pom.xml @@ -48,7 +48,7 @@ scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary -release-0.2.0-incubating +release-0.4.0-incubating http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c8dd9989/runners/apex/pom.xml -- diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml index a47cf74..42d7212 100644 --- a/runners/apex/pom.xml +++ b/runners/apex/pom.xml @@ -15,9 +15,7 @@ See the License for the specific language governing permissions and limitations under the License. --> -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 @@ -267,7 +265,7 @@ - + http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c8dd9989/sdks/java/maven-archetypes/examples-java8/pom.xml -- diff --git a/sdks/java/maven-archetypes/examples-java8/pom.xml b/sdks/java/maven-archetypes/examples-java8/pom.xml index 4aeb627..53ad698 100644 --- a/sdks/java/maven-archetypes/examples-java8/pom.xml +++ b/sdks/java/maven-archetypes/examples-java8/pom.xml @@ -15,9 +15,7 @@ See the License for the specific language governing permissions and limitations under the License. --> -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c8dd9989/sdks/java/maven-archetypes/examples/pom.xml -- diff --git a/sdks/java/maven-archetypes/examples/pom.xml b/sdks/java/maven-archetypes/examples/pom.xml index 72d85f9..3fcd7d4 100644 --- a/sdks/java/maven-archetypes/examples/pom.xml +++ b/sdks/java/maven-archetypes/examples/pom.xml @@ -15,9 +15,7 @@ See the License for the specific language governing permissions and limitations under the License. --> -http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0
[incubator-beam] Git Push Summary
Repository: incubator-beam Updated Branches: refs/heads/release-0.4.0-incubating [created] c8dd9989c
incubator-beam git commit: [maven-release-plugin] prepare for next development iteration
Repository: incubator-beam Updated Branches: refs/heads/master c8dd9989c -> 596f255bc [maven-release-plugin] prepare for next development iteration Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/596f255b Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/596f255b Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/596f255b Branch: refs/heads/master Commit: 596f255bcbfd1d14edbf17aea15d5313b7f34b80 Parents: c8dd998 Author: Davor Bonaci Authored: Tue Dec 13 13:35:15 2016 -0800 Committer: Davor Bonaci Committed: Tue Dec 13 13:35:15 2016 -0800 -- examples/java/pom.xml | 2 +- examples/java8/pom.xml| 2 +- examples/pom.xml | 2 +- pom.xml | 4 ++-- runners/apex/pom.xml | 2 +- runners/core-java/pom.xml | 2 +- runners/direct-java/pom.xml | 2 +- runners/flink/examples/pom.xml| 2 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml | 2 +- runners/google-cloud-dataflow-java/pom.xml| 2 +- runners/pom.xml | 2 +- runners/spark/pom.xml | 2 +- sdks/java/build-tools/pom.xml | 2 +- sdks/java/core/pom.xml| 2 +- sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml | 2 +- sdks/java/extensions/sorter/pom.xml | 2 +- sdks/java/io/google-cloud-platform/pom.xml| 2 +- sdks/java/io/hdfs/pom.xml | 2 +- sdks/java/io/jdbc/pom.xml | 2 +- sdks/java/io/jms/pom.xml | 2 +- sdks/java/io/kafka/pom.xml| 2 +- sdks/java/io/kinesis/pom.xml | 2 +- sdks/java/io/mongodb/pom.xml | 2 +- sdks/java/io/pom.xml | 2 +- sdks/java/java8tests/pom.xml | 2 +- sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +- sdks/java/maven-archetypes/examples/pom.xml | 2 +- sdks/java/maven-archetypes/pom.xml| 2 +- sdks/java/maven-archetypes/starter/pom.xml| 2 +- sdks/java/pom.xml | 2 +- sdks/pom.xml | 2 +- 33 files changed, 34 insertions(+), 34 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/596f255b/examples/java/pom.xml -- diff --git a/examples/java/pom.xml b/examples/java/pom.xml index e541edb..dc86532 100644 --- a/examples/java/pom.xml +++ b/examples/java/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating-SNAPSHOT +0.5.0-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/596f255b/examples/java8/pom.xml -- diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml index 8591955..370d79f 100644 --- a/examples/java8/pom.xml +++ b/examples/java8/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-examples-parent -0.4.0-incubating-SNAPSHOT +0.5.0-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/596f255b/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index eb42861..adfbaa9 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -22,7 +22,7 @@ org.apache.beam beam-parent -0.4.0-incubating-SNAPSHOT +0.5.0-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/596f255b/pom.xml -- diff --git a/pom.xml b/pom.xml index e860c7a..b57be98 100644 --- a/pom.xml +++ b/pom.xml @@ -34,7 +34,7 @@ http://beam.incubator.apache.org 2016 - 0.4.0-incubating-SNAPSHOT + 0.5.0-SNAPSHOT @@ -48,7 +48,7 @@ scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary -release-0.4.0-incubating +release-0.2.0-incubating http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/596f255b/runners/apex/pom.xml -- diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml in
[jira] [Commented] (BEAM-1144) Spark runner fails to deserialize MicrobatchSource in cluster mode
[
https://issues.apache.org/jira/browse/BEAM-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15746099#comment-15746099
]
Davor Bonaci commented on BEAM-1144:
Sounds like it is not "blocking", but nice-to-have. So, I'd say, if it happens,
great, otherwise it will be included in the next release.
> Spark runner fails to deserialize MicrobatchSource in cluster mode
> --
>
> Key: BEAM-1144
> URL: https://issues.apache.org/jira/browse/BEAM-1144
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
>Reporter: Aviem Zur
>Assignee: Amit Sela
>
> When running in cluster mode (yarn), spark runner fails on deserialization of
> {{MicrobatchSource}}
> After changes made in BEAM-921 spark runner fails in cluster mode with the
> following:
> {code}
> 16/12/12 04:27:01 ERROR ApplicationMaster: User class threw exception:
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> com.esotericsoftware.kryo.KryoException: Error during Java deserialization.
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> com.esotericsoftware.kryo.KryoException: Error during Java deserialization.
> at
> org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:72)
> at
> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:115)
> at
> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:101)
> at
> com.paypal.risk.platform.aleph.example.MapOnlyExample.main(MapOnlyExample.java:38)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:559)
> Caused by: com.esotericsoftware.kryo.KryoException: Error during Java
> deserialization.
> at
> com.esotericsoftware.kryo.serializers.JavaSerializer.read(JavaSerializer.java:42)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
> at
> org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:228)
> at
> org.apache.spark.serializer.DeserializationStream.readKey(Serializer.scala:169)
> at
> org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:201)
> at
> org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:198)
> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at
> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
> at
> org.apache.spark.streaming.rdd.MapWithStateRDDRecord$.updateRecordWithData(MapWithStateRDD.scala:55)
> at
> org.apache.spark.streaming.rdd.MapWithStateRDD.compute(MapWithStateRDD.scala:155)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:275)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:275)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.
[jira] [Commented] (BEAM-1144) Spark runner fails to deserialize MicrobatchSource in cluster mode
[
https://issues.apache.org/jira/browse/BEAM-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745865#comment-15745865
]
Davor Bonaci commented on BEAM-1144:
[~amitsela], [~aviemzur], should this be considered release blocking for
0.4.0-incubating?
CC: [~dhalp...@google.com], [~jbonofre].
> Spark runner fails to deserialize MicrobatchSource in cluster mode
> --
>
> Key: BEAM-1144
> URL: https://issues.apache.org/jira/browse/BEAM-1144
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
>Reporter: Aviem Zur
>Assignee: Amit Sela
>
> When running in cluster mode (yarn), spark runner fails on deserialization of
> {{MicrobatchSource}}
> After changes made in BEAM-921 spark runner fails in cluster mode with the
> following:
> {code}
> 16/12/12 04:27:01 ERROR ApplicationMaster: User class threw exception:
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> com.esotericsoftware.kryo.KryoException: Error during Java deserialization.
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> com.esotericsoftware.kryo.KryoException: Error during Java deserialization.
> at
> org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:72)
> at
> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:115)
> at
> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:101)
> at
> com.paypal.risk.platform.aleph.example.MapOnlyExample.main(MapOnlyExample.java:38)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:559)
> Caused by: com.esotericsoftware.kryo.KryoException: Error during Java
> deserialization.
> at
> com.esotericsoftware.kryo.serializers.JavaSerializer.read(JavaSerializer.java:42)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
> at
> org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:228)
> at
> org.apache.spark.serializer.DeserializationStream.readKey(Serializer.scala:169)
> at
> org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:201)
> at
> org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:198)
> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at
> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
> at
> org.apache.spark.streaming.rdd.MapWithStateRDDRecord$.updateRecordWithData(MapWithStateRDD.scala:55)
> at
> org.apache.spark.streaming.rdd.MapWithStateRDD.compute(MapWithStateRDD.scala:155)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:275)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:275)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent
[1/2] incubator-beam git commit: Fix JDom malformed comment in Apex runner.
Repository: incubator-beam Updated Branches: refs/heads/master 91d71d47e -> cd8eeea95 Fix JDom malformed comment in Apex runner. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/db644318 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/db644318 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/db644318 Branch: refs/heads/master Commit: db644318e1117d2a9afc3ed56270ad13e1186cfa Parents: 91d71d4 Author: Jean-Baptiste Onofré Authored: Tue Dec 13 15:09:03 2016 +0100 Committer: Davor Bonaci Committed: Tue Dec 13 10:17:24 2016 -0800 -- runners/apex/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/db644318/runners/apex/pom.xml -- diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml index 9f1455a..74fe0b6 100644 --- a/runners/apex/pom.xml +++ b/runners/apex/pom.xml @@ -75,7 +75,7 @@ runtime - + org.apache.beam beam-sdks-java-core
[2/2] incubator-beam git commit: This closes #1593
This closes #1593 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cd8eeea9 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cd8eeea9 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cd8eeea9 Branch: refs/heads/master Commit: cd8eeea950cc4f4f2c394184022426086c10199c Parents: 91d71d4 db64431 Author: Davor Bonaci Authored: Tue Dec 13 10:17:28 2016 -0800 Committer: Davor Bonaci Committed: Tue Dec 13 10:17:28 2016 -0800 -- runners/apex/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --
[1/2] incubator-beam git commit: [BEAM-927] Fix findbugs and re-enable Maven plugin in JmsIO
Repository: incubator-beam
Updated Branches:
refs/heads/master e776d1d70 -> 91d71d47e
[BEAM-927] Fix findbugs and re-enable Maven plugin in JmsIO
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c1fe7603
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c1fe7603
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c1fe7603
Branch: refs/heads/master
Commit: c1fe760303ac4a69427aafbb40f8796c414daff9
Parents: e776d1d
Author: Jean-Baptiste Onofré
Authored: Thu Nov 17 13:19:42 2016 +0100
Committer: Davor Bonaci
Committed: Tue Dec 13 10:15:30 2016 -0800
--
.../src/main/resources/beam/findbugs-filter.xml | 19 +++
sdks/java/io/jms/pom.xml | 13 -
.../org/apache/beam/sdk/io/jms/JmsRecord.java| 8 +---
3 files changed, 24 insertions(+), 16 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c1fe7603/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
index adfe2dd..a696aeb 100644
--- a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
@@ -44,6 +44,25 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c1fe7603/sdks/java/io/jms/pom.xml
--
diff --git a/sdks/java/io/jms/pom.xml b/sdks/java/io/jms/pom.xml
index aedba9e..93916e7 100644
--- a/sdks/java/io/jms/pom.xml
+++ b/sdks/java/io/jms/pom.xml
@@ -37,19 +37,6 @@
-
-
-
-
- org.codehaus.mojo
- findbugs-maven-plugin
-
-true
-
-
-
-
-
org.apache.maven.plugins
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c1fe7603/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsRecord.java
--
diff --git
a/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsRecord.java
b/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsRecord.java
index 65a9189..15a1109 100644
--- a/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsRecord.java
+++ b/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsRecord.java
@@ -20,6 +20,8 @@ package org.apache.beam.sdk.io.jms;
import java.io.Serializable;
import java.util.Map;
import java.util.Objects;
+
+import javax.annotation.Nullable;
import javax.jms.Destination;
/**
@@ -31,7 +33,7 @@ public class JmsRecord implements Serializable {
private final String jmsMessageID;
private final long jmsTimestamp;
private final String jmsCorrelationID;
- private final Destination jmsReplyTo;
+ @Nullable private final Destination jmsReplyTo;
private final Destination jmsDestination;
private final int jmsDeliveryMode;
private final boolean jmsRedelivered;
@@ -45,7 +47,7 @@ public class JmsRecord implements Serializable {
String jmsMessageID,
long jmsTimestamp,
String jmsCorrelationID,
- Destination jmsReplyTo,
+ @Nullable Destination jmsReplyTo,
Destination jmsDestination,
int jmsDeliveryMode,
boolean jmsRedelivered,
@@ -80,7 +82,7 @@ public class JmsRecord implements Serializable {
return jmsCorrelationID;
}
- public Destination getJmsReplyTo() {
+ public @Nullable Destination getJmsReplyTo() {
return jmsReplyTo;
}
[2/2] incubator-beam git commit: This closes #1357
This closes #1357 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/91d71d47 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/91d71d47 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/91d71d47 Branch: refs/heads/master Commit: 91d71d47e61a3ba132cfa828056eef69e32e680e Parents: e776d1d c1fe760 Author: Davor Bonaci Authored: Tue Dec 13 10:15:34 2016 -0800 Committer: Davor Bonaci Committed: Tue Dec 13 10:15:34 2016 -0800 -- .../src/main/resources/beam/findbugs-filter.xml | 19 +++ sdks/java/io/jms/pom.xml | 13 - .../org/apache/beam/sdk/io/jms/JmsRecord.java| 8 +--- 3 files changed, 24 insertions(+), 16 deletions(-) --
[jira] [Updated] (BEAM-842) dependency.py: package not found when running on Windows
[ https://issues.apache.org/jira/browse/BEAM-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-842: -- Affects Version/s: (was: 0.4.0-incubating) > dependency.py: package not found when running on Windows > > > Key: BEAM-842 > URL: https://issues.apache.org/jira/browse/BEAM-842 > Project: Beam > Issue Type: Bug > Components: sdk-py > Environment: Windows 10, Python 2.7.11 >Reporter: Matthias Baetens >Assignee: Ahmet Altay >Priority: Minor > Labels: newbie > > When having splitting your pipeline into multiple files and configuring your > project according to the Juliaset example > (https://cloud.google.com/dataflow/pipelines/dependencies-python#multiple-file-dependencies), > the Pipeline still crashes when using Windows. > This is caused by setuptools defaulting to a .zip on Windows, and the current > Beam code looks for a .tar.gz (dependency.py, line 400). When changing this > line to: output_files = glob.glob(os.path.join(temp_dir, '*.zip')), it works. > Suggestion: checking the OS would probably solve this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-973) Add end user and developer documentation to gearpump-runner
[ https://issues.apache.org/jira/browse/BEAM-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-973: -- Affects Version/s: (was: 0.4.0-incubating) > Add end user and developer documentation to gearpump-runner > --- > > Key: BEAM-973 > URL: https://issues.apache.org/jira/browse/BEAM-973 > Project: Beam > Issue Type: Sub-task > Components: runner-gearpump >Reporter: Manu Zhang >Assignee: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-840) Add Java SDK extension to support non-distributed sorting
[ https://issues.apache.org/jira/browse/BEAM-840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-840: -- Affects Version/s: (was: 0.4.0-incubating) > Add Java SDK extension to support non-distributed sorting > - > > Key: BEAM-840 > URL: https://issues.apache.org/jira/browse/BEAM-840 > Project: Beam > Issue Type: New Feature > Components: sdk-java-extensions >Reporter: Mitch Shanklin >Assignee: Mitch Shanklin >Priority: Minor > > Add an extension that provides a PTransform which performs > local(non-distributed) sorting. It will sort in memory until the buffer is > full, then flush to disk and use external sorting. > > Consumes a PCollection of KVs from primary key to iterable of secondary key > and value KVs and sorts the iterables. Would probably be called after a > GroupByKey. Uses coders to convert secondary keys and values into byte arrays > and does a lexicographical comparison on the secondary keys. > Uses Hadoop as an external sorting library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-970) Add side-input/side-output support to gearpump-runner
[ https://issues.apache.org/jira/browse/BEAM-970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-970: -- Affects Version/s: (was: 0.4.0-incubating) > Add side-input/side-output support to gearpump-runner > - > > Key: BEAM-970 > URL: https://issues.apache.org/jira/browse/BEAM-970 > Project: Beam > Issue Type: Sub-task > Components: runner-gearpump >Reporter: Manu Zhang >Assignee: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-972) Add basic level of unit testing to gearpump runner
[ https://issues.apache.org/jira/browse/BEAM-972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-972: -- Affects Version/s: (was: 0.4.0-incubating) > Add basic level of unit testing to gearpump runner > -- > > Key: BEAM-972 > URL: https://issues.apache.org/jira/browse/BEAM-972 > Project: Beam > Issue Type: Sub-task > Components: runner-gearpump, testing >Reporter: Manu Zhang >Assignee: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-971) Add support for merging windows in gearpump runner
[ https://issues.apache.org/jira/browse/BEAM-971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-971: -- Affects Version/s: (was: 0.4.0-incubating) > Add support for merging windows in gearpump runner > -- > > Key: BEAM-971 > URL: https://issues.apache.org/jira/browse/BEAM-971 > Project: Beam > Issue Type: Sub-task > Components: runner-gearpump >Reporter: Manu Zhang >Assignee: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-968) Update capability matrix to include gearpump-runner
[ https://issues.apache.org/jira/browse/BEAM-968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-968: -- Affects Version/s: (was: 0.4.0-incubating) > Update capability matrix to include gearpump-runner > --- > > Key: BEAM-968 > URL: https://issues.apache.org/jira/browse/BEAM-968 > Project: Beam > Issue Type: Sub-task > Components: runner-gearpump, website >Reporter: Manu Zhang >Assignee: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1076) Datastore Delete template
[ https://issues.apache.org/jira/browse/BEAM-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-1076: --- Affects Version/s: (was: 0.4.0-incubating) > Datastore Delete template > - > > Key: BEAM-1076 > URL: https://issues.apache.org/jira/browse/BEAM-1076 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Vikas Kedigehalli >Assignee: Vikas Kedigehalli > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-909) Starter archetype's pom doesn't include the right dependencies
[ https://issues.apache.org/jira/browse/BEAM-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745745#comment-15745745 ] Davor Bonaci commented on BEAM-909: --- Re-tagging to fix confusion: to make something release blocking, we normally set Fix Versions field, as opposed to Affects Versions. > Starter archetype's pom doesn't include the right dependencies > -- > > Key: BEAM-909 > URL: https://issues.apache.org/jira/browse/BEAM-909 > Project: Beam > Issue Type: Bug > Components: examples-java >Reporter: Frances Perry >Assignee: Daniel Halperin > Fix For: 0.4.0-incubating > > > Repro: > $ mvn archetype:generate -DarchetypeGroupId=org.apache.beam > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-starter > -DarchetypeVersion=LATEST -DgroupId=org.example > -DartifactId=beam-starter -Dversion="0.1" -DinteractiveMode=false > The resulting pom doesn't seem to have dependencies on any runners or a > profile for enabling them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-909) Starter archetype's pom doesn't include the right dependencies
[ https://issues.apache.org/jira/browse/BEAM-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-909: -- Affects Version/s: (was: 0.4.0-incubating) > Starter archetype's pom doesn't include the right dependencies > -- > > Key: BEAM-909 > URL: https://issues.apache.org/jira/browse/BEAM-909 > Project: Beam > Issue Type: Bug > Components: examples-java >Reporter: Frances Perry >Assignee: Daniel Halperin > Fix For: 0.4.0-incubating > > > Repro: > $ mvn archetype:generate -DarchetypeGroupId=org.apache.beam > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-starter > -DarchetypeVersion=LATEST -DgroupId=org.example > -DartifactId=beam-starter -Dversion="0.1" -DinteractiveMode=false > The resulting pom doesn't seem to have dependencies on any runners or a > profile for enabling them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-909) Starter archetype's pom doesn't include the right dependencies
[ https://issues.apache.org/jira/browse/BEAM-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-909: -- Fix Version/s: 0.4.0-incubating > Starter archetype's pom doesn't include the right dependencies > -- > > Key: BEAM-909 > URL: https://issues.apache.org/jira/browse/BEAM-909 > Project: Beam > Issue Type: Bug > Components: examples-java >Reporter: Frances Perry >Assignee: Daniel Halperin > Fix For: 0.4.0-incubating > > > Repro: > $ mvn archetype:generate -DarchetypeGroupId=org.apache.beam > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-starter > -DarchetypeVersion=LATEST -DgroupId=org.example > -DartifactId=beam-starter -Dversion="0.1" -DinteractiveMode=false > The resulting pom doesn't seem to have dependencies on any runners or a > profile for enabling them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-909) Starter archetype's pom doesn't include the right dependencies
[ https://issues.apache.org/jira/browse/BEAM-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-909: -- Assignee: Daniel Halperin > Starter archetype's pom doesn't include the right dependencies > -- > > Key: BEAM-909 > URL: https://issues.apache.org/jira/browse/BEAM-909 > Project: Beam > Issue Type: Bug > Components: examples-java >Reporter: Frances Perry >Assignee: Daniel Halperin > Fix For: 0.4.0-incubating > > > Repro: > $ mvn archetype:generate -DarchetypeGroupId=org.apache.beam > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-starter > -DarchetypeVersion=LATEST -DgroupId=org.example > -DartifactId=beam-starter -Dversion="0.1" -DinteractiveMode=false > The resulting pom doesn't seem to have dependencies on any runners or a > profile for enabling them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/2] incubator-beam git commit: This closes #1589
This closes #1589 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e776d1d7 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e776d1d7 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e776d1d7 Branch: refs/heads/master Commit: e776d1d705857309107d78abf4eab5b5bc0c27da Parents: f898ff7 592c4d9 Author: Davor Bonaci Authored: Mon Dec 12 21:23:11 2016 -0800 Committer: Davor Bonaci Committed: Mon Dec 12 21:23:11 2016 -0800 -- .../test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-) --
[1/2] incubator-beam git commit: FileBasedSinkTest: fix tests in Windows OS by using IOChannelUtils.resolve().
Repository: incubator-beam
Updated Branches:
refs/heads/master f898ff785 -> e776d1d70
FileBasedSinkTest: fix tests in Windows OS by using IOChannelUtils.resolve().
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/592c4d9f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/592c4d9f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/592c4d9f
Branch: refs/heads/master
Commit: 592c4d9fb9cbca22035e75fdcc77b887d40e86ec
Parents: f898ff7
Author: Pei He
Authored: Mon Dec 12 17:52:07 2016 -0800
Committer: Davor Bonaci
Committed: Mon Dec 12 21:22:47 2016 -0800
--
.../test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java | 7 +++
1 file changed, 3 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/592c4d9f/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
--
diff --git
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
index 930ca29..9f0c424 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
@@ -51,6 +51,7 @@ import org.apache.beam.sdk.io.FileBasedSink.FileResult;
import org.apache.beam.sdk.io.FileBasedSink.WritableByteChannelFactory;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.util.IOChannelUtils;
import
org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream;
import org.junit.Rule;
import org.junit.Test;
@@ -88,8 +89,7 @@ public class FileBasedSinkTest {
@Test
public void testWriter() throws Exception {
String testUid = "testId";
-String expectedFilename =
-getBaseTempDirectory() + "/" + testUid;
+String expectedFilename = IOChannelUtils.resolve(getBaseTempDirectory(),
testUid);
SimpleSink.SimpleWriter writer = buildWriter();
List values = Arrays.asList("sympathetic vulture", "boresome
hummingbird");
@@ -488,8 +488,7 @@ public class FileBasedSinkTest {
.createWriteOperation(null);
final FileBasedWriter writer =
writeOp.createWriter(null);
-final String expectedFilename =
-writeOp.tempDirectory.get() + "/" + testUid;
+final String expectedFilename =
IOChannelUtils.resolve(writeOp.tempDirectory.get(), testUid);
final List expected = new ArrayList<>();
expected.add("header");
[jira] [Resolved] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system
[
https://issues.apache.org/jira/browse/BEAM-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davor Bonaci resolved BEAM-1020.
Resolution: Duplicate
Confirmed duplicate, which is fixed in the short-term. A new JIRA issue to be
filed for the long-term fix.
[~tenghuanhe], thanks for your contribution! I'd love to see you continue
contributing to the project!
> FileBasedSink fails to remove temporary directory on Windows operating system
> -
>
> Key: BEAM-1020
> URL: https://issues.apache.org/jira/browse/BEAM-1020
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Tenghuan He
>Assignee: Tenghuan He
> Fix For: 0.4.0-incubating
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> Beam fails to remove temporary directory after completing pipelines and
> sinking output results on Windows operating system and throws the following
> exception
> {noformat}
> Exception in thread "main"
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
> at
> org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
> at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
> at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
> at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
> at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
> at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
> at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
> at
> org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
> at
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
> at
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
> at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
> {noformat}
> This is due to the use of wildcard * when matching all files in the temporary
> directory, while Windows does not support wildcard * in the file path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (BEAM-863) WordCount on Windows: java.nio.file.InvalidPathException
[ https://issues.apache.org/jira/browse/BEAM-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci resolved BEAM-863. --- Resolution: Fixed This is now fixed, by catching and ignoring the exception. [~pei...@gmail.com], would you mind filing a separate issue for the long term fix? > WordCount on Windows: java.nio.file.InvalidPathException > > > Key: BEAM-863 > URL: https://issues.apache.org/jira/browse/BEAM-863 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Davor Bonaci >Assignee: Pei He > Fix For: 0.4.0-incubating > > > Running basic, unchanged WordCount on Windows with local files results in an > exception during cleanup: > Caused by: java.nio.file.InvalidPathException: Illegal char <\*> at index 46: > C:\\dir\\beam_material\\result-temp-\* at > sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:18) at > org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:392) > More details: > http://stackoverflow.com/questions/40285433/apache-beam-exception-while-running-wordcount-example-in-eclipse > We need to address this for the next release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/2] incubator-beam git commit: FileBasedSink: ignore exceptions when removing temp output files for issues in Windows OS.
Repository: incubator-beam
Updated Branches:
refs/heads/master 5ef34f7ee -> f898ff785
FileBasedSink: ignore exceptions when removing temp output files for issues in
Windows OS.
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c62f0a89
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c62f0a89
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c62f0a89
Branch: refs/heads/master
Commit: c62f0a890dca5229353a9395d9a1f5d6628f7ca0
Parents: 5ef34f7
Author: Pei He
Authored: Mon Dec 12 15:43:14 2016 -0800
Committer: Davor Bonaci
Committed: Mon Dec 12 21:14:06 2016 -0800
--
.../org/apache/beam/sdk/io/FileBasedSink.java| 19 +++
1 file changed, 15 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c62f0a89/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
--
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
index 33296b4..32b8b4f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
@@ -30,7 +30,6 @@ import java.nio.channels.Channels;
import java.nio.channels.WritableByteChannel;
import java.nio.file.Path;
import java.util.ArrayList;
-import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
@@ -477,7 +476,14 @@ public abstract class FileBasedSink extends Sink {
// (produced by successfully completed bundles).
// This may still fail to remove temporary outputs of some failed
bundles, but at least
// the common case (where all bundles succeed) is guaranteed to be fully
addressed.
- Collection matches = factory.match(factory.resolve(tempDir,
"*"));
+ Set matches = new HashSet<>();
+ // TODO: Windows OS cannot resolves and matches '*' in the path,
+ // ignore the exception for now to avoid failing the pipeline.
+ try {
+matches.addAll(factory.match(factory.resolve(tempDir, "*")));
+ } catch (Exception e) {
+LOG.warn("Failed to match temporary files under: [{}].", tempDir);
+ }
Set allMatches = new HashSet<>(matches);
allMatches.addAll(knownFiles);
LOG.debug(
@@ -486,8 +492,13 @@ public abstract class FileBasedSink extends Sink {
tempDir,
matches.size(),
allMatches.size() - matches.size());
- factory.remove(allMatches);
- factory.remove(ImmutableList.of(tempDir));
+ // Deletion of the temporary directory might fail, if not all temporary
files are removed.
+ try {
+factory.remove(allMatches);
+factory.remove(ImmutableList.of(tempDir));
+ } catch (Exception e) {
+LOG.warn("Failed to remove temporary directory: [{}].", tempDir);
+ }
}
/**
[2/2] incubator-beam git commit: This closes #1587
This closes #1587 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f898ff78 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f898ff78 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f898ff78 Branch: refs/heads/master Commit: f898ff785e613394d07bfb3ca266d000524e4adb Parents: 5ef34f7 c62f0a8 Author: Davor Bonaci Authored: Mon Dec 12 21:14:15 2016 -0800 Committer: Davor Bonaci Committed: Mon Dec 12 21:14:15 2016 -0800 -- .../org/apache/beam/sdk/io/FileBasedSink.java| 19 +++ 1 file changed, 15 insertions(+), 4 deletions(-) --
[jira] [Resolved] (BEAM-1142) maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488]
[ https://issues.apache.org/jira/browse/BEAM-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci resolved BEAM-1142. Resolution: Fixed Fix Version/s: 0.4.0-incubating > maven-archetype-starter failed in Windows OS due to [ARCHETYPE-488] > --- > > Key: BEAM-1142 > URL: https://issues.apache.org/jira/browse/BEAM-1142 > Project: Beam > Issue Type: Bug > Components: examples-java >Reporter: Pei He >Assignee: Pei He > Fix For: 0.4.0-incubating > > > mvn clean install -DskipTests failed in maven-archetype-starter > (Windows OS, maven version 3.3.x) > Caused by: https://issues.apache.org/jira/browse/ARCHETYPE-488 > Logs: > [INFO] Invoking post-archetype-generation goals: verify > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 32.082 s > [INFO] Finished at: 2016-12-13T00:21:12+00:00 > [INFO] Final Memory: 32M/78M > [INFO] > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-archetype-plugin:2 > .4:integration-test (default-integration-test) on project > beam-sdks-java-maven-a > rchetypes-starter: > [ERROR] Archetype IT 'basic' failed: Cannot run additions goals. > [ERROR] -> [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal o > rg.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test > (default-int > egration-test) on project beam-sdks-java-maven-archetypes-starter: > Archetype IT 'basic' failed: Cannot run additions goals. > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:212) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje > ct(LifecycleModuleBuilder.java:116) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProje > ct(LifecycleModuleBuilder.java:80) > at > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThre > adedBuilder.build(SingleThreadedBuilder.java:51) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(Lifecycl > eStarter.java:128) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) > at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. > java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces > sorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Laun > cher.java:289) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav > a:229) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La > uncher.java:415) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java: > 356) > Caused by: org.apache.maven.plugin.MojoExecutionException: > Archetype IT 'basic' failed: Cannot run additions goals. > at > org.apache.maven.archetype.mojos.IntegrationTestMojo.execute(Integrat > ionTestMojo.java:269) > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(Default > BuildPluginManager.java:134) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor > .java:207) > ... 20 more > [ERROR] > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > rea > d the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE > xception -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/2] incubator-beam git commit: [BEAM-1142] Upgrade maven-invoker to address maven bug ARCHETYPE-488.
Repository: incubator-beam Updated Branches: refs/heads/master 91cc606b4 -> 5ef34f7ee [BEAM-1142] Upgrade maven-invoker to address maven bug ARCHETYPE-488. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2b84b7bc Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2b84b7bc Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2b84b7bc Branch: refs/heads/master Commit: 2b84b7bc08e4e351074f5cbd9f20e908295bd7d8 Parents: 91cc606 Author: Pei He Authored: Mon Dec 12 17:41:59 2016 -0800 Committer: Davor Bonaci Committed: Mon Dec 12 21:08:45 2016 -0800 -- sdks/java/maven-archetypes/examples-java8/pom.xml | 7 +++ sdks/java/maven-archetypes/examples/pom.xml | 7 +++ sdks/java/maven-archetypes/starter/pom.xml| 7 +++ 3 files changed, 21 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2b84b7bc/sdks/java/maven-archetypes/examples-java8/pom.xml -- diff --git a/sdks/java/maven-archetypes/examples-java8/pom.xml b/sdks/java/maven-archetypes/examples-java8/pom.xml index 5cf094e..4aeb627 100644 --- a/sdks/java/maven-archetypes/examples-java8/pom.xml +++ b/sdks/java/maven-archetypes/examples-java8/pom.xml @@ -50,6 +50,13 @@ maven-archetype-plugin 2.4 + + + org.apache.maven.shared + maven-invoker + 2.2 + +
[2/2] incubator-beam git commit: This closes #1588
This closes #1588 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5ef34f7e Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5ef34f7e Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5ef34f7e Branch: refs/heads/master Commit: 5ef34f7ee8ecb89e4eae3b06fcd19e2102cd367b Parents: 91cc606 2b84b7b Author: Davor Bonaci Authored: Mon Dec 12 21:08:56 2016 -0800 Committer: Davor Bonaci Committed: Mon Dec 12 21:08:56 2016 -0800 -- sdks/java/maven-archetypes/examples-java8/pom.xml | 7 +++ sdks/java/maven-archetypes/examples/pom.xml | 7 +++ sdks/java/maven-archetypes/starter/pom.xml| 7 +++ 3 files changed, 21 insertions(+) --
[3/3] incubator-beam-site git commit: This closes #107
This closes #107 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/b46a41fa Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/b46a41fa Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/b46a41fa Branch: refs/heads/asf-site Commit: b46a41fab2375ced039706dcfeaa6f122cdfe5d7 Parents: a4eb4de d1b19ee Author: Davor Bonaci Authored: Mon Dec 12 17:33:41 2016 -0800 Committer: Davor Bonaci Committed: Mon Dec 12 17:33:41 2016 -0800 -- content/get-started/quickstart/index.html | 3 ++- src/get-started/quickstart.md | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) --
[2/3] incubator-beam-site git commit: Regenerate website
Regenerate website Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/d1b19ee3 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/d1b19ee3 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/d1b19ee3 Branch: refs/heads/asf-site Commit: d1b19ee3997c5fb1518074d715f74153c9f0d2f7 Parents: 5eaa92b Author: Davor Bonaci Authored: Mon Dec 12 17:33:41 2016 -0800 Committer: Davor Bonaci Committed: Mon Dec 12 17:33:41 2016 -0800 -- content/get-started/quickstart/index.html | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/d1b19ee3/content/get-started/quickstart/index.html -- diff --git a/content/get-started/quickstart/index.html b/content/get-started/quickstart/index.html index cc5e1ed..2e2fc6a 100644 --- a/content/get-started/quickstart/index.html +++ b/content/get-started/quickstart/index.html @@ -247,7 +247,8 @@ You can monitor the running job by visiting the Flink dashboard at http://<fl $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ -Dexec.args="--runner=DataflowRunner --gcpTempLocation=gs://<your-gcs-bucket>/tmp \ - --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs://<your-gcs-bucket>/counts" + --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs://<your-gcs-bucket>/counts" \ + -Pdataflow-runner
[1/3] incubator-beam-site git commit: Dataflow quickstart fix
Repository: incubator-beam-site Updated Branches: refs/heads/asf-site a4eb4de53 -> b46a41fab Dataflow quickstart fix Signed-off-by: Jason Kuster Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/5eaa92b2 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/5eaa92b2 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/5eaa92b2 Branch: refs/heads/asf-site Commit: 5eaa92b20f322a4f54214e9976632e5a30d05421 Parents: a4eb4de Author: Jason Kuster Authored: Mon Dec 12 17:29:34 2016 -0800 Committer: Davor Bonaci Committed: Mon Dec 12 17:33:13 2016 -0800 -- src/get-started/quickstart.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5eaa92b2/src/get-started/quickstart.md -- diff --git a/src/get-started/quickstart.md b/src/get-started/quickstart.md index 76777b0..e10f9f3 100644 --- a/src/get-started/quickstart.md +++ b/src/get-started/quickstart.md @@ -105,7 +105,8 @@ $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ ``` $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ -Dexec.args="--runner=DataflowRunner --gcpTempLocation=gs:///tmp \ - --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs:///counts" + --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs:///counts" \ + -Pdataflow-runner ```
[jira] [Updated] (BEAM-975) Issue with MongoDBIO
[ https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-975: -- Fix Version/s: (was: 0.4.0-incubating) > Issue with MongoDBIO > > > Key: BEAM-975 > URL: https://issues.apache.org/jira/browse/BEAM-975 > Project: Beam > Issue Type: Bug > Components: sdk-java-extensions >Reporter: Reza Nouri >Assignee: Jean-Baptiste Onofré > > It appears that there is an issue with MongoDBIO. I am using Apache Beam in a > REST service that reads data from Mongo. After a number of requests, mongoIO > throws the following exception: > com.mongodb.MongoSocketReadException: Prematurely reached end of stream > at com.mongodb.connection.SocketStream.read(SocketStream.java:88) > at > com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491) > at > com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221) > at > com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134) > at > com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121) > at > com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32) > at > com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83) > at > com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43) > at > com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115) > at > com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46) > at > com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381) > at > com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96) > at > com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82) > at > com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72) > at > com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86) > at > com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237) > at > com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212) > at com.mongodb.operation.FindOperation.execute(FindOperation.java:482) > at com.mongodb.operation.FindOperation.execute(FindOperation.java:79) > at com.mongodb.Mongo.execute(Mongo.java:772) > at com.mongodb.Mongo$2.execute(Mongo.java:759) > at com.mongodb.OperationIterable.iterator(OperationIterable.java:47) > at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143) > at > org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359) > at > org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99) > at > org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154) > at > org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > I suppose there must be a problem with Mongo connection which causes this > issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1141) Beam Quickstart doesn't work on DataflowRunner
[
https://issues.apache.org/jira/browse/BEAM-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743694#comment-15743694
]
Davor Bonaci commented on BEAM-1141:
Need to add a profile for the Dataflow runner, as all other runners are doing.
> Beam Quickstart doesn't work on DataflowRunner
> --
>
> Key: BEAM-1141
> URL: https://issues.apache.org/jira/browse/BEAM-1141
> Project: Beam
> Issue Type: Bug
> Components: website
>Reporter: Jason Kuster
>Assignee: Davor Bonaci
>
> Repro steps:
> In new dir, run current archetype:generate command:
> {code}
> mvn archetype:generate \
>
> -DarchetypeRepository=https://repository.apache.org/content/groups/snapshots \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=LATEST \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
> {code}
> Then run the sample command for DataflowRunner:
> {code}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>-Dexec.args="--runner=DataflowRunner
> --gcpTempLocation=gs:///tmp \
> --inputFile=gs://apache-beam-samples/shakespeare/*
> --output=gs:///counts"
> {code}
> Expected outcome: WordCount runs on Dataflow
> Actual outcome:
> {code}
> [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ word-count-beam ---
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Unknown 'runner' specified
> 'DataflowRunner', supported pipeline runners [DirectRunner]
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1610)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:105)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:290)
> at org.apache.beam.examples.WordCount.main(WordCount.java:174)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException: DataflowRunner
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1596)
> ... 9 more
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 17.916 s
> [INFO] Finished at: 2016-12-12T16:38:55-08:00
> [INFO] Final Memory: 29M/266M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java
> (default-cli) on project word-count-beam: An exception occured while
> executing the Java class. null: InvocationTargetException: Unknown 'runner'
> specified 'DataflowRunner', supported pipeline runners [DirectRunner] ->
> [Help 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BEAM-1141) Beam Quickstart doesn't work on DataflowRunner
[
https://issues.apache.org/jira/browse/BEAM-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davor Bonaci updated BEAM-1141:
---
Component/s: website
> Beam Quickstart doesn't work on DataflowRunner
> --
>
> Key: BEAM-1141
> URL: https://issues.apache.org/jira/browse/BEAM-1141
> Project: Beam
> Issue Type: Bug
> Components: website
>Reporter: Jason Kuster
>Assignee: Davor Bonaci
>
> Repro steps:
> In new dir, run current archetype:generate command:
> {code}
> mvn archetype:generate \
>
> -DarchetypeRepository=https://repository.apache.org/content/groups/snapshots \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=LATEST \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
> {code}
> Then run the sample command for DataflowRunner:
> {code}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>-Dexec.args="--runner=DataflowRunner
> --gcpTempLocation=gs:///tmp \
> --inputFile=gs://apache-beam-samples/shakespeare/*
> --output=gs:///counts"
> {code}
> Expected outcome: WordCount runs on Dataflow
> Actual outcome:
> {code}
> [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ word-count-beam ---
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Unknown 'runner' specified
> 'DataflowRunner', supported pipeline runners [DirectRunner]
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1610)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:105)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:290)
> at org.apache.beam.examples.WordCount.main(WordCount.java:174)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException: DataflowRunner
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1596)
> ... 9 more
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 17.916 s
> [INFO] Finished at: 2016-12-12T16:38:55-08:00
> [INFO] Final Memory: 29M/266M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java
> (default-cli) on project word-count-beam: An exception occured while
> executing the Java class. null: InvocationTargetException: Unknown 'runner'
> specified 'DataflowRunner', supported pipeline runners [DirectRunner] ->
> [Help 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-1132) Jenkins JaCoCo plugin for Beam
[ https://issues.apache.org/jira/browse/BEAM-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743538#comment-15743538 ] Davor Bonaci commented on BEAM-1132: Yup -- something to consider. Not urgent, however. > Jenkins JaCoCo plugin for Beam > -- > > Key: BEAM-1132 > URL: https://issues.apache.org/jira/browse/BEAM-1132 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Kenneth Knowles >Assignee: Jason Kuster >Priority: Minor > > Jenkins has a JaCoCo plugin and other Apache projects use it. We should try > it too (and might as well disable coveralls, as I don't know anyone who looks > at it). > If this takes more than 10 minutes to set up, then another option is to just > archive JaCoCo's HTML reports so we can browse them. > Either of these should take just minutes and yield huge benefits in > visibility of where we have really bad coverage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1132) Jenkins JaCoCo plugin for Beam
[ https://issues.apache.org/jira/browse/BEAM-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-1132: --- Priority: Minor (was: Major) > Jenkins JaCoCo plugin for Beam > -- > > Key: BEAM-1132 > URL: https://issues.apache.org/jira/browse/BEAM-1132 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Kenneth Knowles >Assignee: Jason Kuster >Priority: Minor > > Jenkins has a JaCoCo plugin and other Apache projects use it. We should try > it too (and might as well disable coveralls, as I don't know anyone who looks > at it). > If this takes more than 10 minutes to set up, then another option is to just > archive JaCoCo's HTML reports so we can browse them. > Either of these should take just minutes and yield huge benefits in > visibility of where we have really bad coverage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1132) Jenkins JaCoCo plugin for Beam
[ https://issues.apache.org/jira/browse/BEAM-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-1132: --- Assignee: Jason Kuster (was: Davor Bonaci) > Jenkins JaCoCo plugin for Beam > -- > > Key: BEAM-1132 > URL: https://issues.apache.org/jira/browse/BEAM-1132 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Kenneth Knowles >Assignee: Jason Kuster > > Jenkins has a JaCoCo plugin and other Apache projects use it. We should try > it too (and might as well disable coveralls, as I don't know anyone who looks > at it). > If this takes more than 10 minutes to set up, then another option is to just > archive JaCoCo's HTML reports so we can browse them. > Either of these should take just minutes and yield huge benefits in > visibility of where we have really bad coverage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-1135) Make WindowedWordCount example more independent of runner and execution style
[ https://issues.apache.org/jira/browse/BEAM-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci resolved BEAM-1135. Resolution: Fixed [~kenn] just merged a PR that fixes this. > Make WindowedWordCount example more independent of runner and execution style > - > > Key: BEAM-1135 > URL: https://issues.apache.org/jira/browse/BEAM-1135 > Project: Beam > Issue Type: Improvement > Components: examples-java >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > Fix For: 0.4.0-incubating > > > Right now, WindowedWordCount has not received the same nice revision as our > other examples. The PR has been open for a bit I am breaking it out into its > own ticket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1121) Update documentation following rename of PTransform.apply
[ https://issues.apache.org/jira/browse/BEAM-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742628#comment-15742628 ] Davor Bonaci commented on BEAM-1121: [~kenn], can you take a peek? I'm removing the Fix Versions field, since the documentation doesn't get into the release itself. > Update documentation following rename of PTransform.apply > - > > Key: BEAM-1121 > URL: https://issues.apache.org/jira/browse/BEAM-1121 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Daniel Halperin >Assignee: Kenneth Knowles > > Since PTransform#apply does not exist any more, significant website > documentation may be wrong. > Fix version: 0.4.0-incubating really just means this needs to be done as part > of the 0.4.0-incubating release, since this change will make it into said > release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1121) Update documentation following rename of PTransform.apply
[ https://issues.apache.org/jira/browse/BEAM-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-1121: --- Fix Version/s: (was: 0.4.0-incubating) > Update documentation following rename of PTransform.apply > - > > Key: BEAM-1121 > URL: https://issues.apache.org/jira/browse/BEAM-1121 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Daniel Halperin >Assignee: Kenneth Knowles > > Since PTransform#apply does not exist any more, significant website > documentation may be wrong. > Fix version: 0.4.0-incubating really just means this needs to be done as part > of the 0.4.0-incubating release, since this change will make it into said > release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1092) Shade commonly used libraries (e.g. Guava) to avoid class conflicts
[ https://issues.apache.org/jira/browse/BEAM-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742623#comment-15742623 ] Davor Bonaci commented on BEAM-1092: Doesn't sound like a release blocker, so removing the Fix Versions field. This is a larger item, across all project modules -- we should definitely do this, but it requires some thought, I think. > Shade commonly used libraries (e.g. Guava) to avoid class conflicts > --- > > Key: BEAM-1092 > URL: https://issues.apache.org/jira/browse/BEAM-1092 > Project: Beam > Issue Type: Bug > Components: examples-java, sdk-java-extensions >Affects Versions: 0.3.0-incubating >Reporter: Maximilian Michels >Assignee: Frances Perry > > Beam shades away some of its dependencies like Guava to avoid user classes > from clashing with these dependencies. Some of the artifacts, e.g. KafkaIO, > do not shade any classes and directly depend on potentially conflicting > libraries (e.g. Guava). Also, users might manually add such libraries as > dependencies. > Runners who add classes to the classpath (e.g. Hadoop) can run into conflict > with multiple versions of the same class. To prevent that, we should adjust > the Maven archetypes pom files used for the Quickstart to perform shading of > commonly used libraries (again, Guava is often the culprit). > To prevent the problem in the first place, we should expand the shading of > Guava and other libraries to all modules which make use of these. > To solve both dimensions of the issue, we need to address: > 1. Adding shading of commonly used libraries to the archetypes poms > 2. Properly shade all commonly used libraries in the SDK modules > 2) seems to be of highest priority since it affects users who simply use the > provided IO modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1092) Shade commonly used libraries (e.g. Guava) to avoid class conflicts
[ https://issues.apache.org/jira/browse/BEAM-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-1092: --- Fix Version/s: (was: 0.4.0-incubating) > Shade commonly used libraries (e.g. Guava) to avoid class conflicts > --- > > Key: BEAM-1092 > URL: https://issues.apache.org/jira/browse/BEAM-1092 > Project: Beam > Issue Type: Bug > Components: examples-java, sdk-java-extensions >Affects Versions: 0.3.0-incubating >Reporter: Maximilian Michels >Assignee: Frances Perry > > Beam shades away some of its dependencies like Guava to avoid user classes > from clashing with these dependencies. Some of the artifacts, e.g. KafkaIO, > do not shade any classes and directly depend on potentially conflicting > libraries (e.g. Guava). Also, users might manually add such libraries as > dependencies. > Runners who add classes to the classpath (e.g. Hadoop) can run into conflict > with multiple versions of the same class. To prevent that, we should adjust > the Maven archetypes pom files used for the Quickstart to perform shading of > commonly used libraries (again, Guava is often the culprit). > To prevent the problem in the first place, we should expand the shading of > Guava and other libraries to all modules which make use of these. > To solve both dimensions of the issue, we need to address: > 1. Adding shading of commonly used libraries to the archetypes poms > 2. Properly shade all commonly used libraries in the SDK modules > 2) seems to be of highest priority since it affects users who simply use the > provided IO modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system
[
https://issues.apache.org/jira/browse/BEAM-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742619#comment-15742619
]
Davor Bonaci commented on BEAM-1020:
[~pei...@gmail.com], is this a duplicate of BEAM-863? It is currently marked
0.4.0-incubating release blocking. Any thoughts here?
> FileBasedSink fails to remove temporary directory on Windows operating system
> -
>
> Key: BEAM-1020
> URL: https://issues.apache.org/jira/browse/BEAM-1020
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Tenghuan He
>Assignee: Tenghuan He
> Fix For: 0.4.0-incubating
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> Beam fails to remove temporary directory after completing pipelines and
> sinking output results on Windows operating system and throws the following
> exception
> {noformat}
> Exception in thread "main"
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
> at
> org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
> at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
> at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
> at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
> at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
> at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
> at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
> at
> org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
> at
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
> at
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
> at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
> {noformat}
> This is due to the use of wildcard * when matching all files in the temporary
> directory, while Windows does not support wildcard * in the file path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-974) Adds attributes support to PubsubIO
[ https://issues.apache.org/jira/browse/BEAM-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742607#comment-15742607 ] Davor Bonaci commented on BEAM-974: --- This is an improvement, hence it doesn't seem to be 0.4.0-incubating release blocking, right? > Adds attributes support to PubsubIO > --- > > Key: BEAM-974 > URL: https://issues.apache.org/jira/browse/BEAM-974 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Affects Versions: 0.3.0-incubating >Reporter: Reuven Lax >Assignee: Reuven Lax > > Adds support for Pubsub attributes by introducing a new PubsubMessage class > that allows manipulation of payload and attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-974) Adds attributes support to PubsubIO
[ https://issues.apache.org/jira/browse/BEAM-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-974: -- Fix Version/s: (was: 0.4.0-incubating) > Adds attributes support to PubsubIO > --- > > Key: BEAM-974 > URL: https://issues.apache.org/jira/browse/BEAM-974 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Affects Versions: 0.3.0-incubating >Reporter: Reuven Lax >Assignee: Reuven Lax > > Adds support for Pubsub attributes by introducing a new PubsubMessage class > that allows manipulation of payload and attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-863) WordCount on Windows: java.nio.file.InvalidPathException
[ https://issues.apache.org/jira/browse/BEAM-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-863: -- Assignee: Pei He > WordCount on Windows: java.nio.file.InvalidPathException > > > Key: BEAM-863 > URL: https://issues.apache.org/jira/browse/BEAM-863 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Davor Bonaci >Assignee: Pei He > Fix For: 0.4.0-incubating > > > Running basic, unchanged WordCount on Windows with local files results in an > exception during cleanup: > Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 46: > C:\\dir\\beam_material\\result-temp-* at > sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:18) at > org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:392) > More details: > http://stackoverflow.com/questions/40285433/apache-beam-exception-while-running-wordcount-example-in-eclipse > We need to address this for the next release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-863) WordCount on Windows: java.nio.file.InvalidPathException
[ https://issues.apache.org/jira/browse/BEAM-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742597#comment-15742597 ] Davor Bonaci commented on BEAM-863: --- [~pei...@gmail.com], would you mind taking a look here? This seems to be blocking 0.4.0-incubating, and we'd like to start that release as soon as possible. > WordCount on Windows: java.nio.file.InvalidPathException > > > Key: BEAM-863 > URL: https://issues.apache.org/jira/browse/BEAM-863 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Davor Bonaci >Assignee: Pei He > Fix For: 0.4.0-incubating > > > Running basic, unchanged WordCount on Windows with local files results in an > exception during cleanup: > Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 46: > C:\\dir\\beam_material\\result-temp-* at > sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:18) at > org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:392) > More details: > http://stackoverflow.com/questions/40285433/apache-beam-exception-while-running-wordcount-example-in-eclipse > We need to address this for the next release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-969) Add a gearpump runner web page under "learn/runners"
[ https://issues.apache.org/jira/browse/BEAM-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-969: -- Fix Version/s: (was: 0.4.0-incubating) > Add a gearpump runner web page under "learn/runners" > > > Key: BEAM-969 > URL: https://issues.apache.org/jira/browse/BEAM-969 > Project: Beam > Issue Type: Sub-task > Components: runner-gearpump, website >Reporter: Manu Zhang >Assignee: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-969) Add a gearpump runner web page under "learn/runners"
[ https://issues.apache.org/jira/browse/BEAM-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742602#comment-15742602 ] Davor Bonaci commented on BEAM-969: --- Website is not released, hence the issue shouldn't be marked as 0.4.0-incubating blocking. > Add a gearpump runner web page under "learn/runners" > > > Key: BEAM-969 > URL: https://issues.apache.org/jira/browse/BEAM-969 > Project: Beam > Issue Type: Sub-task > Components: runner-gearpump, website >Reporter: Manu Zhang >Assignee: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-843) Use New DoFn Directly in Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742592#comment-15742592 ] Davor Bonaci commented on BEAM-843: --- [~aljoscha], is this a blocker 0.4.0-incubating? Sounds like it shouldn't be, right? > Use New DoFn Directly in Flink Runner > - > > Key: BEAM-843 > URL: https://issues.apache.org/jira/browse/BEAM-843 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Aljoscha Krettek > Fix For: 0.4.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-806) Maven Release Plugin Does Not Set Archetype Versions
[
https://issues.apache.org/jira/browse/BEAM-806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davor Bonaci updated BEAM-806:
--
Fix Version/s: (was: 0.4.0-incubating)
> Maven Release Plugin Does Not Set Archetype Versions
>
>
> Key: BEAM-806
> URL: https://issues.apache.org/jira/browse/BEAM-806
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating, 0.3.0-incubating
>Reporter: Aljoscha Krettek
>Priority: Blocker
>
> When running {{mvn release:prepare}} as described in the new release guide
> this does not update the version of the poms in the archetypes. To be clear,
> the version of the archetype pom is updated, the pom in
> {{archetype-resources}} (the pom of the project generated by the archetype)
> is not updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-806) Maven Release Plugin Does Not Set Archetype Versions
[
https://issues.apache.org/jira/browse/BEAM-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742587#comment-15742587
]
Davor Bonaci commented on BEAM-806:
---
Removing fix version field, as this doesn't seem to block 0.4.0-incubating
release. This is a process issue, and doesn't seem to affect released code. So,
we can probably live with a manual action for this release and try to address
the problem in the near future.
> Maven Release Plugin Does Not Set Archetype Versions
>
>
> Key: BEAM-806
> URL: https://issues.apache.org/jira/browse/BEAM-806
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating, 0.3.0-incubating
>Reporter: Aljoscha Krettek
>Priority: Blocker
>
> When running {{mvn release:prepare}} as described in the new release guide
> this does not update the version of the poms in the archetypes. To be clear,
> the version of the archetype pom is updated, the pom in
> {{archetype-resources}} (the pom of the project generated by the archetype)
> is not updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (BEAM-1126) Expose UnboundedSource split backlog in number of events
[
https://issues.apache.org/jira/browse/BEAM-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15740357#comment-15740357
]
Davor Bonaci edited comment on BEAM-1126 at 12/11/16 9:10 PM:
--
Interesting perspective, thanks [~aviemzur].
I think the primary design goal of the current API was to enable dynamic
optimizations, as opposed to monitoring scenarios. The general idea was that
the source should provide an indication of the amount of pending work, and it
was probably thought that the size in bytes better correlates to "work" than
the size in terms of number of elements. Basically, it was intended that the
consumer of the data is the runner, not the user.
That said, monitoring scenarios are possibly even more important. I think the
idea there was that the source should publish monitoring metrics directly
through Beam abstractions in a runner-independent way. Then, all runners would
get this benefit, with no particular work required, in a metric that makes
sense for that source. (However, I don't think a source can do this today --
but this could a different approach for the same problem.)
Anyways, I'm sure [~dhalp...@google.com] will comment more ;-)
was (Author: davor):
Interesting perspective, thanks [~aviemzur].
I think the primary design goal of the current API was to enable dynamic
optimizations, as opposed to monitoring scenarios. The general idea was that
the source should provide an indication of amount of pending work, and it was
probably thought that the size in bytes better correlates to "work" than the
size in terms of number of elements. Basically, it was intended that the
consumer of the data is the runner, not the user.
That said, monitoring scenarios are possibly even more important. I think the
idea there was that the source should publish monitoring metrics directly
thought Beam abstractions in a runner-independent way. Then, all runners would
get this benefit, with no particular work required, in a metric that makes
sense for that source. (However, I don't think a source can do this today --
but this could a different approach for the same problem.)
Anyways, I'm sure [~dhalp...@google.com] will comment more ;-)
> Expose UnboundedSource split backlog in number of events
>
>
> Key: BEAM-1126
> URL: https://issues.apache.org/jira/browse/BEAM-1126
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Daniel Halperin
>Priority: Minor
>
> Today {{UnboundedSource}} exposes split backlog in bytes via
> {{getSplitBacklogBytes()}}
> There is value in exposing backlog in number of events as well, since this
> number can be more human comprehensible than bytes. something like
> {{getSplitBacklogEvents()}} or {{getSplitBacklogCount()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BEAM-1126) Expose UnboundedSource split backlog in number of events
[
https://issues.apache.org/jira/browse/BEAM-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15740357#comment-15740357
]
Davor Bonaci commented on BEAM-1126:
Interesting perspective, thanks [~aviemzur].
I think the primary design goal of the current API was to enable dynamic
optimizations, as opposed to monitoring scenarios. The general idea was that
the source should provide an indication of amount of pending work, and it was
probably thought that the size in bytes better correlates to "work" than the
size in terms of number of elements. Basically, it was intended that the
consumer of the data is the runner, not the user.
That said, monitoring scenarios are possibly even more important. I think the
idea there was that the source should publish monitoring metrics directly
thought Beam abstractions in a runner-independent way. Then, all runners would
get this benefit, with no particular work required, in a metric that makes
sense for that source. (However, I don't think a source can do this today --
but this could a different approach for the same problem.)
Anyways, I'm sure [~dhalp...@google.com] will comment more ;-)
> Expose UnboundedSource split backlog in number of events
>
>
> Key: BEAM-1126
> URL: https://issues.apache.org/jira/browse/BEAM-1126
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Daniel Halperin
>Priority: Minor
>
> Today {{UnboundedSource}} exposes split backlog in bytes via
> {{getSplitBacklogBytes()}}
> There is value in exposing backlog in number of events as well, since this
> number can be more human comprehensible than bytes. something like
> {{getSplitBacklogEvents()}} or {{getSplitBacklogCount()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[2/2] incubator-beam git commit: This closes #1576
This closes #1576 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0afadf64 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0afadf64 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0afadf64 Branch: refs/heads/master Commit: 0afadf64f85b9495d905fc2a3963219713da3523 Parents: bf8a3cb 36f09c4 Author: Davor Bonaci Authored: Sun Dec 11 12:46:13 2016 -0800 Committer: Davor Bonaci Committed: Sun Dec 11 12:46:13 2016 -0800 -- .../java/org/apache/beam/sdk/io/FileBasedSource.java | 6 -- .../org/apache/beam/sdk/io/FileBasedSourceTest.java | 14 ++ 2 files changed, 18 insertions(+), 2 deletions(-) --
[1/2] incubator-beam git commit: [BEAM-551] Fix toString for FileBasedSource
Repository: incubator-beam
Updated Branches:
refs/heads/master bf8a3cb3a -> 0afadf64f
[BEAM-551] Fix toString for FileBasedSource
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/36f09c4d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/36f09c4d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/36f09c4d
Branch: refs/heads/master
Commit: 36f09c4d13f76d477e3575d8ee5bbfb1825c69ab
Parents: bf8a3cb
Author: Sam McVeety
Authored: Sat Dec 10 15:59:54 2016 -0800
Committer: Davor Bonaci
Committed: Sun Dec 11 12:46:02 2016 -0800
--
.../java/org/apache/beam/sdk/io/FileBasedSource.java | 6 --
.../org/apache/beam/sdk/io/FileBasedSourceTest.java | 14 ++
2 files changed, 18 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/36f09c4d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java
--
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java
index e0fc6b6..d835f9b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java
@@ -411,11 +411,13 @@ public abstract class FileBasedSource extends
OffsetBasedSource {
@Override
public String toString() {
+String fileString = fileOrPatternSpec.isAccessible()
+? fileOrPatternSpec.get() : fileOrPatternSpec.toString();
switch (mode) {
case FILEPATTERN:
-return fileOrPatternSpec.toString();
+return fileString;
case SINGLE_FILE_OR_SUBRANGE:
-return fileOrPatternSpec.toString() + " range " + super.toString();
+return fileString + " range " + super.toString();
default:
throw new IllegalStateException("Unexpected mode: " + mode);
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/36f09c4d/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSourceTest.java
--
diff --git
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSourceTest.java
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSourceTest.java
index dde5d02..a065191 100644
---
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSourceTest.java
+++
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSourceTest.java
@@ -916,4 +916,18 @@ public class FileBasedSourceTest {
TestFileBasedSource source = new TestFileBasedSource(file.getPath(), 1, 0,
file.length(), null);
assertSplitAtFractionExhaustive(source, options);
}
+
+ @Test
+ public void testToStringFile() throws Exception {
+String path = "/tmp/foo";
+TestFileBasedSource source = new TestFileBasedSource(path, 1, 0, 10, null);
+assertEquals(String.format("%s range [0, 10)", path), source.toString());
+ }
+
+ @Test
+ public void testToStringPattern() throws Exception {
+String path = "/tmp/foo/*";
+TestFileBasedSource source = new TestFileBasedSource(path, 1, 0, 10, null);
+assertEquals(String.format("%s range [0, 10)", path), source.toString());
+ }
}
[2/3] incubator-beam-site git commit: Regenerate website
Regenerate website Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/2245a27d Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/2245a27d Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/2245a27d Branch: refs/heads/asf-site Commit: 2245a27d028c967ada0c8b22d0ebf20d2acb8d62 Parents: a5dacb9 Author: Davor Bonaci Authored: Sat Dec 10 13:20:58 2016 -0800 Committer: Davor Bonaci Committed: Sat Dec 10 13:20:58 2016 -0800 -- content/get-started/quickstart/index.html | 18 ++ 1 file changed, 14 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2245a27d/content/get-started/quickstart/index.html -- diff --git a/content/get-started/quickstart/index.html b/content/get-started/quickstart/index.html index 36bf203..cc5e1ed 100644 --- a/content/get-started/quickstart/index.html +++ b/content/get-started/quickstart/index.html @@ -186,7 +186,7 @@ This will create a directory word-count-beam that contains a simple pom.xml and a series of example pipelines that count words in text files. -$ cd beam-word-count/ +$ cd word-count-beam/ $ ls pom.xmlsrc @@ -240,7 +240,8 @@ You can monitor the running job by visiting the Flink dashboard at http://<fl -TODO BEAM-900 +$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ + -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" -Pspark-runner @@ -270,7 +271,7 @@ You can monitor the running job by visiting the Flink dashboard at http://<fl -TODO BEAM-900 +$ ls counts* @@ -326,7 +327,16 @@ Foundation: 1 -TODO BEAM-900 +$ more counts* +beam: 27 +SF: 1 +fat: 1 +job: 1 +limitations: 1 +require: 1 +of: 11 +profile: 10 +...
[3/3] incubator-beam-site git commit: This closes #106
This closes #106 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/a4eb4de5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/a4eb4de5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/a4eb4de5 Branch: refs/heads/asf-site Commit: a4eb4de53aa572569a0ee7188c333f76b351f9be Parents: 59f789b 2245a27 Author: Davor Bonaci Authored: Sat Dec 10 13:20:58 2016 -0800 Committer: Davor Bonaci Committed: Sat Dec 10 13:20:58 2016 -0800 -- content/get-started/quickstart/index.html | 18 ++ src/get-started/quickstart.md | 18 ++ 2 files changed, 28 insertions(+), 8 deletions(-) --
[1/3] incubator-beam-site git commit: [BEAM-900] Spark quickstart instructions.
Repository: incubator-beam-site Updated Branches: refs/heads/asf-site 59f789b97 -> a4eb4de53 [BEAM-900] Spark quickstart instructions. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/a5dacb95 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/a5dacb95 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/a5dacb95 Branch: refs/heads/asf-site Commit: a5dacb95d5d8dff7718a52f1a05055702742b860 Parents: 59f789b Author: Sela Authored: Sat Dec 10 22:43:49 2016 +0200 Committer: Davor Bonaci Committed: Sat Dec 10 13:19:50 2016 -0800 -- src/get-started/quickstart.md | 18 ++ 1 file changed, 14 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/a5dacb95/src/get-started/quickstart.md -- diff --git a/src/get-started/quickstart.md b/src/get-started/quickstart.md index 8c1d5e9..76777b0 100644 --- a/src/get-started/quickstart.md +++ b/src/get-started/quickstart.md @@ -42,7 +42,7 @@ $ mvn archetype:generate \ This will create a directory `word-count-beam` that contains a simple `pom.xml` and a series of example pipelines that count words in text files. ``` -$ cd beam-word-count/ +$ cd word-count-beam/ $ ls pom.xmlsrc @@ -97,7 +97,8 @@ You can monitor the running job by visiting the Flink dashboard at http://
[2/2] incubator-beam git commit: This closes #1545
This closes #1545 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e841b1a2 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e841b1a2 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e841b1a2 Branch: refs/heads/master Commit: e841b1a21d850947a8cda9d897f2f51cac5ee2f1 Parents: a834fb0 9ebc465 Author: Davor Bonaci Authored: Sat Dec 10 11:12:36 2016 -0800 Committer: Davor Bonaci Committed: Sat Dec 10 11:12:36 2016 -0800 -- .../beam/runners/dataflow/DataflowRunner.java | 7 ++- .../runners/dataflow/DataflowRunnerTest.java| 20 +++ .../org/apache/beam/sdk/io/FileBasedSink.java | 62 .../apache/beam/sdk/io/FileBasedSinkTest.java | 6 +- .../java/org/apache/beam/sdk/io/TextIOTest.java | 19 ++ .../org/apache/beam/sdk/io/XmlSinkTest.java | 4 +- 6 files changed, 89 insertions(+), 29 deletions(-) --
[1/2] incubator-beam git commit: [BEAM-551] Fix handling of TextIO.Sink
Repository: incubator-beam
Updated Branches:
refs/heads/master a834fb0eb -> e841b1a21
[BEAM-551] Fix handling of TextIO.Sink
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9ebc4653
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9ebc4653
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9ebc4653
Branch: refs/heads/master
Commit: 9ebc4653af39aa40217635dbe6f22450be78
Parents: a834fb0
Author: Sam McVeety
Authored: Wed Dec 7 13:27:53 2016 -0800
Committer: Davor Bonaci
Committed: Sat Dec 10 11:11:38 2016 -0800
--
.../beam/runners/dataflow/DataflowRunner.java | 7 ++-
.../runners/dataflow/DataflowRunnerTest.java| 20 +++
.../org/apache/beam/sdk/io/FileBasedSink.java | 62
.../apache/beam/sdk/io/FileBasedSinkTest.java | 6 +-
.../java/org/apache/beam/sdk/io/TextIOTest.java | 19 ++
.../org/apache/beam/sdk/io/XmlSinkTest.java | 4 +-
6 files changed, 89 insertions(+), 29 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9ebc4653/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 85318e6..d902ccb 100644
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -2065,8 +2065,11 @@ public class DataflowRunner extends
PipelineRunner {
public PDone expand(PCollection input) {
if (transform.getSink() instanceof FileBasedSink) {
FileBasedSink sink = (FileBasedSink) transform.getSink();
-PathValidator validator = runner.options.getPathValidator();
-
validator.validateOutputFilePrefixSupported(sink.getBaseOutputFilename());
+if (sink.getBaseOutputFilenameProvider().isAccessible()) {
+ PathValidator validator = runner.options.getPathValidator();
+ validator.validateOutputFilePrefixSupported(
+ sink.getBaseOutputFilenameProvider().get());
+}
}
return transform.expand(input);
}
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9ebc4653/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
--
diff --git
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
index 133ae8a..4159b61 100644
---
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
+++
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
@@ -82,6 +82,7 @@ import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptions.CheckEnabled;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.runners.TransformHierarchy;
import org.apache.beam.sdk.runners.dataflow.TestCountingSource;
import org.apache.beam.sdk.testing.ExpectedLogs;
@@ -313,6 +314,25 @@ public class DataflowRunnerTest {
assertValidJob(jobCaptor.getValue());
}
+ /** Options for testing. */
+ public interface RuntimeTestOptions extends PipelineOptions {
+ValueProvider getInput();
+void setInput(ValueProvider value);
+
+ValueProvider getOutput();
+void setOutput(ValueProvider value);
+ }
+
+ @Test
+ public void testTextIOWithRuntimeParameters() throws IOException {
+DataflowPipelineOptions dataflowOptions = buildPipelineOptions();
+RuntimeTestOptions options = dataflowOptions.as(RuntimeTestOptions.class);
+Pipeline p = buildDataflowPipeline(dataflowOptions);
+p
+.apply(TextIO.Read.from(options.getInput()).withoutValidation())
+.apply(TextIO.Write.to(options.getOutput()).withoutValidation());
+ }
+
@Test
public void testRunReturnDifferentRequestId() throws IOException {
DataflowPipelineOptions options = buildPipelineOptions();
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9ebc4653/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.j
[jira] [Commented] (BEAM-1067) apex.examples.WordCountTest.testWordCountExample may be flaky
[
https://issues.apache.org/jira/browse/BEAM-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738294#comment-15738294
]
Davor Bonaci commented on BEAM-1067:
More hits:
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5743/
> apex.examples.WordCountTest.testWordCountExample may be flaky
> -
>
> Key: BEAM-1067
> URL: https://issues.apache.org/jira/browse/BEAM-1067
> Project: Beam
> Issue Type: Bug
> Components: runner-apex
>Reporter: Stas Levin
>Assignee: Thomas Weise
>
> Seems that
> {{org.apache.beam.runners.apex.examples.WordCountTest.testWordCountExample}}
> is flaky.
> For example,
> [this|https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5408/org.apache.beam$beam-runners-apex/testReport/org.apache.beam.runners.apex.examples/WordCountTest/testWordCountExample/
> ] run failed although no changes were made in {{runner-apex}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[1/2] incubator-beam git commit: Update badge for new suite name.
Repository: incubator-beam Updated Branches: refs/heads/master 1fab1521a -> 58000b397 Update badge for new suite name. Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0b38c4bd Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0b38c4bd Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0b38c4bd Branch: refs/heads/master Commit: 0b38c4bd75e96f53d9955a7508e2843c22fd0fbf Parents: 1fab152 Author: jasonkuster Authored: Fri Dec 9 18:41:39 2016 -0800 Committer: GitHub Committed: Fri Dec 9 18:41:39 2016 -0800 -- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0b38c4bd/README.md -- diff --git a/README.md b/README.md index 6175006..d919e5b 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ fully endorsed by the ASF. _**The Apache Beam project is in the process of bootstrapping. This includes the creation of project resources, the refactoring of the initial code submissions, and the formulation of project documentation, planning, and design documents. Please expect a significant amount of churn and breaking changes in the near future.**_ [](https://travis-ci.org/apache/incubator-beam?branch=master) -[](https://builds.apache.org/job/beam_PostCommit_MavenVerify/) +[](https://builds.apache.org/job/beam_PostCommit_MavenVerify/) [](https://coveralls.io/github/apache/incubator-beam?branch=master) ## Overview
[2/2] incubator-beam git commit: This closes #1572
This closes #1572 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/58000b39 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/58000b39 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/58000b39 Branch: refs/heads/master Commit: 58000b39713eef4d96d00512a6b559068d2397c9 Parents: 1fab152 0b38c4b Author: Davor Bonaci Authored: Sat Dec 10 08:41:42 2016 -0800 Committer: Davor Bonaci Committed: Sat Dec 10 08:41:42 2016 -0800 -- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --
[jira] [Commented] (BEAM-1126) Expose UnboundedSource split backlog in number of events
[ https://issues.apache.org/jira/browse/BEAM-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738092#comment-15738092 ] Davor Bonaci commented on BEAM-1126: Assigning to [~dhalp...@google.com] for thoughts here. > Expose UnboundedSource split backlog in number of events > > > Key: BEAM-1126 > URL: https://issues.apache.org/jira/browse/BEAM-1126 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Aviem Zur >Assignee: Daniel Halperin >Priority: Minor > > Today UnboundedSource exposes split backlog in bytes via > getSplitBacklogBytes() > There is value in exposing backlog in number of events as well, since this > number can be more human comprehensible than bytes. something like > getSplitBacklogEvents() or getSplitBacklogCount(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BEAM-1126) Expose UnboundedSource split backlog in number of events
[ https://issues.apache.org/jira/browse/BEAM-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci updated BEAM-1126: --- Assignee: Daniel Halperin (was: Davor Bonaci) > Expose UnboundedSource split backlog in number of events > > > Key: BEAM-1126 > URL: https://issues.apache.org/jira/browse/BEAM-1126 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Aviem Zur >Assignee: Daniel Halperin >Priority: Minor > > Today UnboundedSource exposes split backlog in bytes via > getSplitBacklogBytes() > There is value in exposing backlog in number of events as well, since this > number can be more human comprehensible than bytes. something like > getSplitBacklogEvents() or getSplitBacklogCount(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BEAM-835) add Intellij instructions to the contribution guide
[ https://issues.apache.org/jira/browse/BEAM-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci resolved BEAM-835. --- Resolution: Fixed Fix Version/s: Not applicable > add Intellij instructions to the contribution guide > --- > > Key: BEAM-835 > URL: https://issues.apache.org/jira/browse/BEAM-835 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Frances Perry >Assignee: Minudika Malshan >Priority: Minor > Labels: starter > Fix For: Not applicable > > > Add Intellij-specific instructions to the contribution guide, to go alongside > the Eclipse instructions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/3] incubator-beam-site git commit: Rename PTransform apply to expand throughout
Repository: incubator-beam-site
Updated Branches:
refs/heads/asf-site eb5397b83 -> 37adeb9ba
Rename PTransform apply to expand throughout
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit:
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/b68b6805
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/b68b6805
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/b68b6805
Branch: refs/heads/asf-site
Commit: b68b6805960b8c852a6551256827be33054598b9
Parents: eb5397b
Author: Kenneth Knowles
Authored: Fri Dec 9 14:29:44 2016 -0800
Committer: Kenneth Knowles
Committed: Fri Dec 9 14:29:44 2016 -0800
--
src/get-started/mobile-gaming-example.md | 8
src/get-started/wordcount-example.md | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/b68b6805/src/get-started/mobile-gaming-example.md
--
diff --git a/src/get-started/mobile-gaming-example.md
b/src/get-started/mobile-gaming-example.md
index b51f81a..3aee213 100644
--- a/src/get-started/mobile-gaming-example.md
+++ b/src/get-started/mobile-gaming-example.md
@@ -77,7 +77,7 @@ public static class ExtractAndSumScore
}
@Override
- public PCollection> apply(
+ public PCollection> expand(
PCollection gameInfo) {
return gameInfo
@@ -320,7 +320,7 @@ static class CalculateUserScores
}
@Override
- public PCollection> apply(PCollection
input) {
+ public PCollection> expand(PCollection
input) {
return input.apply("LeaderboardUserGlobalWindow",
Window.into(new GlobalWindows())
// Get periodic results every ten minutes.
@@ -374,7 +374,7 @@ static class CalculateTeamScores
}
@Override
- public PCollection> apply(PCollection
infos) {
+ public PCollection> expand(PCollection
infos) {
return infos.apply("LeaderboardTeamFixedWindows",
Window.into(FixedWindows.of(teamWindowDuration))
// We will get early (speculative) results as well as cumulative
@@ -428,7 +428,7 @@ public static class CalculateSpammyUsers
private static final double SCORE_WEIGHT = 2.5;
@Override
- public PCollection> apply(PCollection> userScores) {
+ public PCollection> expand(PCollection> userScores) {
// Get the sum of scores for each user.
PCollection> sumScores = userScores
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/b68b6805/src/get-started/wordcount-example.md
--
diff --git a/src/get-started/wordcount-example.md
b/src/get-started/wordcount-example.md
index 9b02a2b..7c98a99 100644
--- a/src/get-started/wordcount-example.md
+++ b/src/get-started/wordcount-example.md
@@ -188,7 +188,7 @@ When `CountWords` is defined, we specify its ultimate input
and output; the inpu
public static class CountWords extends PTransform,
PCollection>> {
@Override
- public PCollection> apply(PCollection lines) {
+ public PCollection> expand(PCollection lines) {
// Convert lines of text into individual words.
PCollection words = lines.apply(
[3/3] incubator-beam-site git commit: This closes #104
This closes #104 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/37adeb9b Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/37adeb9b Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/37adeb9b Branch: refs/heads/asf-site Commit: 37adeb9ba1110e0835e845fb07158b306202ff0c Parents: eb5397b c5f8a46 Author: Davor Bonaci Authored: Fri Dec 9 14:44:42 2016 -0800 Committer: Davor Bonaci Committed: Fri Dec 9 14:44:42 2016 -0800 -- content/get-started/mobile-gaming-example/index.html | 8 content/get-started/wordcount-example/index.html | 2 +- src/get-started/mobile-gaming-example.md | 8 src/get-started/wordcount-example.md | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-) --
[2/3] incubator-beam-site git commit: Regenerate website
Regenerate website
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit:
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/c5f8a46e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/c5f8a46e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/c5f8a46e
Branch: refs/heads/asf-site
Commit: c5f8a46e8298bd23565552ed4c1e5deed2e82d25
Parents: b68b680
Author: Davor Bonaci
Authored: Fri Dec 9 14:44:42 2016 -0800
Committer: Davor Bonaci
Committed: Fri Dec 9 14:44:42 2016 -0800
--
content/get-started/mobile-gaming-example/index.html | 8
content/get-started/wordcount-example/index.html | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c5f8a46e/content/get-started/mobile-gaming-example/index.html
--
diff --git a/content/get-started/mobile-gaming-example/index.html
b/content/get-started/mobile-gaming-example/index.html
index ebd4b9f..ada7ce7 100644
--- a/content/get-started/mobile-gaming-example/index.html
+++ b/content/get-started/mobile-gaming-example/index.html
@@ -255,7 +255,7 @@
}
@Override
- public PCollection<KV<String, Integer>> apply(
+ public PCollection<KV<String, Integer>> expand(
PCollection<GameActionInfo> gameInfo) {
return gameInfo
@@ -518,7 +518,7 @@
}
@Override
- public PCollection<KV<String, Integer>> apply(PCollection<GameActionInfo> input) {
+ public PCollection<KV<String, Integer>> expand(PCollection<GameActionInfo> input) {
return input.apply("LeaderboardUserGlobalWindow",
Window.<GameActionInfo>into(new GlobalWindows())
// Get periodic results every ten minutes.
@@ -570,7 +570,7 @@
}
@Override
- public PCollection<KV<String, Integer>> apply(PCollection<GameActionInfo> infos) {
+ public PCollection<KV<String, Integer>> expand(PCollection<GameActionInfo> infos) {
return infos.apply("LeaderboardTeamFixedWindows",
Window.<GameActionInfo>into(FixedWindows.of(teamWindowDuration))
// We will get early (speculative) results as
well as cumulative
@@ -628,7 +628,7 @@
private static final double SCORE_WEIGHT = 2.5;
@Override
- public PCollection<KV<String, Integer>> apply(PCollection<KV<String, Integer>> userScores) {
+ public PCollection<KV<String, Integer>> expand(PCollection<KV<String, Integer>> userScores) {
// Get the sum of scores for each user.
PCollection<KV<String, Integer>> sumScores = userScores
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c5f8a46e/content/get-started/wordcount-example/index.html
--
diff --git a/content/get-started/wordcount-example/index.html
b/content/get-started/wordcount-example/index.html
index 5394081..5132240 100644
--- a/content/get-started/wordcount-example/index.html
+++ b/content/get-started/wordcount-example/index.html
@@ -373,7 +373,7 @@ Figure 1: The pipeline data flow.
public static class CountWords extends PTransform<PCollection<String>,
PCollection<KV<String, Long>>> {
@Override
- public PCollection<KV<String, Long>> apply(PCollection<String> lines) {
+ public PCollection<KV<String, Long>> expand(PCollection<String> lines) {
// Convert lines of text into individual words.
PCollection<String> words
= lines.apply(
[2/2] incubator-beam git commit: This closes #1567
This closes #1567 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/05bb254b Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/05bb254b Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/05bb254b Branch: refs/heads/master Commit: 05bb254b8e64fbb2ac598ff998542c089baf37e6 Parents: 58bb117 65dd62c Author: Davor Bonaci Authored: Fri Dec 9 14:43:49 2016 -0800 Committer: Davor Bonaci Committed: Fri Dec 9 14:43:49 2016 -0800 -- .../main/resources/archetype-resources/pom.xml | 116 --- 1 file changed, 101 insertions(+), 15 deletions(-) --
[1/2] incubator-beam git commit: Copy Runner Profiles to Java8 Archetype POM
Repository: incubator-beam
Updated Branches:
refs/heads/master 58bb1174d -> 05bb254b8
Copy Runner Profiles to Java8 Archetype POM
This enables Java8 examples to be run cross-runner
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/65dd62cd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/65dd62cd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/65dd62cd
Branch: refs/heads/master
Commit: 65dd62cd9b1d6f7925ee60b73a5a3fac1e4b1bf1
Parents: 58bb117
Author: Thomas Groh
Authored: Fri Dec 9 11:36:22 2016 -0800
Committer: Thomas Groh
Committed: Fri Dec 9 14:33:46 2016 -0800
--
.../main/resources/archetype-resources/pom.xml | 116 ---
1 file changed, 101 insertions(+), 15 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/65dd62cd/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
--
diff --git
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
index ece8412..693850a 100644
---
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
+++
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
@@ -26,6 +26,10 @@
jar
+
+0.4.0-incubating-SNAPSHOT
+
+
@@ -71,6 +75,103 @@
+
+
+ direct-runner
+
+true
+
+
+
+
+ org.apache.beam
+ beam-runners-direct-java
+ ${beam.version}
+ runtime
+
+
+
+
+
+ apex-runner
+
+
+
+ org.apache.beam
+ beam-runners-apex
+ ${beam.version}
+ runtime
+
+
+
+
+
+ dataflow-runner
+
+
+
+ org.apache.beam
+ beam-runners-google-cloud-dataflow-java
+ ${beam.version}
+ runtime
+
+
+
+
+
+ flink-runner
+
+
+
+ org.apache.beam
+ beam-runners-flink_2.10
+ ${beam.version}
+ runtime
+
+
+
+
+
+org.apache.maven.plugins
+maven-shade-plugin
+
+
+
+
+
+
+ spark-runner
+
+
+
+ org.apache.beam
+ beam-runners-spark
+ ${beam.version}
+ runtime
+
+
+ org.apache.spark
+ spark-streaming_2.10
+ 1.6.2
+ runtime
+
+
+ org.slf4j
+ jul-to-slf4j
+
+
+
+
+ com.fasterxml.jackson.module
+ jackson-module-scala_2.10
+ 2.7.2
+ runtime
+
+
+
+
+
@@ -79,21 +180,6 @@
0.4.0-incubating-SNAPSHOT
-
-
- org.apache.beam
- beam-runners-direct-java
- 0.4.0-incubating-SNAPSHOT
- runtime
-
-
-
- org.apache.beam
- beam-runners-google-cloud-dataflow-java
- 0.4.0-incubating-SNAPSHOT
- runtime
-
-
org.apache.beam
[2/3] incubator-beam-site git commit: Regenerate website
Regenerate website
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit:
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/6b76c3f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/6b76c3f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/6b76c3f4
Branch: refs/heads/asf-site
Commit: 6b76c3f4e0b15236d433909732453e95b635ce87
Parents: 7a9190c
Author: Davor Bonaci
Authored: Fri Dec 9 12:49:30 2016 -0800
Committer: Davor Bonaci
Committed: Fri Dec 9 12:49:30 2016 -0800
--
content/documentation/runners/spark/index.html | 157 +++-
1 file changed, 156 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/6b76c3f4/content/documentation/runners/spark/index.html
--
diff --git a/content/documentation/runners/spark/index.html
b/content/documentation/runners/spark/index.html
index 521cf5d..23960e0 100644
--- a/content/documentation/runners/spark/index.html
+++ b/content/documentation/runners/spark/index.html
@@ -146,7 +146,162 @@
Using the Apache Spark
Runner
-This page is under construction (https://issues.apache.org/jira/browse/BEAM-507";>BEAM-507).
+The Apache Spark Runner can be used to execute Beam pipelines using http://spark.apache.org/";>Apache Spark.
+The Spark Runner can execute Spark pipelines just like a native Spark
application; deploying a self-contained application for local mode, running on
Sparkâs Standalone RM, or using YARN or Mesos.
+
+The Spark Runner executes Beam pipelines on top of Apache Spark,
providing:
+
+
+ Batch and streaming (and combined) pipelines.
+ The same fault-tolerance http://spark.apache.org/docs/1.6.3/streaming-programming-guide.html#fault-tolerance-semantics";>guarantees
as provided by RDDs and DStreams.
+ The same http://spark.apache.org/docs/1.6.3/security.html";>security features
Spark provides.
+ Built-in metrics reporting using Sparkâs metrics system, which reports
Beam Aggregators as well.
+ Native support for Beam side-inputs via sparkâs Broadcast
variables.
+
+
+The Beam Capability
Matrix documents the currently supported capabilities of the Spark
Runner.
+
+Note: support for the Beam Model in streaming
is currently experimental, follow-up in the mailing list for status updates.
+
+Spark Runner prerequisites and
setup
+
+The Spark runner currently supports Sparkâs 1.6 branch, and more
specifically any version greater than 1.6.0.
+
+You can add a dependency on the latest version of the Spark runner by
adding to your pom.xml the following:
+<dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-runners-spark</artifactId>
+ <version>0.3.0-incubating</version>
+</dependency>
+
+
+
+Deploying Spark with your
application
+
+In some cases, such as running in local mode/Standalone, your
(self-contained) application would be required to pack Spark by explicitly
adding the following dependencies in your pom.xml:
+<dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-core_2.10</artifactId>
+ <version>${spark.version}</version>
+</dependency>
+
+<dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-streaming_2.10</artifactId>
+ <version>${spark.version}</version>
+</dependency>
+
+
+
+And shading the application jar using the maven shade plugin:
+<plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <configuration>
+<createDependencyReducedPom>false</createDependencyReducedPom>
+<filters>
+ <filter>
+<artifact>*:*</artifact>
+<excludes>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+</excludes>
+ </filter>
+</filters>
+ </configuration>
+ <executions>
+<execution>
+ <phase>package</phase>
+ <goals>
+<goal>shade</goal>
+ </goals>
+ <configuration>
+<shadedArtifactAttached>true</shadedArtifactAttached>
+<shadedClassifierName>shaded</shadedClassifierName>
+ </configuration>
+</execution>
+ </executions>
+</plugin>
+
+
+
+After running mvn package, run ls target and you
should see (assuming your artifactId is beam-examples and the version is 1.0.0):
+beam-examples-1.0.0-shaded
