[jira] [Updated] (BEAM-147) Introduce an easy API for pipeline metrics

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-147:
---
Description: 
The existing Aggregators are confusing both because of their name and because 
they serve multiple purposes.

Previous discussions around Aggregators/metrics/etc.
See discussion at 
http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
 and 
http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser 
. Exact name still being bikeshedded.

Design document: http://s.apache.org/beam-metrics-api

  was:
The existing Aggregators are confusing both because of their name and because 
they serve multiple purposes.

Previous discussions around Aggregators/metrics/etc.
See discussion at 
http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
 and 
http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser 
. Exact name still being bikeshedded.


> Introduce an easy API for pipeline metrics
> --
>
> Key: BEAM-147
> URL: https://issues.apache.org/jira/browse/BEAM-147
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Robert Bradshaw
>Assignee: Ben Chambers
>
> The existing Aggregators are confusing both because of their name and because 
> they serve multiple purposes.
> Previous discussions around Aggregators/metrics/etc.
> See discussion at 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
>  and 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser
>  . Exact name still being bikeshedded.
> Design document: http://s.apache.org/beam-metrics-api



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1224) Rename options.py to pipeline_options.py

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784783#comment-15784783
 ] 

ASF GitHub Bot commented on BEAM-1224:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/116


> Rename options.py to pipeline_options.py
> 
>
> Key: BEAM-1224
> URL: https://issues.apache.org/jira/browse/BEAM-1224
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: 0.3.0-incubating
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
>   Original Estimate: 144h
>  Remaining Estimate: 144h
>
> options.py contains the class PipelineOptions (and other classes derived from 
> it) and should be renamed pipeline_options.py for clarity. Its test file is 
> pipeline_options_test.py.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam-site pull request #116: [BEAM-1224] Rename apache_beam.utils.options to...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/116


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam-site git commit: Regenerate website

[dhalperi]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/d9ade058
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/d9ade058
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/d9ade058

Branch: refs/heads/asf-site
Commit: d9ade0584b0c4683594825df233a54ac3132787c
Parents: 57c0a0d
Author: Dan Halperin 
Authored: Wed Dec 28 23:46:00 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 23:46:00 2016 -0800

--
 content/documentation/programming-guide/index.html | 4 ++--
 content/documentation/runners/dataflow/index.html  | 2 +-
 content/documentation/runners/direct/index.html| 2 +-
 content/documentation/runners/flink/index.html | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/d9ade058/content/documentation/programming-guide/index.html
--
diff --git a/content/documentation/programming-guide/index.html 
b/content/documentation/programming-guide/index.html
index 1042062..2549abe 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -233,7 +233,7 @@
 
 The pipeline configuration options determine, among other things, the PipelineRunner that determines where the 
pipeline gets executed: locally, or using a distributed back-end of your 
choice. Depending on where your pipeline gets executed and what your specifed 
Runner requires, the options can also help you specify other aspects of 
execution.
 
-To set your pipeline’s configuration options and create the pipeline, 
create an object of type PipelineOptionshttps://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py";>PipelineOptions
 and pass it to Pipeline.Create(). The 
most common way to do this is by parsing arguments from the command-line:
+To set your pipeline’s configuration options and create the pipeline, 
create an object of type PipelineOptionshttps://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/utils/pipeline_options.py";>PipelineOptions
 and pass it to Pipeline.Create(). The 
most common way to do this is by parsing arguments from the command-line:
 
 public static void main(String[] args) {
// Will parse the arguments passed into the application 
and construct a PipelineOptions
@@ -246,7 +246,7 @@
 
 
 
-from apache_beam.utils.options import PipelineOptions
+from apache_beam.utils.pipeline_options import PipelineOptions
 
 # Will parse the arguments passed into the application and 
construct a PipelineOptions
 # Note that --help will print registered options.

http://git-wip-us.apache.org/repos/asf/beam-site/blob/d9ade058/content/documentation/runners/dataflow/index.html
--
diff --git a/content/documentation/runners/dataflow/index.html 
b/content/documentation/runners/dataflow/index.html
index cc4e510..f61b477 100644
--- a/content/documentation/runners/dataflow/index.html
+++ b/content/documentation/runners/dataflow/index.html
@@ -256,7 +256,7 @@
 
 
 
-See the reference documentation for the  DataflowPipelineOptionshttps://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py";>PipelineOptions
 interface (and its subinterfaces) for the complete list of pipeline 
configuration options.
+See the reference documentation for the  DataflowPipelineOptionshttps://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/utils/pipeline_options.py";>PipelineOptions
 interface (and its subinterfaces) for the complete list of pipeline 
configuration options.
 
 Additional information and 
caveats
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/d9ade058/content/documentation/runners/direct/index.html
--
diff --git a/content/documentation/runners/direct/index.html 
b/content/documentation/runners/direct/index.html
index 53d07db..335632f 100644
--- a/content/documentation/runners/direct/index.html
+++ b/content/documentation/runners/direct/index.html
@@ -180,7 +180,7 @@
 
 When executing your pipeline from the command-line, set runner to direct. The default values for the other 
pipeline options are generally sufficient.
 
-See the reference documentation for the  DirectOptionshttps://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py";>PipelineOptions interface (and its 
subinterfaces) for defaults and the complete list of pipeline configuration 
options.
+See the reference documentation for the  DirectOptionshttps://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/utils/pipeline_options.py";>PipelineOptions interface (and its 
subinterfac

[1/3] beam-site git commit: Rename apache_beam.utils.options to apache_beam.utils.pipeline_options

[dhalperi]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 80fc7499d -> d9ade0584


Rename apache_beam.utils.options to apache_beam.utils.pipeline_options


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/69fead60
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/69fead60
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/69fead60

Branch: refs/heads/asf-site
Commit: 69fead60fdbb721df705170a8557aedd5dd4cc6a
Parents: 80fc749
Author: Maria Garcia Herrero 
Authored: Wed Dec 28 23:12:37 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 23:44:41 2016 -0800

--
 src/documentation/programming-guide.md | 4 ++--
 src/documentation/runners/dataflow.md  | 2 +-
 src/documentation/runners/direct.md| 2 +-
 src/documentation/runners/flink.md | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/69fead60/src/documentation/programming-guide.md
--
diff --git a/src/documentation/programming-guide.md 
b/src/documentation/programming-guide.md
index 15528c5..d743c49 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -77,7 +77,7 @@ To use Beam, your driver program must first create an 
instance of the Beam SDK c
 
 The pipeline configuration options determine, among other things, the 
`PipelineRunner` that determines where the pipeline gets executed: locally, or 
using a distributed back-end of your choice. Depending on where your pipeline 
gets executed and what your specifed Runner requires, the options can also help 
you specify other aspects of execution.
 
-To set your pipeline's configuration options and create the pipeline, create 
an object of type [PipelineOptions]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/sdk/options/PipelineOptions.html)[PipelineOptions](https://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py)
 and pass it to `Pipeline.Create()`. The most common way to do this is by 
parsing arguments from the command-line:
+To set your pipeline's configuration options and create the pipeline, create 
an object of type [PipelineOptions]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/sdk/options/PipelineOptions.html)[PipelineOptions](https://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/utils/pipeline_options.py)
 and pass it to `Pipeline.Create()`. The most common way to do this is by 
parsing arguments from the command-line:
 
 ```java
 public static void main(String[] args) {
@@ -91,7 +91,7 @@ public static void main(String[] args) {
 ```
 
 ```py
-from apache_beam.utils.options import PipelineOptions
+from apache_beam.utils.pipeline_options import PipelineOptions
 
 # Will parse the arguments passed into the application and construct a 
PipelineOptions
 # Note that --help will print registered options.

http://git-wip-us.apache.org/repos/asf/beam-site/blob/69fead60/src/documentation/runners/dataflow.md
--
diff --git a/src/documentation/runners/dataflow.md 
b/src/documentation/runners/dataflow.md
index 14a133c..f707d47 100644
--- a/src/documentation/runners/dataflow.md
+++ b/src/documentation/runners/dataflow.md
@@ -101,7 +101,7 @@ When executing your pipeline with the Cloud Dataflow 
Runner, set these pipeline
 
 
 
-See the reference documentation for the  [DataflowPipelineOptions]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html)[PipelineOptions](https://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py)
 interface (and its subinterfaces) for the complete list of pipeline 
configuration options.
+See the reference documentation for the  [DataflowPipelineOptions]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html)[PipelineOptions](https://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/utils/pipeline_options.py)
 interface (and its subinterfaces) for the complete list of pipeline 
configuration options.
 
 ## Additional information and caveats
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/69fead60/src/documentation/runners/direct.md
--
diff --git a/src/documentation/runners/direct.md 
b/src/documentation/runners/direct.md
index b72b933..c96e7b8 100644
--- a/src/documentation/runners/direct.md
+++ b/src/documentation/runners/direct.md
@@ 

[2/3] beam-site git commit: This closes #116

[dhalperi]

This closes #116


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/57c0a0d2
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/57c0a0d2
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/57c0a0d2

Branch: refs/heads/asf-site
Commit: 57c0a0d212a4882d10e590b275e641db19d39c4b
Parents: 80fc749 69fead6
Author: Dan Halperin 
Authored: Wed Dec 28 23:45:18 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 23:45:18 2016 -0800

--
 src/documentation/programming-guide.md | 4 ++--
 src/documentation/runners/dataflow.md  | 2 +-
 src/documentation/runners/direct.md| 2 +-
 src/documentation/runners/flink.md | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)
--

[jira] [Commented] (BEAM-1224) Rename options.py to pipeline_options.py

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784748#comment-15784748
 ] 

ASF GitHub Bot commented on BEAM-1224:
--

GitHub user mariapython opened a pull request:

https://github.com/apache/beam-site/pull/116

[BEAM-1224] Rename apache_beam.utils.options to 
apache_beam.utils.pipeline_options

Update documentation snippets that reference apache_beam.utils.options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariapython/beam-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/116.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #116


commit e94e15b2bac716bc6b8b1e6b6e72d049f64bfe93
Author: Maria Garcia Herrero 
Date:   2016-12-29T07:12:37Z

Rename apache_beam.utils.options to apache_beam.utils.pipeline_options




> Rename options.py to pipeline_options.py
> 
>
> Key: BEAM-1224
> URL: https://issues.apache.org/jira/browse/BEAM-1224
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: 0.3.0-incubating
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
>   Original Estimate: 144h
>  Remaining Estimate: 144h
>
> options.py contains the class PipelineOptions (and other classes derived from 
> it) and should be renamed pipeline_options.py for clarity. Its test file is 
> pipeline_options_test.py.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam-site pull request #117: Add 0.4.0 release

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/beam-site/pull/117

Add 0.4.0 release



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/beam-site release-0.4.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #117


commit 351ce65df3bb8a3f3b646789d7754e38cbef6518
Author: Jean-Baptiste Onofré 
Date:   2016-12-29T07:26:34Z

Add 0.4.0 release




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #116: [BEAM-1224] Rename apache_beam.utils.options to...

[mariapython]

GitHub user mariapython opened a pull request:

https://github.com/apache/beam-site/pull/116

[BEAM-1224] Rename apache_beam.utils.options to 
apache_beam.utils.pipeline_options

Update documentation snippets that reference apache_beam.utils.options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariapython/beam-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/116.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #116


commit e94e15b2bac716bc6b8b1e6b6e72d049f64bfe93
Author: Maria Garcia Herrero 
Date:   2016-12-29T07:12:37Z

Rename apache_beam.utils.options to apache_beam.utils.pipeline_options




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #109: Publish 0.4.0-incubating release

[jbonofre]

Github user jbonofre closed the pull request at:

https://github.com/apache/beam-site/pull/109


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

svn commit: r17614 - in /dev/beam: 0.4.0-incubating/ 0.4.0/ 0.4.0/apache-beam-0.4.0-source-release.zip 0.4.0/apache-beam-0.4.0-source-release.zip.asc 0.4.0/apache-beam-0.4.0-source-release.zip.md5 0.4

[jbonofre]

Author: jbonofre
Date: Thu Dec 29 07:13:41 2016
New Revision: 17614

Log:
Publish 0.4.0 source release artifact

Added:
dev/beam/0.4.0/
dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip   (with props)
dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.asc
dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.md5
dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.sha1
Removed:
dev/beam/0.4.0-incubating/

Added: dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip
==
Binary file - no diff available.

Propchange: dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip
--
svn:mime-type = application/octet-stream

Added: dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.asc
==
--- dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.asc (added)
+++ dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.asc Thu Dec 29 07:13:41 
2016
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQIcBAABAgAGBQJYZKu9AAoJEL/y7kLIKC52e7EQAJy5ZV6VhsAYQW94O3/TZuxg
+lto7JbMjI1RvpTzlzfVeHOKoriurPD9eZxdpq8h1cgzNDOHSftyiSDmaLqUuaagl
+UTS2q5nhBcHSUe7/UL4KccpM7JKwMsyW7pPhZ15vxPB7jhVmsbgMuheNh4FevCNh
+wU3X+3nVQN6oAwybIKKIFJy1UDAimGXDj1Jyu6kkn/xqbEVEmqiVCVUhVudK5dOk
+ZGdvm/CGO0zphcvts86F3PbUZLRxsC9z9uLfrs/gZ47ffIPqFo07g5zsS4SeNyEj
+6vQ3Y7Ycf6eiHwMcpDbUS8qK28rk88VSIKxqbGvVoOMVr798oZimz+DYywmNpr1J
+dcX3WP3SemMf1SFHMXSJkdrVzSQCu5LjigNKKbs6+tZR/QjtfSKMuOOYwn6c7AWZ
+1th4+G/HDhvH0RZ3nEXkRBjy3ebLJAkrvLeuf08lHi1NxxBXSU+OfTGcbYG3HFu+
+1T4/VPU/A4cBVATAwgYln3mcfQpEcJlG+k3A7AcENZ5PlXa166poxYMrTFMqAo8F
+gniRd8q6/v14q7uUxBY7A14D2kQRLhtxcYQq/kZmP8ouwcLyBltAw5RcvS1aq6X5
+hwOSackAbUl3BTCMPGsnM1zvRCPRXhuZgm3TMjJ5G3biLRgXl5aQqPCiIszt0wJQ
+qD2HUqlWKkY1php0zb1q
+=QOHW
+-END PGP SIGNATURE-

Added: dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.md5
==
--- dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.md5 (added)
+++ dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.md5 Thu Dec 29 07:13:41 
2016
@@ -0,0 +1 @@
+32436b1c276e8bcabaa8cb2d232d0e64  apache-beam-0.4.0-source-release.zip

Added: dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.sha1
==
--- dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.sha1 (added)
+++ dev/beam/0.4.0/apache-beam-0.4.0-source-release.zip.sha1 Thu Dec 29 
07:13:41 2016
@@ -0,0 +1 @@
+f6793a06a8b0c9af4ab478b523a8858fd2327b2d  apache-beam-0.4.0-source-release.zip

[beam] Git Push Summary

[jbonofre]

Repository: beam
Updated Tags:  refs/tags/v0.4.0-RC1 [created] ab73a243c

beam git commit: [maven-release-plugin] prepare release v0.4.0-RC1

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/release-0.4.0 53270cc36 -> 95225f586


[maven-release-plugin] prepare release v0.4.0-RC1


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/95225f58
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/95225f58
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/95225f58

Branch: refs/heads/release-0.4.0
Commit: 95225f5863558e7731b417aaa0baf98d64ff5044
Parents: 53270cc
Author: Jean-Baptiste Onofré 
Authored: Thu Dec 29 07:31:03 2016 +0100
Committer: Jean-Baptiste Onofré 
Committed: Thu Dec 29 07:31:03 2016 +0100

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 33 files changed, 34 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/95225f58/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index ab5bffa..eeb2044 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-SNAPSHOT
+0.4.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/95225f58/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index c80de36..58f4e94 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-SNAPSHOT
+0.4.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/95225f58/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 9d25193..105a5f3 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.4.0-SNAPSHOT
+0.4.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/95225f58/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 6e509ef..3e5517d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.apache.org/
   2016
 
-  0.4.0-SNAPSHOT
+  0.4.0
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 https://git-wip-us.apache.org/repos/asf?p=beam.git;a=summary
-release-0.4.0
+v0.4.0-RC1
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/95225f58/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 3edeaa8..6d9a5b0 100644
--- a/runners/apex/pom.xml
+++ b/runners/apex/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-runners-parent
-0.4.0-SNAPSHOT
+0.4.0
 ../pom.xml
   
 

http://git-wip-us.apache.or

beam git commit: [maven-release-plugin] rollback changes from release preparation of v0.4.0-RC1

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/release-0.4.0 95225f586 -> c66679db5


[maven-release-plugin] rollback changes from release preparation of v0.4.0-RC1


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c66679db
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c66679db
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c66679db

Branch: refs/heads/release-0.4.0
Commit: c66679db57cd0aef54bb60c1d3fdd70d8b50814c
Parents: 95225f5
Author: Jean-Baptiste Onofré 
Authored: Thu Dec 29 07:31:22 2016 +0100
Committer: Jean-Baptiste Onofré 
Committed: Thu Dec 29 07:31:22 2016 +0100

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 33 files changed, 34 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c66679db/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index eeb2044..ab5bffa 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0
+0.4.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/c66679db/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 58f4e94..c80de36 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0
+0.4.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/c66679db/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 105a5f3..9d25193 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.4.0
+0.4.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/c66679db/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 3e5517d..6e509ef 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.apache.org/
   2016
 
-  0.4.0
+  0.4.0-SNAPSHOT
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 https://git-wip-us.apache.org/repos/asf?p=beam.git;a=summary
-v0.4.0-RC1
+release-0.4.0
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/c66679db/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 6d9a5b0..3edeaa8 100644
--- a/runners/apex/pom.xml
+++ b/runners/apex/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-runners-parent
-0.4.0
+0.4.0-SNAPSHOT
 ../pom.xml
   
 

[jira] [Updated] (BEAM-1229) flink KafkaIOExamples submit error

[Fei Feng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fei Feng updated BEAM-1229:
---
Description: 
I change all the beam pom.xml scala to 2.11.8,scala lib to 2.11，compile the 
beam jars。

Submit KafkaIOExamples in runners/FlinkRunner/KafkaIOExamples to flink,main 
class "ReadStringFromKafka"
error occured as follow：
org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error.
at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:524)
at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
at 
org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:80)
at 
org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:274)
at 
org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:95)
at 
org.apache.flink.runtime.webmonitor.handlers.JarPlanHandler.handleRequest(JarPlanHandler.java:42)
at 
org.apache.flink.runtime.webmonitor.RuntimeMonitorHandler.respondAsLeader(RuntimeMonitorHandler.java:88)
at 
org.apache.flink.runtime.webmonitor.RuntimeMonitorHandlerBase.channelRead0(RuntimeMonitorHandlerBase.java:84)
at 
org.apache.flink.runtime.webmonitor.RuntimeMonitorHandlerBase.channelRead0(RuntimeMonitorHandlerBase.java:44)
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at io.netty.handler.codec.http.router.Handler.routed(Handler.java:62)
at 
io.netty.handler.codec.http.router.DualAbstractHandler.channelRead0(DualAbstractHandler.java:57)
at 
io.netty.handler.codec.http.router.DualAbstractHandler.channelRead0(DualAbstractHandler.java:20)
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at 
org.apache.flink.runtime.webmonitor.HttpRequestHandler.channelRead0(HttpRequestHandler.java:105)
at 
org.apache.flink.runtime.webmonitor.HttpRequestHandler.channelRead0(HttpRequestHandler.java:65)
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
at 
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: null
at 
org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:174)
at org.apache.beam.sdk.values.TypedPValue.getCoder(TypedPValue.java:51)
at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:130)
at 
org.apache.beam.sdk.values.TypedPValue.finishSpecifying(TypedPValue.java:90)
at 
org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:95)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:386)
at org.apache.beam.sdk.Pipeline.applyTransform

[jira] [Created] (BEAM-1229) flink KafkaIOExamples submit error

[Fei Feng (JIRA)]

Fei Feng created BEAM-1229:
--

 Summary: flink KafkaIOExamples submit error 
 Key: BEAM-1229
 URL: https://issues.apache.org/jira/browse/BEAM-1229
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 0.5.0
 Environment: Flink:1.1.3_2.11
JDK:Oracle jdk 1.8.0_73
Beam:0.5.0-incubator-SNAPSHOT
OS:Windows 10
Reporter: Fei Feng
Assignee: Maximilian Michels
Priority: Minor


I change all the beam pom.xml scala to 2.11.8,scala lib to 2.11，compile the 
beam jars。

Submit KafkaIOExamples in runners/FlinkRunner/KafkaIOExamples to flink，
error occured as follow：
org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error.
at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:524)
at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
at 
org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:80)
at 
org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:274)
at 
org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:95)
at 
org.apache.flink.runtime.webmonitor.handlers.JarPlanHandler.handleRequest(JarPlanHandler.java:42)
at 
org.apache.flink.runtime.webmonitor.RuntimeMonitorHandler.respondAsLeader(RuntimeMonitorHandler.java:88)
at 
org.apache.flink.runtime.webmonitor.RuntimeMonitorHandlerBase.channelRead0(RuntimeMonitorHandlerBase.java:84)
at 
org.apache.flink.runtime.webmonitor.RuntimeMonitorHandlerBase.channelRead0(RuntimeMonitorHandlerBase.java:44)
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at io.netty.handler.codec.http.router.Handler.routed(Handler.java:62)
at 
io.netty.handler.codec.http.router.DualAbstractHandler.channelRead0(DualAbstractHandler.java:57)
at 
io.netty.handler.codec.http.router.DualAbstractHandler.channelRead0(DualAbstractHandler.java:20)
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at 
org.apache.flink.runtime.webmonitor.HttpRequestHandler.channelRead0(HttpRequestHandler.java:105)
at 
org.apache.flink.runtime.webmonitor.HttpRequestHandler.channelRead0(HttpRequestHandler.java:65)
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
at 
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: null
at 
org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:174)
at org.apache.beam.sdk.values.TypedPValue.getCoder(TypedPValue.java:51)
at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:130)
at 
or

[GitHub] beam pull request #1715: Update dataflow.properties to use 0.4.0 version ins...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1715


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1715

[dhalperi]

This closes #1715


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/53270cc3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/53270cc3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/53270cc3

Branch: refs/heads/release-0.4.0
Commit: 53270cc3690470d4098c644903ef59ea54dc5ca9
Parents: 6ea16f0 2e63133
Author: Dan Halperin 
Authored: Wed Dec 28 22:02:53 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 22:02:53 2016 -0800

--
 .../org/apache/beam/runners/dataflow/dataflow.properties | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[1/2] beam git commit: Update dataflow.properties to use 0.4.0 version instead of 0.4.0-incubating

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/release-0.4.0 6ea16f008 -> 53270cc36


Update dataflow.properties to use 0.4.0 version instead of 0.4.0-incubating


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2e631332
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2e631332
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2e631332

Branch: refs/heads/release-0.4.0
Commit: 2e6313329a8baca2d90db1b171ccfa2f70625f13
Parents: 6ea16f0
Author: Jean-Baptiste Onofré 
Authored: Thu Dec 29 06:53:20 2016 +0100
Committer: Jean-Baptiste Onofré 
Committed: Thu Dec 29 06:53:20 2016 +0100

--
 .../org/apache/beam/runners/dataflow/dataflow.properties | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2e631332/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
 
b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
index abfea0c..2928f0f 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
+++ 
b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
@@ -18,6 +18,6 @@
 
 environment.major.version=6
 
-worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-0.4.0-incubating
+worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-0.4.0
 
-worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-0.4.0-incubating
+worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-0.4.0

[GitHub] beam pull request #1715: Update dataflow.properties to use 0.4.0 version ins...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/beam/pull/1715

Update dataflow.properties to use 0.4.0 version instead of 0.4.0-incubating

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/beam patch-release-0.4.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1715.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1715


commit 2e6313329a8baca2d90db1b171ccfa2f70625f13
Author: Jean-Baptiste Onofré 
Date:   2016-12-29T05:53:20Z

Update dataflow.properties to use 0.4.0 version instead of 0.4.0-incubating




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1224) Rename options.py to pipeline_options.py

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784588#comment-15784588
 ] 

ASF GitHub Bot commented on BEAM-1224:
--

Github user mariapython closed the pull request at:

https://github.com/apache/beam/pull/1714


> Rename options.py to pipeline_options.py
> 
>
> Key: BEAM-1224
> URL: https://issues.apache.org/jira/browse/BEAM-1224
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: 0.3.0-incubating
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
>   Original Estimate: 144h
>  Remaining Estimate: 144h
>
> options.py contains the class PipelineOptions (and other classes derived from 
> it) and should be renamed pipeline_options.py for clarity. Its test file is 
> pipeline_options_test.py.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1714: [BEAM-1224] Rename options.py to pipeline_options.p...

[mariapython]

Github user mariapython closed the pull request at:

https://github.com/apache/beam/pull/1714


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1227) Release 0.4.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784581#comment-15784581
 ] 

ASF GitHub Bot commented on BEAM-1227:
--

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/1713


> Release 0.4.0
> -
>
> Key: BEAM-1227
> URL: https://issues.apache.org/jira/browse/BEAM-1227
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Affects Versions: 0.4.0-incubating
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>
> Umbrella bug for the 0.4.0 (incubating or not) release.
> JB is the release manager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1227) Release 0.4.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784580#comment-15784580
 ] 

ASF GitHub Bot commented on BEAM-1227:
--

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/1712


> Release 0.4.0
> -
>
> Key: BEAM-1227
> URL: https://issues.apache.org/jira/browse/BEAM-1227
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Affects Versions: 0.4.0-incubating
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>
> Umbrella bug for the 0.4.0 (incubating or not) release.
> JB is the release manager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1712: [BEAM-1227] Cherrypick from #1701 to remove incubat...

[dhalperi]

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/1712


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1713

[dhalperi]

This closes #1713


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6ea16f00
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6ea16f00
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6ea16f00

Branch: refs/heads/release-0.4.0
Commit: 6ea16f00874d4895e3445db2d2f09383df8081a6
Parents: 2d59c62 bcb7c78
Author: Dan Halperin 
Authored: Wed Dec 28 21:43:36 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:43:36 2016 -0800

--
 .../examples-java8/src/main/resources/archetype-resources/pom.xml  | 2 +-
 .../examples/src/main/resources/archetype-resources/pom.xml| 2 +-
 .../starter/src/main/resources/archetype-resources/pom.xml | 2 +-
 .../starter/src/test/resources/projects/basic/reference/pom.xml| 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)
--

[GitHub] beam pull request #1713: [BEAM-1227] Update archetype pom.xml to non-incubat...

[dhalperi]

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/1713


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Update archetype pom.xml to non-incubating release

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/release-0.4.0 2d59c62d0 -> 6ea16f008


Update archetype pom.xml to non-incubating release


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bcb7c788
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bcb7c788
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bcb7c788

Branch: refs/heads/release-0.4.0
Commit: bcb7c788a74062835872a454dafaf799b193af10
Parents: 2d59c62
Author: Dan Halperin 
Authored: Wed Dec 28 14:48:12 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:43:33 2016 -0800

--
 .../examples-java8/src/main/resources/archetype-resources/pom.xml  | 2 +-
 .../examples/src/main/resources/archetype-resources/pom.xml| 2 +-
 .../starter/src/main/resources/archetype-resources/pom.xml | 2 +-
 .../starter/src/test/resources/projects/basic/reference/pom.xml| 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bcb7c788/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
index c9645d3..a20a9be 100644
--- 
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
@@ -27,7 +27,7 @@
   jar
 
   
-0.4.0-incubating
+0.4.0
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/bcb7c788/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 62a2de5..91405ac 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -27,7 +27,7 @@
   jar
 
   
-0.4.0-incubating
+0.4.0
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/bcb7c788/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
index 4dab830..64ca4c4 100644
--- 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
@@ -25,7 +25,7 @@
   ${version}
 
   
-0.4.0-incubating
+0.4.0
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/bcb7c788/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
index 5fb2bf6..48703d2 100644
--- 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
@@ -25,7 +25,7 @@
   0.1
 
   
-0.4.0-incubating
+0.4.0
   
 
   

[7/8] beam git commit: Update top-level pom.xml to remove incubation references

[dhalperi]

Update top-level pom.xml to remove incubation references


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e37eaeff
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e37eaeff
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e37eaeff

Branch: refs/heads/release-0.4.0
Commit: e37eaeff0d788ed74c20d159c8492f2221ca828f
Parents: 9528943
Author: Davor Bonaci 
Authored: Tue Dec 27 13:14:01 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:36:44 2016 -0800

--
 pom.xml | 30 +++---
 1 file changed, 15 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e37eaeff/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 78925c0..6e509ef 100644
--- a/pom.xml
+++ b/pom.xml
@@ -31,7 +31,7 @@
   Apache Beam provides a simple, Java-based interface
   for processing virtually any size data. This artifact includes the parent POM
   for other Beam artifacts.
-  http://beam.incubator.apache.org
+  http://beam.apache.org/
   2016
 
   0.4.0-SNAPSHOT
@@ -59,32 +59,32 @@
   
 
   Beam Dev
-  dev-subscr...@beam.incubator.apache.org
-  dev-unsubscr...@beam.incubator.apache.org
-  d...@beam.incubator.apache.org
-  
http://www.mail-archive.com/dev%beam.incubator.apache.org
+  dev-subscr...@beam.apache.org
+  dev-unsubscr...@beam.apache.org
+  d...@beam.apache.org
+  http://www.mail-archive.com/dev%beam.apache.org
 
 
   Beam User
-  user-subscr...@beam.incubator.apache.org
-  user-unsubscr...@beam.incubator.apache.org
-  u...@beam.incubator.apache.org
-  
http://www.mail-archive.com/user%beam.incubator.apache.org
+  user-subscr...@beam.apache.org
+  user-unsubscr...@beam.apache.org
+  u...@beam.apache.org
+  http://www.mail-archive.com/user%beam.apache.org
 
 
   Beam Commits
-  commits-subscr...@beam.incubator.apache.org
-  commits-unsubscr...@beam.incubator.apache.org
-  comm...@beam.incubator.apache.org
-  
http://www.mail-archive.com/commits%beam.incubator.apache.org
+  commits-subscr...@beam.apache.org
+  commits-unsubscr...@beam.apache.org
+  commits@beam.apache.org
+  http://www.mail-archive.com/commits%beam.apache.org
 
   
 
   
 
   The Apache Beam Team
-  d...@beam.incubator.apache.org
-  http://beam.incubator.apache.org
+  d...@beam.apache.org
+  http://beam.apache.org
   Apache Software Foundation
   http://www.apache.org
 

[8/8] beam git commit: This closes #1712

[dhalperi]

This closes #1712


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2d59c62d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2d59c62d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2d59c62d

Branch: refs/heads/release-0.4.0
Commit: 2d59c62d027851b01e983e76bfd1d9f978e0436f
Parents: e77080a 8c7a137
Author: Dan Halperin 
Authored: Wed Dec 28 21:36:46 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:36:46 2016 -0800

--
 .jenkins/common_job_properties.groovy   |  6 +--
 .../job_beam_Release_NightlySnapshot.groovy |  2 +-
 .jenkins/job_seed.groovy|  2 +-
 .travis/README.md   |  2 +-
 DISCLAIMER  | 10 -
 NOTICE  |  2 +-
 README.md   | 46 ++--
 examples/java/README.md | 16 +++
 .../org/apache/beam/examples/WordCount.java |  4 +-
 .../org/apache/beam/examples/complete/README.md | 14 +++---
 .../org/apache/beam/examples/cookbook/README.md | 14 +++---
 .../beam/examples/cookbook/TriggerExample.java  |  4 +-
 pom.xml | 30 ++---
 runners/apex/README.md  |  4 +-
 runners/flink/README.md |  6 +--
 runners/spark/README.md |  8 ++--
 .../beam/sdk/util/state/StateContexts.java  |  4 +-
 sdks/java/extensions/join-library/README.md | 10 -
 .../main/resources/archetype-resources/pom.xml  |  2 +-
 .../resources/projects/basic/reference/pom.xml  |  2 +-
 20 files changed, 74 insertions(+), 114 deletions(-)
--

[3/8] beam git commit: Update README to remove incubation information

[dhalperi]

Update README to remove incubation information


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f5c92f12
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f5c92f12
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f5c92f12

Branch: refs/heads/release-0.4.0
Commit: f5c92f12f42c2b7f7211ca3a3597edfe81b7aa58
Parents: dc2ad93
Author: Davor Bonaci 
Authored: Tue Dec 27 13:03:53 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:36:44 2016 -0800

--
 README.md | 46 +-
 1 file changed, 13 insertions(+), 33 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f5c92f12/README.md
--
diff --git a/README.md b/README.md
index d919e5b..f7492e4 100644
--- a/README.md
+++ b/README.md
@@ -17,30 +17,15 @@
 under the License.
 -->
 
-# Apache Beam (incubating)
+# Apache Beam
 
-[Apache Beam](http://beam.incubator.apache.org) is a unified model for 
defining both batch and streaming data-parallel processing pipelines, as well 
as a set of language-specific SDKs for constructing pipelines and Runners for 
executing them on distributed processing backends like [Apache 
Spark](http://spark.apache.org/), [Apache Flink](http://flink.apache.org), and 
[Google Cloud Dataflow](http://cloud.google.com/dataflow).
-
-```
-Apache Beam is an effort undergoing incubation at the Apache Software
-Foundation (ASF), sponsored by the Apache Incubator PMC.
-
-Incubation is required of all newly accepted projects until a further review
-indicates that the infrastructure, communications, and decision making process
-have stabilized in a manner consistent with other successful ASF projects.
-
-While incubation status is not necessarily a reflection of the completeness
-or stability of the code, it does indicate that the project has yet to be
-fully endorsed by the ASF.
-```
+[Apache Beam](http://beam.apache.org/) is a unified model for defining both 
batch and streaming data-parallel processing pipelines, as well as a set of 
language-specific SDKs for constructing pipelines and Runners for executing 
them on distributed processing backends, including [Apache 
Apex](http://apex.apache.org/), [Apache Flink](http://flink.apache.org/), 
[Apache Spark](http://spark.apache.org/), and [Google Cloud 
Dataflow](http://cloud.google.com/dataflow/).
 
 ## Status
 
-_**The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submissions, and the formulation of project documentation, planning, and design 
documents. Please expect a significant amount of churn and breaking changes in 
the near future.**_
-
-[![Build 
Status](https://api.travis-ci.org/apache/incubator-beam.svg?branch=master)](https://travis-ci.org/apache/incubator-beam?branch=master)
+[![Build 
Status](https://api.travis-ci.org/apache/beam.svg?branch=master)](https://travis-ci.org/apache/beam?branch=master)
 [![Build 
Status](https://builds.apache.org/buildStatus/icon?job=beam_PostCommit_Java_MavenInstall)](https://builds.apache.org/job/beam_PostCommit_MavenVerify/)
-[![Coverage 
Status](https://coveralls.io/repos/github/apache/incubator-beam/badge.svg?branch=master)](https://coveralls.io/github/apache/incubator-beam?branch=master)
+[![Coverage 
Status](https://coveralls.io/repos/github/apache/beam/badge.svg?branch=master)](https://coveralls.io/github/apache/beam?branch=master)
 
 ## Overview
 
@@ -50,7 +35,6 @@ Beam provides a general approach to expressing 
[embarrassingly parallel](https:/
 2. _SDK Writers_: Developing a Beam SDK targeted at a specific user community 
(Java, Python, Scala, Go, R, graphical, etc). These users are language geeks, 
and  would prefer to be shielded from all the details of various runners and 
their implementations.
 3. _Runner Writers_: Have an execution environment for distributed processing 
and would like to support programs written against the Beam Model. Would prefer 
to be shielded from details of multiple SDKs.
 
-
 ### The Beam Model
 
 The model behind Beam evolved from a number of internal Google data processing 
projects, including 
[MapReduce](http://research.google.com/archive/mapreduce.html), 
[FlumeJava](http://research.google.com/pubs/pub35650.html), and 
[Millwheel](http://research.google.com/pubs/pub41378.html). This model was 
originally known as the “[Dataflow 
Model](http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf)”.
@@ -64,16 +48,14 @@ The key concepts in the Beam programming model are:
 * `Pipeline`: manages a directed acyclic graph of PTransforms and PCollections 
that is ready for execution.
 * `PipelineRunner`: specifies where and how the pipeline should execute.
 
-
 ### SDKs
 
 Beam supports mult

[2/8] beam git commit: Remove incubation references from core SDK and archetypes

[dhalperi]

Remove incubation references from core SDK and archetypes


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/59944804
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/59944804
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/59944804

Branch: refs/heads/release-0.4.0
Commit: 59944804ffded82307fca9cab207fb8603e144c7
Parents: ae0055a
Author: Davor Bonaci 
Authored: Tue Dec 27 13:20:30 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:36:44 2016 -0800

--
 .../main/java/org/apache/beam/sdk/util/state/StateContexts.java  | 4 ++--
 .../starter/src/main/resources/archetype-resources/pom.xml   | 2 +-
 .../starter/src/test/resources/projects/basic/reference/pom.xml  | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/59944804/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
index 1d6e949..a9ce0b5 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
@@ -54,7 +54,7 @@ public class StateContexts {
   /**
* @deprecated This exists only for temporary compatibility with Dataflow 
worker and should be
* deleted once a worker image is released that uses runners-core build 
after
-   * https://github.com/apache/incubator-beam/pull/1353.
+   * https://github.com/apache/beam/pull/1353.
*/
   @Deprecated
   public static  StateContext windowOnly(final W 
window) {
@@ -80,7 +80,7 @@ public class StateContexts {
   /**
* @deprecated This exists only for temporary compatibility with Dataflow 
worker and should be
* deleted once a worker image is released that uses runners-core build 
after
-   * https://github.com/apache/incubator-beam/pull/1353.
+   * https://github.com/apache/beam/pull/1353.
*/
   @Deprecated
   public static  StateContext createFromComponents(

http://git-wip-us.apache.org/repos/asf/beam/blob/59944804/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
index 08139b2..4dab830 100644
--- 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
@@ -81,7 +81,7 @@
   to enable development and testing of pipelines. To run on another of the
   Beam runners, add its module to this pom.xml according to the
   runner-specific setup instructions on the Beam website:
-http://beam.incubator.apache.org/documentation/#runners
+http://beam.apache.org/documentation/#runners
 -->
 
   org.apache.beam

http://git-wip-us.apache.org/repos/asf/beam/blob/59944804/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
index 2a668e8..5fb2bf6 100644
--- 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
@@ -81,7 +81,7 @@
   to enable development and testing of pipelines. To run on another of the
   Beam runners, add its module to this pom.xml according to the
   runner-specific setup instructions on the Beam website:
-http://beam.incubator.apache.org/documentation/#runners
+http://beam.apache.org/documentation/#runners
 -->
 
   org.apache.beam

[5/8] beam git commit: Update component READMEs to remove graduation information

[dhalperi]

Update component READMEs to remove graduation information


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ae0055a3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ae0055a3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ae0055a3

Branch: refs/heads/release-0.4.0
Commit: ae0055a38543bd54e95bccac015083710c5235fb
Parents: e37eaef
Author: Davor Bonaci 
Authored: Tue Dec 27 13:17:22 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:36:44 2016 -0800

--
 runners/apex/README.md  |  4 ++--
 runners/flink/README.md |  6 +++---
 runners/spark/README.md |  8 
 sdks/java/extensions/join-library/README.md | 10 --
 4 files changed, 9 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ae0055a3/runners/apex/README.md
--
diff --git a/runners/apex/README.md b/runners/apex/README.md
index 7aa433e..9a82731 100644
--- a/runners/apex/README.md
+++ b/runners/apex/README.md
@@ -37,13 +37,13 @@ The following shows how to run the WordCount example that 
is provided with the s
 To get the latest version of Beam with Apex-Runner, first clone the Beam 
repository:
 
 ```
-git clone https://github.com/apache/incubator‐beam
+git clone https://github.com/apache/beam
 ```
 
 Then switch to the newly created directory and run Maven to build the Apache 
Beam:
 
 ```
-cd incubator‐beam
+cd beam
 mvn clean install ‐DskipTests
 ```
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ae0055a3/runners/flink/README.md
--
diff --git a/runners/flink/README.md b/runners/flink/README.md
index c0b3830..ceb44ea 100644
--- a/runners/flink/README.md
+++ b/runners/flink/README.md
@@ -94,11 +94,11 @@ To get started using the Flink Runner, we first need to 
install the latest versi
 
 To retrieve the latest version of Flink-Runner, run the following command
 
-git clone https://github.com/apache/incubator-beam
+git clone https://github.com/apache/beam
 
 Then switch to the newly created directory and run Maven to build the Beam 
runner:
 
-cd incubator-beam
+cd beam
 mvn clean install -DskipTests
 
 Flink-Runner is now installed in your local maven repository.
@@ -169,7 +169,7 @@ The contents of the root `pom.xml` should be slightly 
changed afterwards (explan
 
   org.apache.beam
   beam-runners-flink_2.10
-  0.2.0-incubating-SNAPSHOT
+  0.2.0-SNAPSHOT
 
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ae0055a3/runners/spark/README.md
--
diff --git a/runners/spark/README.md b/runners/spark/README.md
index aad65b3..15f7081 100644
--- a/runners/spark/README.md
+++ b/runners/spark/README.md
@@ -56,7 +56,7 @@ The Spark runner provides full support for the Beam Model in 
batch processing vi
 
 ### Streaming
 
-Providing full support for the Beam Model in streaming pipelines is under 
development. To follow-up you can subscribe to our [mailing 
list](http://beam.incubator.apache.org/get-started/support/).
+Providing full support for the Beam Model in streaming pipelines is under 
development. To follow-up you can subscribe to our [mailing 
list](http://beam.apache.org/get-started/support/).
 
 ### issue tracking
 
@@ -67,12 +67,12 @@ See [Beam JIRA](https://issues.apache.org/jira/browse/BEAM) 
(runner-spark)
 
 To get the latest version of the Spark Runner, first clone the Beam repository:
 
-git clone https://github.com/apache/incubator-beam
+git clone https://github.com/apache/beam
 
 
 Then switch to the newly created directory and run Maven to build the Apache 
Beam:
 
-cd incubator-beam
+cd beam
 mvn clean install -DskipTests
 
 Now Apache Beam and the Spark Runner are installed in your local maven 
repository.
@@ -116,7 +116,7 @@ Check the output by running:
 __Note: running examples using `mvn exec:exec` only works for Spark local mode 
at the
 moment. See the next section for how to run on a cluster.__
 
-[wc]: 
https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
+[wc]: 
https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/examples/WordCount.java
 ## Running on a Cluster
 
 Spark Beam pipelines can be run on a cluster using the `spark-submit` command.

http://git-wip-us.apache.org/repos/asf/beam/blob/ae0055a3/sdks/java/extensions/join-library/README.md
--
diff --git a/sdks/java/extensions/join-library/README.md 
b/sdks/java/extensions/join-li

[4/8] beam git commit: Remove incubating references from examples

[dhalperi]

Remove incubating references from examples


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9528943a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9528943a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9528943a

Branch: refs/heads/release-0.4.0
Commit: 9528943ac6a4f288668529eaa16423a85a903311
Parents: f5c92f1
Author: Davor Bonaci 
Authored: Tue Dec 27 13:11:41 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:36:44 2016 -0800

--
 examples/java/README.md | 16 
 .../java/org/apache/beam/examples/WordCount.java|  4 ++--
 .../org/apache/beam/examples/complete/README.md | 14 +++---
 .../org/apache/beam/examples/cookbook/README.md | 14 +++---
 .../beam/examples/cookbook/TriggerExample.java  |  4 ++--
 5 files changed, 26 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9528943a/examples/java/README.md
--
diff --git a/examples/java/README.md b/examples/java/README.md
index 2b5edf5..b58153e 100644
--- a/examples/java/README.md
+++ b/examples/java/README.md
@@ -26,21 +26,21 @@ the development of more complex pipelines.
 ## Word Count
 
 A good starting point for new users is our set of
-[word 
count](https://github.com/apache/incubator-beam/blob/master/examples/java/src/main/java/org/apache/beam/examples)
 examples, which computes word frequencies.  This series of four successively 
more detailed pipelines is described in detail in the accompanying 
[walkthrough](https://cloud.google.com/dataflow/examples/wordcount-example).
+[word 
count](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples)
 examples, which computes word frequencies.  This series of four successively 
more detailed pipelines is described in detail in the accompanying 
[walkthrough](https://cloud.google.com/dataflow/examples/wordcount-example).
 
-1. 
[`MinimalWordCount`](https://github.com/apache/incubator-beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java)
 is the simplest word count pipeline and introduces basic concepts like 
[Pipelines](https://cloud.google.com/dataflow/model/pipelines),
+1. 
[`MinimalWordCount`](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java)
 is the simplest word count pipeline and introduces basic concepts like 
[Pipelines](https://cloud.google.com/dataflow/model/pipelines),
 [PCollections](https://cloud.google.com/dataflow/model/pcollection),
 [ParDo](https://cloud.google.com/dataflow/model/par-do),
 and [reading and writing 
data](https://cloud.google.com/dataflow/model/reading-and-writing-data) from 
external storage.
 
-1. 
[`WordCount`](https://github.com/apache/incubator-beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java)
 introduces Dataflow best practices like 
[PipelineOptions](https://cloud.google.com/dataflow/pipelines/constructing-your-pipeline#Creating)
 and custom 
[PTransforms](https://cloud.google.com/dataflow/model/composite-transforms).
+1. 
[`WordCount`](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java)
 introduces Dataflow best practices like 
[PipelineOptions](https://cloud.google.com/dataflow/pipelines/constructing-your-pipeline#Creating)
 and custom 
[PTransforms](https://cloud.google.com/dataflow/model/composite-transforms).
 
-1. 
[`DebuggingWordCount`](https://github.com/apache/incubator-beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java)
+1. 
[`DebuggingWordCount`](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java)
 shows how to view live aggregators in the [Dataflow Monitoring 
Interface](https://cloud.google.com/dataflow/pipelines/dataflow-monitoring-intf),
 get the most out of
 [Cloud Logging](https://cloud.google.com/dataflow/pipelines/logging) 
integration, and start writing
 [good 
tests](https://cloud.google.com/dataflow/pipelines/testing-your-pipeline).
 
-1. 
[`WindowedWordCount`](https://github.com/apache/incubator-beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java)
 shows how to run the same pipeline over either unbounded PCollections in 
streaming mode or bounded PCollections in batch mode.
+1. 
[`WindowedWordCount`](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java)
 shows how to run the same pipeline over either unbounded PCollections in 
streaming mode or bounded PCollections in batch mode.
 
 ## Building an

[6/8] beam git commit: Update top-level NOTICE and remove incubating disclaimer

[dhalperi]

Update top-level NOTICE and remove incubating disclaimer


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/dc2ad934
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/dc2ad934
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/dc2ad934

Branch: refs/heads/release-0.4.0
Commit: dc2ad9341c3f1ae7a6ea6d67c0a72f412704c5c5
Parents: e77080a
Author: Davor Bonaci 
Authored: Tue Dec 27 13:00:09 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:36:44 2016 -0800

--
 DISCLAIMER | 10 --
 NOTICE |  2 +-
 2 files changed, 1 insertion(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/dc2ad934/DISCLAIMER
--
diff --git a/DISCLAIMER b/DISCLAIMER
deleted file mode 100644
index f8568f6..000
--- a/DISCLAIMER
+++ /dev/null
@@ -1,10 +0,0 @@
-Apache Beam is an effort undergoing incubation at the Apache Software
-Foundation (ASF), sponsored by the Apache Incubator PMC.
-
-Incubation is required of all newly accepted projects until a further review
-indicates that the infrastructure, communications, and decision making process
-have stabilized in a manner consistent with other successful ASF projects.
-
-While incubation status is not necessarily a reflection of the completeness
-or stability of the code, it does indicate that the project has yet to be
-fully endorsed by the ASF.

http://git-wip-us.apache.org/repos/asf/beam/blob/dc2ad934/NOTICE
--
diff --git a/NOTICE b/NOTICE
index bd3400c..9b35cd4 100644
--- a/NOTICE
+++ b/NOTICE
@@ -1,4 +1,4 @@
-Apache Beam (incubating)
+Apache Beam
 Copyright 2016 The Apache Software Foundation
 
 This product includes software developed at

[1/8] beam git commit: Update Jenkins and Travis to remove incubation references

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/release-0.4.0 e77080a4d -> 2d59c62d0


Update Jenkins and Travis to remove incubation references


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8c7a1378
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8c7a1378
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8c7a1378

Branch: refs/heads/release-0.4.0
Commit: 8c7a1378986717227db8777f5e3c39903cc19845
Parents: 5994480
Author: Davor Bonaci 
Authored: Tue Dec 27 13:47:55 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:36:44 2016 -0800

--
 .jenkins/common_job_properties.groovy| 6 +++---
 .jenkins/job_beam_Release_NightlySnapshot.groovy | 2 +-
 .jenkins/job_seed.groovy | 2 +-
 .travis/README.md| 2 +-
 4 files changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8c7a1378/.jenkins/common_job_properties.groovy
--
diff --git a/.jenkins/common_job_properties.groovy 
b/.jenkins/common_job_properties.groovy
index f3a8a07..0c89181 100644
--- a/.jenkins/common_job_properties.groovy
+++ b/.jenkins/common_job_properties.groovy
@@ -26,7 +26,7 @@ class common_job_properties {
   def default_timeout = 100) {
 // GitHub project.
 context.properties {
-  githubProjectUrl('https://github.com/apache/incubator-beam/')
+  githubProjectUrl('https://github.com/apache/beam/')
 }
 
 // Set JDK version.
@@ -45,7 +45,7 @@ class common_job_properties {
 context.scm {
   git {
 remote {
-  url('https://github.com/apache/incubator-beam.git')
+  url('https://github.com/apache/beam.git')
   refspec('+refs/heads/*:refs/remotes/origin/* ' +
   '+refs/pull/*:refs/remotes/origin/pr/*')
 }
@@ -149,7 +149,7 @@ class common_job_properties {
   static def setPostCommit(def context,
def build_schedule = '0 */6 * * *',
def scm_schedule = '* * * * *',
-   def notify_address = 
'comm...@beam.incubator.apache.org') {
+   def notify_address = 'commits@beam.apache.org') {
 // Set build triggers
 context.triggers {
   // By default runs every 6 hours.

http://git-wip-us.apache.org/repos/asf/beam/blob/8c7a1378/.jenkins/job_beam_Release_NightlySnapshot.groovy
--
diff --git a/.jenkins/job_beam_Release_NightlySnapshot.groovy 
b/.jenkins/job_beam_Release_NightlySnapshot.groovy
index 496a0c5..533b444 100644
--- a/.jenkins/job_beam_Release_NightlySnapshot.groovy
+++ b/.jenkins/job_beam_Release_NightlySnapshot.groovy
@@ -39,7 +39,7 @@ mavenJob('beam_Release_NightlySnapshot') {
   delegate,
   '0 7 * * *',
   '0 5 31 2 *',
-  'd...@beam.incubator.apache.org')
+  'd...@beam.apache.org')
 
   // Maven goals for this job.
   goals('-B -e clean deploy -P release,dataflow-runner -DskipITs=false 
-DintegrationTestPipelineOptions=\'[ "--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]\'')

http://git-wip-us.apache.org/repos/asf/beam/blob/8c7a1378/.jenkins/job_seed.groovy
--
diff --git a/.jenkins/job_seed.groovy b/.jenkins/job_seed.groovy
index e46e5a4..96277d1 100644
--- a/.jenkins/job_seed.groovy
+++ b/.jenkins/job_seed.groovy
@@ -33,7 +33,7 @@ job('beam_SeedJob_Main') {
   delegate,
   '0 6 * * *',
   '0 5 31 2 *',
-  'd...@beam.incubator.apache.org')
+  'd...@beam.apache.org')
 
   steps {
 dsl {

http://git-wip-us.apache.org/repos/asf/beam/blob/8c7a1378/.travis/README.md
--
diff --git a/.travis/README.md b/.travis/README.md
index 536692d..526995a 100644
--- a/.travis/README.md
+++ b/.travis/README.md
@@ -19,5 +19,5 @@
 
 # Travis Scripts
 
-This directory contains scripts used for [Travis 
CI](https://travis-ci.org/apache/incubator-beam/)
+This directory contains scripts used for [Travis 
CI](https://travis-ci.org/apache/beam/)
 testing.

[GitHub] beam pull request #1710: [BEAM-1227] Cherrypick BigQueryIO fixups from #1400...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1710


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1227) Release 0.4.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784535#comment-15784535
 ] 

ASF GitHub Bot commented on BEAM-1227:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1710


> Release 0.4.0
> -
>
> Key: BEAM-1227
> URL: https://issues.apache.org/jira/browse/BEAM-1227
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Affects Versions: 0.4.0-incubating
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>
> Umbrella bug for the 0.4.0 (incubating or not) release.
> JB is the release manager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] beam git commit: BigQueryIO: fix streaming write, typo in API

[dhalperi]

BigQueryIO: fix streaming write, typo in API

and improve testing


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b1459dea
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b1459dea
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b1459dea

Branch: refs/heads/release-0.4.0
Commit: b1459deacfc7077b49803b9460c5127cd4fb74bf
Parents: c53a332
Author: Sam McVeety 
Authored: Fri Dec 16 18:10:28 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 14:22:34 2016 -0800

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 23 +--
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 72 
 2 files changed, 63 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b1459dea/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 28049ed..7bb1e51 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -370,7 +370,8 @@ public class BigQueryIO {
 }
   }
 
-  private static class TableSpecToTableRef
+  @VisibleForTesting
+  static class TableSpecToTableRef
   implements SerializableFunction {
 @Override
 public TableReference apply(String from) {
@@ -807,6 +808,7 @@ public class BigQueryIO {
   /**
* Returns the query to be read, or {@code null} if reading from a table 
instead.
*/
+  @Nullable
   public String getQuery() {
 return query == null ? null : query.get();
   }
@@ -814,7 +816,8 @@ public class BigQueryIO {
   /**
* Returns the query to be read, or {@code null} if reading from a table 
instead.
*/
-  public ValueProvider getQueryProivder() {
+  @Nullable
+  public ValueProvider getQueryProvider() {
 return query;
   }
 
@@ -2813,7 +2816,8 @@ public class BigQueryIO {
* a randomUUID is generated only once per bucket of data. The actual unique
* id is created by concatenating this randomUUID with a sequential number.
*/
-  private static class TagWithUniqueIdsAndTable
+  @VisibleForTesting
+  static class TagWithUniqueIdsAndTable
   extends DoFn, TableRowInfo>> {
 /** TableSpec to write to. */
 private final ValueProvider tableSpec;
@@ -2830,8 +2834,12 @@ public class BigQueryIO {
   checkArgument(table == null ^ tableRefFunction == null,
   "Exactly one of table or tableRefFunction should be set");
   if (table != null) {
-if (table.isAccessible() && table.get().getProjectId() == null) {
-  
table.get().setProjectId(options.as(BigQueryOptions.class).getProject());
+if (table.isAccessible() && 
Strings.isNullOrEmpty(table.get().getProjectId())) {
+  TableReference tableRef = table.get()
+  .setProjectId(options.as(BigQueryOptions.class).getProject());
+  table = NestedValueProvider.of(
+  StaticValueProvider.of(toJsonString(tableRef)),
+  new JsonTableRefToTableRef());
 }
 this.tableSpec = NestedValueProvider.of(table, new 
TableRefToTableSpec());
   } else {
@@ -2870,6 +2878,11 @@ public class BigQueryIO {
   }
 }
 
+@VisibleForTesting
+ValueProvider getTableSpec() {
+  return tableSpec;
+}
+
 private String tableSpecFromWindow(BigQueryOptions options, BoundedWindow 
window) {
   if (tableSpec != null) {
 return tableSpec.get();

http://git-wip-us.apache.org/repos/asf/beam/blob/b1459dea/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
index b78316f..dc566d2 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
@@ -26,6 +26,7 @@ import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisp
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.hasItem;
 import static org.junit.Assert.assertEqual

[3/3] beam git commit: This closes #1710

[dhalperi]

This closes #1710


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e77080a4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e77080a4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e77080a4

Branch: refs/heads/release-0.4.0
Commit: e77080a4d4358d0149d3c1e554581a254ea26506
Parents: 4820160 b1459de
Author: Dan Halperin 
Authored: Wed Dec 28 21:10:23 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 21:10:23 2016 -0800

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|  71 ++-
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |   7 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 121 -
 .../io/gcp/bigquery/BigQueryTableInserter.java  | 217 -
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 484 +++
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 139 +-
 .../gcp/bigquery/BigQueryTableInserterTest.java | 245 --
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   |  50 +-
 8 files changed, 696 insertions(+), 638 deletions(-)
--

[1/3] beam git commit: [BEAM-1022] Add testing coverage for BigQuery streaming writes

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/release-0.4.0 482016037 -> e77080a4d


[BEAM-1022] Add testing coverage for BigQuery streaming writes


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c53a332e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c53a332e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c53a332e

Branch: refs/heads/release-0.4.0
Commit: c53a332e23bc6a6441a0d52c57c9fbe5e00171b5
Parents: 4820160
Author: Reuven Lax 
Authored: Thu Nov 17 10:57:41 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 14:22:11 2016 -0800

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|  48 +-
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |   7 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 121 -
 .../io/gcp/bigquery/BigQueryTableInserter.java  | 217 -
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 456 +++
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 139 +-
 .../gcp/bigquery/BigQueryTableInserterTest.java | 245 --
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   |  50 +-
 8 files changed, 655 insertions(+), 628 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c53a332e/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 0be8567..28049ed 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -24,7 +24,6 @@ import static com.google.common.base.Preconditions.checkState;
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.google.api.client.json.JsonFactory;
-import com.google.api.services.bigquery.Bigquery;
 import com.google.api.services.bigquery.model.Job;
 import com.google.api.services.bigquery.model.JobConfigurationExtract;
 import com.google.api.services.bigquery.model.JobConfigurationLoad;
@@ -33,6 +32,7 @@ import 
com.google.api.services.bigquery.model.JobConfigurationTableCopy;
 import com.google.api.services.bigquery.model.JobReference;
 import com.google.api.services.bigquery.model.JobStatistics;
 import com.google.api.services.bigquery.model.JobStatus;
+import com.google.api.services.bigquery.model.Table;
 import com.google.api.services.bigquery.model.TableReference;
 import com.google.api.services.bigquery.model.TableRow;
 import com.google.api.services.bigquery.model.TableSchema;
@@ -1796,8 +1796,8 @@ public class BigQueryIO {
* Does not modify this object.
*/
   public Bound withCreateDisposition(CreateDisposition createDisposition) {
-return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema, 
createDisposition,
-writeDisposition, validate, bigQueryServices);
+return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema,
+createDisposition, writeDisposition, validate, bigQueryServices);
   }
 
   /**
@@ -1806,8 +1806,8 @@ public class BigQueryIO {
* Does not modify this object.
*/
   public Bound withWriteDisposition(WriteDisposition writeDisposition) {
-return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema, 
createDisposition,
-writeDisposition, validate, bigQueryServices);
+return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema,
+createDisposition, writeDisposition, validate, bigQueryServices);
   }
 
   /**
@@ -2136,7 +2136,8 @@ public class BigQueryIO {
   /** Returns the table reference, or {@code null}. */
   @Nullable
   public ValueProvider getTable() {
-return NestedValueProvider.of(jsonTableRef, new 
JsonTableRefToTableRef());
+return jsonTableRef == null ? null :
+NestedValueProvider.of(jsonTableRef, new JsonTableRefToTableRef());
   }
 
   /** Returns {@code true} if table validation is enabled. */
@@ -2550,6 +2551,13 @@ public class BigQueryIO {
 }
   }
 
+  /**
+   * Clear the cached map of created tables. Used for testing.
+   */
+  @VisibleForTesting
+  static void clearCreatedTables() {
+StreamingWriteFn.clearCreatedTables();
+  }
   /
 
   /**
@@ -2585,6 +2593,15 @@ public class BigQueryIO {
   this.bqServices = checkNotNull(bqServices, "bqServices");
 }
 
+/**
+ * Clear the cached map 

[3/3] beam git commit: This closes #1714

[dhalperi]

This closes #1714


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8bf4c805
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8bf4c805
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8bf4c805

Branch: refs/heads/python-sdk
Commit: 8bf4c8059a240d187edb1489d266b2ec09b67069
Parents: 89a5b3c cd9c9f8
Author: Dan Halperin 
Authored: Wed Dec 28 20:32:27 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 20:32:27 2016 -0800

--
 .../examples/complete/autocomplete.py   |   4 +-
 .../examples/complete/estimate_pi.py|   4 +-
 .../apache_beam/examples/complete/tfidf.py  |   4 +-
 .../examples/complete/top_wikipedia_sessions.py |   4 +-
 .../examples/cookbook/bigquery_side_input.py|   4 +-
 .../apache_beam/examples/cookbook/bigshuffle.py |   4 +-
 .../apache_beam/examples/cookbook/coders.py |   4 +-
 .../examples/cookbook/custom_ptransform.py  |   2 +-
 .../examples/cookbook/datastore_wordcount.py|   6 +-
 .../examples/cookbook/group_with_coder.py   |   4 +-
 .../examples/cookbook/mergecontacts.py  |   4 +-
 .../examples/cookbook/multiple_output_pardo.py  |   4 +-
 .../apache_beam/examples/snippets/snippets.py   |  50 +-
 .../examples/snippets/snippets_test.py  |   2 +-
 sdks/python/apache_beam/examples/wordcount.py   |   4 +-
 .../apache_beam/examples/wordcount_debugging.py |   4 +-
 .../apache_beam/examples/wordcount_minimal.py   |   4 +-
 sdks/python/apache_beam/internal/apiclient.py   |   8 +-
 .../apache_beam/internal/apiclient_test.py  |   2 +-
 sdks/python/apache_beam/internal/auth.py|   4 +-
 sdks/python/apache_beam/io/bigquery.py  |   2 +-
 sdks/python/apache_beam/io/bigquery_test.py |   2 +-
 sdks/python/apache_beam/pipeline.py |   8 +-
 .../apache_beam/runners/dataflow_runner.py  |   2 +-
 .../runners/direct/transform_evaluator.py   |   2 +-
 sdks/python/apache_beam/runners/runner_test.py  |   2 +-
 .../apache_beam/runners/template_runner_test.py |   2 +-
 .../runners/test/test_dataflow_runner.py|   2 +-
 sdks/python/apache_beam/test_pipeline.py|   2 +-
 sdks/python/apache_beam/test_pipeline_test.py   |   2 +-
 sdks/python/apache_beam/transforms/core.py  |   2 +-
 sdks/python/apache_beam/transforms/display.py   |   2 +-
 .../apache_beam/transforms/display_test.py  |   2 +-
 .../python/apache_beam/transforms/ptransform.py |   2 +-
 .../apache_beam/transforms/ptransform_test.py   |   4 +-
 .../transforms/write_ptransform_test.py |   2 +-
 .../typehints/typed_pipeline_test.py|   4 +-
 sdks/python/apache_beam/utils/dependency.py |   4 +-
 .../python/apache_beam/utils/dependency_test.py |   6 +-
 sdks/python/apache_beam/utils/options.py| 543 ---
 .../apache_beam/utils/pipeline_options.py   | 540 ++
 .../apache_beam/utils/pipeline_options_test.py  |   2 +-
 .../utils/pipeline_options_validator.py |  14 +-
 .../utils/pipeline_options_validator_test.py|   2 +-
 44 files changed, 639 insertions(+), 642 deletions(-)
--

[2/3] beam git commit: Rename options.py -> pipeline_options.py

[dhalperi]

Rename options.py -> pipeline_options.py


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cd9c9f88
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cd9c9f88
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cd9c9f88

Branch: refs/heads/python-sdk
Commit: cd9c9f79d425a990031f28bf3bff63d2087e
Parents: 89a5b3c
Author: Maria Garcia Herrero 
Authored: Wed Dec 28 15:28:37 2016 -0800
Committer: Dan Halperin 
Committed: Wed Dec 28 20:32:24 2016 -0800

--
 .../examples/complete/autocomplete.py   |   4 +-
 .../examples/complete/estimate_pi.py|   4 +-
 .../apache_beam/examples/complete/tfidf.py  |   4 +-
 .../examples/complete/top_wikipedia_sessions.py |   4 +-
 .../examples/cookbook/bigquery_side_input.py|   4 +-
 .../apache_beam/examples/cookbook/bigshuffle.py |   4 +-
 .../apache_beam/examples/cookbook/coders.py |   4 +-
 .../examples/cookbook/custom_ptransform.py  |   2 +-
 .../examples/cookbook/datastore_wordcount.py|   6 +-
 .../examples/cookbook/group_with_coder.py   |   4 +-
 .../examples/cookbook/mergecontacts.py  |   4 +-
 .../examples/cookbook/multiple_output_pardo.py  |   4 +-
 .../apache_beam/examples/snippets/snippets.py   |  50 +-
 .../examples/snippets/snippets_test.py  |   2 +-
 sdks/python/apache_beam/examples/wordcount.py   |   4 +-
 .../apache_beam/examples/wordcount_debugging.py |   4 +-
 .../apache_beam/examples/wordcount_minimal.py   |   4 +-
 sdks/python/apache_beam/internal/apiclient.py   |   8 +-
 .../apache_beam/internal/apiclient_test.py  |   2 +-
 sdks/python/apache_beam/internal/auth.py|   4 +-
 sdks/python/apache_beam/io/bigquery.py  |   2 +-
 sdks/python/apache_beam/io/bigquery_test.py |   2 +-
 sdks/python/apache_beam/pipeline.py |   8 +-
 .../apache_beam/runners/dataflow_runner.py  |   2 +-
 .../runners/direct/transform_evaluator.py   |   2 +-
 sdks/python/apache_beam/runners/runner_test.py  |   2 +-
 .../apache_beam/runners/template_runner_test.py |   2 +-
 .../runners/test/test_dataflow_runner.py|   2 +-
 sdks/python/apache_beam/test_pipeline.py|   2 +-
 sdks/python/apache_beam/test_pipeline_test.py   |   2 +-
 sdks/python/apache_beam/transforms/core.py  |   2 +-
 sdks/python/apache_beam/transforms/display.py   |   2 +-
 .../apache_beam/transforms/display_test.py  |   2 +-
 .../python/apache_beam/transforms/ptransform.py |   2 +-
 .../apache_beam/transforms/ptransform_test.py   |   4 +-
 .../transforms/write_ptransform_test.py |   2 +-
 .../typehints/typed_pipeline_test.py|   4 +-
 sdks/python/apache_beam/utils/dependency.py |   4 +-
 .../python/apache_beam/utils/dependency_test.py |   6 +-
 sdks/python/apache_beam/utils/options.py| 543 ---
 .../apache_beam/utils/pipeline_options.py   | 540 ++
 .../apache_beam/utils/pipeline_options_test.py  |   2 +-
 .../utils/pipeline_options_validator.py |  14 +-
 .../utils/pipeline_options_validator_test.py|   2 +-
 44 files changed, 639 insertions(+), 642 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cd9c9f88/sdks/python/apache_beam/examples/complete/autocomplete.py
--
diff --git a/sdks/python/apache_beam/examples/complete/autocomplete.py 
b/sdks/python/apache_beam/examples/complete/autocomplete.py
index eaa5ca2..87e6c0c 100644
--- a/sdks/python/apache_beam/examples/complete/autocomplete.py
+++ b/sdks/python/apache_beam/examples/complete/autocomplete.py
@@ -24,8 +24,8 @@ import logging
 import re
 
 import apache_beam as beam
-from apache_beam.utils.options import PipelineOptions
-from apache_beam.utils.options import SetupOptions
+from apache_beam.utils.pipeline_options import PipelineOptions
+from apache_beam.utils.pipeline_options import SetupOptions
 
 
 def run(argv=None):

http://git-wip-us.apache.org/repos/asf/beam/blob/cd9c9f88/sdks/python/apache_beam/examples/complete/estimate_pi.py
--
diff --git a/sdks/python/apache_beam/examples/complete/estimate_pi.py 
b/sdks/python/apache_beam/examples/complete/estimate_pi.py
index 682c6d2..d0faefe 100644
--- a/sdks/python/apache_beam/examples/complete/estimate_pi.py
+++ b/sdks/python/apache_beam/examples/complete/estimate_pi.py
@@ -36,8 +36,8 @@ import apache_beam as beam
 from apache_beam.typehints import Any
 from apache_beam.typehints import Iterable
 from apache_beam.typehints import Tuple
-from apache_beam.utils.options import PipelineOptions
-from apache_beam.utils.options import SetupOptions
+from apache_beam.utils.pipeline_options import PipelineOptions
+from apache_beam.utils.pipeline_options import SetupOptions

[1/3] beam git commit: Rename options.py -> pipeline_options.py

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/python-sdk 89a5b3c3e -> 8bf4c8059


http://git-wip-us.apache.org/repos/asf/beam/blob/cd9c9f88/sdks/python/apache_beam/utils/pipeline_options_validator_test.py
--
diff --git a/sdks/python/apache_beam/utils/pipeline_options_validator_test.py 
b/sdks/python/apache_beam/utils/pipeline_options_validator_test.py
index 8878c3f..ffed048 100644
--- a/sdks/python/apache_beam/utils/pipeline_options_validator_test.py
+++ b/sdks/python/apache_beam/utils/pipeline_options_validator_test.py
@@ -21,7 +21,7 @@ import logging
 import unittest
 
 from apache_beam.internal import pickler
-from apache_beam.utils.options import PipelineOptions
+from apache_beam.utils.pipeline_options import PipelineOptions
 from apache_beam.utils.pipeline_options_validator import 
PipelineOptionsValidator
 from hamcrest.core.base_matcher import BaseMatcher
 

[jira] [Commented] (BEAM-773) Implement Metrics support for Flink runner

[JingChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784365#comment-15784365
 ] 

JingChen commented on BEAM-773:
---

when will the issue be resolved? i'm urgent to use metrics in beam-flink, or i 
can submit a merge-quest

> Implement Metrics support for Flink runner
> --
>
> Key: BEAM-773
> URL: https://issues.apache.org/jira/browse/BEAM-773
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Chambers
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-147) Introduce an easy API for pipeline metrics

[JingChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784363#comment-15784363
 ] 

JingChen commented on BEAM-147:
---

when will flink runner support metrics api? i'm urgent to use metrics in 
beam-flink

> Introduce an easy API for pipeline metrics
> --
>
> Key: BEAM-147
> URL: https://issues.apache.org/jira/browse/BEAM-147
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Robert Bradshaw
>Assignee: Ben Chambers
>
> The existing Aggregators are confusing both because of their name and because 
> they serve multiple purposes.
> Previous discussions around Aggregators/metrics/etc.
> See discussion at 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
>  and 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser
>  . Exact name still being bikeshedded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] beam-site git commit: This closes #115

[davor]

This closes #115


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/80fc7499
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/80fc7499
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/80fc7499

Branch: refs/heads/asf-site
Commit: 80fc7499daeab80d12625d73d0c7badaf8562e96
Parents: 1e2528f 6682c44
Author: Davor Bonaci 
Authored: Wed Dec 28 18:42:21 2016 -0800
Committer: Davor Bonaci 
Committed: Wed Dec 28 18:42:21 2016 -0800

--
 content/contribute/maturity-model/index.html |  4 +-
 content/contribute/release-guide/index.html  | 88 ---
 content/get-started/downloads/index.html |  6 +-
 src/contribute/maturity-model.md |  4 +-
 src/contribute/release-guide.md  | 82 -
 src/get-started/downloads.md |  6 +-
 6 files changed, 37 insertions(+), 153 deletions(-)
--

[1/3] beam-site git commit: Update Release Guide as part of the post graduation tasks

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 1e2528f17 -> 80fc7499d


Update Release Guide as part of the post graduation tasks


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/b2ae6a94
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/b2ae6a94
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/b2ae6a94

Branch: refs/heads/asf-site
Commit: b2ae6a9404ee839240c22f7db2ed7c8cce4383b7
Parents: 1e2528f
Author: Jean-Baptiste Onofré 
Authored: Wed Dec 28 21:52:55 2016 +0100
Committer: Davor Bonaci 
Committed: Wed Dec 28 18:40:30 2016 -0800

--
 src/contribute/maturity-model.md |  4 +-
 src/contribute/release-guide.md  | 82 ++-
 src/get-started/downloads.md |  6 +--
 3 files changed, 18 insertions(+), 74 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/b2ae6a94/src/contribute/maturity-model.md
--
diff --git a/src/contribute/maturity-model.md b/src/contribute/maturity-model.md
index 9123c15..3ae90de 100644
--- a/src/contribute/maturity-model.md
+++ b/src/contribute/maturity-model.md
@@ -87,7 +87,7 @@ The following table summarizes project's self-assessment 
against the Apache Matu
 
   RE10
   Releases consist of source code, distributed using standard and 
open archive formats that are expected to stay readable in the long term. 
[6]
-  YES. https://dist.apache.org/repos/dist/release/incubator/beam/";>Source 
releases are distributed via dist.apache.org and linked from the website.
+  YES. https://dist.apache.org/repos/dist/release/beam/";>Source releases are 
distributed via dist.apache.org and linked from the website.
 
 
   RE20
@@ -97,7 +97,7 @@ The following table summarizes project's self-assessment 
against the Apache Matu
 
   RE30
   Releases are signed and/or distributed along with digests that 
can be reliably used to validate the downloaded archives.
-  YES. All releases are signed, and the https://dist.apache.org/repos/dist/release/incubator/beam/KEYS";>KEYS 
file is provided on dist.apache.org.
+  YES. All releases are signed, and the https://dist.apache.org/repos/dist/release/beam/KEYS";>KEYS file is 
provided on dist.apache.org.
 
 
   RE40

http://git-wip-us.apache.org/repos/asf/beam-site/blob/b2ae6a94/src/contribute/release-guide.md
--
diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md
index 73e90a6..bb6a95f 100644
--- a/src/contribute/release-guide.md
+++ b/src/contribute/release-guide.md
@@ -75,7 +75,7 @@ This will list your GPG keys. One of these should reflect 
your Apache account, f
 
 Here, the key ID is the 8-digit hex string in the `pub` line: `845E6689`.
 
-Now, add your Apache GPG key to the Beam’s `KEYS` file both in 
[`dev`](https://dist.apache.org/repos/dist/dev/incubator/beam/KEYS) and 
[`release`](https://dist.apache.org/repos/dist/release/incubator/beam/KEYS) 
repositories at `dist.apache.org`. Follow the instructions listed at the top of 
these files.
+Now, add your Apache GPG key to the Beam’s `KEYS` file both in 
[`dev`](https://dist.apache.org/repos/dist/dev/beam/KEYS) and 
[`release`](https://dist.apache.org/repos/dist/release/beam/KEYS) repositories 
at `dist.apache.org`. Follow the instructions listed at the top of these files.
 
 Configure `git` to use this key when signing code by giving it your key ID, as 
follows:
 
@@ -161,8 +161,8 @@ Check out the version of the codebase from which you start 
the release. For a ne
 
 Set up a few environment variables to simplify Maven commands that follow. (We 
use `bash` Unix syntax in this guide.)
 
-VERSION="1.2.3-incubating"
-NEXT_VERSION="1.2.4-incubating"
+VERSION="1.2.3"
+NEXT_VERSION="1.2.4"
 BRANCH_NAME="release-${VERSION}"
 DEVELOPMENT_VERSION="${NEXT_VERSION}-SNAPSHOT"
 
@@ -235,7 +235,7 @@ Copy the source release to the dev repository of 
`dist.apache.org`.
 
 1. If you have not already, check out the Beam section of the `dev` repository 
on `dist.apache.org` via Subversion. In a fresh directory:
 
-svn co https://dist.apache.org/repos/dist/dev/incubator/beam
+svn co https://dist.apache.org/repos/dist/dev/beam
 
 1. Make a directory for the new release:
 
@@ -253,7 +253,7 @@ Copy the source release to the dev repository of 
`dist.apache.org`.
 svn add beam/${VERSION}
 svn commit
 
-1. Verify that files are 
[present](https://dist.apache.org/repos/dist/dev/incubator/beam).
+1. Verify that files are 
[present](https://dist.apache.org/repos/dist/dev/beam).
 
 ### Build the API reference
 
@@ -291,7 +291,7 @@ Finally, propose a pull reque

[GitHub] beam-site pull request #115: Update Release Guide as part of the post gradua...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/115


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/6682c44a
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/6682c44a
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/6682c44a

Branch: refs/heads/asf-site
Commit: 6682c44abefa98697cfd91442f50218f04faf033
Parents: b2ae6a9
Author: Davor Bonaci 
Authored: Wed Dec 28 18:42:20 2016 -0800
Committer: Davor Bonaci 
Committed: Wed Dec 28 18:42:20 2016 -0800

--
 content/contribute/maturity-model/index.html |  4 +-
 content/contribute/release-guide/index.html  | 88 ---
 content/get-started/downloads/index.html |  6 +-
 3 files changed, 19 insertions(+), 79 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/6682c44a/content/contribute/maturity-model/index.html
--
diff --git a/content/contribute/maturity-model/index.html 
b/content/contribute/maturity-model/index.html
index 10b68a0..be18b0f 100644
--- a/content/contribute/maturity-model/index.html
+++ b/content/contribute/maturity-model/index.html
@@ -237,7 +237,7 @@
 
   RE10
   Releases consist of source code, distributed using standard and 
open archive formats that are expected to stay readable in the long term. 
[6]
-  YES. https://dist.apache.org/repos/dist/release/incubator/beam/";>Source 
releases are distributed via dist.apache.org and linked from the website.
+  YES. https://dist.apache.org/repos/dist/release/beam/";>Source releases are 
distributed via dist.apache.org and linked from the website.
 
 
   RE20
@@ -247,7 +247,7 @@
 
   RE30
   Releases are signed and/or distributed along with digests that 
can be reliably used to validate the downloaded archives.
-  YES. All releases are signed, and the https://dist.apache.org/repos/dist/release/incubator/beam/KEYS";>KEYS 
file is provided on dist.apache.org.
+  YES. All releases are signed, and the https://dist.apache.org/repos/dist/release/beam/KEYS";>KEYS file is 
provided on dist.apache.org.
 
 
   RE40

http://git-wip-us.apache.org/repos/asf/beam-site/blob/6682c44a/content/contribute/release-guide/index.html
--
diff --git a/content/contribute/release-guide/index.html 
b/content/contribute/release-guide/index.html
index 97b9ef5..a1505bb 100644
--- a/content/contribute/release-guide/index.html
+++ b/content/contribute/release-guide/index.html
@@ -196,7 +196,7 @@
   
   Promote the release
   Apache mailing lists
-  ASF 
press release
+  Recordkeeping
   Beam blog
   Social 
media
   Checklist to 
declare the process completed
@@ -279,7 +279,7 @@ sub   2048R/BA4D50BE 2016-02-23
 
 Here, the key ID is the 8-digit hex string in the pub line: 845E6689.
 
-Now, add your Apache GPG key to the Beam’s KEYS file both in https://dist.apache.org/repos/dist/dev/incubator/beam/KEYS";>dev and https://dist.apache.org/repos/dist/release/incubator/beam/KEYS";>release repositories at dist.apache.org. Follow the instructions 
listed at the top of these files.
+Now, add your Apache GPG key to the Beam’s KEYS file both in https://dist.apache.org/repos/dist/dev/beam/KEYS";>dev and https://dist.apache.org/repos/dist/release/beam/KEYS";>release repositories at dist.apache.org. Follow the instructions 
listed at the top of these files.
 
 Configure git to use this key when 
signing code by giving it your key ID, as follows:
 
@@ -382,8 +382,8 @@ export GPG_AGENT_INFO
 
 Set up a few environment variables to simplify Maven commands that follow. 
(We use bash Unix syntax in this 
guide.)
 
-VERSION="1.2.3-incubating"
-NEXT_VERSION="1.2.4-incubating"
+VERSION="1.2.3"
+NEXT_VERSION="1.2.4"
 BRANCH_NAME="release-${VERSION}"
 DEVELOPMENT_VERSION="${NEXT_VERSION}-SNAPSHOT"
 
@@ -474,7 +474,7 @@ TAG="v${VERSION}-RC${RC_NUM}"
   
 If you have not already, check out the Beam section of the dev repository on dist.apache.org via Subversion. In a fresh 
directory:
 
- svn co 
https://dist.apache.org/repos/dist/dev/incubator/beam
+ svn co 
https://dist.apache.org/repos/dist/dev/beam
 
 
   
@@ -504,7 +504,7 @@ TAG="v${VERSION}-RC${RC_NUM}"
 
   
   
-Verify that files are https://dist.apache.org/repos/dist/dev/incubator/beam";>present.
+Verify that files are https://dist.apache.org/repos/dist/dev/beam";>present.
   
 
 
@@ -551,7 +551,7 @@ TAG="v${VERSION}-RC${RC_NUM}"
 
 
   Maven artifacts deployed to the staging repository of https://repository.apache.org/content/repositories/";>repository.apache.org
-  Source distribution deployed to the dev repository of https://dist.apache.org/repos/dist/dev/incubator/beam/";>dist.apache.org
+  Source distri

svn commit: r17612 - /dev/beam/0.3.0-incubating/

[dhalperi]

Author: dhalperi
Date: Thu Dec 29 02:31:04 2016
New Revision: 17612

Log:
Delete the dev version of 0.3.0-incubating release of Beam.

Removed:
dev/beam/0.3.0-incubating/

svn commit: r17611 - /release/beam/

[davor]

Author: davor
Date: Thu Dec 29 02:24:39 2016
New Revision: 17611

Log:
Move release/incubator/beam to release/beam following Beam TLP graduation.


Added:
release/beam/
  - copied from r17610, release/incubator/beam/

svn commit: r17611 - /release/incubator/beam/

[davor]

Author: davor
Date: Thu Dec 29 02:24:39 2016
New Revision: 17611

Log:
Move release/incubator/beam to release/beam following Beam TLP graduation.


Removed:
release/incubator/beam/

svn commit: r17610 - /dev/incubator/beam/

[davor]

Author: davor
Date: Thu Dec 29 02:21:08 2016
New Revision: 17610

Log:
Remove dev/incubator/beam following Beam TLP graduation.


Removed:
dev/incubator/beam/

svn commit: r17609 - in /dev/beam: 0.3.0-incubating/ 0.4.0-incubating/

[davor]

Author: davor
Date: Thu Dec 29 02:18:40 2016
New Revision: 17609

Log:
Move dev/incubator/beam to dev/beam following Beam TLP graduation.


Added:
dev/beam/0.3.0-incubating/
  - copied from r17608, dev/incubator/beam/0.3.0-incubating/
dev/beam/0.4.0-incubating/
  - copied from r17608, dev/incubator/beam/0.4.0-incubating/

svn commit: r17608 - in /dev/beam: ./ KEYS

[davor]

Author: davor
Date: Thu Dec 29 02:12:42 2016
New Revision: 17608

Log:
Add dev/beam/KEYS from dev/incubator/beam/KEYS.


Added:
dev/beam/
dev/beam/KEYS

Added: dev/beam/KEYS
==
--- dev/beam/KEYS (added)
+++ dev/beam/KEYS Thu Dec 29 02:12:42 2016
@@ -0,0 +1,199 @@
+This file contains the PGP keys of various developers.
+
+Users: pgp < KEYS
+   gpg --import KEYS
+Developers:
+pgp -kxa  and append it to this file.
+(pgpk -ll  && pgpk -xa ) >> this file.
+(gpg --list-sigs 
+ && gpg --armor --export ) >> this file.
+
+
+pub   4096R/C8282E76 2009-09-08
+uid  Jean-Baptiste Onofré 
+sig 3C8282E76 2009-09-08  Jean-Baptiste Onofré 
+sub   4096R/9F043BBC 2009-09-08
+sig  C8282E76 2009-09-08  Jean-Baptiste Onofré 
+
+-BEGIN PGP PUBLIC KEY BLOCK-
+Version: GnuPG v1
+
+mQINBEqmJkEBEADAAMOjOidXzoyK4FK9WhhRg2EEGX1gm5lK8PpJtk68Fqmz6xvv
+N8VJXMIJUgeD7M35zZSQUWJY43xEU8Yfn6oLL0KR0dIqVOclxE+7G8vxXFcIbRE9
+ziZFp7Z5yzsdzjiIzXv5MVQMczcAAMev/i0BnjiRy5Cg+k6kHXVpu/Gsn05JKPaG
+s7ZcfSxpboyS99MVKQvoFLE5Z/Shh4gFJn2rFInqK5EgVpoZbVyysF52nx0dti/e
+O0NjraQkrEDBWvsPt3cYZA0oP1gWiZiRvOLfAFIarf3poMDyoWBIwnbqb3Msv09j
+yDAmcGq9wsD3alHFHcRIiJl5SzFUStml1d5x/BvUl/Xc5VfHPi2ObKF3xOPGkyTf
+aZ6mYFLaRCAJ0v2MPW+4/grDXKsP8n8xPbE2VQvHBpxaZklD7q4Omn2d+m2sUOLX
+NRUo4n29NyfowAffBYl7ZqrYBBodR9YngWC9LpgM+APHyiw3HzauZ94bGy5Of3+L
+Yu6/riDcP4OXF6r6IH6KIsVqIkv5xzq7OGxxXmlhWg8ifNPLq5yNRccS0nWXc5BD
+/9q06ta/ceQGNkXL327XPuZC+lstWGAa4dKEosRDgcO0Pv2j2a3h8W8oHyxF+gEe
+O+9s0mGdQFxNiEA+JyeKCg+jvfx9Hv/2Syrlert76NEkfbaTFA7BJ4c3EQARAQAB
+tCtKZWFuLUJhcHRpc3RlIE9ub2Zyw6kgPGpib25vZnJlQGFwYWNoZS5vcmc+iQI2
+BBMBAgAgBQJKpiZBAhsDBgsJCAcDAgQVAggDBBYCAwECHgECF4AACgkQv/LuQsgo
+LnboyRAAguqFIpiKkCCR6TR0Y5UQDFhgEMhBreQKCEW0czbGoFnxfULV9H1kJRSB
+Vt0knecGaYS340WEmz4B7BMpkBCgaszgn66+fhacZTBd+Aff1k2lbhdMgdBvlPcm
+q9vFGtbE515j9bPHzsPRJ2wFWd6ot9wXiLD3RJLV6c7L3Egstu3qTp0tEoFHrQps
+qskGBl+mahhMyz3BUDlusavB0Y0tb6hhXCR79ErhjQrTgU947isztYWpgJlA40lx
+DW0hskZWbuGNXjxUJvTT3pKiYUN32WG+2CDNYHceuhsfRLxO/Wb4BKwwDaHWAlH9
+d5F9/vhdPObSv5GQbuUtmCEzeqADUd65jLLM7WSlvRJ+i4m0/TTeP8y4NfxlVbBP
+WuYrQW4gPmDKEDNvEec6PH6hhBfMLJz3M6o4huwLp2kQrq6wSTMDGIoxOLP0ae3c
+BMIuFM5EavLDJmuATUIWWyZt/c7mmAOOh5TGcFWTugnJ6l4FllOrFPiWyFsjMn+U
+zzzaeSkYmq/xZYxjRTdWjK5Zb5rbVuCx/q5VF9Awdy4EM6UXhaqWo06VyjWNOJ86
+wgres4+bVldB7+TiVi9iO6n80WNlPgIaQJlLc+FRsld4Er21kdXreX5doxFD5Iue
+S4y/pLwftHfx1xxj+p2jPJ49Hb0ddNr+XrsrO5txing2pNJgfH65Ag0ESqYmQQEQ
+AKPoXgIIKnyJiPvks7xBV+FqJPecVAx3SSlLyTfsh/jBat9QLd4hsfiZcv1ANZHB
+n4qDeGlsmJ6uDGv8wnUZQ2Im8Heje1h7dKeLNpNnxfBS9gn6e2bXKhAsJGUE7gip
+qVfijFnEY0Vj6Tztzq+Wyqg2Gbz+bJZMo1JVQiaAYyQeQlrOcoZcQHsA/Ol+y48h
+Le36A1TSIPMOSI4ZAZXkqxXAumEaMaz82EvV8KDH7Ijr23Y0wZjEUJ+dJQM9ssuE
+f9GMLIuCbmM/CJ5MCCwepGJd52ymllvgJTHC7B+BY/jKNMWHwAsMJ1oWcPlLzFQI
+Bmyy5RjKoMifzaoSo/hTWkiwcL2Vc+qU3b3/2eUtnCnBB/nkrZkJNNc+OV5YGBSP
+vNPaN43Gvjbvborv4PBvt7QhVjZYQemtXO2sWx1XWSFsucD2K4kJ8ipNWxVgIqDu
+J8SJOnGigX9hMpsZ2HVAwOeKP/jI90J3voKrCPLaKcL1Ip+b28k0aj7kl44YJqw4
+5pbRSx/v73bH4uleQiXSW+JczA+KLw7hX3tOWJEnLS2+Ig9sNUKYGZOg0nw613bN
+fZy8Cbx/UkT10Lznx9FW6MedGyJPYT4MJMMh/PnnsWv50jFnfu2rtnRXEOUXwujL
+fwrmCYbXHgE3Ka+fmRz8HxsyTmtqIHtPixw8RoqfoFfxABEBAAGJAh8EGAECAAkF
+AkqmJkECGwwACgkQv/LuQsgoLnb8AQ/+POsLFdqNqSKfwBXp1YOIEjNdbVjysQc6
+zC6LlMJXNSxAmUmol2g9bJYh9LdpvOTU3gfFgIanaGytC75U7/NOl0zEsN4IU18j
+CLBNaD5/Or1ciQ3CVrID/lPO8s0Hm0/cUPreEjJPPrrPbXG+i9bweg3Dtfy3+WQl
+PhfpvgudwtUjB3st2gztYipkUhmrH+STbbJZVJN5ZNL8mOoM5M2wGS+9VweOWbKe
+z0QeZ9hIPyQNMzTn1xlvRUVNTu8fz2FGvumrd+zgzYcpTE5VpFkOxxUayr3aWXSf
+Cak+HH0WjUDWc9/lJR4dVpwdjLonJfiC70W07J4CnNodYwnPUaGKTVYq3pvQzAPw
+hjx4u6t5zTZy5CbCAEhZC/9GeQmtuM0rcQhz048don4s4baDrqUPKL+X3C3ev4/o
+00yLrQ5rLX8K4iE/Go5xUyhzT7gqFJUPWdo8neTXXwQGThqqhVQovnn3M6i55rCg
+EeOTd7uW+k3vt6kunWZFKPjzRBxMD4NYovIQXwhPxj0vq6DnE0RQa7Dfm6l3cAV7
+/l3kRQcT69AWXotUJQnpY4bemTuYlxAYWCkTGNLdNNiBhiaqlR7xgYMNXS4XqcgA
+6QtP8ulb2FPR0MWEtvGkbHgAAIayV+Jt1Ed2JkIsdJHGeSZO5WEiupySDQCGn6rZ
+DR2E2zua3tQ=
+=LCyH
+-END PGP PUBLIC KEY BLOCK-
+pub   2048R/8F0D334F 2014-10-31
+uid  Davor Bonaci 
+sig 38F0D334F 2014-10-31  Davor Bonaci 
+sub   2048R/D1B59DCE 2014-10-31
+sig  8F0D334F 2014-10-31  Davor Bonaci 
+
+-BEGIN PGP PUBLIC KEY BLOCK-
+Version: GnuPG v1
+
+mQENBFRUAG4BCADDIXl7xSyNCm/4PiaRWLQXB/fjiAYEDZvrodiA7sYh7ZBeuUzV
+R72RXdzqsL43ZKS0OVfgKIN/qSriVHzz6HfdBJNLbVdzHlIA9/zCTZ/rgPT72P+i
+flU/iKPVuYLd+xOb/iPG1Ha0zMobw3tWWD3ifXIe+N8DEnxo1ywiP7+Uf2MDmK70
+Is0v55/pF/fTridIK/a3Hmt5OtyA/quw0j7k2ga9bi/m+R9BO7W198EpSAn6rZRI
+Ipo+kjdEdF1QKR39dbGjLjzfAsT24Loq2H9WZnSjwaPaAIJWZR82AcFda04qEiJ/
+yfQLn+IXihNErX7IFUd+PcQQ9RjyABwL9LWXABEBAAG0H0Rhdm9yIEJvbmFjaSA8
+ZGF2b3JAZ29vZ2xlLmNvbT6JATgEEwECACIFAlRUAG4CGwMGCwkIBwMCBhUIAgkK
+CwQWAgMBAh4BAheAAAoJEMkEN+GPDTNPpVUIALNRhjG0gwdE3OuHTWo/YUNUJQIa
+CU4TwgfXBcgIrhCzlfeiIlD7yOtpsSzChVR44cwZVzGRT/5GOxhnm34G+aNSQM0c
+6PbCNnyj6DZ/MGv1swc+xhBGPSBPBAeE58LLZPLXEXt6V9etqSJRgCYNzYc/ntSf
+y/V9YuFanhghe1TKoqUvlOqX+weTz7UTpTg2aaGulrPl9BNlea6G

Jenkins build is back to normal : beam_PostCommit_Python_Verify #947

[Apache Jenkins Server]

See 

[jira] [Reopened] (BEAM-430) Introducing gcpTempLocation that default to tempLocation

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-430:
---

(reopening temporarily to add the components tag to the issue)

> Introducing gcpTempLocation that default to tempLocation
> 
>
> Key: BEAM-430
> URL: https://issues.apache.org/jira/browse/BEAM-430
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: 0.2.0-incubating
>
>
> Currently, DataflowPipelineOptions.stagingLocation default to tempLocation. 
> And, it requires tempLocation to be a gcs path.
> Another case is BigQueryIO uses tempLocation and also requires it to be on 
> gcs.
> So, users cannot set tempLocation to a non-gcs path with DataflowRunner or 
> BigQueryIO.
> However, tempLocation could be on any file system. For example, WordCount 
> defaults to output to tempLocation.
> The proposal is to add gcpTempLocation. And, it defaults to tempLocation if 
> tempLocation is a gcs path.
> StagingLocation and BigQueryIO will use gcpTempLocation by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-430) Introducing gcpTempLocation that default to tempLocation

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-430:
--
Component/s: sdk-java-gcp

> Introducing gcpTempLocation that default to tempLocation
> 
>
> Key: BEAM-430
> URL: https://issues.apache.org/jira/browse/BEAM-430
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: 0.2.0-incubating
>
>
> Currently, DataflowPipelineOptions.stagingLocation default to tempLocation. 
> And, it requires tempLocation to be a gcs path.
> Another case is BigQueryIO uses tempLocation and also requires it to be on 
> gcs.
> So, users cannot set tempLocation to a non-gcs path with DataflowRunner or 
> BigQueryIO.
> However, tempLocation could be on any file system. For example, WordCount 
> defaults to output to tempLocation.
> The proposal is to add gcpTempLocation. And, it defaults to tempLocation if 
> tempLocation is a gcs path.
> StagingLocation and BigQueryIO will use gcpTempLocation by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-430) Introducing gcpTempLocation that default to tempLocation

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-430.
---
Resolution: Fixed

> Introducing gcpTempLocation that default to tempLocation
> 
>
> Key: BEAM-430
> URL: https://issues.apache.org/jira/browse/BEAM-430
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: 0.2.0-incubating
>
>
> Currently, DataflowPipelineOptions.stagingLocation default to tempLocation. 
> And, it requires tempLocation to be a gcs path.
> Another case is BigQueryIO uses tempLocation and also requires it to be on 
> gcs.
> So, users cannot set tempLocation to a non-gcs path with DataflowRunner or 
> BigQueryIO.
> However, tempLocation could be on any file system. For example, WordCount 
> defaults to output to tempLocation.
> The proposal is to add gcpTempLocation. And, it defaults to tempLocation if 
> tempLocation is a gcs path.
> StagingLocation and BigQueryIO will use gcpTempLocation by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-754) WordCountIT Flake -- Incorrect checksum

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-754:
---

(reopening temporarily to add the components tag to the issue)

> WordCountIT Flake -- Incorrect checksum
> ---
>
> Key: BEAM-754
> URL: https://issues.apache.org/jira/browse/BEAM-754
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Jason Kuster
>Assignee: Mark Liu
>Priority: Minor
> Fix For: Not applicable
>
>
> WordCountIT flaked in Jenkins PostCommit -- 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-examples-java/1532/testReport/junit/org.apache.beam.examples/WordCountIT/
>  -- due to an incorrect checksum. Can we add some additional debug output so 
> we can understand what happened in these cases?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-875) Inconsistency between time types between the inprocess and dataflow runner for BQ

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-875:
--
Component/s: sdk-py

> Inconsistency between time types between the inprocess and dataflow runner 
> for BQ
> -
>
> Key: BEAM-875
> URL: https://issues.apache.org/jira/browse/BEAM-875
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> investigate if we support datatypes TIMESTAMP, DATE, TIME, and DATETIME 
> properly and in a consistent matter across InProcess and Dataflow runners for 
> BQ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-875) Inconsistency between time types between the inprocess and dataflow runner for BQ

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-875.
-
Resolution: Fixed

> Inconsistency between time types between the inprocess and dataflow runner 
> for BQ
> -
>
> Key: BEAM-875
> URL: https://issues.apache.org/jira/browse/BEAM-875
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> investigate if we support datatypes TIMESTAMP, DATE, TIME, and DATETIME 
> properly and in a consistent matter across InProcess and Dataflow runners for 
> BQ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-754) WordCountIT Flake -- Incorrect checksum

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-754:
--
Component/s: examples-java

> WordCountIT Flake -- Incorrect checksum
> ---
>
> Key: BEAM-754
> URL: https://issues.apache.org/jira/browse/BEAM-754
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Jason Kuster
>Assignee: Mark Liu
>Priority: Minor
> Fix For: Not applicable
>
>
> WordCountIT flaked in Jenkins PostCommit -- 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-examples-java/1532/testReport/junit/org.apache.beam.examples/WordCountIT/
>  -- due to an incorrect checksum. Can we add some additional debug output so 
> we can understand what happened in these cases?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-875) Inconsistency between time types between the inprocess and dataflow runner for BQ

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-875:
---

(reopening temporarily to add the components tag to the issue)

> Inconsistency between time types between the inprocess and dataflow runner 
> for BQ
> -
>
> Key: BEAM-875
> URL: https://issues.apache.org/jira/browse/BEAM-875
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> investigate if we support datatypes TIMESTAMP, DATE, TIME, and DATETIME 
> properly and in a consistent matter across InProcess and Dataflow runners for 
> BQ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-754) WordCountIT Flake -- Incorrect checksum

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-754.
---
Resolution: Fixed

> WordCountIT Flake -- Incorrect checksum
> ---
>
> Key: BEAM-754
> URL: https://issues.apache.org/jira/browse/BEAM-754
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Jason Kuster
>Assignee: Mark Liu
>Priority: Minor
> Fix For: Not applicable
>
>
> WordCountIT flaked in Jenkins PostCommit -- 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-examples-java/1532/testReport/junit/org.apache.beam.examples/WordCountIT/
>  -- due to an incorrect checksum. Can we add some additional debug output so 
> we can understand what happened in these cases?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-538) Create Bigtop Beam packages

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-538.
---
Resolution: Invalid

> Create Bigtop Beam packages
> ---
>
> Key: BEAM-538
> URL: https://issues.apache.org/jira/browse/BEAM-538
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-ideas
>Reporter: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>
> Considering that Apache Beam is a unified model to write both Batch and 
> streaming jobs and that at runtime it depends on already packaged Bigtop 
> dependencies, such as Apache Spark and Apache Flink, It is probably not hard 
> to create ready to go Beam packages to be included in Bigtop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-411) Avoid calling BigQuery directly during validation.

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-411:
---

(reopening temporarily to add the components tag to the issue)

> Avoid calling BigQuery directly during validation.
> --
>
> Key: BEAM-411
> URL: https://issues.apache.org/jira/browse/BEAM-411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
> Fix For: Not applicable
>
>
> BigQueryIO calls the service directly. This makes tests hard to write and 
> slow.
> We should replace BigQuery direct calls with BigQueryServices abstraction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-411) Avoid calling BigQuery directly during validation.

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-411:
--
Component/s: sdk-java-gcp

> Avoid calling BigQuery directly during validation.
> --
>
> Key: BEAM-411
> URL: https://issues.apache.org/jira/browse/BEAM-411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
> Fix For: Not applicable
>
>
> BigQueryIO calls the service directly. This makes tests hard to write and 
> slow.
> We should replace BigQuery direct calls with BigQueryServices abstraction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-411) Avoid calling BigQuery directly during validation.

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-411.
---
Resolution: Fixed

> Avoid calling BigQuery directly during validation.
> --
>
> Key: BEAM-411
> URL: https://issues.apache.org/jira/browse/BEAM-411
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
> Fix For: Not applicable
>
>
> BigQueryIO calls the service directly. This makes tests hard to write and 
> slow.
> We should replace BigQuery direct calls with BigQueryServices abstraction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-538) Create Bigtop Beam packages

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-538:
--
Component/s: sdk-ideas

> Create Bigtop Beam packages
> ---
>
> Key: BEAM-538
> URL: https://issues.apache.org/jira/browse/BEAM-538
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-ideas
>Reporter: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>
> Considering that Apache Beam is a unified model to write both Batch and 
> streaming jobs and that at runtime it depends on already packaged Bigtop 
> dependencies, such as Apache Spark and Apache Flink, It is probably not hard 
> to create ready to go Beam packages to be included in Bigtop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-574) KafkaIO prints superfluous information

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-574:
---

(reopening temporarily to add the components tag to the issue)

> KafkaIO prints superfluous information
> --
>
> Key: BEAM-574
> URL: https://issues.apache.org/jira/browse/BEAM-574
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>
> If you create an unbounded pipeline that reads data from KafkaIO and there is 
> not new information in the Kafka topic, The KafkaIO keeps logging infinitely 
> a useless message:
> [pool-1-thread-4] WARN org.apache.beam.sdk.io.kafka.KafkaIO - Reader-0: 
> getWatermark() : no records have been read yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-429) minor: remove an obsolete comment in KafakIOTest.java

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-429:
--
Component/s: sdk-java-extensions

> minor: remove an obsolete comment in KafakIOTest.java
> -
>
> Key: BEAM-429
> URL: https://issues.apache.org/jira/browse/BEAM-429
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: Not applicable
>
>
> see https://github.com/apache/incubator-beam/pull/606



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-729) Does sdks/java/io/google-cloud-platform/pom.xml need javadoc plugin?

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-729:
--
Component/s: sdk-java-gcp

> Does sdks/java/io/google-cloud-platform/pom.xml need javadoc plugin?
> 
>
> Key: BEAM-729
> URL: https://issues.apache.org/jira/browse/BEAM-729
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Eugene Kirpichov
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> I would like to generate javadoc for that module by running the respective 
> plugin in IntelliJ's view of that maven module (not the root module).
> As-is, this maven module doesn't have a javadoc plugin (at least it doesn't 
> show up in IntelliJ). However, most other modules have it - e.g.:
> $ grep "maven-javadoc" **/pom.xml
>   
>   examples/java8/pom.xml: 
>maven-javadoc-plugin
> examples/java/pom.xml:maven-javadoc-plugin
> pom.xml:  maven-javadoc-plugin
> pom.xml:  maven-javadoc-plugin
> runners/core-java/pom.xml:
> maven-javadoc-plugin
> runners/direct-java/pom.xml:
> maven-javadoc-plugin
> runners/flink/examples/pom.xml:
> maven-javadoc-plugin
> runners/flink/runner/pom.xml:
> maven-javadoc-plugin
> runners/google-cloud-dataflow-java/pom.xml:
> maven-javadoc-plugin
> runners/spark/pom.xml:maven-javadoc-plugin
> sdks/java/core/pom.xml:  maven-javadoc-plugin
> sdks/java/core/pom.xml:maven-javadoc-plugin
> sdks/java/io/kafka/pom.xml:
> maven-javadoc-plugin
> sdks/java/io/kinesis/pom.xml:
> maven-javadoc-plugin
> sdks/java/io/mongodb/pom.xml:
> maven-javadoc-plugin
> Arguably, this module should have it too, then.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-429) minor: remove an obsolete comment in KafakIOTest.java

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-429:
---

(reopening temporarily to add the components tag to the issue)

> minor: remove an obsolete comment in KafakIOTest.java
> -
>
> Key: BEAM-429
> URL: https://issues.apache.org/jira/browse/BEAM-429
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: Not applicable
>
>
> see https://github.com/apache/incubator-beam/pull/606



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-574) KafkaIO prints superfluous information

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-574:
--
Component/s: sdk-java-extensions

> KafkaIO prints superfluous information
> --
>
> Key: BEAM-574
> URL: https://issues.apache.org/jira/browse/BEAM-574
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>
> If you create an unbounded pipeline that reads data from KafkaIO and there is 
> not new information in the Kafka topic, The KafkaIO keeps logging infinitely 
> a useless message:
> [pool-1-thread-4] WARN org.apache.beam.sdk.io.kafka.KafkaIO - Reader-0: 
> getWatermark() : no records have been read yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-538) Create Bigtop Beam packages

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-538:
---

(reopening temporarily to add the components tag to the issue)

> Create Bigtop Beam packages
> ---
>
> Key: BEAM-538
> URL: https://issues.apache.org/jira/browse/BEAM-538
> Project: Beam
>  Issue Type: Wish
>Reporter: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>
> Considering that Apache Beam is a unified model to write both Batch and 
> streaming jobs and that at runtime it depends on already packaged Bigtop 
> dependencies, such as Apache Spark and Apache Flink, It is probably not hard 
> to create ready to go Beam packages to be included in Bigtop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-484) Datastore Source should support Dynamic Splitting

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-484:
---

(reopening temporarily to add the components tag to the issue)

> Datastore Source should support Dynamic Splitting
> -
>
> Key: BEAM-484
> URL: https://issues.apache.org/jira/browse/BEAM-484
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
> Fix For: 0.2.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-446) Improve IOChannelUtils.resolve() to accept multiple paths at once

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-446.
---
Resolution: Fixed

> Improve IOChannelUtils.resolve() to accept multiple paths at once
> -
>
> Key: BEAM-446
> URL: https://issues.apache.org/jira/browse/BEAM-446
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> Currently, IOChannelUtils.resolve() method can only resolve one path against 
> base path. 
> It's useful to have another method with arguments that includes one base path 
> and multiple others. The return string will be a directory that start with 
> base path and append rests which are separated by file separator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-429) minor: remove an obsolete comment in KafakIOTest.java

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-429.
---
Resolution: Fixed

> minor: remove an obsolete comment in KafakIOTest.java
> -
>
> Key: BEAM-429
> URL: https://issues.apache.org/jira/browse/BEAM-429
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: Not applicable
>
>
> see https://github.com/apache/incubator-beam/pull/606



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-446) Improve IOChannelUtils.resolve() to accept multiple paths at once

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-446:
---

> Improve IOChannelUtils.resolve() to accept multiple paths at once
> -
>
> Key: BEAM-446
> URL: https://issues.apache.org/jira/browse/BEAM-446
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> Currently, IOChannelUtils.resolve() method can only resolve one path against 
> base path. 
> It's useful to have another method with arguments that includes one base path 
> and multiple others. The return string will be a directory that start with 
> base path and append rests which are separated by file separator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-574) KafkaIO prints superfluous information

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-574.
---
Resolution: Won't Fix

> KafkaIO prints superfluous information
> --
>
> Key: BEAM-574
> URL: https://issues.apache.org/jira/browse/BEAM-574
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>
> If you create an unbounded pipeline that reads data from KafkaIO and there is 
> not new information in the Kafka topic, The KafkaIO keeps logging infinitely 
> a useless message:
> [pool-1-thread-4] WARN org.apache.beam.sdk.io.kafka.KafkaIO - Reader-0: 
> getWatermark() : no records have been read yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-854) ReifyTimestampAndWindows explodes windows

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-854:
--
Component/s: runner-core

> ReifyTimestampAndWindows explodes windows
> -
>
> Key: BEAM-854
> URL: https://issues.apache.org/jira/browse/BEAM-854
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
> Fix For: 0.4.0-incubating
>
>
> Commit 
> https://github.com/apache/incubator-beam/commit/b2350417f73ae6c34f849ff0e93d5bd93df3088d
>  changed the fn from accessing the collection of windows to accessing the 
> windows one-by-one, which causes this DoFn to explode them, which (I suppose) 
> hurts performance, since this Fn is quite important: it's a part of 
> GroupByKey in the Spark runner and Direct runner, part of Splittable DoFn, 
> part of GBKViaGBKO, and part of GatherAllPanes.
> I can see a couple of ways to fix this:
> - Make this be a primitive transform, with runner-specific support to keep 
> the windows compressed
> - Allow any DoFn to access the compressed representation explicitly using a 
> Collection parameter
> I'm personally in favor of the latter because it automatically benefits all 
> runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-446) Improve IOChannelUtils.resolve() to accept multiple paths at once

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-446:
--
Component/s: sdk-java-core

> Improve IOChannelUtils.resolve() to accept multiple paths at once
> -
>
> Key: BEAM-446
> URL: https://issues.apache.org/jira/browse/BEAM-446
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> Currently, IOChannelUtils.resolve() method can only resolve one path against 
> base path. 
> It's useful to have another method with arguments that includes one base path 
> and multiple others. The return string will be a directory that start with 
> base path and append rests which are separated by file separator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-729) Does sdks/java/io/google-cloud-platform/pom.xml need javadoc plugin?

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-729.
---
Resolution: Fixed

> Does sdks/java/io/google-cloud-platform/pom.xml need javadoc plugin?
> 
>
> Key: BEAM-729
> URL: https://issues.apache.org/jira/browse/BEAM-729
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Eugene Kirpichov
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> I would like to generate javadoc for that module by running the respective 
> plugin in IntelliJ's view of that maven module (not the root module).
> As-is, this maven module doesn't have a javadoc plugin (at least it doesn't 
> show up in IntelliJ). However, most other modules have it - e.g.:
> $ grep "maven-javadoc" **/pom.xml
>   
>   examples/java8/pom.xml: 
>maven-javadoc-plugin
> examples/java/pom.xml:maven-javadoc-plugin
> pom.xml:  maven-javadoc-plugin
> pom.xml:  maven-javadoc-plugin
> runners/core-java/pom.xml:
> maven-javadoc-plugin
> runners/direct-java/pom.xml:
> maven-javadoc-plugin
> runners/flink/examples/pom.xml:
> maven-javadoc-plugin
> runners/flink/runner/pom.xml:
> maven-javadoc-plugin
> runners/google-cloud-dataflow-java/pom.xml:
> maven-javadoc-plugin
> runners/spark/pom.xml:maven-javadoc-plugin
> sdks/java/core/pom.xml:  maven-javadoc-plugin
> sdks/java/core/pom.xml:maven-javadoc-plugin
> sdks/java/io/kafka/pom.xml:
> maven-javadoc-plugin
> sdks/java/io/kinesis/pom.xml:
> maven-javadoc-plugin
> sdks/java/io/mongodb/pom.xml:
> maven-javadoc-plugin
> Arguably, this module should have it too, then.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-484) Datastore Source should support Dynamic Splitting

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-484.
---
Resolution: Fixed

> Datastore Source should support Dynamic Splitting
> -
>
> Key: BEAM-484
> URL: https://issues.apache.org/jira/browse/BEAM-484
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
> Fix For: 0.2.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-854) ReifyTimestampAndWindows explodes windows

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-854:
---

(reopening temporarily to add the components tag to the issue)

> ReifyTimestampAndWindows explodes windows
> -
>
> Key: BEAM-854
> URL: https://issues.apache.org/jira/browse/BEAM-854
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
> Fix For: 0.4.0-incubating
>
>
> Commit 
> https://github.com/apache/incubator-beam/commit/b2350417f73ae6c34f849ff0e93d5bd93df3088d
>  changed the fn from accessing the collection of windows to accessing the 
> windows one-by-one, which causes this DoFn to explode them, which (I suppose) 
> hurts performance, since this Fn is quite important: it's a part of 
> GroupByKey in the Spark runner and Direct runner, part of Splittable DoFn, 
> part of GBKViaGBKO, and part of GatherAllPanes.
> I can see a couple of ways to fix this:
> - Make this be a primitive transform, with runner-specific support to keep 
> the windows compressed
> - Allow any DoFn to access the compressed representation explicitly using a 
> Collection parameter
> I'm personally in favor of the latter because it automatically benefits all 
> runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-854) ReifyTimestampAndWindows explodes windows

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-854.
---
Resolution: Fixed

> ReifyTimestampAndWindows explodes windows
> -
>
> Key: BEAM-854
> URL: https://issues.apache.org/jira/browse/BEAM-854
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
> Fix For: 0.4.0-incubating
>
>
> Commit 
> https://github.com/apache/incubator-beam/commit/b2350417f73ae6c34f849ff0e93d5bd93df3088d
>  changed the fn from accessing the collection of windows to accessing the 
> windows one-by-one, which causes this DoFn to explode them, which (I suppose) 
> hurts performance, since this Fn is quite important: it's a part of 
> GroupByKey in the Spark runner and Direct runner, part of Splittable DoFn, 
> part of GBKViaGBKO, and part of GatherAllPanes.
> I can see a couple of ways to fix this:
> - Make this be a primitive transform, with runner-specific support to keep 
> the windows compressed
> - Allow any DoFn to access the compressed representation explicitly using a 
> Collection parameter
> I'm personally in favor of the latter because it automatically benefits all 
> runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-484) Datastore Source should support Dynamic Splitting

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-484:
--
Component/s: sdk-java-gcp

> Datastore Source should support Dynamic Splitting
> -
>
> Key: BEAM-484
> URL: https://issues.apache.org/jira/browse/BEAM-484
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
> Fix For: 0.2.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-729) Does sdks/java/io/google-cloud-platform/pom.xml need javadoc plugin?

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-729:
---

(reopening temporarily to add the components tag to the issue)

> Does sdks/java/io/google-cloud-platform/pom.xml need javadoc plugin?
> 
>
> Key: BEAM-729
> URL: https://issues.apache.org/jira/browse/BEAM-729
> Project: Beam
>  Issue Type: Bug
>Reporter: Eugene Kirpichov
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> I would like to generate javadoc for that module by running the respective 
> plugin in IntelliJ's view of that maven module (not the root module).
> As-is, this maven module doesn't have a javadoc plugin (at least it doesn't 
> show up in IntelliJ). However, most other modules have it - e.g.:
> $ grep "maven-javadoc" **/pom.xml
>   
>   examples/java8/pom.xml: 
>maven-javadoc-plugin
> examples/java/pom.xml:maven-javadoc-plugin
> pom.xml:  maven-javadoc-plugin
> pom.xml:  maven-javadoc-plugin
> runners/core-java/pom.xml:
> maven-javadoc-plugin
> runners/direct-java/pom.xml:
> maven-javadoc-plugin
> runners/flink/examples/pom.xml:
> maven-javadoc-plugin
> runners/flink/runner/pom.xml:
> maven-javadoc-plugin
> runners/google-cloud-dataflow-java/pom.xml:
> maven-javadoc-plugin
> runners/spark/pom.xml:maven-javadoc-plugin
> sdks/java/core/pom.xml:  maven-javadoc-plugin
> sdks/java/core/pom.xml:maven-javadoc-plugin
> sdks/java/io/kafka/pom.xml:
> maven-javadoc-plugin
> sdks/java/io/kinesis/pom.xml:
> maven-javadoc-plugin
> sdks/java/io/mongodb/pom.xml:
> maven-javadoc-plugin
> Arguably, this module should have it too, then.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-493) All Runners Run WordCount in Precommit

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-493.
---
Resolution: Fixed

> All Runners Run WordCount in Precommit
> --
>
> Key: BEAM-493
> URL: https://issues.apache.org/jira/browse/BEAM-493
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Jason Kuster
>Assignee: Jason Kuster
> Fix For: 0.3.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-493) All Runners Run WordCount in Precommit

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-493:
--
Component/s: project-management

> All Runners Run WordCount in Precommit
> --
>
> Key: BEAM-493
> URL: https://issues.apache.org/jira/browse/BEAM-493
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Jason Kuster
>Assignee: Jason Kuster
> Fix For: 0.3.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-492) Spark runner should call Pipeline.run() instead of SparkRunner.run()

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-492.
---
Resolution: Fixed

> Spark runner should call Pipeline.run() instead of SparkRunner.run()
> 
>
> Key: BEAM-492
> URL: https://issues.apache.org/jira/browse/BEAM-492
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.1.0-incubating
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.3.0-incubating
>
>
> Currently, in some places (streaming examples, etc.), the Pipeline executes 
> by calling SparkRunner.run() instead of calling Pipeline.run() which is kind 
> of "backwards" to the Beam API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-492) Spark runner should call Pipeline.run() instead of SparkRunner.run()

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-492:
--
Component/s: runner-spark

> Spark runner should call Pipeline.run() instead of SparkRunner.run()
> 
>
> Key: BEAM-492
> URL: https://issues.apache.org/jira/browse/BEAM-492
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.1.0-incubating
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.3.0-incubating
>
>
> Currently, in some places (streaming examples, etc.), the Pipeline executes 
> by calling SparkRunner.run() instead of calling Pipeline.run() which is kind 
> of "backwards" to the Beam API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-492) Spark runner should call Pipeline.run() instead of SparkRunner.run()

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-492:
---

> Spark runner should call Pipeline.run() instead of SparkRunner.run()
> 
>
> Key: BEAM-492
> URL: https://issues.apache.org/jira/browse/BEAM-492
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.1.0-incubating
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.3.0-incubating
>
>
> Currently, in some places (streaming examples, etc.), the Pipeline executes 
> by calling SparkRunner.run() instead of calling Pipeline.run() which is kind 
> of "backwards" to the Beam API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-493) All Runners Run WordCount in Precommit

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reopened BEAM-493:
---

> All Runners Run WordCount in Precommit
> --
>
> Key: BEAM-493
> URL: https://issues.apache.org/jira/browse/BEAM-493
> Project: Beam
>  Issue Type: Improvement
>Reporter: Jason Kuster
>Assignee: Jason Kuster
> Fix For: 0.3.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1194) DataflowRunner should canonicalize gcpTempLocation/stagingLocation/etc formats.

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1194:
--
Description: 
Cloud Dataflow has a minor history of small bugs related to various code paths 
expecting there to be or not be a trailing forward-slash in these location 
fields. The way that Beam's integration tests are set up, we are likely to only 
have one of these two cases tested (there is a single set of integration test 
pipeline options).

We should add a dedicated DataflowRunner integration test to handle this case.

Actually, we should probably canonicalize the URLs so that we only ever produce 
one version.

  was:
Cloud Dataflow has a minor history of small bugs related to various code paths 
expecting there to be or not be a trailing forward-slash in these location 
fields. The way that Beam's integration tests are set up, we are likely to only 
have one of these two cases tested (there is a single set of integration test 
pipeline options).

We should add a dedicated DataflowRunner integration test to handle this case.


> DataflowRunner should canonicalize gcpTempLocation/stagingLocation/etc 
> formats.
> ---
>
> Key: BEAM-1194
> URL: https://issues.apache.org/jira/browse/BEAM-1194
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>
> Cloud Dataflow has a minor history of small bugs related to various code 
> paths expecting there to be or not be a trailing forward-slash in these 
> location fields. The way that Beam's integration tests are set up, we are 
> likely to only have one of these two cases tested (there is a single set of 
> integration test pipeline options).
> We should add a dedicated DataflowRunner integration test to handle this case.
> Actually, we should probably canonicalize the URLs so that we only ever 
> produce one version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1194) DataflowRunner should canonicalize gcpTempLocation/stagingLocation/etc formats.

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1194:
--
Summary: DataflowRunner should canonicalize 
gcpTempLocation/stagingLocation/etc formats.  (was: DataflowRunner should test 
a variety of valid tempLocation/stagingLocation/etc formats.)

> DataflowRunner should canonicalize gcpTempLocation/stagingLocation/etc 
> formats.
> ---
>
> Key: BEAM-1194
> URL: https://issues.apache.org/jira/browse/BEAM-1194
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>
> Cloud Dataflow has a minor history of small bugs related to various code 
> paths expecting there to be or not be a trailing forward-slash in these 
> location fields. The way that Beam's integration tests are set up, we are 
> likely to only have one of these two cases tested (there is a single set of 
> integration test pipeline options).
> We should add a dedicated DataflowRunner integration test to handle this case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1194) DataflowRunner should test a variety of valid tempLocation/stagingLocation/etc formats.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784131#comment-15784131
 ] 

ASF GitHub Bot commented on BEAM-1194:
--

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/1671


> DataflowRunner should test a variety of valid 
> tempLocation/stagingLocation/etc formats.
> ---
>
> Key: BEAM-1194
> URL: https://issues.apache.org/jira/browse/BEAM-1194
> Project: Beam
>  Issue Type: Test
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>
> Cloud Dataflow has a minor history of small bugs related to various code 
> paths expecting there to be or not be a trailing forward-slash in these 
> location fields. The way that Beam's integration tests are set up, we are 
> likely to only have one of these two cases tested (there is a single set of 
> integration test pipeline options).
> We should add a dedicated DataflowRunner integration test to handle this case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1671: [BEAM-1194] Add DataflowLocationIT

[dhalperi]

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/1671


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---