Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2890

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3814

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2793) Include all versions in top-level pom

[Jason Kuster (JIRA)]

Jason Kuster created BEAM-2793:
--

 Summary: Include all versions in top-level pom
 Key: BEAM-2793
 URL: https://issues.apache.org/jira/browse/BEAM-2793
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Jason Kuster
Assignee: Davor Bonaci


Top-level pom is missing at least

kafka version
cloudlogging version



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2889

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2724) MSEC counters should support Structured Names in Dataflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137640#comment-16137640
 ] 

ASF GitHub Bot commented on BEAM-2724:
--

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/3747

[BEAM-2724] Support for IOTargetName like in updated IO dataflow metrics



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam add-iotnm

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3747.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3747


commit 7beb2abf8b9ed33882961ef9e993a3f53a9363df
Author: Pablo 
Date:   2017-08-22T23:53:32Z

Adding IOTargetName and unittests for CounterName




> MSEC counters should support Structured Names in Dataflow
> -
>
> Key: BEAM-2724
> URL: https://issues.apache.org/jira/browse/BEAM-2724
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3747: [BEAM-2724] Support for IOTargetName like in update...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/3747

[BEAM-2724] Support for IOTargetName like in updated IO dataflow metrics



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam add-iotnm

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3747.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3747


commit 7beb2abf8b9ed33882961ef9e993a3f53a9363df
Author: Pablo 
Date:   2017-08-22T23:53:32Z

Adding IOTargetName and unittests for CounterName




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

svn commit: r21270 - /dev/beam/2.1.0/

[altay]

Author: altay
Date: Tue Aug 22 23:58:55 2017
New Revision: 21270

Log:
Remove all release candidates for Apache Beam, version 2.1.0 following the 
release of RC3.

Removed:
dev/beam/2.1.0/

svn commit: r21269 - /release/beam/2.1.0/

[altay]

Author: altay
Date: Tue Aug 22 23:55:37 2017
New Revision: 21269

Log:
Release Apache Beam, version 2.1.0

Added:
release/beam/2.1.0/
release/beam/2.1.0/apache-beam-2.1.0-python.zip   (with props)
release/beam/2.1.0/apache-beam-2.1.0-python.zip.asc
release/beam/2.1.0/apache-beam-2.1.0-python.zip.md5
release/beam/2.1.0/apache-beam-2.1.0-python.zip.sha1
release/beam/2.1.0/apache-beam-2.1.0-source-release.zip   (with props)
release/beam/2.1.0/apache-beam-2.1.0-source-release.zip.asc
release/beam/2.1.0/apache-beam-2.1.0-source-release.zip.md5
release/beam/2.1.0/apache-beam-2.1.0-source-release.zip.sha1

Added: release/beam/2.1.0/apache-beam-2.1.0-python.zip
==
Binary file - no diff available.

Propchange: release/beam/2.1.0/apache-beam-2.1.0-python.zip
--
svn:mime-type = application/octet-stream

Added: release/beam/2.1.0/apache-beam-2.1.0-python.zip.asc
==
--- release/beam/2.1.0/apache-beam-2.1.0-python.zip.asc (added)
+++ release/beam/2.1.0/apache-beam-2.1.0-python.zip.asc Tue Aug 22 23:55:37 2017
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQIcBAABAgAGBQJZipYNAAoJEL/y7kLIKC52GroP/jtquMaM0d6rlJndGow1dg9y
+LzuKKjU1O8ftBhbgTiMs/7XWT4LDnN63Ql2LtDkISDb8fPNtOTlBO972ztwxJ1aq
+qzxKHvvh0UwZ/chFYkdNVUeFrZoJaCrIwd9F1rSNADhusoPlkPXPwGWLp4IUVoEl
+s7+XLcfux6zZ5YIaja8prohq9gQKx9Ob2ZJSp0YGorxEUbHu/rVNg0Dt67MEXuW8
+OblHSlY0vmU6DSqw4ck1kVMb4nu0KeN5zdSwRj9BWq+VFDO/vOzEqHuTmVUo+A/r
+dslzC6MwDTH5MN+b3cY+eHbmT5Vd08/yG0ZuLxlYHbYZ2lptLE9lt+EpNwBPnscp
+cXc7OFp2d0F4JlIG+pRO1tmRjbjood5xRPgS+PRTfsuY6SrtEd9Opt6X+P9wcitK
+EzG7gfss/XbfB0sU6yT+jLREhdtP/+uDZcLN91T8D2RPzwgKPiinjJT3b3PXCmuq
+MSjTJ557Xh/jD9/z17la0vssiGtUtbuBvIkp+GozR74eHBpbXVyRZhqg8luqj+uO
+BgeQJ/HS1DI6tr733GEJnoY0bAzBkMbnGRtjAyS49TBPc0odLM/jX6PJrY54rjg8
+8iPxeNfOM7+mxzjrLAlA2LxjYcxzPVbYikeRrCD6g3QfFG22KcKSlXNdkRp6X4FR
+fMOcP5JAD5JHbdy/mRQ+
+=P4qe
+-END PGP SIGNATURE-

Added: release/beam/2.1.0/apache-beam-2.1.0-python.zip.md5
==
--- release/beam/2.1.0/apache-beam-2.1.0-python.zip.md5 (added)
+++ release/beam/2.1.0/apache-beam-2.1.0-python.zip.md5 Tue Aug 22 23:55:37 2017
@@ -0,0 +1 @@
+0122b39da8f5b73334d75298ff8e37af  apache-beam-2.1.0-python.zip

Added: release/beam/2.1.0/apache-beam-2.1.0-python.zip.sha1
==
--- release/beam/2.1.0/apache-beam-2.1.0-python.zip.sha1 (added)
+++ release/beam/2.1.0/apache-beam-2.1.0-python.zip.sha1 Tue Aug 22 23:55:37 
2017
@@ -0,0 +1 @@
+cc5da6cecc538540a5f08098a03acc72a0c322c1  apache-beam-2.1.0-python.zip

Added: release/beam/2.1.0/apache-beam-2.1.0-source-release.zip
==
Binary file - no diff available.

Propchange: release/beam/2.1.0/apache-beam-2.1.0-source-release.zip
--
svn:mime-type = application/octet-stream

Added: release/beam/2.1.0/apache-beam-2.1.0-source-release.zip.asc
==
--- release/beam/2.1.0/apache-beam-2.1.0-source-release.zip.asc (added)
+++ release/beam/2.1.0/apache-beam-2.1.0-source-release.zip.asc Tue Aug 22 
23:55:37 2017
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQIcBAABAgAGBQJZifpwAAoJEL/y7kLIKC52WVIQAKtW68nr0RYHs+JkKBrOiBuf
+l2lzeamYm3kkf3TPS2CTFdWbvjLkyrdl/92UbqhRzeCrQy4hps++VhlZpEpmalM5
+oUTceupFYRD1Hqf7zIsbw0RR61EamGalSDNKdIdRld557KepjiwlQy4ves50rlC2
+7D9l9ZF+QOGdX+OKUQBqAYJPmRcJB2IKZux3hmhN0bdhE8CHrbuS92KkPsBVXFZB
+mi1U/L43znwxHSsK7YEUdqmdJfSs1Kyfv+zZQNbcIf/tmoJt+pjmJkkIpEkcOcvf
+BrYnxYXv++UoV800tRp+VebT02Y9llXC4xB8zXIgl57CZ9RhTA9XoQmxG7FyBP9/
+C0OAQ/pwmFi41XIvl6lNmFBJnK63pT7ajMwYu+umUfzJJ9BCm4PstsAC2hA5ESUb
+ttuS1MMeyHkhxP3G/3WJFk4LoKeNY12J+DV4cyr6FNCtoH4FfIhLp990LM11IcpT
+kU7NuXqMttsqqXFcMYTSx0OECSuGLDhNGCUrCsZWlNE864Dj+Xa0sa/PjijFm9i6
+Zden2Zaqa2NCCF56k1r81nB1lAWMym7NVqIsf8HiCixR+HzHomLH+DaAX6tDanAj
+4DYfiX/UCySrVZR64vO807Gwgen+eR+eAUw/5Z8IcQgTx3s057UdfjeZ3C22a2PF
++X1UGJlSpYo2LnA0Com+
+=4U1w
+-END PGP SIGNATURE-

Added: release/beam/2.1.0/apache-beam-2.1.0-source-release.zip.md5
==
--- release/beam/2.1.0/apache-beam-2.1.0-source-release.zip.md5 (added)
+++ release/beam/2.1.0/apache-beam-2.1.0-source-release.zip.md5 Tue Aug 22 
23:55:37 2017
@@ -0,0 +1 @@
+f069e7a692b8a40c4aa05cb31b2347fa  apache-beam-2.1.0-source-release.zip

Added: release/beam/2.1.0/apache-beam-2.1.0-source-release.zip.sha1
==
--- release/beam/2.1.0/apache-beam-2.1.0-source-release.z

[beam] Git Push Summary

[altay]

Repository: beam
Updated Tags:  refs/tags/v2.1.0 [created] 58e820e54

[beam-site] 01/03: Fix quote

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 74483a00bdf70ae5db68c6f5f0991232b79031b2
Author: Rafael Fernandez 
AuthorDate: Fri Aug 18 11:32:37 2017 -0700

Fix quote

Typo
---
 src/index.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/index.md b/src/index.md
index 80c661c..25e99e3 100644
--- a/src/index.md
+++ b/src/index.md
@@ -27,7 +27,7 @@ pillars:
 cards:
 - quote: "A framework that delivers the flexibility and advanced functionality 
our customers need."
   name: –Talend
-- quote: "Apache Beam [has powerful semantics that solve real-world challenges 
of stream processing."
+- quote: "Apache Beam has powerful semantics that solve real-world challenges 
of stream processing."
   name: –PayPal
 - quote: "Apache Beam represents a principled approach for analyzing data 
streams."
   name: –data Artisans

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3813

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2792) Populate All Runner API Components from the Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137259#comment-16137259
 ] 

ASF GitHub Bot commented on BEAM-2792:
--

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3746

[BEAM-2792] Translate basic coders through the Runner API

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam runner-api-coders

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3746.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3746


commit 7999eecc881bce1e95622ea2bc2a12cd050f27ff
Author: Robert Bradshaw 
Date:   2017-08-22T17:01:40Z

Add URN registration mechanism for coders.

commit 313de698594054b030f49a010586ecfca222c435
Author: Robert Bradshaw 
Date:   2017-08-22T17:54:21Z

Runner API encoding of common coders.




> Populate All Runner API Components from the Python SDK
> --
>
> Key: BEAM-2792
> URL: https://issues.apache.org/jira/browse/BEAM-2792
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3746: [BEAM-2792] Translate basic coders through the Runn...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3746

[BEAM-2792] Translate basic coders through the Runner API

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam runner-api-coders

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3746.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3746


commit 7999eecc881bce1e95622ea2bc2a12cd050f27ff
Author: Robert Bradshaw 
Date:   2017-08-22T17:01:40Z

Add URN registration mechanism for coders.

commit 313de698594054b030f49a010586ecfca222c435
Author: Robert Bradshaw 
Date:   2017-08-22T17:54:21Z

Runner API encoding of common coders.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2792) Populate All Runner API Components from the Python SDK

[Robert Bradshaw (JIRA)]

Robert Bradshaw created BEAM-2792:
-

 Summary: Populate All Runner API Components from the Python SDK
 Key: BEAM-2792
 URL: https://issues.apache.org/jira/browse/BEAM-2792
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Robert Bradshaw
Assignee: Robert Bradshaw






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4619

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2791) DirectRunner shuts down when there are pending event time timers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137174#comment-16137174
 ] 

ASF GitHub Bot commented on BEAM-2791:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3745

[BEAM-2791] Add a test for an event time timer loop in ParDo

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh adding this test to indicate the issue and link with JIRA, and 
will try to find time to fix it later.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam timer-loop

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3745.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3745


commit 2a4d086c9e41b1bfcc495a166dac43e68bb82dbe
Author: Kenneth Knowles 
Date:   2017-08-22T18:34:59Z

Add a test for an event time timer loop in ParDo




> DirectRunner shuts down when there are pending event time timers
> 
>
> Key: BEAM-2791
> URL: https://issues.apache.org/jira/browse/BEAM-2791
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> When there are pending event time timers for a window by definition they are 
> within the window so it should not be possible to GC the window nor to shut 
> down the pipeline.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3745: [BEAM-2791] Add a test for an event time timer loop...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3745

[BEAM-2791] Add a test for an event time timer loop in ParDo

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh adding this test to indicate the issue and link with JIRA, and 
will try to find time to fix it later.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam timer-loop

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3745.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3745


commit 2a4d086c9e41b1bfcc495a166dac43e68bb82dbe
Author: Kenneth Knowles 
Date:   2017-08-22T18:34:59Z

Add a test for an event time timer loop in ParDo




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2791) DirectRunner shuts down when there are pending event time timers

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2791:
-

 Summary: DirectRunner shuts down when there are pending event time 
timers
 Key: BEAM-2791
 URL: https://issues.apache.org/jira/browse/BEAM-2791
 Project: Beam
  Issue Type: Bug
  Components: runner-direct
Reporter: Kenneth Knowles
Assignee: Thomas Groh


When there are pending event time timers for a window by definition they are 
within the window so it should not be possible to GC the window nor to shut 
down the pipeline.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2888

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2785) Allow termination of Python DirectRunner execution with Ctrl-C

[Charles Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charles Chen resolved BEAM-2785.

Resolution: Fixed

> Allow termination of Python DirectRunner execution with Ctrl-C
> --
>
> Key: BEAM-2785
> URL: https://issues.apache.org/jira/browse/BEAM-2785
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.0.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Minor
> Fix For: 2.2.0
>
>
> We should allow termination of Python DirectRunner execution with a Ctrl-C 
> KeyboardInterrupt.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #2887

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3812

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4618

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3811

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2790) Error while reading from Amazon S3 via Hadoop File System

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-2790:
---
Description: 
If you try to use hadoop-aws with Beam to read from AWS S3 it breaks because 
S3AInputStream (the implementation of Hadoop's FSDataInputStream) is not 
ByteBufferReadable. 

{code}
Exception in thread "main" java.lang.UnsupportedOperationException: Byte-buffer 
read unsupported by input stream
at 
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:146)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem$HadoopSeekableByteChannel.read(HadoopFileSystem.java:192)
at 
org.apache.beam.sdk.io.TextSource$TextBasedReader.tryToEnsureNumberOfBytesInBuffer(TextSource.java:232)
at 
org.apache.beam.sdk.io.TextSource$TextBasedReader.findSeparatorBounds(TextSource.java:166)
at 
org.apache.beam.sdk.io.TextSource$TextBasedReader.readNextRecord(TextSource.java:198)
at 
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:481)
at 
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:476)
at 
org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:261)
{code}

  was:
If you try to use hadoop-aws with Beam to read from AWS S3 it breaks because 
S3AInputStream (the implementation of Hadoop's FSDataInputStream) is not 
ByteBufferReadable. 

[code]
Exception in thread "main" java.lang.UnsupportedOperationException: Byte-buffer 
read unsupported by input stream
at 
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:146)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem$HadoopSeekableByteChannel.read(HadoopFileSystem.java:192)
at 
org.apache.beam.sdk.io.TextSource$TextBasedReader.tryToEnsureNumberOfBytesInBuffer(TextSource.java:232)
at 
org.apache.beam.sdk.io.TextSource$TextBasedReader.findSeparatorBounds(TextSource.java:166)
at 
org.apache.beam.sdk.io.TextSource$TextBasedReader.readNextRecord(TextSource.java:198)
at 
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:481)
at 
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:476)
at 
org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:261)
[code]


> Error while reading from Amazon S3 via Hadoop File System
> -
>
> Key: BEAM-2790
> URL: https://issues.apache.org/jira/browse/BEAM-2790
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> If you try to use hadoop-aws with Beam to read from AWS S3 it breaks because 
> S3AInputStream (the implementation of Hadoop's FSDataInputStream) is not 
> ByteBufferReadable. 
> {code}
> Exception in thread "main" java.lang.UnsupportedOperationException: 
> Byte-buffer read unsupported by input stream
>   at 
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:146)
>   at 
> org.apache.beam.sdk.io.hdfs.HadoopFileSystem$HadoopSeekableByteChannel.read(HadoopFileSystem.java:192)
>   at 
> org.apache.beam.sdk.io.TextSource$TextBasedReader.tryToEnsureNumberOfBytesInBuffer(TextSource.java:232)
>   at 
> org.apache.beam.sdk.io.TextSource$TextBasedReader.findSeparatorBounds(TextSource.java:166)
>   at 
> org.apache.beam.sdk.io.TextSource$TextBasedReader.readNextRecord(TextSource.java:198)
>   at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:481)
>   at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:476)
>   at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:261)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136759#comment-16136759
 ] 

Ismaël Mejía commented on BEAM-2500:


I created BEAM-2790 to move the discussion on the issue with the read of S3 
using HadoopFileSystem into its own thread. Remember this JIRA was created for 
the 'native' (non hadoop-based) implementation of S3.
I basically applied a minimal version of the patch that [~igosuki] contributed 
(thanks!), if any further issue with the Hadoop based implementation please 
report in the new JIRA or create a new issue.

> Add support for S3 as a Apache Beam FileSystem
> --
>
> Key: BEAM-2500
> URL: https://issues.apache.org/jira/browse/BEAM-2500
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Priority: Minor
> Attachments: hadoop_fs_patch.patch
>
>
> Note that this is for providing direct integration with S3 as an Apache Beam 
> FileSystem.
> There is already support for using the Hadoop S3 connector by depending on 
> the Hadoop File System module[1], configuring HadoopFileSystemOptions[2] with 
> a S3 configuration[3].
> 1: https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system
> 2: 
> https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java#L53
> 3: https://wiki.apache.org/hadoop/AmazonS3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2790) Error while reading from Amazon S3 via Hadoop File System

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136748#comment-16136748
 ] 

ASF GitHub Bot commented on BEAM-2790:
--

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/3744

[BEAM-2790] Use byte[] instead of ByteBuffer to read from Hadoop FS

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-2790-fix-s3-read

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3744.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3744


commit 76f2d91bc9f612d4e98fe7d8099dffd36c47ff85
Author: Ismaël Mejía 
Date:   2017-08-22T12:50:12Z

[BEAM-2790] Use byte[] instead of ByteBuffer to read from Hadoop FS




> Error while reading from Amazon S3 via Hadoop File System
> -
>
> Key: BEAM-2790
> URL: https://issues.apache.org/jira/browse/BEAM-2790
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>
> If you try to use hadoop-aws with Beam to read from AWS S3 it breaks because 
> S3AInputStream (the implementation of Hadoop's FSDataInputStream) is not 
> ByteBufferReadable. 
> [code]
> Exception in thread "main" java.lang.UnsupportedOperationException: 
> Byte-buffer read unsupported by input stream
>   at 
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:146)
>   at 
> org.apache.beam.sdk.io.hdfs.HadoopFileSystem$HadoopSeekableByteChannel.read(HadoopFileSystem.java:192)
>   at 
> org.apache.beam.sdk.io.TextSource$TextBasedReader.tryToEnsureNumberOfBytesInBuffer(TextSource.java:232)
>   at 
> org.apache.beam.sdk.io.TextSource$TextBasedReader.findSeparatorBounds(TextSource.java:166)
>   at 
> org.apache.beam.sdk.io.TextSource$TextBasedReader.readNextRecord(TextSource.java:198)
>   at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:481)
>   at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:476)
>   at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:261)
> [code]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3744: [BEAM-2790] Use byte[] instead of ByteBuffer to rea...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/3744

[BEAM-2790] Use byte[] instead of ByteBuffer to read from Hadoop FS

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-2790-fix-s3-read

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3744.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3744


commit 76f2d91bc9f612d4e98fe7d8099dffd36c47ff85
Author: Ismaël Mejía 
Date:   2017-08-22T12:50:12Z

[BEAM-2790] Use byte[] instead of ByteBuffer to read from Hadoop FS




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2790) Error while reading from Amazon S3 via Hadoop File System

[JIRA]

Ismaël Mejía created BEAM-2790:
--

 Summary: Error while reading from Amazon S3 via Hadoop File System
 Key: BEAM-2790
 URL: https://issues.apache.org/jira/browse/BEAM-2790
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Affects Versions: 2.0.0, 2.1.0
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía


If you try to use hadoop-aws with Beam to read from AWS S3 it breaks because 
S3AInputStream (the implementation of Hadoop's FSDataInputStream) is not 
ByteBufferReadable. 

[code]
Exception in thread "main" java.lang.UnsupportedOperationException: Byte-buffer 
read unsupported by input stream
at 
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:146)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem$HadoopSeekableByteChannel.read(HadoopFileSystem.java:192)
at 
org.apache.beam.sdk.io.TextSource$TextBasedReader.tryToEnsureNumberOfBytesInBuffer(TextSource.java:232)
at 
org.apache.beam.sdk.io.TextSource$TextBasedReader.findSeparatorBounds(TextSource.java:166)
at 
org.apache.beam.sdk.io.TextSource$TextBasedReader.readNextRecord(TextSource.java:198)
at 
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:481)
at 
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:476)
at 
org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:261)
[code]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2886

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2789) Watermark can become unavailable for executors while it's updated with new values

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin updated BEAM-2789:
-
Summary: Watermark can become unavailable for executors while it's updated 
with new values  (was: Watermarks can become unavailable for executors while 
it's updated with new values)

> Watermark can become unavailable for executors while it's updated with new 
> values
> -
>
> Key: BEAM-2789
> URL: https://issues.apache.org/jira/browse/BEAM-2789
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Stas Levin
>Assignee: Amit Sela
>
> The watermark is updated by the driver like so:
> {code:java}
> blockManager.removeBlock(WATERMARKS_BLOCK_ID, true);
> blockManager.putSingle(WATERMARKS_BLOCK_ID, newValues, 
> StorageLevel.MEMORY_ONLY(), 
> true);
> {code}
> However, these operations are neither synchronous nor atomic, so if an 
> executor requests the watermark values before they are actually put but after 
> they have been removed, it may get a {{null}} as a response, which will make 
> it default to negative infinity as the watermark. This can result in an 
> erroneous results.
> To overcome this in tests, a workaround which assumes a single JVM setting is 
> used. In such a setting the watermark values are stored in a static member, 
> accessible by both the driver and the executors, bypassing the 
> {{BlockManager#putSingle(...)}} {{BlockManager#removeBlock(...)}} APIs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2789) Watermarks can become unavailable for executors while it's updated with new values

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin updated BEAM-2789:
-
Summary: Watermarks can become unavailable for executors while it's updated 
with new values  (was: Watermark can become unavailable for executors while 
it's updated with new values)

> Watermarks can become unavailable for executors while it's updated with new 
> values
> --
>
> Key: BEAM-2789
> URL: https://issues.apache.org/jira/browse/BEAM-2789
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Stas Levin
>Assignee: Amit Sela
>
> The watermark is updated by the driver like so:
> {code:java}
> blockManager.removeBlock(WATERMARKS_BLOCK_ID, true);
> blockManager.putSingle(WATERMARKS_BLOCK_ID, newValues, 
> StorageLevel.MEMORY_ONLY(), 
> true);
> {code}
> However, these operations are neither synchronous nor atomic, so if an 
> executor requests the watermark values before they are actually put but after 
> they have been removed, it may get a {{null}} as a response, which will make 
> it default to negative infinity as the watermark. This can result in an 
> erroneous results.
> To overcome this in tests, a workaround which assumes a single JVM setting is 
> used. In such a setting the watermark values are stored in a static member, 
> accessible by both the driver and the executors, bypassing the 
> {{BlockManager#putSingle(...)}} {{BlockManager#removeBlock(...)}} APIs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2789) Watermark can become unavailable for executors while it's updated with new values

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin updated BEAM-2789:
-
Description: 
The watermark is updated by the driver like so:

{code:java}
blockManager.removeBlock(WATERMARKS_BLOCK_ID, true);
blockManager.putSingle(WATERMARKS_BLOCK_ID, newValues, 
StorageLevel.MEMORY_ONLY(), 
true);
{code}

However, these operations are neither synchronous nor atomic, so if an executor 
requests the watermark values before they are actually put but after they have 
been removed, it may get a {{null}} as a response, which will make it default 
to negative infinity as the watermark. This can result in an erroneous results.

To overcome this in tests, a workaround which assumes a single JVM setting is 
used. In such a setting the watermark values are stored in a static member, 
accessible by both the driver and the executors, bypassing the 
{{BlockManager#putSingle(...)}} {{BlockManager#removeBlock(...)}} APIs.

  was:
The watermark is updated by the driver like so:

{code:java}
blockManager.removeBlock(WATERMARKS_BLOCK_ID, true);
blockManager.putSingle(WATERMARKS_BLOCK_ID, newValues, 
StorageLevel.MEMORY_ONLY(), 
true);
{code}

However, these operations are neither synchronous nor atomic, so if an executor 
requests the watermark values before they are actually put but after they have 
been removed, it may get a {{null}} as a response, which will make it default 
to negative infinity as the watermark. This can result in an erroneous results.


> Watermark can become unavailable for executors while it's updated with new 
> values
> -
>
> Key: BEAM-2789
> URL: https://issues.apache.org/jira/browse/BEAM-2789
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Stas Levin
>Assignee: Amit Sela
>
> The watermark is updated by the driver like so:
> {code:java}
> blockManager.removeBlock(WATERMARKS_BLOCK_ID, true);
> blockManager.putSingle(WATERMARKS_BLOCK_ID, newValues, 
> StorageLevel.MEMORY_ONLY(), 
> true);
> {code}
> However, these operations are neither synchronous nor atomic, so if an 
> executor requests the watermark values before they are actually put but after 
> they have been removed, it may get a {{null}} as a response, which will make 
> it default to negative infinity as the watermark. This can result in an 
> erroneous results.
> To overcome this in tests, a workaround which assumes a single JVM setting is 
> used. In such a setting the watermark values are stored in a static member, 
> accessible by both the driver and the executors, bypassing the 
> {{BlockManager#putSingle(...)}} {{BlockManager#removeBlock(...)}} APIs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2789) Watermark can become unavailable for executors while it's updated with new values

[Stas Levin (JIRA)]

Stas Levin created BEAM-2789:


 Summary: Watermark can become unavailable for executors while it's 
updated with new values
 Key: BEAM-2789
 URL: https://issues.apache.org/jira/browse/BEAM-2789
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Affects Versions: 2.0.0, 2.1.0
Reporter: Stas Levin
Assignee: Amit Sela


The watermark is updated by the driver like so:

{code:java}
blockManager.removeBlock(WATERMARKS_BLOCK_ID, true);
blockManager.putSingle(WATERMARKS_BLOCK_ID, newValues, 
StorageLevel.MEMORY_ONLY(), 
true);
{code}

However, these operations are neither synchronous nor atomic, so if an executor 
requests the watermark values before they are actually put but after they have 
been removed, it may get a {{null}} as a response, which will make it default 
to negative infinity as the watermark. This can result in an erroneous results.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2885

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1534) Create a dockerized developer environment for Beam

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136676#comment-16136676
 ] 

ASF GitHub Bot commented on BEAM-1534:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3651


> Create a dockerized developer environment for Beam
> --
>
> Key: BEAM-1534
> URL: https://issues.apache.org/jira/browse/BEAM-1534
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-java
>Reporter: Nitin Lamba
>Assignee: Ekrem Aksoy
>
> This will help create a repeatable developer environment setup.
> Other Apache projects can be used as a reference:
> https://github.com/apache/ambari/tree/trunk/dev-support/docker
> https://github.com/apache/geode/blob/develop/docker/Dockerfile



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: [BEAM-1534] Create a dockerized developer environment for Beam

[iemejia]

Repository: beam
Updated Branches:
  refs/heads/master 62294e64c -> 9b175ccc2


[BEAM-1534] Create a dockerized developer environment for Beam


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/68091296
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/68091296
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/68091296

Branch: refs/heads/master
Commit: 68091296f12ad51523c816954ec593e720733c37
Parents: 62294e6
Author: Ismaël Mejía 
Authored: Tue Jul 25 11:57:09 2017 +0200
Committer: Ismaël Mejía 
Committed: Tue Aug 22 10:21:39 2017 +0200

--
 .gitignore  |  3 ++
 .../resources/docker/file/openjdk7/Dockerfile   | 49 ++
 .../docker/file/openjdk7/docker-entrypoint.sh   | 24 +
 .../resources/docker/file/openjdk8/Dockerfile   | 49 ++
 .../docker/file/openjdk8/docker-entrypoint.sh   | 24 +
 .../resources/docker/git/openjdk8/Dockerfile| 53 
 .../docker/git/openjdk8/docker-entrypoint.sh| 22 
 .../resources/docker/release/python2/Dockerfile | 21 
 8 files changed, 245 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/68091296/.gitignore
--
diff --git a/.gitignore b/.gitignore
index 36c5cc8..f996dfd 100644
--- a/.gitignore
+++ b/.gitignore
@@ -42,6 +42,9 @@ sdks/python/apache_beam/portability/api/*pb2*.*
 .apt_generated/
 .settings/
 
+# Ignore Visual Studio Code files.
+.vscode/
+
 # The build process generates the dependency-reduced POM, but it shouldn't be
 # committed.
 dependency-reduced-pom.xml

http://git-wip-us.apache.org/repos/asf/beam/blob/68091296/sdks/java/build-tools/src/main/resources/docker/file/openjdk7/Dockerfile
--
diff --git 
a/sdks/java/build-tools/src/main/resources/docker/file/openjdk7/Dockerfile 
b/sdks/java/build-tools/src/main/resources/docker/file/openjdk7/Dockerfile
new file mode 100644
index 000..35d164a
--- /dev/null
+++ b/sdks/java/build-tools/src/main/resources/docker/file/openjdk7/Dockerfile
@@ -0,0 +1,49 @@
+###
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+###
+
+FROM maven:3-jdk-7
+
+# Download OS dependencies
+RUN apt-get update && \
+apt-get install -y \
+  libsnappy1 \
+  python-pip \
+  python-virtualenv \
+  python-dev \
+  rsync \
+  && rm -rf /var/lib/apt/lists/*
+
+# Add the entrypoint script that downloads the beam sources on run
+COPY docker-entrypoint.sh /usr/local/bin/
+RUN ln -s usr/local/bin/docker-entrypoint.sh /entrypoint.sh # backwards compat
+ENTRYPOINT ["docker-entrypoint.sh"]
+
+# Create beam user to validate the build on user space
+ENV USER=user \
+UID= \
+HOME=/home/user
+RUN groupadd --system --gid=$UID $USER; \
+useradd --system --uid=$UID --gid $USER $USER;
+RUN mkdir -p $HOME; \
+chown -R $USER:$USER $HOME;
+USER $USER
+WORKDIR $HOME
+
+ENV URL=https://github.com/apache/beam/archive/master.zip
+ENV SRC_FILE=master.zip
+ENV SRC_DIR=beam-master

http://git-wip-us.apache.org/repos/asf/beam/blob/68091296/sdks/java/build-tools/src/main/resources/docker/file/openjdk7/docker-entrypoint.sh
--
diff --git 
a/sdks/java/build-tools/src/main/resources/docker/file/openjdk7/docker-entrypoint.sh
 
b/sdks/java/build-tools/src/main/resources/docker/file/openjdk7/docker-entrypoint.sh
new file mode 100755
index 000..589e6ba
--- /dev/null
+++ 
b/sdks/java/build-tools/src/main/resources/docker/file/openjdk7/docker-entrypoint.sh
@@ -0,0 +1,24 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright 

[GitHub] beam pull request #3651: [BEAM-1534] Create a dockerized developer environme...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3651


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3651

[iemejia]

This closes #3651


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9b175ccc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9b175ccc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9b175ccc

Branch: refs/heads/master
Commit: 9b175ccc2f88d805ead3131c56129c24bdf1045e
Parents: 62294e6 6809129
Author: Ismaël Mejía 
Authored: Tue Aug 22 13:48:42 2017 +0200
Committer: Ismaël Mejía 
Committed: Tue Aug 22 13:48:42 2017 +0200

--
 .gitignore  |  3 ++
 .../resources/docker/file/openjdk7/Dockerfile   | 49 ++
 .../docker/file/openjdk7/docker-entrypoint.sh   | 24 +
 .../resources/docker/file/openjdk8/Dockerfile   | 49 ++
 .../docker/file/openjdk8/docker-entrypoint.sh   | 24 +
 .../resources/docker/git/openjdk8/Dockerfile| 53 
 .../docker/git/openjdk8/docker-entrypoint.sh| 22 
 .../resources/docker/release/python2/Dockerfile | 21 
 8 files changed, 245 insertions(+)
--

[jira] [Assigned] (BEAM-2671) CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark runner

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin reassigned BEAM-2671:


Assignee: Stas Levin  (was: Jean-Baptiste Onofré)

> CreateStreamTest.testFirstElementLate validatesRunner test fails on Spark 
> runner
> 
>
> Key: BEAM-2671
> URL: https://issues.apache.org/jira/browse/BEAM-2671
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Etienne Chauchot
>Assignee: Stas Levin
> Fix For: 2.2.0
>
>
> Error message:
> Flatten.Iterables/FlattenIterables/FlatMap/ParMultiDo(Anonymous).out0: 
> Expected: iterable over [] in any order
>  but: Not matched: "late"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #2884

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #2883

[Apache Jenkins Server]

See 


--
[...truncated 595.29 KB...]
2017-08-22T10:03:16.041 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar
2017-08-22T10:03:16.066 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/module/jackson-module-scala_2.10/2.8.9/jackson-module-scala_2.10-2.8.9.jar
 (494 KB at 454.5 KB/sec)
2017-08-22T10:03:16.066 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-client/0.8.2/tachyon-client-0.8.2.jar
2017-08-22T10:03:16.090 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-all/4.0.29.Final/netty-all-4.0.29.Final.jar
 (2007 KB at 1807.9 KB/sec)
2017-08-22T10:03:16.090 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-hdfs/0.8.2/tachyon-underfs-hdfs-0.8.2.jar
2017-08-22T10:03:16.116 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-hdfs/0.8.2/tachyon-underfs-hdfs-0.8.2.jar
 (11 KB at 9.5 KB/sec)
2017-08-22T10:03:16.116 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-s3/0.8.2/tachyon-underfs-s3-0.8.2.jar
2017-08-22T10:03:16.193 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar 
(1253 KB at 1032.5 KB/sec)
2017-08-22T10:03:16.193 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-local/0.8.2/tachyon-underfs-local-0.8.2.jar
2017-08-22T10:03:16.197 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-s3/0.8.2/tachyon-underfs-s3-0.8.2.jar
 (494 KB at 405.5 KB/sec)
2017-08-22T10:03:16.197 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/razorvine/pyrolite/4.9/pyrolite-4.9.jar
2017-08-22T10:03:16.218 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-local/0.8.2/tachyon-underfs-local-0.8.2.jar
 (8 KB at 5.7 KB/sec)
2017-08-22T10:03:16.218 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/sf/py4j/py4j/0.9/py4j-0.9.jar
2017-08-22T10:03:16.228 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/razorvine/pyrolite/4.9/pyrolite-4.9.jar
 (92 KB at 73.1 KB/sec)
2017-08-22T10:03:16.228 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar
2017-08-22T10:03:16.248 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.10.6/scala-reflect-2.10.6.jar
 (3132 KB at 2469.3 KB/sec)
2017-08-22T10:03:16.248 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-streaming_2.10/1.6.3/spark-streaming_2.10-1.6.3.jar
2017-08-22T10:03:16.250 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/sf/py4j/py4j/0.9/py4j-0.9.jar (83 KB 
at 64.9 KB/sec)
2017-08-22T10:03:16.250 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-network-common_2.10/1.6.3/spark-network-common_2.10-1.6.3.jar
2017-08-22T10:03:16.251 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-client/0.8.2/tachyon-client-0.8.2.jar
 (2238 KB at 1760.4 KB/sec)
2017-08-22T10:03:16.251 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.7.3/hadoop-common-2.7.3.jar
2017-08-22T10:03:16.253 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar
 (3 KB at 2.1 KB/sec)
2017-08-22T10:03:16.253 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-annotations/2.7.3/hadoop-annotations-2.7.3.jar
2017-08-22T10:03:16.298 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-annotations/2.7.3/hadoop-annotations-2.7.3.jar
 (40 KB at 30.3 KB/sec)
2017-08-22T10:03:16.298 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/xmlenc/xmlenc/0.52/xmlenc-0.52.jar
2017-08-22T10:03:16.324 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/xmlenc/xmlenc/0.52/xmlenc-0.52.jar (15 KB 
at 10.9 KB/sec)
2017-08-22T10:03:16.324 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.4/commons-codec-1.4.jar
2017-08-22T10:03:16.354 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.4/commons-codec-1.4.jar
 (57 KB at 41.3 KB/sec)
2017-08-22T10:03:16.354 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar
2017-08-22T10:03:16.417 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-streaming_2.10/1.6.3/spark-streaming_2.10-1.6.3.jar
 (2005 KB at 1394.9 KB/sec)
2017-08-22T10:03:16.417 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar
2017-08-22T10:03:16.442 [INFO] Downloaded: 
htt

Jenkins build is back to normal : beam_PostCommit_Python_Verify #2967

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3810

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #2966

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-2727] Make jar generation reproducible

[iemejia] [BEAM-2727] Disable timestamps from the Javadoc generation

[iemejia] Fix leaking directories/files on ApexYarnLauncherTest

--
[...truncated 733.86 KB...]
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Group/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s28"
}, 
"serialized_fn": 
"%0A%9B%05%22%98%05%0A%1Dref_Coder_GlobalWindowCoder_1%12%F6%04%0A%F3%04%0A%F0%04%0A%1Ebeam%3Acoder%3Apickled_python%3Av0.1%12%C0%02%0A.type.googleapis.com/google.protobuf.BytesValue%12%8D%02%0A%8A%02GlobalWindowCoder%24eNplzLEOgjAQBmAFQS3qc%2BggISCCs4NhlziSFmogoZRri0wk%2BuaCMSbE6S73/989tRTXOM1pQihmdsozKuR3oEvJCS5vRZXx9jycEEx2L5h2oMVm8sBlQ0EfeSVwJe9cMGm3Hzb6gWDWc6MDcxsbUmGhYD7ijSpKaauC0T5lNbr%2BNlj0ctkBik1WpIJLsKL9yXU9L3Ad7xj6hyDwQydEksQ6rTJY5YNYd7DJrei/GPTFhkhivwEtGlPK%1A%8A%02GlobalWindowCoder%24eNplzLEOgjAQBmAFQS3qc%2BggISCCs4NhlziSFmogoZRri0wk%2BuaCMSbE6S73/989tRTXOM1pQihmdsozKuR3oEvJCS5vRZXx9jycEEx2L5h2oMVm8sBlQ0EfeSVwJe9cMGm3Hzb6gWDWc6MDcxsbUmGhYD7ijSpKaauC0T5lNbr%2BNlj0ctkBik1WpIJLsKL9yXU9L3Ad7xj6hyDwQydEksQ6rTJY5YNYd7DJrei/GPTFhkhivwEtGlPKjJ%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00",
 
"user_name": "assert:even/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s30", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert:even/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s29"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s31", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2882

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2784) Fix issues from automated conversion to allow Python 2 functionality

[holdenk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136481#comment-16136481
 ] 

holdenk commented on BEAM-2784:
---

I've started working on this (it's down under a 100 errors :p)  
(https://github.com/holdenk/beam/tree/py2t3-plus-minal-fixes has the WIP 
branch).

> Fix issues from automated conversion to allow Python 2 functionality
> 
>
> Key: BEAM-2784
> URL: https://issues.apache.org/jira/browse/BEAM-2784
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: holdenk
>Assignee: Ahmet Altay
>
> As part of BEAM-1251 we want to move to support a Python2/3 code base. To do 
> this we can use futurize but futurize will break some Python2 elements. A 
> good intermediate checkpoint is contiuing to support Python 2 after 
> futurization from which we can build Python 3 support on top of.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2727) Make Beam released artifacts reproducible

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136476#comment-16136476
 ] 

ASF GitHub Bot commented on BEAM-2727:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3685


> Make Beam released artifacts reproducible
> -
>
> Key: BEAM-2727
> URL: https://issues.apache.org/jira/browse/BEAM-2727
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>
> There is a recent movement to create a verifiable path from source code to 
> binary releases in different open source projects. See 
> http://reproducible-builds.org/
> For the case of Java binaries (classes and jars) if we want to have a 
> reproducible build we need to have a consistent build environment so we can 
> reproduce those.
> For Beam we need to create an environment with a fixed version of the Java 
> compiler, maven and its plugins. This is addressed by BEAM-1534. But also by 
> including the (JDK/Maven versions) used to generate the jar artifacts in the 
> information to validate at the vote. Additionally we need to fix the Jar 
> files because by default the maven-jar-plugin include the timestamp of the 
> build in the files which breaks the reproducibility of the process, 
> fortunately there is a Maven plugin that fixes this and that we can include 
> in the release build.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3685: [BEAM-2727] Make beam builds reproducible

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3685


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/4] beam git commit: This closes #3685

[iemejia]

This closes #3685


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/62294e64
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/62294e64
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/62294e64

Branch: refs/heads/master
Commit: 62294e64cb3f68e37cd0ed133070fb617334c09e
Parents: d8d3f30 10a49a5
Author: Ismaël Mejía 
Authored: Tue Aug 22 09:53:10 2017 +0200
Committer: Ismaël Mejía 
Committed: Tue Aug 22 09:53:10 2017 +0200

--
 pom.xml | 20 
 .../beam/runners/apex/ApexYarnLauncherTest.java |  9 +
 2 files changed, 25 insertions(+), 4 deletions(-)
--

[1/4] beam git commit: Fix leaking directories/files on ApexYarnLauncherTest

[iemejia]

Repository: beam
Updated Branches:
  refs/heads/master d8d3f3093 -> 62294e64c


Fix leaking directories/files on ApexYarnLauncherTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/10a49a5a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/10a49a5a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/10a49a5a

Branch: refs/heads/master
Commit: 10a49a5a9401c5b7ddf57fd6463b6888e5ecbaf0
Parents: c58c42c
Author: Ismaël Mejía 
Authored: Mon Aug 21 18:07:07 2017 +0200
Committer: Ismaël Mejía 
Committed: Tue Aug 22 09:52:45 2017 +0200

--
 .../org/apache/beam/runners/apex/ApexYarnLauncherTest.java  | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/10a49a5a/runners/apex/src/test/java/org/apache/beam/runners/apex/ApexYarnLauncherTest.java
--
diff --git 
a/runners/apex/src/test/java/org/apache/beam/runners/apex/ApexYarnLauncherTest.java
 
b/runners/apex/src/test/java/org/apache/beam/runners/apex/ApexYarnLauncherTest.java
index 68ec2ea..adaf67b 100644
--- 
a/runners/apex/src/test/java/org/apache/beam/runners/apex/ApexYarnLauncherTest.java
+++ 
b/runners/apex/src/test/java/org/apache/beam/runners/apex/ApexYarnLauncherTest.java
@@ -43,12 +43,15 @@ import org.apache.apex.api.Launcher.LaunchMode;
 import org.apache.commons.io.FileUtils;
 import org.apache.hadoop.conf.Configuration;
 import org.junit.Assert;
+import org.junit.Rule;
 import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
 
 /**
  * Test for dependency resolution for pipeline execution on YARN.
  */
 public class ApexYarnLauncherTest {
+  @Rule public TemporaryFolder tmpFolder = new TemporaryFolder();
 
   @Test
   public void testGetYarnDeployDependencies() throws Exception {
@@ -119,10 +122,9 @@ public class ApexYarnLauncherTest {
 
   @Test
   public void testCreateJar() throws Exception {
-File baseDir = new File("./target/testCreateJar");
-File srcDir = new File(baseDir, "src");
+File baseDir = tmpFolder.newFolder("target", "testCreateJar");
+File srcDir = tmpFolder.newFolder("target", "testCreateJar", "src");
 String file1 = "file1";
-FileUtils.forceMkdir(srcDir);
 FileUtils.write(new File(srcDir, file1), "file1");
 
 File jarFile = new File(baseDir, "test.jar");
@@ -134,6 +136,5 @@ public class ApexYarnLauncherTest {
   Assert.assertTrue("manifest", 
Files.isRegularFile(zipfs.getPath(JarFile.MANIFEST_NAME)));
   Assert.assertTrue("file1", Files.isRegularFile(zipfs.getPath(file1)));
 }
-
   }
 }

[3/4] beam git commit: [BEAM-2727] Make jar generation reproducible

[iemejia]

[BEAM-2727] Make jar generation reproducible

This enables the Maven reproducible-build plugin that strips tiemstamps
and other information to achieve reproducible builds.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a72b6286
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a72b6286
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a72b6286

Branch: refs/heads/master
Commit: a72b62864fc42b0f7596ea4af2378edbd756fc11
Parents: d8d3f30
Author: Ismaël Mejía 
Authored: Mon Aug 21 17:19:46 2017 +0200
Committer: Ismaël Mejía 
Committed: Tue Aug 22 09:52:45 2017 +0200

--
 pom.xml | 19 +++
 1 file changed, 19 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a72b6286/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 1bdaa97..29f329f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -165,6 +165,7 @@
 3.0.0-M1
 3.0.2
 3.0.0
+
0.3
 
 -Werror
 
-Xpkginfo:always
@@ -271,6 +272,11 @@
 org.apache.maven.plugins
 maven-source-plugin
   
+
+  
+io.github.zlika
+reproducible-build-maven-plugin
+  
 
   
 
@@ -1772,6 +1778,19 @@
 
   
 
+
+
+  io.github.zlika
+  reproducible-build-maven-plugin
+  ${reproducible-build-maven-plugin.version}
+  
+
+  
+strip-jar
+  
+
+  
+
   
 
 

[2/4] beam git commit: [BEAM-2727] Disable timestamps from the Javadoc generation

[iemejia]

[BEAM-2727] Disable timestamps from the Javadoc generation

This makes the javadoc reproducible.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c58c42cc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c58c42cc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c58c42cc

Branch: refs/heads/master
Commit: c58c42cc978c7ee1f66d708b55eb9fcd6478e912
Parents: a72b628
Author: Ismaël Mejía 
Authored: Mon Aug 21 17:19:59 2017 +0200
Committer: Ismaël Mejía 
Committed: Tue Aug 22 09:52:45 2017 +0200

--
 pom.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c58c42cc/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 29f329f..5fd1297 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1416,6 +1416,7 @@
 Apache Beam SDK for Java, version 
${project.version}
 false
 true
+true
   
 
 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3809

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3743: [BEAM-2788] Use SerializablePipelineOptions to serd...

[huafengw]

GitHub user huafengw opened a pull request:

https://github.com/apache/beam/pull/3743

[BEAM-2788] Use SerializablePipelineOptions to serde PipelineOptions …

…in Gearpump runner

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/huafengw/beam minor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3743.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3743


commit a2366084af9fe748200f630eece9e12df6480b10
Author: huafengw 
Date:   2017-08-22T07:15:29Z

[BEAM-2788] Use SerializablePipelineOptions to serde PipelineOptions in 
Gearpump runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2788) Use SerializablePipelineOptions to serde PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136446#comment-16136446
 ] 

ASF GitHub Bot commented on BEAM-2788:
--

GitHub user huafengw opened a pull request:

https://github.com/apache/beam/pull/3743

[BEAM-2788] Use SerializablePipelineOptions to serde PipelineOptions …

…in Gearpump runner

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/huafengw/beam minor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3743.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3743


commit a2366084af9fe748200f630eece9e12df6480b10
Author: huafengw 
Date:   2017-08-22T07:15:29Z

[BEAM-2788] Use SerializablePipelineOptions to serde PipelineOptions in 
Gearpump runner




> Use SerializablePipelineOptions to serde PipelineOptions
> 
>
> Key: BEAM-2788
> URL: https://issues.apache.org/jira/browse/BEAM-2788
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-gearpump
>Affects Versions: 2.0.0
>Reporter: Manu Zhang
>Assignee: Huafeng Wang
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)