[jira] [Created] (BEAM-1121) Update documentation following rename of PTransform.apply

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1121:
-

 Summary: Update documentation following rename of PTransform.apply
 Key: BEAM-1121
 URL: https://issues.apache.org/jira/browse/BEAM-1121
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Daniel Halperin
Assignee: Kenneth Knowles
 Fix For: 0.4.0-incubating


Since PTransform#apply does not exist any more, significant website 
documentation may be wrong.

Fix version: 0.4.0-incubating really just means this needs to be done as part 
of the 0.4.0-incubating release, since this change will make it into said 
release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1561: [BEAM-498] Remove misc uses of OldDoFn

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1561

[BEAM-498] Remove misc uses of OldDoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @amitsela 

Just wiping these out in my idle moments


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam wipe-OldDoFn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1561.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1561


commit de8ed03621586314c09e39015e074e6a4a7894cc
Author: Kenneth Knowles 
Date:   2016-12-09T07:33:40Z

Remove misc uses of OldDoFn




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734589#comment-15734589
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1561

[BEAM-498] Remove misc uses of OldDoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @amitsela 

Just wiping these out in my idle moments


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam wipe-OldDoFn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1561.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1561


commit de8ed03621586314c09e39015e074e6a4a7894cc
Author: Kenneth Knowles 
Date:   2016-12-09T07:33:40Z

Remove misc uses of OldDoFn




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1120) Prepare Dataflow runner for 0.4.0 release

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1120:
-

 Summary: Prepare Dataflow runner for 0.4.0 release
 Key: BEAM-1120
 URL: https://issues.apache.org/jira/browse/BEAM-1120
 Project: Beam
  Issue Type: Improvement
Reporter: Daniel Halperin
Assignee: Daniel Halperin
 Fix For: 0.4.0-incubating






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1560: DataflowRunner: bump environment major ve...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1560

DataflowRunner: bump environment major version

R: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam dataflow-upgrade

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1560.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1560


commit 9bcba398c7516437c00517e03d75e27544b01166
Author: Dan Halperin 
Date:   2016-12-09T07:15:19Z

DataflowRunner: bump environment major version




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1119) Add a trigger that fires when the watermark passes timestamp

[Manu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manu Zhang updated BEAM-1119:
-
Description: 
For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}}, 

* "a" would be emitted if watermark passes 1
* "a" and "b" would be emitted if watermark passes 4
* all the elements would be emitted if watermark passes 5
* if the watermark jumps from 0 to 7 then all the elements would be emitted
* if "d" comes in late with timestamp 3, then it will be emitted if it's within 
the allowed lateness. 

This usually means the trigger fires in the middle of a window and is useful 
when firing at the end of window misses the latency requirement.

  was:
For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}}, 

* "a" would be emitted if watermark passes 1
* "a" and "b" would be emitted if watermark passes 4
* all the elements would be emitted if watermark passes 5 or the end of the 
window
* if the watermark jumps from 0 to 7 then all the elements would be emitted
* if "d" comes in late with timestamp 3, then it will be emitted if it's within 
the allowed lateness. 


> Add a trigger that fires when the watermark passes timestamp
> 
>
> Key: BEAM-1119
> URL: https://issues.apache.org/jira/browse/BEAM-1119
> Project: Beam
>  Issue Type: Improvement
>Reporter: Manu Zhang
>
> For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}}, 
> * "a" would be emitted if watermark passes 1
> * "a" and "b" would be emitted if watermark passes 4
> * all the elements would be emitted if watermark passes 5
> * if the watermark jumps from 0 to 7 then all the elements would be emitted
> * if "d" comes in late with timestamp 3, then it will be emitted if it's 
> within the allowed lateness. 
> This usually means the trigger fires in the middle of a window and is useful 
> when firing at the end of window misses the latency requirement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1119) Add a trigger that fires when the watermark passes timestamp

[Manu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manu Zhang updated BEAM-1119:
-
Assignee: Kenneth Knowles

> Add a trigger that fires when the watermark passes timestamp
> 
>
> Key: BEAM-1119
> URL: https://issues.apache.org/jira/browse/BEAM-1119
> Project: Beam
>  Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kenneth Knowles
>
> For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}}, 
> * "a" would be emitted if watermark passes 1
> * "a" and "b" would be emitted if watermark passes 4
> * all the elements would be emitted if watermark passes 5
> * if the watermark jumps from 0 to 7 then all the elements would be emitted
> * if "d" comes in late with timestamp 3, then it will be emitted if it's 
> within the allowed lateness. 
> This usually means the trigger fires in the middle of a window and is useful 
> when firing at the end of window misses the latency requirement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1119) Add a trigger that fires when the watermark passes timestamp

[Manu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manu Zhang updated BEAM-1119:
-
Summary: Add a trigger that fires when the watermark passes timestamp  
(was: Add a trigger that fires in the middle of window)

> Add a trigger that fires when the watermark passes timestamp
> 
>
> Key: BEAM-1119
> URL: https://issues.apache.org/jira/browse/BEAM-1119
> Project: Beam
>  Issue Type: Improvement
>Reporter: Manu Zhang
>
> For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}}, 
> * "a" would be emitted if watermark passes 1
> * "a" and "b" would be emitted if watermark passes 4
> * all the elements would be emitted if watermark passes 5 or the end of the 
> window
> * if the watermark jumps from 0 to 7 then all the elements would be emitted
> * if "d" comes in late with timestamp 3, then it will be emitted if it's 
> within the allowed lateness. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1119) Add a trigger that fires in the middle of window

[Manu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manu Zhang updated BEAM-1119:
-
Description: 
For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}}, 

* "a" would be emitted if watermark passes 1
* "a" and "b" would be emitted if watermark passes 4
* all the elements would be emitted if watermark passes 5 or the end of the 
window
* if the watermark jumps from 0 to 7 then all the elements would be emitted
* if "d" comes in late with timestamp 3, then it will be emitted if it's within 
the allowed lateness. 

> Add a trigger that fires in the middle of window
> 
>
> Key: BEAM-1119
> URL: https://issues.apache.org/jira/browse/BEAM-1119
> Project: Beam
>  Issue Type: Improvement
>Reporter: Manu Zhang
>
> For a pane with elements and timestamps {{[(a, 1), (b, 4), (c, 5)]}}, 
> * "a" would be emitted if watermark passes 1
> * "a" and "b" would be emitted if watermark passes 4
> * all the elements would be emitted if watermark passes 5 or the end of the 
> window
> * if the watermark jumps from 0 to 7 then all the elements would be emitted
> * if "d" comes in late with timestamp 3, then it will be emitted if it's 
> within the allowed lateness. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1119) Add a trigger that fires in the middle of window

[Manu Zhang (JIRA)]

Manu Zhang created BEAM-1119:


 Summary: Add a trigger that fires in the middle of window
 Key: BEAM-1119
 URL: https://issues.apache.org/jira/browse/BEAM-1119
 Project: Beam
  Issue Type: Improvement
Reporter: Manu Zhang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1115) Support for new Timer API in Spark runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1115:
--
Component/s: (was: runner-apex)
 runner-spark

> Support for new Timer API in Spark runner
> -
>
> Key: BEAM-1115
> URL: https://issues.apache.org/jira/browse/BEAM-1115
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-1117) Support for new Timer API in Direct runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-1117:
-

Assignee: Kenneth Knowles  (was: Thomas Groh)

> Support for new Timer API in Direct runner
> --
>
> Key: BEAM-1117
> URL: https://issues.apache.org/jira/browse/BEAM-1117
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1116) Support for new Timer API in Flink runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1116:
--
Assignee: (was: Maximilian Michels)

> Support for new Timer API in Flink runner
> -
>
> Key: BEAM-1116
> URL: https://issues.apache.org/jira/browse/BEAM-1116
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1118) Support for new Timer API in Gearpump runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1118:
-

 Summary: Support for new Timer API in Gearpump runner
 Key: BEAM-1118
 URL: https://issues.apache.org/jira/browse/BEAM-1118
 Project: Beam
  Issue Type: New Feature
  Components: runner-gearpump
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1117) Support for new Timer API in Direct runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1117:
-

 Summary: Support for new Timer API in Direct runner
 Key: BEAM-1117
 URL: https://issues.apache.org/jira/browse/BEAM-1117
 Project: Beam
  Issue Type: New Feature
  Components: runner-direct
Reporter: Kenneth Knowles
Assignee: Thomas Groh






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1116) Support for new Timer API in Flink runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1116:
-

 Summary: Support for new Timer API in Flink runner
 Key: BEAM-1116
 URL: https://issues.apache.org/jira/browse/BEAM-1116
 Project: Beam
  Issue Type: New Feature
  Components: runner-flink
Reporter: Kenneth Knowles
Assignee: Maximilian Michels






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1114) Support for new Timer API in Apex runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1114:
-

 Summary: Support for new Timer API in Apex runner
 Key: BEAM-1114
 URL: https://issues.apache.org/jira/browse/BEAM-1114
 Project: Beam
  Issue Type: New Feature
  Components: runner-apex
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1115) Support for new Timer API in Spark runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1115:
-

 Summary: Support for new Timer API in Spark runner
 Key: BEAM-1115
 URL: https://issues.apache.org/jira/browse/BEAM-1115
 Project: Beam
  Issue Type: New Feature
  Components: runner-apex
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1113) Support for new Timer API in Dataflow runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1113:
-

 Summary: Support for new Timer API in Dataflow runner
 Key: BEAM-1113
 URL: https://issues.apache.org/jira/browse/BEAM-1113
 Project: Beam
  Issue Type: New Feature
  Components: runner-dataflow
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Spark #362

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Spark #362

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Flink #1052

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Flink #1052

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2042

[Apache Jenkins Server]

See 


Changes:

[lcwik] fixup! Fix CountingInput naming

--
[...truncated 2448 lines...]
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-debug-all/5.0.1/asm-debug-all-5.0.1.pom
 (2 KB at 71.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-parent/5.0.1/asm-parent-5.0.1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-parent/5.0.1/asm-parent-5.0.1.pom
 (6 KB at 191.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.pom
 (2 KB at 47.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.1.2/doxia-core-1.1.2.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.8.1/xercesImpl-2.8.1.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.2/commons-codec-1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.2/commons-codec-1.2.jar
 (30 KB at 625.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.1.2/doxia-site-renderer-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar
 (298 KB at 5727.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.1.2/doxia-decoration-model-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar (232 KB 
at 4139.0 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.1.2/doxia-core-1.1.2.jar
 (155 KB at 2862.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.1.2/doxia-module-xhtml-1.1.2.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.1.2/doxia-module-fml-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.1.2/doxia-site-renderer-1.1.2.jar
 (50 KB at 656.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2/commons-collections-3.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.1.2/doxia-module-xhtml-1.1.2.jar
 (15 KB at 183.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.agent/0.7.5.201505241946/org.jacoco.agent-0.7.5.201505241946-runtime.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.1.2/doxia-module-fml-1.1.2.jar
 (37 KB at 424.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.core/0.7.5.201505241946/org.jacoco.core-0.7.5.201505241946.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.1.2/doxia-decoration-model-1.1.2.jar
 (52 KB at 587.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-debug-all/5.0.1/asm-debug-all-5.0.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.8.1/xercesImpl-2.8.1.jar
 (1185 KB at 12875.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.jar
 (137 KB at 1086.7 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.core/0.7.5.201505241946/org.jacoco.core-0.7.5.201505241946.jar
 (131 KB at 995.1 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.agent/0.7.5.201505241946/org.jacoco.agent-0.7.5.201505241946-runtime.jar
 (282 KB at 2133.1 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2/commons-collections-3.2.jar
 (558 KB at 3928.7 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-debug-all/5.0.1/asm-debug-all-5.0.1.jar
 (372 KB at 2597.1 KB/sec)
[INFO] argLine set to 

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Flink #1051

[Apache Jenkins Server]

See 


Changes:

[lcwik] fixup! Fix CountingInput naming

--
[...truncated 1739 lines...]
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.pom
 (2 KB at 51.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.5.6/plexus-utils-1.5.6.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.1/maven-reporting-impl-2.1.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/classworlds/classworlds/1.1-alpha-2/classworlds-1.1-alpha-2.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.1.2/doxia-core-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/classworlds/classworlds/1.1-alpha-2/classworlds-1.1-alpha-2.jar
 (37 KB at 1110.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.8.1/xercesImpl-2.8.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.1/maven-reporting-impl-2.1.jar
 (17 KB at 437.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar (232 KB 
at 5150.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.2/commons-codec-1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.5.6/plexus-utils-1.5.6.jar
 (245 KB at 5437.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.1.2/doxia-site-renderer-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.1.2/doxia-core-1.1.2.jar
 (155 KB at 2534.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.1.2/doxia-decoration-model-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.2/commons-codec-1.2.jar
 (30 KB at 413.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.1.2/doxia-module-xhtml-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.1.2/doxia-site-renderer-1.1.2.jar
 (50 KB at 665.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.1.2/doxia-module-fml-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.1.2/doxia-decoration-model-1.1.2.jar
 (52 KB at 573.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2/commons-collections-3.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar
 (298 KB at 3309.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-validator/commons-validator/1.2.0/commons-validator-1.2.0.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.1.2/doxia-module-xhtml-1.1.2.jar
 (15 KB at 148.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.1.2/doxia-module-fml-1.1.2.jar
 (37 KB at 354.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.agent/0.7.5.201505241946/org.jacoco.agent-0.7.5.201505241946-runtime.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-validator/commons-validator/1.2.0/commons-validator-1.2.0.jar
 (89 KB at 723.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.core/0.7.5.201505241946/org.jacoco.core-0.7.5.201505241946.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar
 (107 KB at 814.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-debug-all/5.0.1/asm-debug-all-5.0.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.8.1/xercesImpl-2.8.1.jar
 (1185 KB at 8709.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2/commons-collections-3.2.jar
 (558 KB at 3769.4 

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Flink #1051

[Apache Jenkins Server]

See 


Changes:

[lcwik] fixup! Fix CountingInput naming

--
[...truncated 1738 lines...]
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.pom
 (2 KB at 51.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.5.6/plexus-utils-1.5.6.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.1/maven-reporting-impl-2.1.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/classworlds/classworlds/1.1-alpha-2/classworlds-1.1-alpha-2.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.1.2/doxia-core-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/classworlds/classworlds/1.1-alpha-2/classworlds-1.1-alpha-2.jar
 (37 KB at 1110.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.8.1/xercesImpl-2.8.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-impl/2.1/maven-reporting-impl-2.1.jar
 (17 KB at 437.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar (232 KB 
at 5150.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.2/commons-codec-1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/1.5.6/plexus-utils-1.5.6.jar
 (245 KB at 5437.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.1.2/doxia-site-renderer-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.1.2/doxia-core-1.1.2.jar
 (155 KB at 2534.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.1.2/doxia-decoration-model-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.2/commons-codec-1.2.jar
 (30 KB at 413.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.1.2/doxia-module-xhtml-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.1.2/doxia-site-renderer-1.1.2.jar
 (50 KB at 665.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.1.2/doxia-module-fml-1.1.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.1.2/doxia-decoration-model-1.1.2.jar
 (52 KB at 573.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2/commons-collections-3.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar
 (298 KB at 3309.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-validator/commons-validator/1.2.0/commons-validator-1.2.0.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.1.2/doxia-module-xhtml-1.1.2.jar
 (15 KB at 148.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.1.2/doxia-module-fml-1.1.2.jar
 (37 KB at 354.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.agent/0.7.5.201505241946/org.jacoco.agent-0.7.5.201505241946-runtime.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-validator/commons-validator/1.2.0/commons-validator-1.2.0.jar
 (89 KB at 723.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.core/0.7.5.201505241946/org.jacoco.core-0.7.5.201505241946.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar
 (107 KB at 814.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-debug-all/5.0.1/asm-debug-all-5.0.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.8.1/xercesImpl-2.8.1.jar
 (1185 KB at 8709.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jacoco/org.jacoco.report/0.7.5.201505241946/org.jacoco.report-0.7.5.201505241946.jar

incubator-beam git commit: fixup! Fix extraneous brace

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5169e4925 -> 9bab78b55


fixup! Fix extraneous brace


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9bab78b5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9bab78b5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9bab78b5

Branch: refs/heads/master
Commit: 9bab78b55fe4661e9a221969441d8a2a4716f7e1
Parents: 5169e49
Author: Luke Cwik 
Authored: Thu Dec 8 19:20:22 2016 -0800
Committer: Luke Cwik 
Committed: Thu Dec 8 19:20:22 2016 -0800

--
 .../src/test/java/org/apache/beam/sdk/io/CountingInputTest.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9bab78b5/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java
index 063bead..dfc4919 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java
@@ -71,7 +71,7 @@ public class CountingInputTest {
   public void testBoundedInput() {
 Pipeline p = TestPipeline.create();
 long numElements = 1000;
-PCollection input = p.apply(CountingInput.upTo(numElements)));
+PCollection input = p.apply(CountingInput.upTo(numElements));
 
 addCountingAsserts(input, 0, numElements);
 p.run();

[jira] [Updated] (BEAM-1086) Upgrade to latest Gearpump snapshot

[Manu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manu Zhang updated BEAM-1086:
-
Summary: Upgrade to latest Gearpump snapshot  (was: Upgrade to latest 
GEARPUMP snapshots)

> Upgrade to latest Gearpump snapshot
> ---
>
> Key: BEAM-1086
> URL: https://issues.apache.org/jira/browse/BEAM-1086
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-gearpump
>Reporter: Manu Zhang
>Assignee: Manu Zhang
>
> The latest Gearpump snapshot version is available under [apache repo | 
> https://repository.apache.org/content/repositories/snapshots/org/apache/gearpump/gearpump-core_2.11/0.8.3-SNAPSHOT/].
>  To support the Gearpump runner, we need to continuously evolve Gearpump and 
> bring in Beam capabilities. Depending on a snapshot version will greatly 
> accelerate the integration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-beam git commit: fixup! Fix CountingInput naming

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ddb59125a -> 5169e4925


fixup! Fix CountingInput naming


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5169e492
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5169e492
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5169e492

Branch: refs/heads/master
Commit: 5169e492578a3759e20b50e36ace61bc86636ad2
Parents: ddb5912
Author: Luke Cwik 
Authored: Thu Dec 8 19:17:21 2016 -0800
Committer: Luke Cwik 
Committed: Thu Dec 8 19:17:21 2016 -0800

--
 .../test/java/org/apache/beam/sdk/io/CountingInputTest.java| 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5169e492/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java
index 4349f66..063bead 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java
@@ -71,7 +71,7 @@ public class CountingInputTest {
   public void testBoundedInput() {
 Pipeline p = TestPipeline.create();
 long numElements = 1000;
-PCollection input = 
p.apply(Read.from(CountingSource.upTo(numElements)));
+PCollection input = p.apply(CountingInput.upTo(numElements)));
 
 addCountingAsserts(input, 0, numElements);
 p.run();
@@ -79,7 +79,7 @@ public class CountingInputTest {
 
   @Test
   @Category(RunnableOnService.class)
-  public void testEmptyBoundedSource() {
+  public void testEmptyBoundedInput() {
 Pipeline p = TestPipeline.create();
 PCollection input = p.apply(CountingInput.upTo(0));
 
@@ -89,7 +89,7 @@ public class CountingInputTest {
 
   @Test
   @Category(RunnableOnService.class)
-  public void testEmptyBoundedSourceUsingRange() {
+  public void testEmptyBoundedInputSubrange() {
 Pipeline p = TestPipeline.create();
 PCollection input = p.apply(CountingInput.forSubrange(42, 42));
 

[jira] [Updated] (BEAM-1086) Upgrade to latest GEARPUMP snapshots

[Manu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manu Zhang updated BEAM-1086:
-
Description: The latest Gearpump snapshot version is available under 
[apache repo | 
https://repository.apache.org/content/repositories/snapshots/org/apache/gearpump/gearpump-core_2.11/0.8.3-SNAPSHOT/].
 To support the Gearpump runner, we need to continuously evolve Gearpump and 
bring in Beam capabilities. Depending on a snapshot version will greatly 
accelerate the integration.

> Upgrade to latest GEARPUMP snapshots
> 
>
> Key: BEAM-1086
> URL: https://issues.apache.org/jira/browse/BEAM-1086
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-gearpump
>Reporter: Manu Zhang
>Assignee: Manu Zhang
>
> The latest Gearpump snapshot version is available under [apache repo | 
> https://repository.apache.org/content/repositories/snapshots/org/apache/gearpump/gearpump-core_2.11/0.8.3-SNAPSHOT/].
>  To support the Gearpump runner, we need to continuously evolve Gearpump and 
> bring in Beam capabilities. Depending on a snapshot version will greatly 
> accelerate the integration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1086) Upgrade to latest GEARPUMP snapshots

[Manu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manu Zhang updated BEAM-1086:
-
Summary: Upgrade to latest GEARPUMP snapshots  (was: Upgrade to 
Gearpump-0.8.2-incubating)

> Upgrade to latest GEARPUMP snapshots
> 
>
> Key: BEAM-1086
> URL: https://issues.apache.org/jira/browse/BEAM-1086
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-gearpump
>Reporter: Manu Zhang
>Assignee: Manu Zhang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Python_Verify #849

[Apache Jenkins Server]

See 

--
[...truncated 2969 lines...]
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.dataflow_test.DataflowTest)'

==
ERROR: test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.dataflow_test.DataflowTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 147, in test_par_do_with_multiple_outputs_and_using_yield
pipeline.run()
  File 
"
 line 159, in run
return self.runner.run(self)
  File 
"
 line 175, in run
pipeline.options, job_version)
  File 
"
 line 360, in __init__
credentials = get_service_credentials()
  File 
"
 line 171, in get_service_credentials
credentials.get_access_token()
  File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 677, in 
get_access_token
self.refresh(http)
  File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 560, in 
refresh
self._refresh(http.request)
  File 
"
 line 113, in _refresh
output, _ = gcloud_process.communicate()
  File "/usr/lib/python2.7/subprocess.py", line 791, in communicate
stdout = _eintr_retry_call(self.stdout.read)
  File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call
return func(*args)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.dataflow_test.DataflowTest)'

==
ERROR: test_par_do_with_side_input_as_arg 
(apache_beam.dataflow_test.DataflowTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 94, in test_par_do_with_side_input_as_arg
pipeline.run()
  File 
"
 line 159, in run
return self.runner.run(self)
  File 

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1769

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1769

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1557: Add support for having an empty CountingI...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1557


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Add support for having an empty CountingInput/CountingSource

[lcwik]

Add support for having an empty CountingInput/CountingSource

This closes #1557


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ddb59125
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ddb59125
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ddb59125

Branch: refs/heads/master
Commit: ddb59125aeacb809b7695c203fe8b1a40e36aed2
Parents: 40bd276 30ff1ee
Author: Luke Cwik 
Authored: Thu Dec 8 18:41:42 2016 -0800
Committer: Luke Cwik 
Committed: Thu Dec 8 18:41:42 2016 -0800

--
 .../org/apache/beam/sdk/io/CountingInput.java   | 12 ++
 .../org/apache/beam/sdk/io/CountingSource.java  | 12 ++
 .../apache/beam/sdk/io/CountingInputTest.java   | 23 +++-
 .../apache/beam/sdk/io/CountingSourceTest.java  | 10 +
 4 files changed, 48 insertions(+), 9 deletions(-)
--

[1/2] incubator-beam git commit: Add support for having an empty CountingInput/CountingSource

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 40bd27602 -> ddb59125a


Add support for having an empty CountingInput/CountingSource


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/30ff1ee1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/30ff1ee1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/30ff1ee1

Branch: refs/heads/master
Commit: 30ff1ee17bb290f2b50fd082d8cb63d48280c5c2
Parents: 40bd276
Author: Luke Cwik 
Authored: Thu Dec 8 15:22:35 2016 -0800
Committer: Luke Cwik 
Committed: Thu Dec 8 18:41:17 2016 -0800

--
 .../org/apache/beam/sdk/io/CountingInput.java   | 12 ++
 .../org/apache/beam/sdk/io/CountingSource.java  | 12 ++
 .../apache/beam/sdk/io/CountingInputTest.java   | 23 +++-
 .../apache/beam/sdk/io/CountingSourceTest.java  | 10 +
 4 files changed, 48 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/30ff1ee1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java
index 3148d8d..ac70aca 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java
@@ -75,17 +75,21 @@ public class CountingInput {
* from {@code 0} to {@code numElements - 1}.
*/
   public static BoundedCountingInput upTo(long numElements) {
-checkArgument(numElements > 0, "numElements (%s) must be greater than 0", 
numElements);
+checkArgument(numElements >= 0,
+"numElements (%s) must be greater than or equal to 0",
+numElements);
 return new BoundedCountingInput(numElements);
   }
 
   /**
* Creates a {@link BoundedCountingInput} that will produce elements
-   * starting from {@code startIndex} to {@code endIndex - 1}.
+   * starting from {@code startIndex} (inclusive) to {@code endIndex} 
(exclusive).
+   * If {@code startIndex == endIndex}, then no elements will be produced.
*/
   public static BoundedCountingInput forSubrange(long startIndex, long 
endIndex) {
-checkArgument(endIndex > startIndex, "endIndex (%s) must be greater than 
startIndex (%s)",
-endIndex, startIndex);
+checkArgument(endIndex >= startIndex,
+"endIndex (%s) must be greater than or equal to startIndex (%s)",
+endIndex, startIndex);
 return new BoundedCountingInput(startIndex, endIndex);
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/30ff1ee1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java
index bc7fb78..9752dba 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java
@@ -78,17 +78,21 @@ public class CountingSource {
*/
   @Deprecated
   public static BoundedSource upTo(long numElements) {
-checkArgument(numElements > 0, "numElements (%s) must be greater than 0", 
numElements);
+checkArgument(numElements >= 0,
+"numElements (%s) must be greater than or equal to 0",
+numElements);
 return new BoundedCountingSource(0, numElements);
   }
 
   /**
* Creates a {@link BoundedSource} that will produce elements
-   * from {@code startIndex} to {@code endIndex - 1}.
+   * starting from {@code startIndex} (inclusive) to {@code endIndex} 
(exclusive).
+   * If {@code startIndex == endIndex}, then no elements will be produced.
*/
   static BoundedSource createSourceForSubrange(long startIndex, long 
endIndex) {
-checkArgument(endIndex > startIndex, "endIndex (%s) must be greater than 
startIndex (%s)",
-endIndex, startIndex);
+checkArgument(endIndex >= startIndex,
+"endIndex (%s) must be greater than or equal to startIndex (%s)",
+endIndex, startIndex);
 
 return new BoundedCountingSource(startIndex, endIndex);
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/30ff1ee1/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CountingInputTest.java
index 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1768

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1768

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1093) Confusing Javadocs in StateInternals

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733981#comment-15733981
 ] 

ASF GitHub Bot commented on BEAM-1093:
--

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/1559

[BEAM-1093] Change Dataflow to Beam in StateInternals' Javadocs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam BEAM-1093

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1559.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1559


commit 79c7e7a7e818e1b9e3e835935497e3e460295f3e
Author: manuzhang 
Date:   2016-12-09T01:49:49Z

[BEAM-1093] Change Dataflow to Beam in StateInternals' Javadocs




> Confusing Javadocs in StateInternals
> 
>
> Key: BEAM-1093
> URL: https://issues.apache.org/jira/browse/BEAM-1093
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Manu Zhang
>Assignee: Manu Zhang
>Priority: Minor
>
> At last but one line of  StateInternals' Javadocs, it says "This is a 
> low-level API intended for use by the Dataflow SDK". Not sure what is 
> "Dataflow SDK".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1559: [BEAM-1093] Change Dataflow to Beam in St...

[manuzhang]

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/1559

[BEAM-1093] Change Dataflow to Beam in StateInternals' Javadocs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam BEAM-1093

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1559.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1559


commit 79c7e7a7e818e1b9e3e835935497e3e460295f3e
Author: manuzhang 
Date:   2016-12-09T01:49:49Z

[BEAM-1093] Change Dataflow to Beam in StateInternals' Javadocs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1093) Confusing Javadocs in StateInternals

[Manu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manu Zhang reassigned BEAM-1093:


Assignee: Manu Zhang  (was: Ben Chambers)

> Confusing Javadocs in StateInternals
> 
>
> Key: BEAM-1093
> URL: https://issues.apache.org/jira/browse/BEAM-1093
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Manu Zhang
>Assignee: Manu Zhang
>Priority: Minor
>
> At last but one line of  StateInternals' Javadocs, it says "This is a 
> low-level API intended for use by the Dataflow SDK". Not sure what is 
> "Dataflow SDK".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1525

[tgroh]

This closes #1525


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/40bd2760
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/40bd2760
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/40bd2760

Branch: refs/heads/master
Commit: 40bd27602ebe2269ccaba2685addb8e5e3ba533e
Parents: 409b5df bf1fba4
Author: Thomas Groh 
Authored: Thu Dec 8 17:17:56 2016 -0800
Committer: Thomas Groh 
Committed: Thu Dec 8 17:17:56 2016 -0800

--
 .../beam/runners/core/SplittableParDo.java  |  9 ++-
 .../org/apache/beam/sdk/transforms/ParDo.java   |  8 ++-
 .../apache/beam/sdk/transforms/ParDoTest.java   |  6 +-
 .../beam/sdk/transforms/SplittableDoFnTest.java | 58 +++-
 4 files changed, 73 insertions(+), 8 deletions(-)
--

[GitHub] incubator-beam pull request #1525: Fixes a bug in SDF checkpointing.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1525


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Fix a bug in SplittableDoFn Checkpointing

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 409b5dfcf -> 40bd27602


Fix a bug in SplittableDoFn Checkpointing

Call checkpoint() only once if the SDF emits output several times per
claim call.

Calling checkpoint multiple times would clobber an existing checkpoint,
and the second call would only ever return an empty residual, losing all
of the initial residual.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bf1fba45
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bf1fba45
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bf1fba45

Branch: refs/heads/master
Commit: bf1fba450e6b5fd6c98d006b381472eee8db7b72
Parents: 409b5df
Author: Eugene Kirpichov 
Authored: Tue Dec 6 18:00:03 2016 -0800
Committer: Thomas Groh 
Committed: Thu Dec 8 17:16:30 2016 -0800

--
 .../beam/runners/core/SplittableParDo.java  |  9 ++-
 .../org/apache/beam/sdk/transforms/ParDo.java   |  8 ++-
 .../apache/beam/sdk/transforms/ParDoTest.java   |  6 +-
 .../beam/sdk/transforms/SplittableDoFnTest.java | 58 +++-
 4 files changed, 73 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bf1fba45/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
index 0bf882b..8a9bfcd 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
@@ -590,9 +590,14 @@ public class SplittableParDo
 }
 
 private void noteOutput() {
-  if (++numOutputs >= MAX_OUTPUTS_PER_BUNDLE) {
+  // Take the checkpoint only if it hasn't been taken yet, because:
+  // 1) otherwise we'd lose the previous checkpoint stored in 
residualRestrictionHolder
+  // 2) it's not allowed to checkpoint a RestrictionTracker twice, 
since the first call
+  // by definition already maximally narrows its restriction, so a 
second checkpoint would
+  // have produced a useless empty residual restriction anyway.
+  if (++numOutputs >= MAX_OUTPUTS_PER_BUNDLE && 
residualRestrictionHolder[0] == null) {
 // Request a checkpoint. The fn *may* produce more output, but 
hopefully not too much.
-residualRestrictionHolder[0] = tracker.checkpoint();
+residualRestrictionHolder[0] = checkNotNull(tracker.checkpoint());
   }
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bf1fba45/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index e60c536..167f5fa 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -787,7 +787,9 @@ public class ParDo {
 @Override
 public PCollection expand(PCollection input) {
   checkArgument(
-  !isSplittable(getOldFn()), "Splittable DoFn not supported by the 
current runner");
+  !isSplittable(getOldFn()),
+  "%s does not support Splittable DoFn",
+  input.getPipeline().getOptions().getRunner().getName());
   validateWindowType(input, fn);
   return PCollection.createPrimitiveOutputInternal(
   input.getPipeline(),
@@ -1044,7 +1046,9 @@ public class ParDo {
 @Override
 public PCollectionTuple expand(PCollection input) {
   checkArgument(
-  !isSplittable(getOldFn()), "Splittable DoFn not supported by the 
current runner");
+  !isSplittable(getOldFn()),
+  "%s does not support Splittable DoFn",
+  input.getPipeline().getOptions().getRunner().getName());
   validateWindowType(input, fn);
   PCollectionTuple outputs = PCollectionTuple.ofPrimitiveOutputsInternal(
   input.getPipeline(),

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bf1fba45/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
index 

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1767

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1767

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #2039

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2038

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-59) IOChannelFactory rethinking/redesign

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733783#comment-15733783
 ] 

ASF GitHub Bot commented on BEAM-59:


GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1558

[BEAM-59] initial interfaces and classes of Beam FileSystem.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam file-system-init

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1558.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1558


commit 05ae42bb728c3a3314b5df47c13d0121d7e31444
Author: Pei He 
Date:   2016-12-08T01:35:23Z

[BEAM-59] initial interfaces and classes of Beam FileSystem.




> IOChannelFactory rethinking/redesign
> 
>
> Key: BEAM-59
> URL: https://issues.apache.org/jira/browse/BEAM-59
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Pei He
>
> Right now, FileBasedSource and FileBasedSink communication is mediated by 
> IOChannelFactory. There are a number of issues:
> * Global configuration -- e.g., all 'gs://' URIs use the same credentials. 
> This should be per-source/per-sink/etc.
> * Supported APIs -- currently IOChannelFactory is in the "non-public API" 
> util package and subject to change. We need users to be able to add new 
> backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be 
> broken.
> * Per-backend features: e.g., creating buckets in GCS/s3, setting expiration 
> time, etc.
> Updates:
> Design docs posted on dev@ list:
> Part 1: IOChannelFactory Redesign: 
> https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit#
> Part 2: Configurable BeamFileSystem:
> https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1558: [BEAM-59] initial interfaces and classes ...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1558

[BEAM-59] initial interfaces and classes of Beam FileSystem.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam file-system-init

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1558.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1558


commit 05ae42bb728c3a3314b5df47c13d0121d7e31444
Author: Pei He 
Date:   2016-12-08T01:35:23Z

[BEAM-59] initial interfaces and classes of Beam FileSystem.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1359: [BEAM-978] Support bulk get file size in ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1359


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-978] Support bulk get file size in GcsUtil.

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master e292032cb -> 409b5dfcf


[BEAM-978] Support bulk get file size in GcsUtil.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4a12cd30
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4a12cd30
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4a12cd30

Branch: refs/heads/master
Commit: 4a12cd303df6d4c49425d38a6290f496349cc312
Parents: e292032
Author: Pei He 
Authored: Thu Nov 10 21:53:57 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 15:47:47 2016 -0800

--
 .../java/org/apache/beam/sdk/util/GcsUtil.java  |  90 +++-
 .../org/apache/beam/sdk/util/GcsUtilTest.java   | 106 ++-
 2 files changed, 121 insertions(+), 75 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4a12cd30/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
index 54a963b..2edb1d6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java
@@ -93,8 +93,10 @@ public class GcsUtil {
 public GcsUtil create(PipelineOptions options) {
   LOG.debug("Creating new GcsUtil");
   GcsOptions gcsOptions = options.as(GcsOptions.class);
-  return new GcsUtil(Transport.newStorageClient(gcsOptions).build(),
-  gcsOptions.getExecutorService(), 
gcsOptions.getGcsUploadBufferSizeBytes());
+  return new GcsUtil(
+  Transport.newStorageClient(gcsOptions).build(),
+  gcsOptions.getExecutorService(),
+  gcsOptions.getGcsUploadBufferSizeBytes());
 }
   }
 
@@ -153,7 +155,8 @@ public class GcsUtil {
   }
 
   private GcsUtil(
-  Storage storageClient, ExecutorService executorService,
+  Storage storageClient,
+  ExecutorService executorService,
   @Nullable Integer uploadBufferSizeBytes) {
 this.storageClient = storageClient;
 this.uploadBufferSizeBytes = uploadBufferSizeBytes;
@@ -264,10 +267,7 @@ public class GcsUtil {
* if the resource does not exist.
*/
   public long fileSize(GcsPath path) throws IOException {
-return fileSize(
-path,
-BACKOFF_FACTORY.backoff(),
-Sleeper.DEFAULT);
+return fileSizes(ImmutableList.of(path)).get(0);
   }
 
   /**
@@ -275,23 +275,15 @@ public class GcsUtil {
* if the resource does not exist.
*/
   @VisibleForTesting
-  long fileSize(GcsPath path, BackOff backoff, Sleeper sleeper) throws 
IOException {
-  Storage.Objects.Get getObject =
-  storageClient.objects().get(path.getBucket(), path.getObject());
-  try {
-StorageObject object = ResilientOperation.retry(
-ResilientOperation.getGoogleRequestCallable(getObject),
-backoff,
-RetryDeterminer.SOCKET_ERRORS,
-IOException.class,
-sleeper);
-return object.getSize().longValue();
-  } catch (Exception e) {
-if (e instanceof IOException && 
errorExtractor.itemNotFound((IOException) e)) {
-  throw new FileNotFoundException(path.toString());
-}
-throw new IOException("Unable to get file size", e);
- }
+  List fileSizes(Collection paths) throws IOException {
+List results = Lists.newArrayList();
+executeBatches(makeGetBatches(paths, results));
+
+ImmutableList.Builder ret = ImmutableList.builder();
+for (long[] result : results) {
+  ret.add(result[0]);
+}
+return ret.build();
   }
 
   /**
@@ -483,12 +475,39 @@ public class GcsUtil {
   Thread.currentThread().interrupt();
   throw new IOException("Interrupted while executing batch GCS request", 
e);
 } catch (ExecutionException e) {
+  if (e.getCause() instanceof FileNotFoundException) {
+throw (FileNotFoundException) e.getCause();
+  }
   throw new IOException("Error executing batch GCS request", e);
 } finally {
   executor.shutdown();
 }
   }
 
+  /**
+   * Makes get {@link BatchRequest BatchRequests}.
+   *
+   * @param paths {@link GcsPath GcsPaths}.
+   * @param results mutable {@link List} for return values.
+   * @return {@link BatchRequest BatchRequests} to execute.
+   * @throws IOException
+   */
+  @VisibleForTesting
+  List makeGetBatches(
+  Collection paths,
+  List results) throws IOException {
+List batches = new LinkedList<>();
+for (List filesToGet :
+Lists.partition(Lists.newArrayList(paths), MAX_REQUESTS_PER_BATCH)) 

[jira] [Commented] (BEAM-978) Support bulk get file sizes in GcsUtil

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733757#comment-15733757
 ] 

ASF GitHub Bot commented on BEAM-978:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1359


> Support bulk get file sizes in GcsUtil
> --
>
> Key: BEAM-978
> URL: https://issues.apache.org/jira/browse/BEAM-978
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>
> Bulk get-file-sizes could be useful in FileBasedSource (It currently does 
> sampling and total size estimation with one RPC request per file.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1359

[kenn]

This closes #1359


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/409b5dfc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/409b5dfc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/409b5dfc

Branch: refs/heads/master
Commit: 409b5dfcf6e9a9699eff43b041e9726bbb979b89
Parents: e292032 4a12cd3
Author: Kenneth Knowles 
Authored: Thu Dec 8 15:47:48 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 15:47:48 2016 -0800

--
 .../java/org/apache/beam/sdk/util/GcsUtil.java  |  90 +++-
 .../org/apache/beam/sdk/util/GcsUtilTest.java   | 106 ++-
 2 files changed, 121 insertions(+), 75 deletions(-)
--

[GitHub] incubator-beam pull request #1557: Add support for having an empty CountingI...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/1557

Add support for having an empty CountingInput/CountingSource

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam counting_source

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1557.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1557


commit dff45ea3ce332a93f44c1ba511287ce900e9e6ef
Author: Luke Cwik 
Date:   2016-12-08T23:22:35Z

Add support for having an empty CountingInput/CountingSource




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1542: Fix a typo in SplitQueryFn error handling

[vikkyrk]

Github user vikkyrk closed the pull request at:

https://github.com/apache/incubator-beam/pull/1542


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1112) Python E2E Integration Test Framework - Batch Only

[Mark Liu (JIRA)]

Mark Liu created BEAM-1112:
--

 Summary: Python E2E Integration Test Framework - Batch Only
 Key: BEAM-1112
 URL: https://issues.apache.org/jira/browse/BEAM-1112
 Project: Beam
  Issue Type: Task
  Components: sdk-py, testing
Reporter: Mark Liu
Assignee: Mark Liu


Parity with Java. 

Build e2e integration test framework that can configure and run batch pipeline 
with specified test runner, wait for pipeline execution and verify results with 
given verifiers in the end.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-23) User-facing state and timers for ParDo(DoFn)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733507#comment-15733507
 ] 

ASF GitHub Bot commented on BEAM-23:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1556

[BEAM-23] Refactors to support key-preserving in direct runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @tgroh 

I've peeled these commits off since they make some sense on their own. The 
intent is that just tweaking the logic in `KeyedPValueTrackingVisitor` (in 
potentially hacky ways :-) we can start to allow key preservation.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
KeyedPValueTrackingVisitor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1556.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1556


commit d350b3d3708b83508d3e1df56638e0104ec28b83
Author: Kenneth Knowles 
Date:   2016-12-08T19:49:15Z

Add some key-preserving to KeyedPValueTrackingVisitor

commit 93e3af2dba38dea4a708ce4d85d29cf74e029de4
Author: Kenneth Knowles 
Date:   2016-12-08T21:28:44Z

Move responsibility for knowing about keyedness into EvaluationContext

This will allow transform evaluators to inquire about whether
various collections are keyed.

commit 1c756f85c8a04422d28e6063ba4072dd4fb0ea73
Author: Kenneth Knowles 
Date:   2016-12-08T19:44:48Z

Propagate key through ParDo if DoFn is key-preserving




> User-facing state and timers for ParDo(DoFn)
> 
>
> Key: BEAM-23
> URL: https://issues.apache.org/jira/browse/BEAM-23
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
> Fix For: Not applicable
>
>
> To a key-partitioned ParDo, add the ability for a user's DoFn to, on a 
> per-key-and-window basis:
>  - read and write consistent state
>  - set a timer to request a callback at a particular moment (in any time 
> domain)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Python_Verify #848

[Apache Jenkins Server]

See 

--
[...truncated 2999 lines...]
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.dataflow_test.DataflowTest)'

==
ERROR: test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.dataflow_test.DataflowTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 147, in test_par_do_with_multiple_outputs_and_using_yield
pipeline.run()
  File 
"
 line 159, in run
return self.runner.run(self)
  File 
"
 line 175, in run
pipeline.options, job_version)
  File 
"
 line 360, in __init__
credentials = get_service_credentials()
  File 
"
 line 171, in get_service_credentials
credentials.get_access_token()
  File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 677, in 
get_access_token
self.refresh(http)
  File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 560, in 
refresh
self._refresh(http.request)
  File 
"
 line 113, in _refresh
output, _ = gcloud_process.communicate()
  File "/usr/lib/python2.7/subprocess.py", line 791, in communicate
stdout = _eintr_retry_call(self.stdout.read)
  File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call
return func(*args)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.dataflow_test.DataflowTest)'

==
ERROR: test_par_do_with_side_input_as_arg 
(apache_beam.dataflow_test.DataflowTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 94, in test_par_do_with_side_input_as_arg
pipeline.run()
  File 
"
 line 159, in run
return self.runner.run(self)
  File 

[jira] [Closed] (BEAM-1077) @ValidatesRunner test in Python postcommit

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu closed BEAM-1077.
--
   Resolution: Done
Fix Version/s: Not applicable

> @ValidatesRunner test in Python postcommit
> --
>
> Key: BEAM-1077
> URL: https://issues.apache.org/jira/browse/BEAM-1077
> Project: Beam
>  Issue Type: Test
>  Components: sdk-py, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
> Fix For: Not applicable
>
>
> Modify run_postcommit.sh to have @ValidatesRunner tests running on service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1077) @ValidatesRunner test in Python postcommit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733396#comment-15733396
 ] 

ASF GitHub Bot commented on BEAM-1077:
--

Github user markflyhigh closed the pull request at:

https://github.com/apache/incubator-beam/pull/1492


> @ValidatesRunner test in Python postcommit
> --
>
> Key: BEAM-1077
> URL: https://issues.apache.org/jira/browse/BEAM-1077
> Project: Beam
>  Issue Type: Test
>  Components: sdk-py, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
> Fix For: Not applicable
>
>
> Modify run_postcommit.sh to have @ValidatesRunner tests running on service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1492: [BEAM-1077] @ValidatesRunner Test in Pyth...

[markflyhigh]

Github user markflyhigh closed the pull request at:

https://github.com/apache/incubator-beam/pull/1492


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Flink #1045

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Flink #1045

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: This closes #1555

[kenn]

This closes #1555


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e292032c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e292032c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e292032c

Branch: refs/heads/master
Commit: e292032cbe3defc5c61fb171a00dd8391a27afd8
Parents: 0bfa02d a11eb63
Author: Kenneth Knowles 
Authored: Thu Dec 8 12:46:37 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 12:46:37 2016 -0800

--
 runners/flink/runner/pom.xml | 1 +
 1 file changed, 1 insertion(+)
--

[GitHub] incubator-beam pull request #1555: [BEAM-27] Fix exclusion of UsesTimersInPa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1555


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Fix exclusion of UsesTimersInParDo for FlinkRunner streaming tests

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 0bfa02dd2 -> e292032cb


Fix exclusion of UsesTimersInParDo for FlinkRunner streaming tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a11eb637
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a11eb637
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a11eb637

Branch: refs/heads/master
Commit: a11eb637f6f1c7bdc41b24154cb3893bb194b096
Parents: 0bfa02d
Author: Kenneth Knowles 
Authored: Thu Dec 8 12:42:03 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 12:42:03 2016 -0800

--
 runners/flink/runner/pom.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a11eb637/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 9509476..37de899 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -85,6 +85,7 @@
   
org.apache.beam.sdk.testing.RunnableOnService
   
 org.apache.beam.sdk.testing.UsesStatefulParDo,
+org.apache.beam.sdk.testing.UsesTimersInParDo,
 org.apache.beam.sdk.testing.UsesSplittableParDo
   
   none

[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733347#comment-15733347
 ] 

ASF GitHub Bot commented on BEAM-27:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1555

[BEAM-27] Fix exclusion of UsesTimersInParDo for FlinkRunner streaming tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: anyone (fixing postcommit) ping @aljoscha @mxm 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam FlinkRunner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1555.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1555


commit a11eb637f6f1c7bdc41b24154cb3893bb194b096
Author: Kenneth Knowles 
Date:   2016-12-08T20:42:03Z

Fix exclusion of UsesTimersInParDo for FlinkRunner streaming tests




> Add user-ready API for interacting with timers
> --
>
> Key: BEAM-27
> URL: https://issues.apache.org/jira/browse/BEAM-27
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> Pipeline authors will benefit from a different factorization of interaction 
> with underlying timers. The current APIs are targeted at runner implementers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1555: [BEAM-27] Fix exclusion of UsesTimersInPa...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1555

[BEAM-27] Fix exclusion of UsesTimersInParDo for FlinkRunner streaming tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: anyone (fixing postcommit) ping @aljoscha @mxm 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam FlinkRunner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1555.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1555


commit a11eb637f6f1c7bdc41b24154cb3893bb194b096
Author: Kenneth Knowles 
Date:   2016-12-08T20:42:03Z

Fix exclusion of UsesTimersInParDo for FlinkRunner streaming tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1044

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1044

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-1111) Reject timers for ParDo in SparkRunner streaming evaluators.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-.
-
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> Reject timers for ParDo in SparkRunner streaming evaluators.
> 
>
> Key: BEAM-
> URL: https://issues.apache.org/jira/browse/BEAM-
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1111) Reject timers for ParDo in SparkRunner streaming evaluators.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733242#comment-15733242
 ] 

ASF GitHub Bot commented on BEAM-:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1553


> Reject timers for ParDo in SparkRunner streaming evaluators.
> 
>
> Key: BEAM-
> URL: https://issues.apache.org/jira/browse/BEAM-
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1553

[kenn]

This closes #1553


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0bfa02dd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0bfa02dd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0bfa02dd

Branch: refs/heads/master
Commit: 0bfa02dd26a7fb80753da1ed130acff1265d093a
Parents: 96f9fce 95e2c53
Author: Kenneth Knowles 
Authored: Thu Dec 8 11:54:33 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 11:54:33 2016 -0800

--
 .../spark/translation/TransformTranslator.java  | 28 +
 .../spark/translation/TranslationUtils.java | 33 
 .../streaming/StreamingTransformTranslator.java |  6 
 3 files changed, 40 insertions(+), 27 deletions(-)
--

[GitHub] incubator-beam pull request #1553: [BEAM-1111] Reject timers for ParDo in Sp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1553


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-1111] Reject timers for ParDo in SparkRunner streaming evaluators

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 96f9fce78 -> 0bfa02dd2


[BEAM-] Reject timers for ParDo in SparkRunner streaming evaluators


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/95e2c53d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/95e2c53d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/95e2c53d

Branch: refs/heads/master
Commit: 95e2c53db535952aaf0c335e0d3d27a721c6b55d
Parents: 96f9fce
Author: Sela 
Authored: Thu Dec 8 20:29:35 2016 +0200
Committer: Kenneth Knowles 
Committed: Thu Dec 8 11:54:19 2016 -0800

--
 .../spark/translation/TransformTranslator.java  | 28 +
 .../spark/translation/TranslationUtils.java | 33 
 .../streaming/StreamingTransformTranslator.java |  6 
 3 files changed, 40 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/95e2c53d/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
index 8170366..964eb37 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
@@ -23,6 +23,7 @@ import static 
org.apache.beam.runners.spark.io.hadoop.ShardNameBuilder.getOutput
 import static 
org.apache.beam.runners.spark.io.hadoop.ShardNameBuilder.getOutputFilePrefix;
 import static 
org.apache.beam.runners.spark.io.hadoop.ShardNameBuilder.getOutputFileTemplate;
 import static 
org.apache.beam.runners.spark.io.hadoop.ShardNameBuilder.replaceShardCount;
+import static 
org.apache.beam.runners.spark.translation.TranslationUtils.rejectStateAndTimers;
 
 import com.google.common.collect.Maps;
 import java.io.IOException;
@@ -32,7 +33,6 @@ import org.apache.avro.mapred.AvroKey;
 import org.apache.avro.mapreduce.AvroJob;
 import org.apache.avro.mapreduce.AvroKeyInputFormat;
 import org.apache.beam.runners.core.AssignWindowsDoFn;
-import org.apache.beam.runners.spark.SparkRunner;
 import org.apache.beam.runners.spark.aggregators.NamedAggregators;
 import org.apache.beam.runners.spark.aggregators.SparkAggregators;
 import org.apache.beam.runners.spark.coders.CoderHelpers;
@@ -58,8 +58,6 @@ import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
-import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
-import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
@@ -228,30 +226,6 @@ public final class TransformTranslator {
 };
   }
 
-  private static void rejectStateAndTimers(DoFn doFn) {
-DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
-
-if (signature.stateDeclarations().size() > 0) {
-  throw new UnsupportedOperationException(
-  String.format(
-  "Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
-  DoFn.StateId.class.getSimpleName(),
-  doFn.getClass().getName(),
-  DoFn.class.getSimpleName(),
-  SparkRunner.class.getSimpleName()));
-}
-
-if (signature.timerDeclarations().size() > 0) {
-  throw new UnsupportedOperationException(
-  String.format(
-  "Found %s annotations on %s, but %s cannot yet be used with 
timers in the %s.",
-  DoFn.TimerId.class.getSimpleName(),
-  doFn.getClass().getName(),
-  DoFn.class.getSimpleName(),
-  SparkRunner.class.getSimpleName()));
-}
-  }
-
   private static  TransformEvaluator> parDo() {
 return new TransformEvaluator>() {
   @Override

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/95e2c53d/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java
index 

Build failed in Jenkins: beam_PostCommit_Python_Verify #846

[Apache Jenkins Server]

See 

Changes:

[robertwb] Handle empty batches in GcsIO batch methods

[robertwb] Fix a typo in query split error handling

--
[...truncated 3106 lines...]
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 94, in test_par_do_with_side_input_as_arg
pipeline.run()
  File 
"
 line 159, in run
return self.runner.run(self)
  File 
"
 line 175, in run
pipeline.options, job_version)
  File 
"
 line 360, in __init__
credentials = get_service_credentials()
  File 
"
 line 171, in get_service_credentials
credentials.get_access_token()
  File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 677, in 
get_access_token
self.refresh(http)
  File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 560, in 
refresh
self._refresh(http.request)
  File 
"
 line 109, in _refresh
['gcloud', 'auth', 'print-access-token'], stdout=processes.PIPE)
  File 
"
 line 52, in Popen
return subprocess.Popen(*args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1223, in _execute_child
self.pid = os.fork()
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_par_do_with_side_input_as_arg 
(apache_beam.dataflow_test.DataflowTest)'

==
ERROR: test_par_do_with_side_input_as_keyword_arg 
(apache_beam.dataflow_test.DataflowTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 108, in test_par_do_with_side_input_as_keyword_arg
pipeline.run()
  File 
"
 line 159, in run
return self.runner.run(self)
  File 
"
 line 175, in run
pipeline.options, job_version)
  File 
"
 line 360, in __init__
credentials = get_service_credentials()
  File 
"
 line 171, in get_service_credentials

Build failed in Jenkins: beam_PostCommit_Python_Verify #845

[Apache Jenkins Server]

See 

Changes:

[robertwb] Add reference to the >> and | operators for pipelines.

--
[...truncated 2966 lines...]
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.dataflow_test.DataflowTest)'

==
ERROR: test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.dataflow_test.DataflowTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 147, in test_par_do_with_multiple_outputs_and_using_yield
pipeline.run()
  File 
"
 line 159, in run
return self.runner.run(self)
  File 
"
 line 175, in run
pipeline.options, job_version)
  File 
"
 line 360, in __init__
credentials = get_service_credentials()
  File 
"
 line 171, in get_service_credentials
credentials.get_access_token()
  File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 677, in 
get_access_token
self.refresh(http)
  File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 560, in 
refresh
self._refresh(http.request)
  File 
"
 line 113, in _refresh
output, _ = gcloud_process.communicate()
  File "/usr/lib/python2.7/subprocess.py", line 791, in communicate
stdout = _eintr_retry_call(self.stdout.read)
  File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call
return func(*args)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.dataflow_test.DataflowTest)'

==
ERROR: test_par_do_with_side_input_as_arg 
(apache_beam.dataflow_test.DataflowTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 94, in test_par_do_with_side_input_as_arg
pipeline.run()
  File 
"
 line 159, in run
return 

[2/2] incubator-beam git commit: Fix a typo in query split error handling

[robertwb]

Fix a typo in query split error handling


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d6afb906
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d6afb906
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d6afb906

Branch: refs/heads/python-sdk
Commit: d6afb90690f5be2f3cb38d68dc1d49a1b551118e
Parents: 1392f70
Author: Vikas Kedigehalli 
Authored: Wed Dec 7 14:19:26 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 8 11:22:31 2016 -0800

--
 .../apache_beam/io/datastore/v1/datastoreio.py  |  2 +-
 .../io/datastore/v1/datastoreio_test.py | 29 
 2 files changed, 30 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d6afb906/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
--
diff --git a/sdks/python/apache_beam/io/datastore/v1/datastoreio.py 
b/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
index fc3e813..a86bb0b 100644
--- a/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
+++ b/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
@@ -181,7 +181,7 @@ class ReadFromDatastore(PTransform):
   except Exception:
 logging.warning("Unable to parallelize the given query: %s", query,
 exc_info=True)
-query_splits = [(key, query)]
+query_splits = [query]
 
   sharded_query_splits = []
   for split_query in query_splits:

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d6afb906/sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py
--
diff --git a/sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py 
b/sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py
index 2ac7ffb..f80a320 100644
--- a/sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py
+++ b/sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py
@@ -122,6 +122,35 @@ class DatastoreioTest(unittest.TestCase):
   self.assertEqual(1, len(returned_split_queries))
   self.assertEqual(0, len(self._mock_datastore.method_calls))
 
+  def test_SplitQueryFn_with_exception(self):
+"""A test that verifies that no split is performed when failures occur."""
+with patch.object(helper, 'get_datastore',
+  return_value=self._mock_datastore):
+  # Force SplitQueryFn to compute the number of query splits
+  num_splits = 0
+  expected_num_splits = 1
+  entity_bytes = (expected_num_splits *
+  ReadFromDatastore._DEFAULT_BUNDLE_SIZE_BYTES)
+  with patch.object(ReadFromDatastore, 'get_estimated_size_bytes',
+return_value=entity_bytes):
+
+with patch.object(query_splitter, 'get_splits',
+  side_effect=ValueError("Testing query split error")):
+  split_query_fn = ReadFromDatastore.SplitQueryFn(
+  self._PROJECT, self._query, None, num_splits)
+  mock_context = MagicMock()
+  mock_context.element = self._query
+  split_query_fn.start_bundle(mock_context)
+  returned_split_queries = []
+  for split_query in split_query_fn.process(mock_context):
+returned_split_queries.append(split_query)
+
+  self.assertEqual(len(returned_split_queries), expected_num_splits)
+  self.assertEqual(returned_split_queries[0][1], self._query)
+  self.assertEqual(0,
+   len(self._mock_datastore.run_query.call_args_list))
+  self.verify_unique_keys(returned_split_queries)
+
   def test_DatastoreWriteFn_with_emtpy_batch(self):
 self.check_DatastoreWriteFn(0)
 

[1/2] incubator-beam git commit: Closes #1542

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 1392f70b6 -> 44c1586f3


Closes #1542


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/44c1586f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/44c1586f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/44c1586f

Branch: refs/heads/python-sdk
Commit: 44c1586f3815f957a43037309f2a46a1766c6516
Parents: 1392f70 d6afb90
Author: Robert Bradshaw 
Authored: Thu Dec 8 11:22:31 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 8 11:22:31 2016 -0800

--
 .../apache_beam/io/datastore/v1/datastoreio.py  |  2 +-
 .../io/datastore/v1/datastoreio_test.py | 29 
 2 files changed, 30 insertions(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Handle empty batches in GcsIO batch methods

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 75be6e974 -> 1392f70b6


Handle empty batches in GcsIO batch methods


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3ef83b33
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3ef83b33
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3ef83b33

Branch: refs/heads/python-sdk
Commit: 3ef83b3396a4574b3283b29ba1f878b31badd612
Parents: 75be6e9
Author: Charles Chen 
Authored: Wed Dec 7 15:03:01 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 8 11:18:51 2016 -0800

--
 sdks/python/apache_beam/io/gcsio.py  | 4 
 sdks/python/apache_beam/io/gcsio_test.py | 4 
 2 files changed, 8 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3ef83b33/sdks/python/apache_beam/io/gcsio.py
--
diff --git a/sdks/python/apache_beam/io/gcsio.py 
b/sdks/python/apache_beam/io/gcsio.py
index 748465f..f150c4c 100644
--- a/sdks/python/apache_beam/io/gcsio.py
+++ b/sdks/python/apache_beam/io/gcsio.py
@@ -204,6 +204,8 @@ class GcsIO(object):
  argument, where exception is None if the operation succeeded or
  the relevant exception if the operation failed.
 """
+if not paths:
+  return []
 batch_request = BatchApiRequest(
 retryable_codes=retry.SERVER_ERROR_OR_TIMEOUT_CODES)
 for path in paths:
@@ -264,6 +266,8 @@ class GcsIO(object):
  src_dest_pairs argument, where exception is None if the operation
  succeeded or the relevant exception if the operation failed.
 """
+if not src_dest_pairs:
+  return []
 batch_request = BatchApiRequest(
 retryable_codes=retry.SERVER_ERROR_OR_TIMEOUT_CODES)
 for src, dest in src_dest_pairs:

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3ef83b33/sdks/python/apache_beam/io/gcsio_test.py
--
diff --git a/sdks/python/apache_beam/io/gcsio_test.py 
b/sdks/python/apache_beam/io/gcsio_test.py
index 5af13c6..bd7eb51 100644
--- a/sdks/python/apache_beam/io/gcsio_test.py
+++ b/sdks/python/apache_beam/io/gcsio_test.py
@@ -265,6 +265,10 @@ class TestGCSIO(unittest.TestCase):
 with self.assertRaises(ValueError):
   self.gcs.open(file_name, 'r+b')
 
+  def test_empty_batches(self):
+self.assertEqual([], self.gcs.copy_batch([]))
+self.assertEqual([], self.gcs.delete_batch([]))
+
   def test_delete(self):
 file_name = 'gs://gcsio-test/delete_me'
 file_size = 1024

[2/2] incubator-beam git commit: Closes #1544

[robertwb]

Closes #1544


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1392f70b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1392f70b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1392f70b

Branch: refs/heads/python-sdk
Commit: 1392f70b6ee2db76e333fad16efe0cfdd00e7175
Parents: 75be6e9 3ef83b3
Author: Robert Bradshaw 
Authored: Thu Dec 8 11:18:52 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 8 11:18:52 2016 -0800

--
 sdks/python/apache_beam/io/gcsio.py  | 4 
 sdks/python/apache_beam/io/gcsio_test.py | 4 
 2 files changed, 8 insertions(+)
--

[1/2] incubator-beam git commit: Add reference to the >> and | operators for pipelines.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 43057960a -> 75be6e974


Add reference to the >> and | operators for pipelines.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3510ff99
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3510ff99
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3510ff99

Branch: refs/heads/python-sdk
Commit: 3510ff99da9cd149e67e8fdb12b942689374b2d7
Parents: 4305796
Author: Maria Garcia Herrero 
Authored: Tue Dec 6 21:05:47 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 8 11:14:51 2016 -0800

--
 sdks/python/README.md | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3510ff99/sdks/python/README.md
--
diff --git a/sdks/python/README.md b/sdks/python/README.md
index cff497c..820084d 100644
--- a/sdks/python/README.md
+++ b/sdks/python/README.md
@@ -163,8 +163,9 @@ The following examples demonstrate some basic, fundamental 
concepts for using Ap
 
 A basic pipeline will take as input an iterable, apply the
 beam.Create `PTransform`, and produce a `PCollection` that can
-be written to a file or modified by further `PTransform`s. The
-pipe operator allows to chain `PTransform`s.
+be written to a file or modified by further `PTransform`s.
+The `>>` operator is used to label `PTransform`s and
+the `|` operator is used to chain them.
 
 ```python
 # Standard imports

[2/2] incubator-beam git commit: Closes #1521

[robertwb]

Closes #1521


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/75be6e97
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/75be6e97
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/75be6e97

Branch: refs/heads/python-sdk
Commit: 75be6e974831f73ea935fe1f52fd7091a03c8928
Parents: 4305796 3510ff9
Author: Robert Bradshaw 
Authored: Thu Dec 8 11:14:52 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 8 11:14:52 2016 -0800

--
 sdks/python/README.md | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--

[GitHub] incubator-beam pull request #1554: Change Dataflow profiling option to saveP...

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/incubator-beam/pull/1554

Change Dataflow profiling option to saveProfilesToGcs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [*] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [*] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [*] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/incubator-beam profiling

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1554.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1554


commit a57bf6a147ad1d05ed1f0d3f5981a6296b50bd35
Author: bchambers 
Date:   2016-12-08T18:40:17Z

Change Dataflow profiling option to saveProfilesToGcs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1553: [BEAM-1111] Reject timers for ParDo in Sp...

[amitsela]

GitHub user amitsela opened a pull request:

https://github.com/apache/incubator-beam/pull/1553

[BEAM-] Reject timers for ParDo in SparkRunner streaming evaluators.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/incubator-beam BEAM-

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1553.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1553


commit 0c948c5dce65b9651c576f34f17922184816a703
Author: Sela 
Date:   2016-12-08T18:29:35Z

[BEAM-] Reject timers for ParDo in SparkRunner streaming evaluators.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1111) Reject timers for ParDo in SparkRunner streaming evaluators.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733017#comment-15733017
 ] 

ASF GitHub Bot commented on BEAM-:
--

GitHub user amitsela opened a pull request:

https://github.com/apache/incubator-beam/pull/1553

[BEAM-] Reject timers for ParDo in SparkRunner streaming evaluators.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/incubator-beam BEAM-

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1553.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1553


commit 0c948c5dce65b9651c576f34f17922184816a703
Author: Sela 
Date:   2016-12-08T18:29:35Z

[BEAM-] Reject timers for ParDo in SparkRunner streaming evaluators.




> Reject timers for ParDo in SparkRunner streaming evaluators.
> 
>
> Key: BEAM-
> URL: https://issues.apache.org/jira/browse/BEAM-
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1111) Reject timers for ParDo in SparkRunner streaming evaluators.

[Amit Sela (JIRA)]

Amit Sela created BEAM-:
---

 Summary: Reject timers for ParDo in SparkRunner streaming 
evaluators.
 Key: BEAM-
 URL: https://issues.apache.org/jira/browse/BEAM-
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Amit Sela
Assignee: Amit Sela






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Flink #1043

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Flink #1043

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1550: [BEAM-27] Reject timers for ParDo in each...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1550


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732940#comment-15732940
 ] 

ASF GitHub Bot commented on BEAM-27:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1550


> Add user-ready API for interacting with timers
> --
>
> Key: BEAM-27
> URL: https://issues.apache.org/jira/browse/BEAM-27
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> Pipeline authors will benefit from a different factorization of interaction 
> with underlying timers. The current APIs are targeted at runner implementers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[04/13] incubator-beam git commit: Reject timers for ParDo in DirectRunner

[kenn]

Reject timers for ParDo in DirectRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/274f17f0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/274f17f0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/274f17f0

Branch: refs/heads/master
Commit: 274f17f0c0df08785a78d9a60c22d5556e46584a
Parents: 29f3af3
Author: Kenneth Knowles 
Authored: Wed Dec 7 20:37:33 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 09:53:08 2016 -0800

--
 .../runners/direct/ParDoMultiOverrideFactory.java | 14 +++---
 1 file changed, 11 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/274f17f0/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
index 8c96e9b..4e7914f 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
@@ -50,9 +50,17 @@ class ParDoMultiOverrideFactory
 DoFnSignature signature = DoFnSignatures.getSignature(fn.getClass());
 if (signature.processElement().isSplittable()) {
   return new SplittableParDo(transform);
-} else if (signature.stateDeclarations().size() > 0
-|| signature.timerDeclarations().size() > 0) {
-
+} else if (signature.timerDeclarations().size() > 0) {
+  // Temporarily actually reject timers
+  throw new UnsupportedOperationException(
+  String.format(
+  "Found %s annotations on %s, but %s cannot yet be used with 
timers in the %s.",
+  DoFn.TimerId.class.getSimpleName(),
+  fn.getClass().getName(),
+  DoFn.class.getSimpleName(),
+  DirectRunner.class.getSimpleName()));
+
+} else if (signature.stateDeclarations().size() > 0) {
   // Based on the fact that the signature is stateful, DoFnSignatures 
ensures
   // that it is also keyed
   ParDo.BoundMulti keyedTransform =

[12/13] incubator-beam git commit: Reject timers for ParDo in FlinkRunner

[kenn]

Reject timers for ParDo in FlinkRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/69e0ea25
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/69e0ea25
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/69e0ea25

Branch: refs/heads/master
Commit: 69e0ea25f24597b84c93137dd94e2f25a9b88a15
Parents: 18db3ac
Author: Kenneth Knowles 
Authored: Wed Dec 7 20:34:59 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 09:53:08 2016 -0800

--
 .../FlinkBatchTransformTranslators.java | 46 
 .../FlinkStreamingTransformTranslators.java | 45 +++
 2 files changed, 54 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/69e0ea25/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
index 474d4e3..9ac907f 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
@@ -56,6 +56,7 @@ import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.join.RawUnionValue;
 import org.apache.beam.sdk.transforms.join.UnionCoder;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
@@ -483,6 +484,30 @@ class FlinkBatchTransformTranslators {
 }
   }
 
+  private static void rejectStateAndTimers(DoFn doFn) {
+DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
+
+if (signature.stateDeclarations().size() > 0) {
+  throw new UnsupportedOperationException(
+  String.format(
+  "Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
+  DoFn.StateId.class.getSimpleName(),
+  doFn.getClass().getName(),
+  DoFn.class.getSimpleName(),
+  FlinkRunner.class.getSimpleName()));
+}
+
+if (signature.timerDeclarations().size() > 0) {
+  throw new UnsupportedOperationException(
+  String.format(
+  "Found %s annotations on %s, but %s cannot yet be used with 
timers in the %s.",
+  DoFn.TimerId.class.getSimpleName(),
+  doFn.getClass().getName(),
+  DoFn.class.getSimpleName(),
+  FlinkRunner.class.getSimpleName()));
+}
+  }
+
   private static class ParDoBoundTranslatorBatch
   implements FlinkBatchPipelineTranslator.BatchTransformTranslator<
   ParDo.Bound> {
@@ -493,15 +518,7 @@ class FlinkBatchTransformTranslators {
 
 FlinkBatchTranslationContext context) {
   DoFn doFn = transform.getNewFn();
-  if 
(DoFnSignatures.getSignature(doFn.getClass()).stateDeclarations().size() > 0) {
-throw new UnsupportedOperationException(
-String.format(
-"Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
-DoFn.StateId.class.getSimpleName(),
-doFn.getClass().getName(),
-DoFn.class.getSimpleName(),
-FlinkRunner.class.getSimpleName()));
-  }
+  rejectStateAndTimers(doFn);
 
   DataSet inputDataSet =
   context.getInputDataSet(context.getInput(transform));
@@ -549,16 +566,7 @@ class FlinkBatchTransformTranslators {
 ParDo.BoundMulti transform,
 FlinkBatchTranslationContext context) {
   DoFn doFn = transform.getNewFn();
-  if 
(DoFnSignatures.getSignature(doFn.getClass()).stateDeclarations().size() > 0) {
-throw new UnsupportedOperationException(
-String.format(
-"Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
-DoFn.StateId.class.getSimpleName(),
-doFn.getClass().getName(),
-DoFn.class.getSimpleName(),
-FlinkRunner.class.getSimpleName()));
-  }
-
+  rejectStateAndTimers(doFn);
   DataSet inputDataSet =
   

[05/13] incubator-beam git commit: Reject timers for ParDo in ApexRunner

[kenn]

Reject timers for ParDo in ApexRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/18db3ace
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/18db3ace
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/18db3ace

Branch: refs/heads/master
Commit: 18db3ace77e89203d7ec3f342fe6ce24a2119226
Parents: 50ffc7b
Author: Kenneth Knowles 
Authored: Wed Dec 7 20:34:34 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 09:53:08 2016 -0800

--
 .../apex/translation/ParDoBoundMultiTranslator.java | 16 +++-
 .../apex/translation/ParDoBoundTranslator.java  | 16 +++-
 2 files changed, 30 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/18db3ace/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
index fed5f4b..706482a 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
@@ -33,6 +33,7 @@ import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
 import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
@@ -54,7 +55,9 @@ class ParDoBoundMultiTranslator
   @Override
   public void translate(ParDo.BoundMulti transform, 
TranslationContext context) {
 DoFn doFn = transform.getNewFn();
-if 
(DoFnSignatures.getSignature(doFn.getClass()).stateDeclarations().size() > 0) {
+DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
+
+if (signature.stateDeclarations().size() > 0) {
   throw new UnsupportedOperationException(
   String.format(
   "Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
@@ -63,6 +66,17 @@ class ParDoBoundMultiTranslator
   DoFn.class.getSimpleName(),
   ApexRunner.class.getSimpleName()));
 }
+
+if (signature.timerDeclarations().size() > 0) {
+  throw new UnsupportedOperationException(
+  String.format(
+  "Found %s annotations on %s, but %s cannot yet be used with 
timers in the %s.",
+  DoFn.TimerId.class.getSimpleName(),
+  doFn.getClass().getName(),
+  DoFn.class.getSimpleName(),
+  ApexRunner.class.getSimpleName()));
+}
+
 OldDoFn oldDoFn = transform.getFn();
 PCollectionTuple output = context.getOutput();
 PCollection input = context.getInput();

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/18db3ace/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
index 7a918a7..b5a50f6 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
@@ -25,6 +25,7 @@ import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
 import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
@@ -41,7 +42,9 @@ class ParDoBoundTranslator
   @Override
   public void translate(ParDo.Bound transform, 
TranslationContext context) {
 DoFn doFn = transform.getNewFn();
-if 
(DoFnSignatures.getSignature(doFn.getClass()).stateDeclarations().size() > 0) {
+DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
+

[10/13] incubator-beam git commit: Reject timers for ParDo in SparkRunner

[kenn]

Reject timers for ParDo in SparkRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/29f3af30
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/29f3af30
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/29f3af30

Branch: refs/heads/master
Commit: 29f3af30a4b871244e14998d670b8ca26bd8de94
Parents: 69e0ea2
Author: Kenneth Knowles 
Authored: Wed Dec 7 20:35:08 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 09:53:08 2016 -0800

--
 .../spark/translation/TransformTranslator.java  | 46 
 1 file changed, 27 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/29f3af30/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
index e033ab1..8170366 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
@@ -58,6 +58,7 @@ import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.Window;
@@ -81,7 +82,6 @@ import org.apache.spark.api.java.JavaRDD;
 import org.apache.spark.api.java.JavaSparkContext;
 import org.apache.spark.api.java.function.Function;
 import org.apache.spark.api.java.function.PairFunction;
-
 import scala.Tuple2;
 
 
@@ -228,20 +228,36 @@ public final class TransformTranslator {
 };
   }
 
+  private static void rejectStateAndTimers(DoFn doFn) {
+DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
+
+if (signature.stateDeclarations().size() > 0) {
+  throw new UnsupportedOperationException(
+  String.format(
+  "Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
+  DoFn.StateId.class.getSimpleName(),
+  doFn.getClass().getName(),
+  DoFn.class.getSimpleName(),
+  SparkRunner.class.getSimpleName()));
+}
+
+if (signature.timerDeclarations().size() > 0) {
+  throw new UnsupportedOperationException(
+  String.format(
+  "Found %s annotations on %s, but %s cannot yet be used with 
timers in the %s.",
+  DoFn.TimerId.class.getSimpleName(),
+  doFn.getClass().getName(),
+  DoFn.class.getSimpleName(),
+  SparkRunner.class.getSimpleName()));
+}
+  }
+
   private static  TransformEvaluator> parDo() {
 return new TransformEvaluator>() {
   @Override
   public void evaluate(ParDo.Bound transform, 
EvaluationContext context) {
 DoFn doFn = transform.getNewFn();
-if 
(DoFnSignatures.getSignature(doFn.getClass()).stateDeclarations().size() > 0) {
-  throw new UnsupportedOperationException(
-  String.format(
-  "Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
-  DoFn.StateId.class.getSimpleName(),
-  doFn.getClass().getName(),
-  DoFn.class.getSimpleName(),
-  SparkRunner.class.getSimpleName()));
-}
+rejectStateAndTimers(doFn);
 @SuppressWarnings("unchecked")
 JavaRDD inRDD =
 ((BoundedDataset) 
context.borrowDataset(transform)).getRDD();
@@ -265,15 +281,7 @@ public final class TransformTranslator {
   @Override
   public void evaluate(ParDo.BoundMulti transform, 
EvaluationContext context) {
 DoFn doFn = transform.getNewFn();
-if 
(DoFnSignatures.getSignature(doFn.getClass()).stateDeclarations().size() > 0) {
-  throw new UnsupportedOperationException(
-  String.format(
-  "Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
-  DoFn.StateId.class.getSimpleName(),
-  doFn.getClass().getName(),
-  DoFn.class.getSimpleName(),
-

[09/13] incubator-beam git commit: No longer reject timers in ParDo

[kenn]

No longer reject timers in ParDo


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c22e2a43
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c22e2a43
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c22e2a43

Branch: refs/heads/master
Commit: c22e2a435113c9653b58f1040a4e9266059767f4
Parents: 274f17f
Author: Kenneth Knowles 
Authored: Wed Dec 7 20:04:51 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 09:53:08 2016 -0800

--
 .../main/java/org/apache/beam/sdk/transforms/ParDo.java   | 10 --
 1 file changed, 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c22e2a43/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index 4f7491e..e60c536 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -594,16 +594,6 @@ public class ParDo {
   private static  void validate(DoFn fn) {
 DoFnSignature signature = DoFnSignatures.getSignature((Class) 
fn.getClass());
 
-// To be removed when the features are complete and runners have their own 
adequate
-// rejection logic
-if (!signature.timerDeclarations().isEmpty()) {
-  throw new UnsupportedOperationException(
-  String.format("Found %s annotations on %s, but %s cannot yet be used 
with timers.",
-  DoFn.TimerId.class.getSimpleName(),
-  fn.getClass().getName(),
-  DoFn.class.getSimpleName()));
-}
-
 // State is semantically incompatible with splitting
 if (!signature.stateDeclarations().isEmpty() && 
signature.processElement().isSplittable()) {
   throw new UnsupportedOperationException(

[08/13] incubator-beam git commit: Disable tests for timers in ParDo for Dataflow runner

[kenn]

Disable tests for timers in ParDo for Dataflow runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5a8f9565
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5a8f9565
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5a8f9565

Branch: refs/heads/master
Commit: 5a8f9565d82dd813428f08d97f975c69b28a58ee
Parents: 320f888
Author: Kenneth Knowles 
Authored: Tue Dec 6 20:53:16 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 09:53:08 2016 -0800

--
 runners/google-cloud-dataflow-java/pom.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5a8f9565/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 9ead74a..1543c0e 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -78,6 +78,7 @@
 runnable-on-service-tests
 
   
+org.apache.beam.sdk.testing.UsesTimersInParDo,
 org.apache.beam.sdk.testing.UsesSplittableParDo
   
   

[07/13] incubator-beam git commit: Disables tests for timers in ParDo for Spark runner

[kenn]

Disables tests for timers in ParDo for Spark runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/320f8881
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/320f8881
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/320f8881

Branch: refs/heads/master
Commit: 320f8881e91628fa843e0eb2918d2de7a7aae329
Parents: 2f13fe4
Author: Kenneth Knowles 
Authored: Tue Dec 6 20:53:05 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 8 09:53:08 2016 -0800

--
 runners/spark/pom.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/320f8881/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index 9a3adf6..d1ef225 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -74,6 +74,7 @@
   
org.apache.beam.sdk.testing.RunnableOnService
   
 org.apache.beam.sdk.testing.UsesStatefulParDo,
+org.apache.beam.sdk.testing.UsesTimersInParDo,
 org.apache.beam.sdk.testing.UsesSplittableParDo
   
   1