Jenkins build is back to stable : beam_PostCommit_RunnableOnService_SparkLocal #117

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-959) JdbcIO validation messages need improvement

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-959:
-

Assignee: Jean-Baptiste Onofré  (was: James Malone)

> JdbcIO validation messages need improvement
> ---
>
> Key: BEAM-959
> URL: https://issues.apache.org/jira/browse/BEAM-959
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>
> The error messages upon validation of {{JdbcIO}} should be sentences that 
> explain to a user how to rectify the situation. As it is, they look like 
> internal SDK errors.
> Here is where they are: 
> https://github.com/apache/incubator-beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L319
> And here is how a user sees them: 
> http://stackoverflow.com/questions/40525269/jdbc-fetch-from-oracle-with-beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-960) JdbcIO should infer coder when possible (such as via RowMapper)

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656256#comment-15656256
 ] 

Jean-Baptiste Onofré commented on BEAM-960:
---

Thanks [~kenn], it makes sense. I will do the improvement (I think we can 
consider as an improvement more than a bug in Jira, right ?).

> JdbcIO should infer coder when possible (such as via RowMapper)
> ---
>
> Key: BEAM-960
> URL: https://issues.apache.org/jira/browse/BEAM-960
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>
> There are a few ways that we typically do this:
> # Inspect the static args of the class implementing {{RowMapper}} to 
> determine the output type, then consult the coder registry. This method is 
> more limited and not lambda-friendly.
> # For the a "functional interface" version, like {{RowMapper}} that allows 
> setting the output type descriptor.
> # Also provide an abstract class-based version, for example accepting a 
> {{SimpleFunction}} that will automatically have a rich output type descriptor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-960) JdbcIO should infer coder when possible (such as via RowMapper)

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-960:
-

Assignee: Jean-Baptiste Onofré  (was: James Malone)

> JdbcIO should infer coder when possible (such as via RowMapper)
> ---
>
> Key: BEAM-960
> URL: https://issues.apache.org/jira/browse/BEAM-960
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>
> There are a few ways that we typically do this:
> # Inspect the static args of the class implementing {{RowMapper}} to 
> determine the output type, then consult the coder registry. This method is 
> more limited and not lambda-friendly.
> # For the a "functional interface" version, like {{RowMapper}} that allows 
> setting the output type descriptor.
> # Also provide an abstract class-based version, for example accepting a 
> {{SimpleFunction}} that will automatically have a rich output type descriptor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-912) Range join in Beam

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-912:
-
Assignee: (was: Frances Perry)

> Range join in Beam
> --
>
> Key: BEAM-912
> URL: https://issues.apache.org/jira/browse/BEAM-912
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Jingsong Lee
> Attachments: betweenJoin.png
>
>
> 1.We can support some data-driven trigger, so we need expose data in 
> OnElementContext of onElement method. 
> 2.We can support more flexible join, so we need expose buffer tag in 
> TriggerContext, now this buffer tag is in SystemReduceFn.
> for example: SELECT STREAM * FROM Orders AS o JOIN Shipments AS s
> ON o.orderId = s.orderId AND s.rowtime BETWEEN o.rowtime AND o.rowtime + 
> INTERVAL '1' HOUR;
> link: https://issues.apache.org/jira/browse/BEAM-101



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-922) runner-core does not pass under findbugs

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-922.
--
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> runner-core does not pass under findbugs
> 
>
> Key: BEAM-922
> URL: https://issues.apache.org/jira/browse/BEAM-922
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
> Fix For: 0.4.0-incubating
>
>
> @Kenn because mostly in state machines. Since Findbugs doesn't grok 
> reflection, probably just need suppressions.
> {code}
> INFO] Dead store to invoker rather than field with same name in new 
> org.apache.beam.runners.core.SplittableParDo$ProcessFn(DoFn, Coder, Coder, 
> Coder) [org.apache.beam.runners.core.SplittableParDo$ProcessFn] At 
> SplittableParDo.java:[line 242]
> [INFO] Possible null pointer dereference of 
> UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource.reader
>  in 
> org.apache.beam.runners.core.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource.advance()
>  
> [org.apache.beam.runners.core.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource,
>  
> org.apache.beam.runners.core.UnboundedReadFromBoundedSource$BoundedToUnboundedSourceAdapter$ResidualSource]
>  Dereferenced at UnboundedReadFromBoundedSource.java:[line 476]Known null at 
> UnboundedReadFromBoundedSource.java:[line 472]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterAll)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[lines 130-135]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterEach)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[lines 154-161]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterFirst)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[lines 121-126]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterPane)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[line 139]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterProcessingTime)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[line 175]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterSynchronizedProcessingTime)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[line 117]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterWatermark$AfterWatermarkEarlyAndLate)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[lines 144-150]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(AfterWatermark$FromEndOfWindow)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[line 109]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(DefaultTrigger)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[line 101]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(Never$NeverTrigger)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[line 113]
> [INFO] Private method 
> org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter.evaluateSpecific(OrFinallyTrigger)
>  is never called 
> [org.apache.beam.runners.core.triggers.TriggerStateMachines$StateMachineConverter]
>  At TriggerStateMachines.java:[lines 169-171]
> [INFO] Private method 
> 

[jira] [Updated] (BEAM-912) Range join in Beam

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-912:
-
Assignee: Frances Perry  (was: Kenneth Knowles)

> Range join in Beam
> --
>
> Key: BEAM-912
> URL: https://issues.apache.org/jira/browse/BEAM-912
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Jingsong Lee
>Assignee: Frances Perry
> Attachments: betweenJoin.png
>
>
> 1.We can support some data-driven trigger, so we need expose data in 
> OnElementContext of onElement method. 
> 2.We can support more flexible join, so we need expose buffer tag in 
> TriggerContext, now this buffer tag is in SystemReduceFn.
> for example: SELECT STREAM * FROM Orders AS o JOIN Shipments AS s
> ON o.orderId = s.orderId AND s.rowtime BETWEEN o.rowtime AND o.rowtime + 
> INTERVAL '1' HOUR;
> link: https://issues.apache.org/jira/browse/BEAM-101



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-947) WindowedWordCountIT fails due to not setting "--output" pipeline options

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-947.
--
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> WindowedWordCountIT fails due to not setting "--output" pipeline options
> 
>
> Key: BEAM-947
> URL: https://issues.apache.org/jira/browse/BEAM-947
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-java
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.4.0-incubating
>
>
> Example failure: 
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4793/org.apache.beam$beam-examples-java/testReport/junit/org.apache.beam.examples/WindowedWordCountIT/testWindowedWordCountInStreaming/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-688) Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency error

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-688:
-
Description: 
The starter archetype has flaky dependencies. It is reported to fail reliably 
on repeatedly install.

{noformat}
[INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
beam-sdks-java-maven-archetypes-starter ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
{noformat}


  was:
When building repeatedly using {{mvn install}}, the second build will fail on 
target beam-sdks-java-maven-archetypes-starter with error:

{noformat}
[INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
beam-sdks-java-maven-archetypes-starter ---
[WARNING] Used undeclared dependencies found:
[WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
{noformat}

For some reason this only happens on the second build.


> Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency 
> error
> ---
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> The starter archetype has flaky dependencies. It is reported to fail reliably 
> on repeatedly install.
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-688) Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency error

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-688:
-
Priority: Major  (was: Minor)

> Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency 
> error
> ---
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>
> The starter archetype has flaky dependencies. It is reported to fail reliably 
> on repeatedly install.
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-688) Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency error

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-688:
-
Summary: Failure of beam-sdks-java-maven-archetypes-starter with undeclared 
dependency error  (was: Repeated 'mvn install' fails on 
beam-sdks-java-maven-archetypes-starter with undeclared dependency error)

> Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency 
> error
> ---
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> When building repeatedly using {{mvn install}}, the second build will fail on 
> target beam-sdks-java-maven-archetypes-starter with error:
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}
> For some reason this only happens on the second build.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-688) Repeated 'mvn install' fails on beam-sdks-java-maven-archetypes-starter with undeclared dependency error

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656070#comment-15656070
 ] 

Kenneth Knowles commented on BEAM-688:
--

I don't believe it has to be repeated. It causes flakes on Travis CI very 
frequently. Example: 
https://travis-ci.org/apache/incubator-beam/builds/174930789

> Repeated 'mvn install' fails on beam-sdks-java-maven-archetypes-starter with 
> undeclared dependency error
> 
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> When building repeatedly using {{mvn install}}, the second build will fail on 
> target beam-sdks-java-maven-archetypes-starter with error:
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}
> For some reason this only happens on the second build.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1347: [BEAM-25] Allow stateful DoFn. Implement ...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1347

[BEAM-25] Allow stateful DoFn. Implement for DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam DirectRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1347.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1347


commit e81390b39854f0835d22e9e0cdea6cb297d47643
Author: Kenneth Knowles 
Date:   2016-11-10T05:10:51Z

Add State parameter support to SimpleDoFnRunner

commit 67bc4b2e0c2e75969b16a471a4aa79bd9209bd9b
Author: Kenneth Knowles 
Date:   2016-11-10T05:59:15Z

Schedule window expiration state purge in DirectRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1346: DirectPipelineRunner bug fixes

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1346

DirectPipelineRunner bug fixes

- Execute empty [] | pipelines to the end.
- use pickler to serialize/deserialize DoFns instead of deepcopy similar to 
the othe execution environments.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1346.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1346


commit dd5f0114bddd4d0527d52daf71a2b3ae0301c68d
Author: Ahmet Altay 
Date:   2016-11-11T01:37:45Z

DirectPipelineRunner bug fixes.
- Execute empty [] | pipelines to the end.
- use pickler to serialize/deserialize DoFns instead of deepcopy
  similar to the othe execution environments.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #116

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1789

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-945) BigQueryIO.parseTableSpec is package-private

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655595#comment-15655595
 ] 

ASF GitHub Bot commented on BEAM-945:
-

GitHub user andrewsmartin opened a pull request:

https://github.com/apache/incubator-beam/pull/1345

[BEAM-945] Make BigQueryIO.parseTableSpec public

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewsmartin/incubator-beam 
fix-parse-table-spec

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1345.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1345


commit 2ba3ab20363a6160d3bf42b8c321fc299babfcef
Author: Andrew Martin 
Date:   2016-11-08T20:09:50Z

Make BigQueryIO.parseTableSpec public




> BigQueryIO.parseTableSpec is package-private
> 
>
> Key: BEAM-945
> URL: https://issues.apache.org/jira/browse/BEAM-945
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 0.3.0-incubating
>Reporter: Andrew Martin
>Assignee: Daniel Halperin
>Priority: Blocker
>
> It was changed in this commit. 
> https://github.com/apache/incubator-beam/commit/dbbcbe604e167b306feac2443bec85f2da3c1dd6.
> Is there a good reason for it to not be public? If not I will submit a PR to 
> change it back.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1345: [BEAM-945] Make BigQueryIO.parseTableSpec...

[andrewsmartin]

GitHub user andrewsmartin opened a pull request:

https://github.com/apache/incubator-beam/pull/1345

[BEAM-945] Make BigQueryIO.parseTableSpec public

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewsmartin/incubator-beam 
fix-parse-table-spec

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1345.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1345


commit 2ba3ab20363a6160d3bf42b8c321fc299babfcef
Author: Andrew Martin 
Date:   2016-11-08T20:09:50Z

Make BigQueryIO.parseTableSpec public




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #115

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1788

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-194) Create a walkthrough of Beam examples in mobile gaming domain

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655453#comment-15655453
 ] 

ASF GitHub Bot commented on BEAM-194:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1344

[BEAM-194] Add Java 8 examples archetype including mobile gaming example, 
with generator script

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam moar-examples

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1344.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1344


commit f4782aa2da8e5bbbfa70cb80db07419f717458d7
Author: Kenneth Knowles 
Date:   2016-11-10T22:47:37Z

Add script to generate the java8-examples archetype

commit f783c4790b69b3cc6e220df1beacbcc3a5642b69
Author: Kenneth Knowles 
Date:   2016-11-10T22:47:56Z

Create java8-examples archetype module

commit ccaae328545f2492a8194c89a58011299d1164e6
Author: Kenneth Knowles 
Date:   2016-11-10T22:48:59Z

Generate java8-examples archetype contents




> Create a walkthrough of Beam examples in mobile gaming domain
> -
>
> Key: BEAM-194
> URL: https://issues.apache.org/jira/browse/BEAM-194
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Devin Donnelly
>Assignee: Hadar Hod
>
> The Beam SDKs provide a series of example pipelines in the mobile gaming 
> domain. The Dataflow documentation contains an detailed walkthrough of these 
> examples, explaining the use case, pipeline design, and some of the code.
> Port these examples to the Beam website for Beam users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1344: [BEAM-194] Add Java 8 examples archetype ...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1344

[BEAM-194] Add Java 8 examples archetype including mobile gaming example, 
with generator script

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam moar-examples

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1344.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1344


commit f4782aa2da8e5bbbfa70cb80db07419f717458d7
Author: Kenneth Knowles 
Date:   2016-11-10T22:47:37Z

Add script to generate the java8-examples archetype

commit f783c4790b69b3cc6e220df1beacbcc3a5642b69
Author: Kenneth Knowles 
Date:   2016-11-10T22:47:56Z

Create java8-examples archetype module

commit ccaae328545f2492a8194c89a58011299d1164e6
Author: Kenneth Knowles 
Date:   2016-11-10T22:48:59Z

Generate java8-examples archetype contents




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1343: Support @ValidatesRunner(RunnableOnServic...

[markflyhigh]

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/1343

Support @ValidatesRunner(RunnableOnService) in Python

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - introduce annotation "ValidatesRunner" (same as RunnableOnService in 
Java) for tests that run against runner.
 - support "test_options" argument in command line used to generate 
customized pipeline options for tests with @ValidatesRunner tag.
 - potentially support capability metrix testing using parametrized 
marker(annotation).
 - new test dependency Pytest is introduced to handle marker(annotation) 
and test suites.

TODO: modify some existing unit tests to @ValidatesRunner tests. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
support-validate-runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1343.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1343


commit 50fbce83e77481ea960ef9d190c396d02d090cae
Author: Mark Liu 
Date:   2016-11-10T19:57:21Z

Support @ValidatesRunner Tests

commit b172892a8d1cb5a9a14aa7dc46531d147862a723
Author: Mark Liu 
Date:   2016-11-10T21:34:36Z

Add pytest to setup.py test dependency list

commit fe9306ee3239739efaf9c0603ffc7236083bc4dc
Author: Mark Liu 
Date:   2016-11-10T22:19:10Z

fixup! Fix pylint errors




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-856) IO test should use a free port for embedded resource instance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655373#comment-15655373
 ] 

ASF GitHub Bot commented on BEAM-856:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1262


> IO test should use a free port for embedded resource instance
> -
>
> Key: BEAM-856
> URL: https://issues.apache.org/jira/browse/BEAM-856
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 0.3.0-incubating
>Reporter: Sergio Fernández
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> {{MongoDbIOTest fails}} (addr already in use) when a MongoDB service is 
> locally running. I'd say the port should be random in the test suite.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-856] Use free network port for the Derby test instance

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3f8db06bd -> 48917845b


[BEAM-856] Use free network port for the Derby test instance


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/67215786
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/67215786
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/67215786

Branch: refs/heads/master
Commit: 67215786f99d640283c730e41c70179ae1a5519a
Parents: 3f8db06
Author: Jean-Baptiste Onofré 
Authored: Wed Nov 2 19:22:16 2016 +0100
Committer: Davor Bonaci 
Committed: Thu Nov 10 14:30:01 2016 -0800

--
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 25 +---
 1 file changed, 17 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/67215786/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
--
diff --git 
a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java 
b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
index c7b2de8..fe574af 100644
--- 
a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
+++ 
b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
@@ -22,6 +22,7 @@ import static org.junit.Assert.assertTrue;
 
 import java.io.Serializable;
 import java.net.InetAddress;
+import java.net.ServerSocket;
 import java.sql.Connection;
 import java.sql.PreparedStatement;
 import java.sql.ResultSet;
@@ -59,18 +60,26 @@ public class JdbcIOTest implements Serializable {
   private static NetworkServerControl derbyServer;
   private static ClientDataSource dataSource;
 
+  private static int port;
+
   @BeforeClass
   public static void startDatabase() throws Exception {
+ServerSocket socket = new ServerSocket(0);
+port = socket.getLocalPort();
+socket.close();
+
+LOGGER.info("Starting Derby database on {}", port);
+
 System.setProperty("derby.stream.error.file", "target/derby.log");
 
-derbyServer = new NetworkServerControl(InetAddress.getByName("localhost"), 
1527);
+derbyServer = new NetworkServerControl(InetAddress.getByName("localhost"), 
port);
 derbyServer.start(null);
 
 dataSource = new ClientDataSource();
 dataSource.setCreateDatabase("create");
 dataSource.setDatabaseName("target/beam");
 dataSource.setServerName("localhost");
-dataSource.setPortNumber(1527);
+dataSource.setPortNumber(port);
 
 try (Connection connection = dataSource.getConnection()) {
   try (Statement statement = connection.createStatement()) {
@@ -129,7 +138,7 @@ public class JdbcIOTest implements Serializable {
   public void testDataSourceConfigurationDriverAndUrl() throws Exception {
 JdbcIO.DataSourceConfiguration config = 
JdbcIO.DataSourceConfiguration.create(
 "org.apache.derby.jdbc.ClientDriver",
-"jdbc:derby://localhost:1527/target/beam");
+"jdbc:derby://localhost:" + port + "/target/beam");
 try (Connection conn = config.getConnection()) {
   assertTrue(conn.isValid(0));
 }
@@ -139,7 +148,7 @@ public class JdbcIOTest implements Serializable {
   public void testDataSourceConfigurationUsernameAndPassword() throws 
Exception {
 JdbcIO.DataSourceConfiguration config = 
JdbcIO.DataSourceConfiguration.create(
 "org.apache.derby.jdbc.ClientDriver",
-"jdbc:derby://localhost:1527/target/beam")
+"jdbc:derby://localhost:" + port + "/target/beam")
 .withUsername("sa")
 .withPassword("sa");
 try (Connection conn = config.getConnection()) {
@@ -151,7 +160,7 @@ public class JdbcIOTest implements Serializable {
   public void testDataSourceConfigurationNullPassword() throws Exception {
 JdbcIO.DataSourceConfiguration config = 
JdbcIO.DataSourceConfiguration.create(
 "org.apache.derby.jdbc.ClientDriver",
-"jdbc:derby://localhost:1527/target/beam")
+"jdbc:derby://localhost:" + port + "/target/beam")
 .withUsername("sa")
 .withPassword(null);
 try (Connection conn = config.getConnection()) {
@@ -163,7 +172,7 @@ public class JdbcIOTest implements Serializable {
   public void testDataSourceConfigurationNullUsernameAndPassword() throws 
Exception {
 JdbcIO.DataSourceConfiguration config = 
JdbcIO.DataSourceConfiguration.create(
 "org.apache.derby.jdbc.ClientDriver",
-"jdbc:derby://localhost:1527/target/beam")
+"jdbc:derby://localhost:" + port + "/target/beam")
 .withUsername(null)
 .withPassword(null);
 try (Connection conn = config.getConnection()) {
@@ -256,7 +265,7 @@ public class JdbcIOTest implements 

[GitHub] incubator-beam pull request #1262: [BEAM-856] Use free network port for the ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1262


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1262

[davor]

This closes #1262


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/48917845
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/48917845
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/48917845

Branch: refs/heads/master
Commit: 48917845bde25a13309d4eb1d06e9b4fe91b33aa
Parents: 3f8db06 6721578
Author: Davor Bonaci 
Authored: Thu Nov 10 14:30:05 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Nov 10 14:30:05 2016 -0800

--
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 25 +---
 1 file changed, 17 insertions(+), 8 deletions(-)
--

[jira] [Updated] (BEAM-953) Consider moving control of output location out of WordCountIT and WindowedWordCountIT

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-953:
--
Priority: Major  (was: Minor)

> Consider moving control of output location out of WordCountIT and 
> WindowedWordCountIT
> -
>
> Key: BEAM-953
> URL: https://issues.apache.org/jira/browse/BEAM-953
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>
> See comment by [~davor] at 
> https://github.com/apache/incubator-beam/pull/1320#discussion_r87308740
> The two WordCount integration tests build their output location based on 
> `tempRoot` in the Java code for the integration test. Perhaps this should be 
> controlled externally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-941) Replace Xyz.INSTANCE throughout the SDK with either static methods or XyzFactory.default()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655338#comment-15655338
 ] 

ASF GitHub Bot commented on BEAM-941:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1309


> Replace Xyz.INSTANCE throughout the SDK with either static methods or 
> XyzFactory.default()
> --
>
> Key: BEAM-941
> URL: https://issues.apache.org/jira/browse/BEAM-941
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #114

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1309: [BEAM-941] Refactor DoFnSignatures & DoFn...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1309


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-747) Text checksum verifier is not resilient to eventually consistent filesystems

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655332#comment-15655332
 ] 

Kenneth Knowles commented on BEAM-747:
--

While we wait to get a reliable fix in, there are only a few combinations of 
files that can occur. For for anyone who lands here, one such checksum is 
{{920486dc2879da2de4d551fdb7c444fef911ca74}}. If you saw this checksum, you 
were probably bitten by this flake.

> Text checksum verifier is not resilient to eventually consistent filesystems
> 
>
> Key: BEAM-747
> URL: https://issues.apache.org/jira/browse/BEAM-747
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> Example 1: 
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/3934/org.apache.beam$beam-examples-java/console
> Here it looks like we need to retry listing files, at least a little bit, if 
> none are found. They did show up:
> {code}
> gsutil ls 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results\*
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-0-of-3
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-1-of-3
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-2-of-3
> {code}
> Example 2: 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-examples-java/1525/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> Here it looks like we need to fill in the shard template if the filesystem 
> does not give us a consistent result:
> {code}
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> readLines
> INFO: [0 of 1] Read 162 lines from file: 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-14-00-25-55-609/output/results-0-of-3
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> readLines
> INFO: [1 of 1] Read 144 lines from file: 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-14-00-25-55-609/output/results-2-of-3
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> matchesSafely
> INFO: Generated checksum for output data: 
> aec68948b2515e6ea35fd1ed7649c267a10a01e5
> {code}
> We missed shard 1-of-3 and hence got the wrong checksum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/4] incubator-beam git commit: Remove DoFnSignatures.INSTANCE

[kenn]

Remove DoFnSignatures.INSTANCE


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/efad9d47
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/efad9d47
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/efad9d47

Branch: refs/heads/master
Commit: efad9d47f34694dc730f2d9e663cb61cba307679
Parents: f7745dc
Author: Kenneth Knowles 
Authored: Mon Nov 7 22:36:09 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Nov 10 14:18:07 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java |  2 +-
 .../beam/runners/core/SplittableParDo.java  |  2 +-
 .../runners/direct/ParDoOverrideFactory.java|  2 +-
 .../beam/sdk/transforms/DoFnAdapters.java   |  2 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |  6 +-
 .../sdk/transforms/reflect/DoFnInvokers.java|  2 +-
 .../sdk/transforms/reflect/DoFnSignatures.java  | 11 ++--
 .../sdk/transforms/reflect/OnTimerInvokers.java |  2 +-
 .../DoFnSignaturesProcessElementTest.java   | 12 ++--
 .../DoFnSignaturesSplittableDoFnTest.java   | 34 +-
 .../transforms/reflect/DoFnSignaturesTest.java  | 66 ++--
 11 files changed, 70 insertions(+), 71 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/efad9d47/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index 3abb06b..2c5a850 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -96,7 +96,7 @@ public class SimpleDoFnRunner implements 
DoFnRunner fn) {
 checkNotNull(fn, "fn must not be null");
 this.fn = fn;
-this.signature = DoFnSignatures.INSTANCE.getSignature(fn.getClass());
+this.signature = DoFnSignatures.getSignature(fn.getClass());
 checkArgument(signature.processElement().isSplittable(), "fn must be a 
splittable DoFn");
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/efad9d47/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoOverrideFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoOverrideFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoOverrideFactory.java
index 6052a41..27941f8 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoOverrideFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoOverrideFactory.java
@@ -44,7 +44,7 @@ class ParDoOverrideFactory
   // This is an OldDoFn, hence not splittable.
   return transform;
 }
-DoFnSignature signature = 
DoFnSignatures.INSTANCE.getSignature(fn.getClass());
+DoFnSignature signature = DoFnSignatures.getSignature(fn.getClass());
 if (!signature.processElement().isSplittable()) {
   return transform;
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/efad9d47/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java 

[1/4] incubator-beam git commit: Remove OnTimerInvokers.INSTANCE; deprecate DoFnInvokers.INSTANCE

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master f7745dc29 -> 3f8db06bd


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/14a71e43/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
index 455e49b..64454e4 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokers.java
@@ -17,74 +17,34 @@
  */
 package org.apache.beam.sdk.transforms.reflect;
 
-import static com.google.common.base.Preconditions.checkArgument;
-
+import java.io.Serializable;
 import java.lang.reflect.Constructor;
-import java.lang.reflect.InvocationTargetException;
-import java.lang.reflect.Method;
-import java.util.ArrayList;
 import java.util.LinkedHashMap;
-import java.util.List;
 import java.util.Map;
-import javax.annotation.Nullable;
-import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.NamingStrategy;
-import net.bytebuddy.description.field.FieldDescription;
-import net.bytebuddy.description.method.MethodDescription;
-import net.bytebuddy.description.modifier.FieldManifestation;
-import net.bytebuddy.description.modifier.Visibility;
-import net.bytebuddy.description.type.TypeDescription;
-import net.bytebuddy.description.type.TypeList;
-import net.bytebuddy.dynamic.DynamicType;
-import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
-import net.bytebuddy.dynamic.scaffold.InstrumentedType;
-import net.bytebuddy.dynamic.scaffold.subclass.ConstructorStrategy;
-import net.bytebuddy.implementation.ExceptionMethod;
-import net.bytebuddy.implementation.FixedValue;
-import net.bytebuddy.implementation.Implementation;
-import net.bytebuddy.implementation.Implementation.Context;
-import net.bytebuddy.implementation.MethodDelegation;
-import 
net.bytebuddy.implementation.bind.annotation.TargetMethodAnnotationDrivenBinder;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
-import net.bytebuddy.implementation.bytecode.StackManipulation;
-import net.bytebuddy.implementation.bytecode.Throw;
-import net.bytebuddy.implementation.bytecode.assign.Assigner;
-import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
-import net.bytebuddy.implementation.bytecode.constant.TextConstant;
-import net.bytebuddy.implementation.bytecode.member.FieldAccess;
-import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
-import net.bytebuddy.implementation.bytecode.member.MethodReturn;
-import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
-import net.bytebuddy.jar.asm.Label;
-import net.bytebuddy.jar.asm.MethodVisitor;
-import net.bytebuddy.jar.asm.Opcodes;
-import net.bytebuddy.jar.asm.Type;
-import net.bytebuddy.matcher.ElementMatchers;
-import org.apache.beam.sdk.coders.CannotProvideCoderException;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderRegistry;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.DoFn.ExtraContextFactory;
-import org.apache.beam.sdk.transforms.DoFn.ProcessElement;
 import org.apache.beam.sdk.transforms.DoFnAdapters;
 import org.apache.beam.sdk.transforms.OldDoFn;
-import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.Cases;
-import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.InputProviderParameter;
-import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.OutputReceiverParameter;
-import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionTrackerParameter;
-import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter;
-import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParameter;
-import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter;
 import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
-import org.apache.beam.sdk.util.Timer;
 import org.apache.beam.sdk.util.UserCodeException;
-import org.apache.beam.sdk.values.TypeDescriptor;
 
-/** Dynamically generates {@link DoFnInvoker} instances for invoking a {@link 
DoFn}. */
+/** Static utilities for working with {@link DoFnInvoker}. */
 public class DoFnInvokers {
-  public static final DoFnInvokers INSTANCE = new DoFnInvokers();
 
-  private static final String FN_DELEGATE_FIELD_NAME = "delegate";
+  /**
+   * Returns an {@link DoFnInvoker} for the given {@link DoFn}, using a 
default choice of {@link
+   * DoFnInvokerFactory}.
+   *
+   * The default is permitted to change at any time. Users of this method 
may not depend on any
+   * details {@link DoFnInvokerFactory}-specific details of the invoker. Today 
it is {@link
+   * 

[2/4] incubator-beam git commit: Remove OnTimerInvokers.INSTANCE; deprecate DoFnInvokers.INSTANCE

[kenn]

Remove OnTimerInvokers.INSTANCE; deprecate DoFnInvokers.INSTANCE


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/14a71e43
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/14a71e43
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/14a71e43

Branch: refs/heads/master
Commit: 14a71e435acd9435ce02afe774df3adebd7355f0
Parents: efad9d4
Author: Kenneth Knowles 
Authored: Mon Nov 7 23:03:46 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Nov 10 14:18:07 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java |   2 +-
 .../beam/runners/core/SplittableParDo.java  |  10 +-
 .../runners/direct/DoFnLifecycleManager.java|   4 +-
 .../beam/sdk/transforms/DoFnAdapters.java   |   4 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java| 828 +++
 .../reflect/ByteBuddyOnTimerInvokerFactory.java | 279 +++
 .../transforms/reflect/DoFnInvokerFactory.java  |  27 +
 .../sdk/transforms/reflect/DoFnInvokers.java| 711 +---
 .../reflect/OnTimerInvokerFactory.java  |  36 +
 .../sdk/transforms/reflect/OnTimerInvokers.java | 243 +-
 .../transforms/reflect/DoFnInvokersTest.java|  24 +-
 .../transforms/reflect/OnTimerInvokersTest.java |   2 +-
 .../transforms/DoFnInvokersBenchmark.java   |   2 +-
 13 files changed, 1227 insertions(+), 945 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/14a71e43/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index 2c5a850..3b784d1 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -97,7 +97,7 @@ public class SimpleDoFnRunner implements 
DoFnRunner> splitCoder =
 ElementAndRestrictionCoder.of(input.getCoder(), restrictionCoder);
@@ -166,7 +166,7 @@ public class SplittableParDo<
 
 @Setup
 public void setup() {
-  invoker = DoFnInvokers.INSTANCE.newByteBuddyInvoker(fn);
+  invoker = DoFnInvokers.invokerFor(fn);
 }
 
 @ProcessElement
@@ -246,7 +246,7 @@ public class SplittableParDo<
 
 @Override
 public void setup() throws Exception {
-  invoker = DoFnInvokers.INSTANCE.newByteBuddyInvoker(fn);
+  invoker = DoFnInvokers.invokerFor(fn);
 }
 
 @Override
@@ -460,7 +460,7 @@ public class SplittableParDo<
 
 @Setup
 public void setup() {
-  invoker = DoFnInvokers.INSTANCE.newByteBuddyInvoker(splittableFn);
+  invoker = DoFnInvokers.invokerFor(splittableFn);
 }
 
 @ProcessElement

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/14a71e43/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DoFnLifecycleManager.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DoFnLifecycleManager.java
 

[jira] [Commented] (BEAM-962) Fix games example pipeline options default values conflicts.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655330#comment-15655330
 ] 

ASF GitHub Bot commented on BEAM-962:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1340


> Fix games example pipeline options default values conflicts.
> 
>
> Key: BEAM-962
> URL: https://issues.apache.org/jira/browse/BEAM-962
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Pei He
>Assignee: Pei He
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1340: [BEAM-962] Fix games example pipeline opt...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1340


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is unstable: beam_PostCommit_MavenVerify #1787

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: This closes #1340

[davor]

This closes #1340


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f7745dc2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f7745dc2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f7745dc2

Branch: refs/heads/master
Commit: f7745dc295b0c17e5fbafc263577457a6bb52a17
Parents: 503b407 7da1242
Author: Davor Bonaci 
Authored: Thu Nov 10 14:16:24 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Nov 10 14:16:24 2016 -0800

--
 .../org/apache/beam/examples/complete/game/GameStats.java |  8 
 .../beam/examples/complete/game/HourlyTeamScore.java  |  8 +++-
 .../apache/beam/examples/complete/game/LeaderBoard.java   | 10 --
 .../org/apache/beam/examples/complete/game/UserScore.java |  6 +++---
 .../apache/beam/examples/complete/game/GameStatsTest.java |  5 +
 .../beam/examples/complete/game/HourlyTeamScoreTest.java  |  5 +
 .../beam/examples/complete/game/LeaderBoardTest.java  |  6 ++
 7 files changed, 30 insertions(+), 18 deletions(-)
--

[1/2] incubator-beam git commit: [BEAM-962] Fix games example pipeline options default values conflicts.

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 503b40717 -> f7745dc29


[BEAM-962] Fix games example pipeline options default values conflicts.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7da12429
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7da12429
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7da12429

Branch: refs/heads/master
Commit: 7da124293f1726efb716cb8432fd2c87bab54bca
Parents: 503b407
Author: Pei He 
Authored: Thu Nov 10 13:52:52 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Nov 10 14:16:18 2016 -0800

--
 .../org/apache/beam/examples/complete/game/GameStats.java |  8 
 .../beam/examples/complete/game/HourlyTeamScore.java  |  8 +++-
 .../apache/beam/examples/complete/game/LeaderBoard.java   | 10 --
 .../org/apache/beam/examples/complete/game/UserScore.java |  6 +++---
 .../apache/beam/examples/complete/game/GameStatsTest.java |  5 +
 .../beam/examples/complete/game/HourlyTeamScoreTest.java  |  5 +
 .../beam/examples/complete/game/LeaderBoardTest.java  |  6 ++
 7 files changed, 30 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7da12429/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
index 2b5255f..f352252 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
@@ -181,8 +181,8 @@ public class GameStats extends LeaderBoard {
 
 @Description("Prefix used for the BigQuery table names")
 @Default.String("game_stats")
-String getTablePrefix();
-void setTablePrefix(String value);
+String getGameStatsTablePrefix();
+void setGameStatsTablePrefix(String value);
   }
 
 
@@ -301,7 +301,7 @@ public class GameStats extends LeaderBoard {
   // Write the result to BigQuery
   .apply("WriteTeamSums",
   new WriteWindowedToBigQuery>(
-  options.getTablePrefix() + "_team", configureWindowedWrite()));
+  options.getGameStatsTablePrefix() + "_team", 
configureWindowedWrite()));
 
 
 // [START DocInclude_SessionCalc]
@@ -328,7 +328,7 @@ public class GameStats extends LeaderBoard {
   // Write this info to a BigQuery table.
   .apply("WriteAvgSessionLength",
  new WriteWindowedToBigQuery(
-options.getTablePrefix() + "_sessions", 
configureSessionWindowWrite()));
+options.getGameStatsTablePrefix() + "_sessions", 
configureSessionWindowWrite()));
 // [END DocInclude_Rewindow]
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7da12429/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
index 3a8d2ad..b905d61 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java
@@ -107,12 +107,10 @@ public class HourlyTeamScore extends UserScore {
 String getStopMin();
 void setStopMin(String value);
 
-@Override
 @Description("The BigQuery table name. Should not already exist.")
 @Default.String("hourly_team_score")
-String getTableName();
-@Override
-void setTableName(String value);
+String getHourlyTeamScoreTableName();
+void setHourlyTeamScoreTableName(String value);
   }
 
   /**
@@ -187,7 +185,7 @@ public class HourlyTeamScore extends UserScore {
   // Extract and sum teamname/score pairs from the event data.
   .apply("ExtractTeamScore", new ExtractAndSumScore("team"))
   .apply("WriteTeamScoreSums",
-new WriteWindowedToBigQuery>(options.getTableName(),
+new WriteWindowedToBigQuery>(options.getHourlyTeamScoreTableName(),
 configureWindowedTableWrite()));
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7da12429/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
--
diff --git 

[jira] [Commented] (BEAM-802) Support Dynamic PipelineOptions for python

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655314#comment-15655314
 ] 

ASF GitHub Bot commented on BEAM-802:
-

GitHub user mariapython opened a pull request:

https://github.com/apache/incubator-beam/pull/1342

[BEAM-802] -- Add ability to stage a job and/or stop the execution

Modify create_job to allow staging the job and not submitting it to the 
service.
- Modularize create_job in create job description, stage job, and send for 
execution.
- Modify --dataflow_job_file to stage the job and continue submitting it to 
the service.
- Add --template_location to stage the job but not submit it to the service.
- Add tests for both, including making them mutually exclusive (following 
Java SDK decision).
- Add template_runner_test.py with integration tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariapython/incubator-beam python-sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1342.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1342


commit 3b46dbc7729bd4a3c83e2aa985c7e20b8bc2210b
Author: Maria Garcia Herrero 
Date:   2016-11-02T16:14:48Z

Modify create_job to allow staging the job and not submitting it to the 
service.

- Modularize create_job in create job description, stage job, and send for 
execution.
- Modify --dataflow_job_file to stage the job and continue submitting it to 
the service.
- Add --template_location to stage the job but not submit it to the service.
- Add tests for both, including making them mutually exclusive (following 
Java SDK decision).
- Add template_runner_test.py with integration tests.




> Support Dynamic PipelineOptions for python
> --
>
> Key: BEAM-802
> URL: https://issues.apache.org/jira/browse/BEAM-802
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
>   Original Estimate: 1,680h
>  Remaining Estimate: 1,680h
>
> Goal:  Enable users to run pipelines from templates filled via CL (pipeline 
> options)
> Background: Currently, the Runner creates the JSON pipeline description which 
> can be sent to the worker as is, since everything is already defined there 
> (with links to gs:// for input and binaries). With the parametrized approach, 
> those descriptions are empty and filled by the user or defaulted, so the 
> pipeline needs to be stored somewhere first until the values become available.
> Tasks:
> 1- Create template-style pipeline description (TemplateRunner)
> The graph description is now a template (some parts are not filled) that 
> needs to be saved.
> 2- Define values to inject to the template (ValueProviders API)
> The placeholders can be filled with default values (static) or with dynamic 
> key/value pairs provided at runtime (dynamic)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1342: [BEAM-802] -- Add ability to stage a job ...

[mariapython]

GitHub user mariapython opened a pull request:

https://github.com/apache/incubator-beam/pull/1342

[BEAM-802] -- Add ability to stage a job and/or stop the execution

Modify create_job to allow staging the job and not submitting it to the 
service.
- Modularize create_job in create job description, stage job, and send for 
execution.
- Modify --dataflow_job_file to stage the job and continue submitting it to 
the service.
- Add --template_location to stage the job but not submit it to the service.
- Add tests for both, including making them mutually exclusive (following 
Java SDK decision).
- Add template_runner_test.py with integration tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariapython/incubator-beam python-sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1342.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1342


commit 3b46dbc7729bd4a3c83e2aa985c7e20b8bc2210b
Author: Maria Garcia Herrero 
Date:   2016-11-02T16:14:48Z

Modify create_job to allow staging the job and not submitting it to the 
service.

- Modularize create_job in create job description, stage job, and send for 
execution.
- Modify --dataflow_job_file to stage the job and continue submitting it to 
the service.
- Add --template_location to stage the job but not submit it to the service.
- Add tests for both, including making them mutually exclusive (following 
Java SDK decision).
- Add template_runner_test.py with integration tests.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1338: [BEAM-664] Update examples archetype

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1338


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-664) Port Dataflow SDK WordCount walkthrough to Beam site

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655301#comment-15655301
 ] 

ASF GitHub Bot commented on BEAM-664:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1338


> Port Dataflow SDK WordCount walkthrough to Beam site
> 
>
> Key: BEAM-664
> URL: https://issues.apache.org/jira/browse/BEAM-664
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
>
> Port the WordCount walkthrough from Dataflow docs to Beam website. 
> * Copy prose (translate from html to md, remove Dataflow references, etc)
> * Add accurate "How to Run" instructions for each of the WC examples
> * Include code snippets from real examples



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/4] incubator-beam git commit: Update examples archetype to match examples

[kenn]

Update examples archetype to match examples


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1e132ee8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1e132ee8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1e132ee8

Branch: refs/heads/master
Commit: 1e132ee83d5f393498c12003a328e51d0e93bd06
Parents: 9f78c44
Author: Kenneth Knowles 
Authored: Thu Nov 10 12:06:42 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Nov 10 14:06:39 2016 -0800

--
 .../src/main/java/DebuggingWordCount.java   | 69 +---
 .../src/main/java/MinimalWordCount.java | 52 +++
 .../src/main/java/WindowedWordCount.java|  6 +-
 .../src/main/java/WordCount.java| 64 +++---
 .../common/ExampleBigQueryTableOptions.java |  2 +-
 .../src/main/java/common/ExampleOptions.java|  5 ++
 ...xamplePubsubTopicAndSubscriptionOptions.java |  2 +-
 .../java/common/ExamplePubsubTopicOptions.java  |  2 +-
 .../src/main/java/common/ExampleUtils.java  |  3 +-
 .../src/test/java/DebuggingWordCountTest.java   | 15 -
 .../src/test/java/WordCountTest.java|  7 +-
 11 files changed, 91 insertions(+), 136 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1e132ee8/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
index 9727379..99ae796 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src/main/java/DebuggingWordCount.java
@@ -50,10 +50,9 @@ import org.slf4j.LoggerFactory;
  *
  * New Concepts:
  * 
- *   1. Logging to Cloud Logging
- *   2. Controlling worker log levels
- *   3. Creating a custom aggregator
- *   4. Testing your Pipeline via PAssert
+ *   1. Logging using SLF4J, even in a distributed environment
+ *   2. Creating a custom aggregator (runners have varying levels of support)
+ *   3. Testing your Pipeline via PAssert
  * 
  *
  * To execute this pipeline locally, specify general pipeline configuration:
@@ -68,51 +67,20 @@ import org.slf4j.LoggerFactory;
  * }
  * 
  *
- * To use the additional logging discussed below, specify:
- * {@code
- *   --workerLogLevelOverrides={"org.apache.beam.examples":"DEBUG"}
- * }
- * 
- *
- * Note that when you run via mvn exec, you may need to escape
- * the quotations as appropriate for your shell. For example, in 
bash:
- * 
- * mvn compile exec:java ... \
- *   -Dexec.args="... \
- * 
--workerLogLevelOverrides={\\\"org.apache.beam.examples\\\":\\\"DEBUG\\\"}"
- * 
+ * The input file defaults to a public data set containing the text of of 
King Lear,
+ * by William Shakespeare. You can override it and choose your own input with 
{@code --inputFile}.
  *
- * Concept #2: Dataflow workers which execute user code are configured to 
log to Cloud
- * Logging by default at "INFO" log level and higher. One may override log 
levels for specific
- * logging namespaces by specifying:
- * 
- *   --workerLogLevelOverrides={"Name1":"Level1","Name2":"Level2",...}
- * 
- * For example, by specifying:
- * 
- *   --workerLogLevelOverrides={"org.apache.beam.examples":"DEBUG"}
- * 
- * when executing this pipeline using the Dataflow service, Cloud Logging 
would contain only
- * "DEBUG" or higher level logs for the {@code org.apache.beam.examples} 
package in
- * addition to the default "INFO" or higher level logs. In addition, the 
default Dataflow worker
- * logging configuration can be overridden by specifying
- * {@code --defaultWorkerLogLevel=}. 
For example,
- * by specifying {@code --defaultWorkerLogLevel=DEBUG} when executing this 
pipeline with
- * the Dataflow service, Cloud Logging would contain all "DEBUG" or higher 
level logs. Note
- * that changing the default worker log level to TRACE or DEBUG will 
significantly increase
- * the amount of logs output.
- *
- * The input file defaults to {@code 
gs://apache-beam-samples/shakespeare/kinglear.txt}
- * and can be overridden with {@code --inputFile}.
  */
 public class DebuggingWordCount {
   /** A DoFn that filters for a specific key based upon a regular expression. 
*/
   public static class FilterTextFn extends DoFn, KV> {
 /**
- * Concept #1: The logger below uses the fully qualified 

[4/4] incubator-beam git commit: This closes #1338

[kenn]

This closes #1338


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/503b4071
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/503b4071
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/503b4071

Branch: refs/heads/master
Commit: 503b40717ee75e7ef7ed687f7ac797dfa4543db1
Parents: e43a383 cd5eca8
Author: Kenneth Knowles 
Authored: Thu Nov 10 14:06:40 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Nov 10 14:06:40 2016 -0800

--
 .../src/main/java/DebuggingWordCount.java   | 69 +---
 .../src/main/java/MinimalWordCount.java | 52 +++
 .../src/main/java/WindowedWordCount.java|  6 +-
 .../src/main/java/WordCount.java| 64 +++---
 .../common/ExampleBigQueryTableOptions.java |  2 +-
 .../src/main/java/common/ExampleOptions.java|  5 ++
 ...xamplePubsubTopicAndSubscriptionOptions.java |  2 +-
 .../java/common/ExamplePubsubTopicOptions.java  |  2 +-
 .../src/main/java/common/ExampleUtils.java  |  3 +-
 .../src/test/java/DebuggingWordCountTest.java   | 15 -
 .../src/test/java/WordCountTest.java|  7 +-
 .../update-examples-archetype.sh| 59 +
 12 files changed, 150 insertions(+), 136 deletions(-)
--

[1/4] incubator-beam git commit: fixup! Add script

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master e43a38355 -> 503b40717


fixup! Add script


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cd5eca8d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cd5eca8d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cd5eca8d

Branch: refs/heads/master
Commit: cd5eca8df1d38a1c8ab2a44581bc0272b20bf609
Parents: 1e132ee
Author: Kenneth Knowles 
Authored: Thu Nov 10 12:53:15 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Nov 10 14:06:39 2016 -0800

--
 .../update-examples-archetype.sh| 26 +++-
 1 file changed, 25 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cd5eca8d/sdks/java/maven-archetypes/update-examples-archetype.sh
--
diff --git a/sdks/java/maven-archetypes/update-examples-archetype.sh 
b/sdks/java/maven-archetypes/update-examples-archetype.sh
index e61c916..3c222be 100755
--- a/sdks/java/maven-archetypes/update-examples-archetype.sh
+++ b/sdks/java/maven-archetypes/update-examples-archetype.sh
@@ -1,8 +1,32 @@
-#!/bin/bash -ex
+#!/bin/bash -e
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
 
+# Updates the examples archetype to match the examples module
+#
+# Usage: Invoke with no arguments from any working directory.
+
+# The directory of this script. Assumes root of the maven-archetypes module.
 HERE="$(dirname $0)"
 
+# The directory of the examples-java module
 EXAMPLES_ROOT="${HERE}/../../../examples/java"
+
+# The root of the examples archetype
 ARCHETYPE_ROOT="${HERE}/examples/src/main/resources/archetype-resources"
 
 #

[3/4] incubator-beam git commit: Add script to update examples archetype automatically

[kenn]

Add script to update examples archetype automatically


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9f78c443
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9f78c443
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9f78c443

Branch: refs/heads/master
Commit: 9f78c443d70cb4e01d0b6b9d0cb64902d70c9b87
Parents: e43a383
Author: Kenneth Knowles 
Authored: Thu Nov 10 11:46:24 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Nov 10 14:06:39 2016 -0800

--
 .../update-examples-archetype.sh| 35 
 1 file changed, 35 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9f78c443/sdks/java/maven-archetypes/update-examples-archetype.sh
--
diff --git a/sdks/java/maven-archetypes/update-examples-archetype.sh 
b/sdks/java/maven-archetypes/update-examples-archetype.sh
new file mode 100755
index 000..e61c916
--- /dev/null
+++ b/sdks/java/maven-archetypes/update-examples-archetype.sh
@@ -0,0 +1,35 @@
+#!/bin/bash -ex
+
+HERE="$(dirname $0)"
+
+EXAMPLES_ROOT="${HERE}/../../../examples/java"
+ARCHETYPE_ROOT="${HERE}/examples/src/main/resources/archetype-resources"
+
+#
+# Copy the Java subset of the examples project verbatim. 
+#
+rsync -a --exclude cookbook --exclude complete  \
+"${EXAMPLES_ROOT}"/src/main/java/org/apache/beam/examples/  \
+"${ARCHETYPE_ROOT}/src/main/java"
+
+rsync -a --exclude cookbook --exclude complete --exclude '*IT.java'  \
+"${EXAMPLES_ROOT}"/src/test/java/org/apache/beam/examples/\
+"${ARCHETYPE_ROOT}/src/test/java"
+
+#
+# Replace 'package org.apache.beam.examples' with 'package ${package}' in all 
Java code
+#
+find "${ARCHETYPE_ROOT}/src/main/java" -name '*.java' -print0 \
+| xargs -0 sed -i 's/^package org\.apache\.beam\.examples/package 
${package}/g'
+
+find "${ARCHETYPE_ROOT}/src/test/java" -name '*.java' -print0 \
+| xargs -0 sed -i 's/^package org\.apache\.beam\.examples/package 
${package}/g'
+
+#
+# Replace 'import org.apache.beam.examples.' with 'import ${package}.' in all 
Java code
+#
+find "${ARCHETYPE_ROOT}/src/main/java" -name '*.java' -print0 \
+| xargs -0 sed -i 's/^import org\.apache\.beam\.examples/import 
${package}/g'
+
+find "${ARCHETYPE_ROOT}/src/test/java" -name '*.java' -print0 \
+| xargs -0 sed -i 's/^import org\.apache\.beam\.examples/import 
${package}/g'

[2/2] incubator-beam git commit: Closes #1330

[robertwb]

Closes #1330


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/778194f2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/778194f2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/778194f2

Branch: refs/heads/python-sdk
Commit: 778194f22ae850195c270991499f988f8fe50972
Parents: ec00c53 4827ae8
Author: Robert Bradshaw 
Authored: Thu Nov 10 14:05:02 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Nov 10 14:05:02 2016 -0800

--
 sdks/python/run_postcommit.sh | 3 +++
 1 file changed, 3 insertions(+)
--

[1/2] incubator-beam git commit: Remove tox cache from previous workspace

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk ec00c530c -> 778194f22


Remove tox cache from previous workspace

Jenkins doesn't cleanup the previous workspace, and since .tox
is in the gitignore, we must explicitly delete it.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4827ae84
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4827ae84
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4827ae84

Branch: refs/heads/python-sdk
Commit: 4827ae840797eb43aa7e4265dad7112804b5fb85
Parents: ec00c53
Author: Vikas Kedigehalli 
Authored: Wed Nov 9 17:25:15 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Nov 10 14:05:01 2016 -0800

--
 sdks/python/run_postcommit.sh | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4827ae84/sdks/python/run_postcommit.sh
--
diff --git a/sdks/python/run_postcommit.sh b/sdks/python/run_postcommit.sh
index 23dd516..2cd40da 100755
--- a/sdks/python/run_postcommit.sh
+++ b/sdks/python/run_postcommit.sh
@@ -31,6 +31,9 @@ set -v
 # pip install --user installation location.
 LOCAL_PATH=$HOME/.local/bin/
 
+# Remove any tox cache from previous workspace
+rm -rf sdks/python/.tox
+
 # INFRA does not install virtualenv
 pip install virtualenv --user
 

[GitHub] incubator-beam pull request #1341: Simplify FileOperations interface by refa...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1341

Simplify FileOperations interface by refactoring removeDirectoryAndFiles() 
into FileBasedSink.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam file-operations-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1341.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1341


commit 29e6106d1a9c0d913770312dbaef0ab7accd5717
Author: Pei He 
Date:   2016-11-10T20:50:16Z

Simplify FileOperations interface by refactoring removeDirectoryAndFiles() 
into FileBasedSink.

commit d28bd06e9972ceb05c162708431eead0c5ec40e0
Author: Pei He 
Date:   2016-11-10T21:03:38Z

FileIOChannelFactory.match() returns empty list instead of throw.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [jira] [Created] (BEAM-961) CountingInput could have starting number

[Kenneth Knowles]

I'm not particular about whether the source itself does it versus replacing
uses of the source with a new PTransform encapsulating them. As long as
there is some object with `startingAt` and `upTo`. Either way should be
easy? Left it with starter tag as it would potentially be a fun initial
dive into the codebase.

On Thu, Nov 10, 2016 at 1:23 PM, Dan Halperin  wrote:

> Why not support this in a follow-on pardo that shifts the range?
>
> On Thu, Nov 10, 2016 at 1:22 PM, Kenneth Knowles (JIRA) 
> wrote:
>
>> Kenneth Knowles created BEAM-961:
>> 
>>
>>  Summary: CountingInput could have starting number
>>  Key: BEAM-961
>>  URL: https://issues.apache.org/jira/browse/BEAM-961
>>  Project: Beam
>>   Issue Type: New Feature
>>   Components: sdk-java-core
>> Reporter: Kenneth Knowles
>> Priority: Trivial
>>
>>
>> TL;DR: Add {{startingAt}} to {{CountingInput}}.
>>
>> Right now you can have {{CountingInput.upTo(someNumber)}} but it came up
>> in a test that if you want to have, say, one PCollection that is 1 through
>> 10 and another that is 11 through 20 - so you know they are disjoint - then
>> it requires some boilerplate to add 10 to every element. That boilerplate
>> should be part of the {{CountingInput}}
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.3.4#6332)
>>
>
>

[jira] [Commented] (BEAM-962) Fix games example pipeline options default values conflicts.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655280#comment-15655280
 ] 

ASF GitHub Bot commented on BEAM-962:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1340

[BEAM-962] Fix games example pipeline options default values conflicts.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam leader-board

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1340.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1340


commit b8b5aa2872a062f09ea9df5e1b67101079f6b8f8
Author: Pei He 
Date:   2016-11-10T21:52:52Z

[BEAM-962] Fix games example pipeline options default values conflicts.




> Fix games example pipeline options default values conflicts.
> 
>
> Key: BEAM-962
> URL: https://issues.apache.org/jira/browse/BEAM-962
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Pei He
>Assignee: Pei He
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1340: [BEAM-962] Fix games example pipeline opt...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1340

[BEAM-962] Fix games example pipeline options default values conflicts.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam leader-board

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1340.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1340


commit b8b5aa2872a062f09ea9df5e1b67101079f6b8f8
Author: Pei He 
Date:   2016-11-10T21:52:52Z

[BEAM-962] Fix games example pipeline options default values conflicts.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-310) Exercise splitIntoBundles/generateInitialSplits in the Direct Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655261#comment-15655261
 ] 

ASF GitHub Bot commented on BEAM-310:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1339

[BEAM-310] Actually Split Root Transforms

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Permit the ExecutorServiceParallelExecutor to control its own
ExecutorService by passing only a TargetParallelism parameter. Split
roots into the greater of 3 or the target parallelism.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam actually_split

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1339.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1339


commit b18b876de91bfc01c82cf10bf53eb27a5aef3b09
Author: Thomas Groh 
Date:   2016-11-10T21:47:40Z

Actually Split Root Transforms

Permit the ExecutorServiceParallelExecutor to control its own
ExecutorService by passing only a TargetParallelism parameter. Split
roots into the greater of 3 or the target parallelism.




> Exercise splitIntoBundles/generateInitialSplits in the Direct Runner
> 
>
> Key: BEAM-310
> URL: https://issues.apache.org/jira/browse/BEAM-310
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: 0.3.0-incubating
>
>
> BoundedSource#splitIntoBundles and UnboundedSource#generateInitialSplits are 
> the methods by which sources can be accessed in parallel. Exercising these 
> methods allows reads (and all transforms downstream) to be executed in 
> parallel both pre and post a GroupByKey



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1339: [BEAM-310] Actually Split Root Transforms

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1339

[BEAM-310] Actually Split Root Transforms

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Permit the ExecutorServiceParallelExecutor to control its own
ExecutorService by passing only a TargetParallelism parameter. Split
roots into the greater of 3 or the target parallelism.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam actually_split

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1339.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1339


commit b18b876de91bfc01c82cf10bf53eb27a5aef3b09
Author: Thomas Groh 
Date:   2016-11-10T21:47:40Z

Actually Split Root Transforms

Permit the ExecutorServiceParallelExecutor to control its own
ExecutorService by passing only a TargetParallelism parameter. Split
roots into the greater of 3 or the target parallelism.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-944.

Resolution: Fixed

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:587)
>   at 
> 

[jira] [Created] (BEAM-962) Fix games example pipeline options default values conflicts.

[Pei He (JIRA)]

Pei He created BEAM-962:
---

 Summary: Fix games example pipeline options default values 
conflicts.
 Key: BEAM-962
 URL: https://issues.apache.org/jira/browse/BEAM-962
 Project: Beam
  Issue Type: Bug
  Components: examples-java
Reporter: Pei He
Assignee: Pei He






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655240#comment-15655240
 ] 

Pei He commented on BEAM-944:
-

Filed https://issues.apache.org/jira/browse/BEAM-962
And, I will send a PR soon.

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> 

[1/2] incubator-beam git commit: [BEAM-944] Spark runner causes an exception when creating pipeline options. Create a SparkContextOptions for context-ware options.

[amitsela]

Repository: incubator-beam
Updated Branches:
  refs/heads/master cd3f61cf8 -> e43a38355


[BEAM-944] Spark runner causes an exception when creating pipeline options.
Create a SparkContextOptions for context-ware options.

Move UsesProvidedSparkContext property to SparkPipelineOptions so it's 
available from command-line
as well.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/121bff46
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/121bff46
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/121bff46

Branch: refs/heads/master
Commit: 121bff46d950e319eebf10e3a42bdd890edfb0c5
Parents: cd3f61c
Author: Sela 
Authored: Tue Nov 8 23:05:13 2016 +0200
Committer: Sela 
Committed: Thu Nov 10 23:27:17 2016 +0200

--
 .../beam/runners/spark/SparkContextOptions.java | 64 
 .../runners/spark/SparkPipelineOptions.java | 36 +++
 .../spark/translation/SparkContextFactory.java  | 19 +++---
 .../SparkRunnerStreamingContextFactory.java |  3 +-
 .../runners/spark/ProvidedSparkContextTest.java |  6 +-
 .../streaming/KafkaStreamingTest.java   |  4 +-
 6 files changed, 91 insertions(+), 41 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/121bff46/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkContextOptions.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkContextOptions.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkContextOptions.java
new file mode 100644
index 000..98f7492
--- /dev/null
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkContextOptions.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.spark;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.DefaultValueFactory;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.streaming.api.java.JavaStreamingListener;
+
+
+
+/**
+ * A custom {@link PipelineOptions} to work with properties related to {@link 
JavaSparkContext}.
+ *
+ * This can only be used programmatically (as opposed to passing command 
line arguments),
+ * since the properties here are context-aware and should not be propagated to 
workers.
+ *
+ * Separating this from {@link SparkPipelineOptions} is needed so the 
context-aware properties,
+ * which link to Spark dependencies, won't be scanned by {@link 
PipelineOptions}
+ * reflective instantiation.
+ * Note that {@link SparkContextOptions} is not registered with {@link 
SparkRunnerRegistrar}.
+ */
+public interface SparkContextOptions extends SparkPipelineOptions {
+
+  @Description("Provided Java Spark Context")
+  @JsonIgnore
+  JavaSparkContext getProvidedSparkContext();
+  void setProvidedSparkContext(JavaSparkContext jsc);
+
+  @Description("Spark streaming listeners")
+  @Default.InstanceFactory(EmptyListenersList.class)
+  @JsonIgnore
+  List getListeners();
+  void setListeners(List listeners);
+
+  /** Returns an empty list, top avoid handling null. */
+  class EmptyListenersList implements 
DefaultValueFactory {
+@Override
+public List create(PipelineOptions options) {
+  return new ArrayList<>();
+}
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/121bff46/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
index 

[GitHub] incubator-beam pull request #1316: [BEAM-944] Spark runner causes an excepti...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1316


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1316

[amitsela]

This closes #1316


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e43a3835
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e43a3835
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e43a3835

Branch: refs/heads/master
Commit: e43a383559cb825a498c7427d58ce0a56b3f5245
Parents: cd3f61c 121bff4
Author: Sela 
Authored: Thu Nov 10 23:27:55 2016 +0200
Committer: Sela 
Committed: Thu Nov 10 23:27:55 2016 +0200

--
 .../beam/runners/spark/SparkContextOptions.java | 64 
 .../runners/spark/SparkPipelineOptions.java | 36 +++
 .../spark/translation/SparkContextFactory.java  | 19 +++---
 .../SparkRunnerStreamingContextFactory.java |  3 +-
 .../runners/spark/ProvidedSparkContextTest.java |  6 +-
 .../streaming/KafkaStreamingTest.java   |  4 +-
 6 files changed, 91 insertions(+), 41 deletions(-)
--

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655237#comment-15655237
 ] 

ASF GitHub Bot commented on BEAM-944:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1316


> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> 

Re: [jira] [Created] (BEAM-961) CountingInput could have starting number

[Dan Halperin]

Why not support this in a follow-on pardo that shifts the range?

On Thu, Nov 10, 2016 at 1:22 PM, Kenneth Knowles (JIRA) 
wrote:

> Kenneth Knowles created BEAM-961:
> 
>
>  Summary: CountingInput could have starting number
>  Key: BEAM-961
>  URL: https://issues.apache.org/jira/browse/BEAM-961
>  Project: Beam
>   Issue Type: New Feature
>   Components: sdk-java-core
> Reporter: Kenneth Knowles
> Priority: Trivial
>
>
> TL;DR: Add {{startingAt}} to {{CountingInput}}.
>
> Right now you can have {{CountingInput.upTo(someNumber)}} but it came up
> in a test that if you want to have, say, one PCollection that is 1 through
> 10 and another that is 11 through 20 - so you know they are disjoint - then
> it requires some boilerplate to add 10 to every element. That boilerplate
> should be part of the {{CountingInput}}
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

[jira] [Created] (BEAM-961) CountingInput could have starting number

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-961:


 Summary: CountingInput could have starting number
 Key: BEAM-961
 URL: https://issues.apache.org/jira/browse/BEAM-961
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-core
Reporter: Kenneth Knowles
Priority: Trivial


TL;DR: Add {{startingAt}} to {{CountingInput}}.

Right now you can have {{CountingInput.upTo(someNumber)}} but it came up in a 
test that if you want to have, say, one PCollection that is 1 through 10 and 
another that is 11 through 20 - so you know they are disjoint - then it 
requires some boilerplate to add 10 to every element. That boilerplate should 
be part of the {{CountingInput}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-944) Spark runner causes an exception when creating pipeline options

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655172#comment-15655172
 ] 

Davor Bonaci commented on BEAM-944:
---

Looks like another, unrelated regression. [~pei...@gmail.com], can you take a 
peek? (Probably worth a separate JIRA.)

> Spark runner causes an exception when creating pipeline options
> ---
>
> Key: BEAM-944
> URL: https://issues.apache.org/jira/browse/BEAM-944
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.3.0-incubating
>Reporter: Davor Bonaci
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> Running any example with *any* runner throws an exception as soon as it tries 
> to construct PipelineOptions as long as SparkRunner is on the class path:
> {code}
> mvn compile exec:java 
> -Dexec.mainClass=org.apache.beam.examples.complete.game.LeaderBoard 
> -Dexec.args="--runner=DirectRunner"
> [INFO] Scanning for projects...
> [INFO]
>  
> [INFO] 
> 
> [INFO] Building Tutorial 0.0.1-SNAPSHOT
> [INFO] 
> 
> [INFO] 
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> Tutorial ---
> [WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory 
> /Users/fjp/dev/mobile/src/main/resources
> [INFO] 
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ Tutorial ---
> [INFO] No sources to compile
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ Tutorial ---
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/fjp/.m2/repository/org/slf4j/slf4j-jdk14/1.7.5/slf4j-jdk14-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ExceptionInInitializerError
>   at 
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:179)
>   ... 6 more
> Caused by: java.lang.TypeNotPresentException: Type 
> org.apache.spark.streaming.api.java.JavaStreamingListener not present
>   at 
> sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.visitor.Reifier.reifyTypeArguments(Reifier.java:68)
>   at 
> sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:138)
>   at 
> sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
>   at 
> sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
>   at 
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:284)
>   at java.lang.reflect.Method.getGenericParameterTypes(Method.java:282)
>   at 
> java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
>   at 
> java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:653)
>   at 
> java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:326)
>   at java.beans.PropertyDescriptor.(PropertyDescriptor.java:138)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.getPropertyDescriptors(PipelineOptionsFactory.java:896)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:992)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
>   at 
> org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
>   at 
> 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_SparkLocal #113

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-960) JdbcIO should infer coder when possible (such as via RowMapper)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-960:
-
Summary: JdbcIO should infer coder when possible (such as via RowMapper)  
(was: JdbcIO should infer coder based on RowMapper when possible)

> JdbcIO should infer coder when possible (such as via RowMapper)
> ---
>
> Key: BEAM-960
> URL: https://issues.apache.org/jira/browse/BEAM-960
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: James Malone
>
> There are a few ways that we typically do this:
> # Inspect the static args of the class implementing {{RowMapper}} to 
> determine the output type, then consult the coder registry. This method is 
> more limited and not lambda-friendly.
> # For the a "functional interface" version, like {{RowMapper}} that allows 
> setting the output type descriptor.
> # Also provide an abstract class-based version, for example accepting a 
> {{SimpleFunction}} that will automatically have a rich output type descriptor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-958) desiredNumWorkers in Dataflow is too low

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-958:
-
Affects Version/s: Not applicable

> desiredNumWorkers in Dataflow is too low
> 
>
> Key: BEAM-958
> URL: https://issues.apache.org/jira/browse/BEAM-958
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>  Labels: backward-incompatible
>
> {{desiredNumWorkers}} in [UnboundedSource 
> API|https://github.com/apache/incubator-beam/blob/v0.3.0-incubating-RC1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L69]
>  is a suggestion to a source about how many splits it should create. KafkaIO 
> currently takes this literally and only creates up to this many splits.
> The main draw back is that it is very low in Dataflow. It is calculated as 
>   * {{1 * maxNumWorkers}} if {{--maxNumWorkers}} is specified, otherwise
>   * {{3 * numWorkers}}.
> That implies there is only single reader per worker (which is usually a 4 
> core VM). That can leave CPU under utilized on many pipelines.
> Even 3x in case of fixes number of workers seems low to me. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-958) desiredNumWorkers in Dataflow is too low

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-958:
-
Affects Version/s: (was: 0.3.0-incubating)

> desiredNumWorkers in Dataflow is too low
> 
>
> Key: BEAM-958
> URL: https://issues.apache.org/jira/browse/BEAM-958
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>  Labels: backward-incompatible
>
> {{desiredNumWorkers}} in [UnboundedSource 
> API|https://github.com/apache/incubator-beam/blob/v0.3.0-incubating-RC1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L69]
>  is a suggestion to a source about how many splits it should create. KafkaIO 
> currently takes this literally and only creates up to this many splits.
> The main draw back is that it is very low in Dataflow. It is calculated as 
>   * {{1 * maxNumWorkers}} if {{--maxNumWorkers}} is specified, otherwise
>   * {{3 * numWorkers}}.
> That implies there is only single reader per worker (which is usually a 4 
> core VM). That can leave CPU under utilized on many pipelines.
> Even 3x in case of fixes number of workers seems low to me. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-958) desiredNumWorkers in Dataflow is too low

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-958:
-
Labels: backward-incompatible  (was: breaking_change)

> desiredNumWorkers in Dataflow is too low
> 
>
> Key: BEAM-958
> URL: https://issues.apache.org/jira/browse/BEAM-958
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: 0.3.0-incubating
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>  Labels: backward-incompatible
>
> {{desiredNumWorkers}} in [UnboundedSource 
> API|https://github.com/apache/incubator-beam/blob/v0.3.0-incubating-RC1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L69]
>  is a suggestion to a source about how many splits it should create. KafkaIO 
> currently takes this literally and only creates up to this many splits.
> The main draw back is that it is very low in Dataflow. It is calculated as 
>   * {{1 * maxNumWorkers}} if {{--maxNumWorkers}} is specified, otherwise
>   * {{3 * numWorkers}}.
> That implies there is only single reader per worker (which is usually a 4 
> core VM). That can leave CPU under utilized on many pipelines.
> Even 3x in case of fixes number of workers seems low to me. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-958) desiredNumWorkers in Dataflow is too low

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-958:
--
Assignee: Raghu Angadi  (was: Davor Bonaci)

> desiredNumWorkers in Dataflow is too low
> 
>
> Key: BEAM-958
> URL: https://issues.apache.org/jira/browse/BEAM-958
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: 0.3.0-incubating
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>  Labels: breaking_change
>
> {{desiredNumWorkers}} in [UnboundedSource 
> API|https://github.com/apache/incubator-beam/blob/v0.3.0-incubating-RC1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L69]
>  is a suggestion to a source about how many splits it should create. KafkaIO 
> currently takes this literally and only creates up to this many splits.
> The main draw back is that it is very low in Dataflow. It is calculated as 
>   * {{1 * maxNumWorkers}} if {{--maxNumWorkers}} is specified, otherwise
>   * {{3 * numWorkers}}.
> That implies there is only single reader per worker (which is usually a 4 
> core VM). That can leave CPU under utilized on many pipelines.
> Even 3x in case of fixes number of workers seems low to me. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-960) JdbcIO should infer coder based on RowMapper when possible

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-960:


 Summary: JdbcIO should infer coder based on RowMapper when possible
 Key: BEAM-960
 URL: https://issues.apache.org/jira/browse/BEAM-960
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Kenneth Knowles
Assignee: James Malone


There are a few ways that we typically do this:

# Inspect the static args of the class implementing {{RowMapper}} to determine 
the output type, then consult the coder registry. This method is more limited 
and not lambda-friendly.
# For the a "functional interface" version, like {{RowMapper}} that allows 
setting the output type descriptor.
# Also provide an abstract class-based version, for example accepting a 
{{SimpleFunction}} that will automatically have a rich output type descriptor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-959) JdbcIO validation messages need improvement

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-959:


 Summary: JdbcIO validation messages need improvement
 Key: BEAM-959
 URL: https://issues.apache.org/jira/browse/BEAM-959
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Kenneth Knowles
Assignee: James Malone


The error messages upon validation of {{JdbcIO}} should be sentences that 
explain to a user how to rectify the situation. As it is, they look like 
internal SDK errors.

Here is where they are: 
https://github.com/apache/incubator-beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L319

And here is how a user sees them: 
http://stackoverflow.com/questions/40525269/jdbc-fetch-from-oracle-with-beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-939) New credentials code broke Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655094#comment-15655094
 ] 

ASF GitHub Bot commented on BEAM-939:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1336


> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Update BigtableWriteIT to set Credentials

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master aec7a0a76 -> cd3f61cf8


Update BigtableWriteIT to set Credentials


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/97c883de
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/97c883de
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/97c883de

Branch: refs/heads/master
Commit: 97c883de70cb8eea16c437fc64c6411de7daeea8
Parents: aec7a0a
Author: Thomas Groh 
Authored: Thu Nov 10 11:32:55 2016 -0800
Committer: Thomas Groh 
Committed: Thu Nov 10 11:33:36 2016 -0800

--
 .../beam/sdk/io/gcp/bigtable/BigtableWriteIT.java| 15 ++-
 1 file changed, 10 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/97c883de/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteIT.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteIT.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteIT.java
index 9afcd60..cae19cf 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteIT.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteIT.java
@@ -30,6 +30,7 @@ import com.google.bigtable.v2.Row;
 import com.google.bigtable.v2.RowRange;
 import com.google.bigtable.v2.RowSet;
 import com.google.cloud.bigtable.config.BigtableOptions;
+import com.google.cloud.bigtable.config.CredentialOptions;
 import com.google.cloud.bigtable.config.RetryOptions;
 import com.google.cloud.bigtable.grpc.BigtableSession;
 import com.google.cloud.bigtable.grpc.BigtableTableAdminClient;
@@ -44,6 +45,7 @@ import java.util.List;
 import java.util.Map;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.io.CountingInput;
+import org.apache.beam.sdk.options.GcpOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.DoFn;
@@ -83,11 +85,14 @@ public class BigtableWriteIT implements Serializable {
 retryOptionsBuilder.setStreamingBatchSize(
 retryOptionsBuilder.build().getStreamingBufferSize() / 2);
 
-BigtableOptions.Builder bigtableOptionsBuilder = new 
BigtableOptions.Builder()
-.setProjectId(options.getProjectId())
-.setInstanceId(options.getInstanceId())
-.setUserAgent("apache-beam-test")
-.setRetryOptions(retryOptionsBuilder.build());
+BigtableOptions.Builder bigtableOptionsBuilder =
+new BigtableOptions.Builder()
+.setProjectId(options.getProjectId())
+.setInstanceId(options.getInstanceId())
+.setUserAgent("apache-beam-test")
+.setRetryOptions(retryOptionsBuilder.build())
+.setCredentialOptions(
+
CredentialOptions.credential(options.as(GcpOptions.class).getGcpCredential()));
 bigtableOptions = bigtableOptionsBuilder.build();
 
 session = new BigtableSession(bigtableOptions);

[GitHub] incubator-beam pull request #1336: [BEAM-939] Update BigtableWriteIT to set ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1336


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1336

[davor]

This closes #1336


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cd3f61cf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cd3f61cf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cd3f61cf

Branch: refs/heads/master
Commit: cd3f61cf81208cf84f3f66209e2cd3ade5f4c3ea
Parents: aec7a0a 97c883d
Author: Davor Bonaci 
Authored: Thu Nov 10 12:44:07 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Nov 10 12:44:07 2016 -0800

--
 .../beam/sdk/io/gcp/bigtable/BigtableWriteIT.java| 15 ++-
 1 file changed, 10 insertions(+), 5 deletions(-)
--

[jira] [Commented] (BEAM-958) desiredNumWorkers in Dataflow is too low

[Raghu Angadi (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655020#comment-15655020
 ] 

Raghu Angadi commented on BEAM-958:
---

A change to this policy can break Dataflow job update depending the source as 
update requires number of sources to remain same across an update. Native 
pubsub source is not affected.  

> desiredNumWorkers in Dataflow is too low
> 
>
> Key: BEAM-958
> URL: https://issues.apache.org/jira/browse/BEAM-958
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: 0.3.0-incubating
>Reporter: Raghu Angadi
>Assignee: Davor Bonaci
>  Labels: breaking_change
>
> {{desiredNumWorkers}} in [UnboundedSource 
> API|https://github.com/apache/incubator-beam/blob/v0.3.0-incubating-RC1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L69]
>  is a suggestion to a source about how many splits it should create. KafkaIO 
> currently takes this literally and only creates up to this many splits.
> The main draw back is that it is very low in Dataflow. It is calculated as 
>   * {{1 * maxNumWorkers}} if {{--maxNumWorkers}} is specified, otherwise
>   * {{3 * numWorkers}}.
> That implies there is only single reader per worker (which is usually a 4 
> core VM). That can leave CPU under utilized on many pipelines.
> Even 3x in case of fixes number of workers seems low to me. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-958) desiredNumWorkers in Dataflow is too low

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-958:
--
Labels: breaking_change  (was: )

> desiredNumWorkers in Dataflow is too low
> 
>
> Key: BEAM-958
> URL: https://issues.apache.org/jira/browse/BEAM-958
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: 0.3.0-incubating
>Reporter: Raghu Angadi
>Assignee: Davor Bonaci
>  Labels: breaking_change
>
> {{desiredNumWorkers}} in [UnboundedSource 
> API|https://github.com/apache/incubator-beam/blob/v0.3.0-incubating-RC1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L69]
>  is a suggestion to a source about how many splits it should create. KafkaIO 
> currently takes this literally and only creates up to this many splits.
> The main draw back is that it is very low in Dataflow. It is calculated as 
>   * {{1 * maxNumWorkers}} if {{--maxNumWorkers}} is specified, otherwise
>   * {{3 * numWorkers}}.
> That implies there is only single reader per worker (which is usually a 4 
> core VM). That can leave CPU under utilized on many pipelines.
> Even 3x in case of fixes number of workers seems low to me. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-958) desiredNumWorkers in Dataflow is too low

[Raghu Angadi (JIRA)]

Raghu Angadi created BEAM-958:
-

 Summary: desiredNumWorkers in Dataflow is too low
 Key: BEAM-958
 URL: https://issues.apache.org/jira/browse/BEAM-958
 Project: Beam
  Issue Type: Improvement
  Components: runner-dataflow
Affects Versions: 0.3.0-incubating
Reporter: Raghu Angadi
Assignee: Davor Bonaci


{{desiredNumWorkers}} in [UnboundedSource 
API|https://github.com/apache/incubator-beam/blob/v0.3.0-incubating-RC1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L69]
 is a suggestion to a source about how many splits it should create. KafkaIO 
currently takes this literally and only creates up to this many splits.

The main draw back is that it is very low in Dataflow. It is calculated as 
  * {{1 * maxNumWorkers}} if {{--maxNumWorkers}} is specified, otherwise
  * {{3 * numWorkers}}.

That implies there is only single reader per worker (which is usually a 4 core 
VM). That can leave CPU under utilized on many pipelines.
Even 3x in case of fixes number of workers seems low to me. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-664) Port Dataflow SDK WordCount walkthrough to Beam site

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655006#comment-15655006
 ] 

ASF GitHub Bot commented on BEAM-664:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1338

[BEAM-664] Update examples archetype

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

@davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam examples-archetype

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1338.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1338


commit 985df7591946c32875ebf09aef648b84758e0034
Author: Kenneth Knowles 
Date:   2016-11-10T19:46:24Z

Add script to update examples archetype automatically

commit 87493d48ba3dc80211aba84d3d9105e7ee3e3688
Author: Kenneth Knowles 
Date:   2016-11-10T20:06:42Z

Update examples archetype to match examples




> Port Dataflow SDK WordCount walkthrough to Beam site
> 
>
> Key: BEAM-664
> URL: https://issues.apache.org/jira/browse/BEAM-664
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
>
> Port the WordCount walkthrough from Dataflow docs to Beam website. 
> * Copy prose (translate from html to md, remove Dataflow references, etc)
> * Add accurate "How to Run" instructions for each of the WC examples
> * Include code snippets from real examples



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1338: [BEAM-664] Update examples archetype

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1338

[BEAM-664] Update examples archetype

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

@davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam examples-archetype

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1338.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1338


commit 985df7591946c32875ebf09aef648b84758e0034
Author: Kenneth Knowles 
Date:   2016-11-10T19:46:24Z

Add script to update examples archetype automatically

commit 87493d48ba3dc80211aba84d3d9105e7ee3e3688
Author: Kenneth Knowles 
Date:   2016-11-10T20:06:42Z

Update examples archetype to match examples




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_RunnableOnService_SparkLocal #112

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1337: Use batch GCS operations during FileSink ...

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/incubator-beam/pull/1337

Use batch GCS operations during FileSink write finalization



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/incubator-beam 
gcs-batch-write-finalization

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1337.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1337


commit f3b6ba48bbcc4addf2386961f2194e1eebd486b4
Author: Charles Chen 
Date:   2016-11-10T19:45:06Z

Use batch GCS operations during FileSink write finalization




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1336: [BEAM-939] Update BigtableWriteIT to set ...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1336

[BEAM-939] Update BigtableWriteIT to set Credentials

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam triage

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1336.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1336


commit 97c883de70cb8eea16c437fc64c6411de7daeea8
Author: Thomas Groh 
Date:   2016-11-10T19:32:55Z

Update BigtableWriteIT to set Credentials




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-939) New credentials code broke Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654909#comment-15654909
 ] 

ASF GitHub Bot commented on BEAM-939:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1336

[BEAM-939] Update BigtableWriteIT to set Credentials

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam triage

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1336.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1336


commit 97c883de70cb8eea16c437fc64c6411de7daeea8
Author: Thomas Groh 
Date:   2016-11-10T19:32:55Z

Update BigtableWriteIT to set Credentials




> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1785

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_SparkLocal #111

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1784

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1783

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-664) Port Dataflow SDK WordCount walkthrough to Beam site

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654854#comment-15654854
 ] 

ASF GitHub Bot commented on BEAM-664:
-

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1275


> Port Dataflow SDK WordCount walkthrough to Beam site
> 
>
> Key: BEAM-664
> URL: https://issues.apache.org/jira/browse/BEAM-664
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
>
> Port the WordCount walkthrough from Dataflow docs to Beam website. 
> * Copy prose (translate from html to md, remove Dataflow references, etc)
> * Add accurate "How to Run" instructions for each of the WC examples
> * Include code snippets from real examples



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1335: [BEAM-957] Use a Multiset to track Pendin...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1335

[BEAM-957] Use a Multiset to track Pending Bundles

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

A bundle may be consumed by a Flatten multiple times. The last instance
completing should remove the hold caused by the bundle, not the first.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
flatten_multi_input_flakes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1335.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1335


commit 3c9c36243eb325c40de069c7e58aa0ed6855acd4
Author: Thomas Groh 
Date:   2016-11-10T19:03:34Z

Use a Multiset to track Pending Bundles

A bundle may be consumed by a Flatten multiple times. The last instance
completing should remove the hold caused by the bundle, not the first.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1275: [BEAM-664] Revise WindowedWordCount to be...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1275


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-957) Flake in FlattenTest under DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654843#comment-15654843
 ] 

ASF GitHub Bot commented on BEAM-957:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1335

[BEAM-957] Use a Multiset to track Pending Bundles

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

A bundle may be consumed by a Flatten multiple times. The last instance
completing should remove the hold caused by the bundle, not the first.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
flatten_multi_input_flakes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1335.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1335


commit 3c9c36243eb325c40de069c7e58aa0ed6855acd4
Author: Thomas Groh 
Date:   2016-11-10T19:03:34Z

Use a Multiset to track Pending Bundles

A bundle may be consumed by a Flatten multiple times. The last instance
completing should remove the hold caused by the bundle, not the first.




> Flake in FlattenTest under DirectRunner
> ---
>
> Key: BEAM-957
> URL: https://issues.apache.org/jira/browse/BEAM-957
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Example occurrence: 
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4843/consoleFull



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1320: [BEAM-947] Fix breakage in WindowedWordCo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1320


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1278: [BEAM-896] FileBasedSink removes temp dir...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1278


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-896) FileBasedSink swap to temp directory broke Flink postcommit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654821#comment-15654821
 ] 

ASF GitHub Bot commented on BEAM-896:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1278


> FileBasedSink swap to temp directory broke Flink postcommit
> ---
>
> Key: BEAM-896
> URL: https://issues.apache.org/jira/browse/BEAM-896
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, testing
>Reporter: Daniel Halperin
>Assignee: Maximilian Michels
> Fix For: Not applicable
>
>
> The three new failures on this build 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1720/
> org.apache.beam.runners.flink.ReadSourceITCase.testJobWithoutObjectReuse
> org.apache.beam.runners.flink.ReadSourceITCase.testJobWithObjectReuse
> org.apache.beam.runners.flink.ReadSourceITCase.testJobCollectionExecution
> were caused by
>Makes FileBasedSink use a temporary directory (commit: 8c1008c) (detail)
>https://github.com/apache/incubator-beam/pull/1050
> {code}
> /tmp/org.apache.beam.runners.flink.ReadSourceITCase-result/temp-beam-part-2016-11-308_20-02-14
>  (Is a directory)
> java.io.FileNotFoundException: 
> /tmp/org.apache.beam.runners.flink.ReadSourceITCase-result/temp-beam-part-2016-11-308_20-02-14
>  (Is a directory)
>   at java.io.FileInputStream.open0(Native Method)
>   at java.io.FileInputStream.open(FileInputStream.java:195)
>   at java.io.FileInputStream.(FileInputStream.java:138)
>   at java.io.FileReader.(FileReader.java:72)
>   at 
> org.apache.flink.test.util.TestBaseUtils.getResultReader(TestBaseUtils.java:244)
>   at 
> org.apache.flink.test.util.TestBaseUtils.readAllResultLines(TestBaseUtils.java:276)
>   at 
> org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:304)
>   at 
> org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:298)
>   at 
> org.apache.beam.runners.flink.ReadSourceITCase.postSubmit(ReadSourceITCase.java:58)
>   at 
> org.apache.flink.test.util.JavaProgramTestBase.testJobCollectionExecution(JavaProgramTestBase.java:240)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
> {code}
> My guess is the test is wrong here because it's reading temp files rather 
> than final files? Or something?
> cc [~jkff] [~kirpichov] [~aljoscha] [~mxm]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: [BEAM-896] FileBasedSink removes temp directory

[dhalperi]

[BEAM-896] FileBasedSink removes temp directory

Also fixes RAT plugin in pom.xml to ignore .idea
(found while running integration tests locally).


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/07a3c2c0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/07a3c2c0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/07a3c2c0

Branch: refs/heads/master
Commit: 07a3c2c06c80598faa9d33507ad6fbaa8b70e8c2
Parents: 11eaed1
Author: Eugene Kirpichov 
Authored: Thu Nov 3 15:29:49 2016 -0700
Committer: Dan Halperin 
Committed: Thu Nov 10 10:56:00 2016 -0800

--
 pom.xml |   1 +
 .../org/apache/beam/sdk/io/FileBasedSink.java   | 134 ---
 .../beam/sdk/util/FileIOChannelFactory.java |   3 +-
 .../apache/beam/sdk/io/FileBasedSinkTest.java   |  28 ++--
 .../org/apache/beam/sdk/io/XmlSinkTest.java |   4 +-
 5 files changed, 104 insertions(+), 66 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/07a3c2c0/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 749ca9c..22897e3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -987,6 +987,7 @@
   **/hs_err_pid*.log
   .github/**/*
   **/*.iml
+  **/.idea/**/*
   **/package-list
   **/user.avsc
   **/test/resources/**/*.txt

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/07a3c2c0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
index 101ff61..e6c37de 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
@@ -38,6 +38,7 @@ import java.util.ArrayList;
 import java.util.Collection;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Set;
 import java.util.zip.GZIPOutputStream;
 
 import javax.annotation.Nullable;
@@ -79,6 +80,8 @@ import org.slf4j.LoggerFactory;
  * @param  the type of values written to the sink.
  */
 public abstract class FileBasedSink extends Sink {
+  private static final Logger LOG = 
LoggerFactory.getLogger(FileBasedSink.class);
+
   /**
* Directly supported file output compression types.
*/
@@ -262,11 +265,11 @@ public abstract class FileBasedSink extends Sink {
* FileBasedSinkWriter.
*
* Temporary and Output File Naming: During the write, bundles are 
written to temporary
-   * files using the baseTemporaryFilename that can be provided via the 
constructor of
+   * files using the tempDirectory that can be provided via the constructor of
* FileBasedWriteOperation. These temporary files will be named
-   * {@code {baseTemporaryFilename}-temp-{bundleId}}, where bundleId is the 
unique id of the bundle.
-   * For example, if baseTemporaryFilename is "gs://my-bucket/my_temp_output", 
the output for a
-   * bundle with bundle id 15723 will be 
"gs://my-bucket/my_temp_output-temp-15723".
+   * {@code {tempDirectory}/{bundleId}}, where bundleId is the unique id of 
the bundle.
+   * For example, if tempDirectory is "gs://my-bucket/my_temp_output", the 
output for a
+   * bundle with bundle id 15723 will be "gs://my-bucket/my_temp_output/15723".
*
* Final output files are written to baseOutputFilename with the format
* {@code {baseOutputFilename}-i-of-n.{extension}} where n is the 
total number of bundles
@@ -290,8 +293,6 @@ public abstract class FileBasedSink extends Sink {
* @param  the type of values written to the sink.
*/
   public abstract static class FileBasedWriteOperation extends 
WriteOperation {
-private static final Logger LOG = 
LoggerFactory.getLogger(FileBasedWriteOperation.class);
-
 /**
  * Options for handling of temporary output files.
  */
@@ -310,27 +311,21 @@ public abstract class FileBasedSink extends Sink {
  */
 protected final TemporaryFileRetention temporaryFileRetention;
 
-/**
- * Base filename used for temporary output files. Default is the 
baseOutputFilename.
- */
-protected final String baseTemporaryFilename;
+/** Directory for temporary output files. */
+protected final String tempDirectory;
 
-/**
- * Build a temporary filename using the temporary filename separator with 
the given prefix and
- * suffix.
- */
-protected static final String buildTemporaryFilename(String prefix, String 
suffix) {
-  try {

[1/2] incubator-beam git commit: Closes #1278

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 11eaed19a -> aec7a0a76


Closes #1278


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/aec7a0a7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/aec7a0a7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/aec7a0a7

Branch: refs/heads/master
Commit: aec7a0a767db1df61f89d1fdfac9deedfd6b3a8a
Parents: 11eaed1 07a3c2c
Author: Dan Halperin 
Authored: Thu Nov 10 10:56:00 2016 -0800
Committer: Dan Halperin 
Committed: Thu Nov 10 10:56:00 2016 -0800

--
 pom.xml |   1 +
 .../org/apache/beam/sdk/io/FileBasedSink.java   | 134 ---
 .../beam/sdk/util/FileIOChannelFactory.java |   3 +-
 .../apache/beam/sdk/io/FileBasedSinkTest.java   |  28 ++--
 .../org/apache/beam/sdk/io/XmlSinkTest.java |   4 +-
 5 files changed, 104 insertions(+), 66 deletions(-)
--