[jira] [Commented] (BEAM-1094) Spark runner should define Kafka IO dependency with test scope

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728014#comment-15728014
 ] 

ASF GitHub Bot commented on BEAM-1094:
--

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1531

[BEAM-1094] Set test scope for Kafka IO and junit

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-1094

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1531.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1531


commit 346c0b528297ab39bfa021ee052dcee48f56953d
Author: Jean-Baptiste Onofré 
Date:   2016-12-07T07:37:33Z

[BEAM-1094] Set test scope for Kafka IO and junit




> Spark runner should define Kafka IO dependency with test scope
> --
>
> Key: BEAM-1094
> URL: https://issues.apache.org/jira/browse/BEAM-1094
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> Spark runner uses Kafka IO for testing purpose. However, the Kafka IO 
> dependency is with compile scope whereas it should be test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1531: [BEAM-1094] Set test scope for Kafka IO a...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1531

[BEAM-1094] Set test scope for Kafka IO and junit

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-1094

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1531.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1531


commit 346c0b528297ab39bfa021ee052dcee48f56953d
Author: Jean-Baptiste Onofré 
Date:   2016-12-07T07:37:33Z

[BEAM-1094] Set test scope for Kafka IO and junit




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (BEAM-1102) Flink Batch Runner does not populate aggregator values

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-1102.
--
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> Flink Batch Runner does not populate aggregator values
> --
>
> Key: BEAM-1102
> URL: https://issues.apache.org/jira/browse/BEAM-1102
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.3.0-incubating
>Reporter: Daniel Halperin
>Assignee: Aljoscha Krettek
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> Running the quickstart gives 0 for emptyLines.
> Running with {{--streaming=true}} gives the correct value (for my input file, 
> the default examples archetype {{pom.xml}}, the true value is 27 at the time 
> of writing).
> Streaming output:
> {code}
> Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: Final aggregator values:
> Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: DroppedDueToLateness : 0
> Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: emptyLines : 27
> Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: DroppedDueToClosedWindow : 0
> {code}
> Non-streaming output:
> {code}
> Dec 07, 2016 12:58:07 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: Final aggregator values:
> Dec 07, 2016 12:58:07 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: emptyLines : 0
> {code}
> (Note also that the lateness etc. aggregators are missing entirely, may be 
> expected).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1530: [FLINK-1102] Fix Aggregator Registration ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1530


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [FLINK-1102] Fix Aggregator Registration in Flink Batch Runner

[aljoscha]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 0a2ed832c -> b41a46e86


[FLINK-1102] Fix Aggregator Registration in Flink Batch Runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/869b2710
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/869b2710
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/869b2710

Branch: refs/heads/master
Commit: 869b2710efdb90bc3ce5b6e9d4f3b49a3a804a63
Parents: 0a2ed83
Author: Aljoscha Krettek 
Authored: Wed Dec 7 13:28:13 2016 +0800
Committer: Aljoscha Krettek 
Committed: Wed Dec 7 15:25:44 2016 +0800

--
 .../functions/FlinkProcessContextBase.java  | 21 +---
 1 file changed, 9 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/869b2710/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkProcessContextBase.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkProcessContextBase.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkProcessContextBase.java
index 42607dd..6afca38 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkProcessContextBase.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkProcessContextBase.java
@@ -20,7 +20,6 @@ package org.apache.beam.runners.flink.translation.functions;
 import static com.google.common.base.Preconditions.checkNotNull;
 
 import com.google.common.collect.Iterables;
-import java.io.Serializable;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.Iterator;
@@ -39,7 +38,6 @@ import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
-import org.apache.flink.api.common.accumulators.Accumulator;
 import org.apache.flink.api.common.functions.RuntimeContext;
 import org.joda.time.Instant;
 
@@ -256,15 +254,14 @@ abstract class FlinkProcessContextBase
   @Override
   protected  Aggregator
   createAggregatorInternal(String name, Combine.CombineFn combiner) {
-SerializableFnAggregatorWrapper wrapper =
-new SerializableFnAggregatorWrapper<>(combiner);
-Accumulator existingAccum =
-(Accumulator) 
runtimeContext.getAccumulator(name);
-if (existingAccum != null) {
-  return wrapper;
-} else {
-  runtimeContext.addAccumulator(name, wrapper);
+@SuppressWarnings("unchecked")
+SerializableFnAggregatorWrapper result =
+(SerializableFnAggregatorWrapper)
+runtimeContext.getAccumulator(name);
+
+if (result == null) {
+  result = new SerializableFnAggregatorWrapper<>(combiner);
+  runtimeContext.addAccumulator(name, result);
 }
-return wrapper;
-  }
+return result;  }
 }

[2/2] incubator-beam git commit: This closes #1530

[aljoscha]

This closes #1530


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b41a46e8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b41a46e8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b41a46e8

Branch: refs/heads/master
Commit: b41a46e86fd38c4a887f31bdf6cb75969f4750d3
Parents: 0a2ed83 869b271
Author: Aljoscha Krettek 
Authored: Wed Dec 7 15:26:02 2016 +0800
Committer: Aljoscha Krettek 
Committed: Wed Dec 7 15:26:02 2016 +0800

--
 .../functions/FlinkProcessContextBase.java  | 21 +---
 1 file changed, 9 insertions(+), 12 deletions(-)
--

[jira] [Created] (BEAM-1103) Add Tests For Aggregators in Flink Runner

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created BEAM-1103:
--

 Summary: Add Tests For Aggregators in Flink Runner
 Key: BEAM-1103
 URL: https://issues.apache.org/jira/browse/BEAM-1103
 Project: Beam
  Issue Type: Improvement
  Components: runner-flink
Reporter: Aljoscha Krettek


We currently don't have tests that verify that aggregator values are correctly 
forwarded to Flink.

They didn't work correctly in the Batch Flink runner, as seen in BEAM-1102.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2013

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1530: [FLINK-1102] Fix Aggregator Registration ...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/incubator-beam/pull/1530

[FLINK-1102] Fix Aggregator Registration in Flink Batch Runner

R: @dhalperi 

This is a quick fix now but we definitely need tests for aggregators, both 
in streaming and batch mode.

Should we merge this now and create a new Issue for adding tests or block 
this until we have the tests?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/incubator-beam 
flink-fix-batch-aggregators

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1530.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1530


commit 3792bf1140512f58f61527712b21c74a96cfdc7a
Author: Aljoscha Krettek 
Date:   2016-12-07T05:28:13Z

[FLINK-1102] Fix Aggregator Registration in Flink Batch Runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1102) Flink Batch Runner does not populate aggregator values

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727742#comment-15727742
 ] 

Aljoscha Krettek commented on BEAM-1102:


The problem is this part in {{FlinkProcessContextBase}}:

{code}
  @Override
  protected  Aggregator
  createAggregatorInternal(String name, Combine.CombineFn combiner) {
SerializableFnAggregatorWrapper wrapper =
new SerializableFnAggregatorWrapper<>(combiner);
Accumulator existingAccum =
(Accumulator) 
runtimeContext.getAccumulator(name);
if (existingAccum != null) {
  return wrapper;
} else {
  runtimeContext.addAccumulator(name, wrapper);
}
return wrapper;
  }
{code}

Notice how the newly created wrapper is returned if the accumulator already 
exists.

> Flink Batch Runner does not populate aggregator values
> --
>
> Key: BEAM-1102
> URL: https://issues.apache.org/jira/browse/BEAM-1102
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.3.0-incubating
>Reporter: Daniel Halperin
>Assignee: Aljoscha Krettek
>Priority: Minor
>
> Running the quickstart gives 0 for emptyLines.
> Running with {{--streaming=true}} gives the correct value (for my input file, 
> the default examples archetype {{pom.xml}}, the true value is 27 at the time 
> of writing).
> Streaming output:
> {code}
> Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: Final aggregator values:
> Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: DroppedDueToLateness : 0
> Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: emptyLines : 27
> Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: DroppedDueToClosedWindow : 0
> {code}
> Non-streaming output:
> {code}
> Dec 07, 2016 12:58:07 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: Final aggregator values:
> Dec 07, 2016 12:58:07 PM org.apache.beam.runners.flink.FlinkRunner run
> INFO: emptyLines : 0
> {code}
> (Note also that the lateness etc. aggregators are missing entirely, may be 
> expected).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-812) Shade guava in beam-sdks-java-io-google-cloud-platform

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727720#comment-15727720
 ] 

Daniel Halperin commented on BEAM-812:
--

This is not likely to be fixed in the 0.4.0 release because Bigtable has not 
yet pushed a release of their jar with the API surface cleanup integrated.

> Shade guava in beam-sdks-java-io-google-cloud-platform
> --
>
> Key: BEAM-812
> URL: https://issues.apache.org/jira/browse/BEAM-812
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> Looking at 0.3.0-incubating RC1, we are not properly shading Guava.
> https://repository.apache.org/content/repositories/staging/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.3.0-incubating/beam-sdks-java-io-google-cloud-platform-0.3.0-incubating.pom
> has 
> {code}
> 
>   com.google.guava
>   guava
>  {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-812) Shade guava in beam-sdks-java-io-google-cloud-platform

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-812:
-
Fix Version/s: (was: 0.4.0-incubating)

> Shade guava in beam-sdks-java-io-google-cloud-platform
> --
>
> Key: BEAM-812
> URL: https://issues.apache.org/jira/browse/BEAM-812
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> Looking at 0.3.0-incubating RC1, we are not properly shading Guava.
> https://repository.apache.org/content/repositories/staging/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.3.0-incubating/beam-sdks-java-io-google-cloud-platform-0.3.0-incubating.pom
> has 
> {code}
> 
>   com.google.guava
>   guava
>  {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1102) Flink Batch Runner does not populate aggregator values

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1102:
-

 Summary: Flink Batch Runner does not populate aggregator values
 Key: BEAM-1102
 URL: https://issues.apache.org/jira/browse/BEAM-1102
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 0.3.0-incubating
Reporter: Daniel Halperin
Assignee: Aljoscha Krettek
Priority: Minor


Running the quickstart gives 0 for emptyLines.

Running with {{--streaming=true}} gives the correct value (for my input file, 
the default examples archetype {{pom.xml}}, the true value is 27 at the time of 
writing).

Streaming output:

{code}
Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
INFO: Final aggregator values:
Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
INFO: DroppedDueToLateness : 0
Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
INFO: emptyLines : 27
Dec 07, 2016 1:00:53 PM org.apache.beam.runners.flink.FlinkRunner run
INFO: DroppedDueToClosedWindow : 0
{code}

Non-streaming output:

{code}
Dec 07, 2016 12:58:07 PM org.apache.beam.runners.flink.FlinkRunner run
INFO: Final aggregator values:
Dec 07, 2016 12:58:07 PM org.apache.beam.runners.flink.FlinkRunner run
INFO: emptyLines : 0
{code}

(Note also that the lateness etc. aggregators are missing entirely, may be 
expected).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727709#comment-15727709
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1529

[BEAM-498] Port ParDoTest from OldDoFn to new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aljoscha seems about to time to make this leap?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam ParDoTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1529.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1529


commit 794b54ec6b7284ffabd0a117263a9cf0143c0b72
Author: Kenneth Knowles 
Date:   2016-08-05T22:21:23Z

Add OldDoFn -> DoFn adapter and expose raw DoFn in ParDo

Previously, ParDo.of(DoFn) would immediately wrap it into
an OldDoFn so the new DoFn could not be recovered, since

After this change, the new DoFn is temporarily available
via #getNewFn(), which will eventually overwrite #getFn().
When ParDo is instantiated via ParDo.of(OldDoFn), a wrapper
in the reverse direction will create a new DoFn that can
be accessed via the same method. So a ParDo transform
temporarily always carries both.

commit 06e4df97210b44081aba5288beb25446a8802191
Author: Kenneth Knowles 
Date:   2016-08-08T20:27:00Z

Deprecate more uses of OldDoFn




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1529: [BEAM-498] Port ParDoTest from OldDoFn to...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1529

[BEAM-498] Port ParDoTest from OldDoFn to new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aljoscha seems about to time to make this leap?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam ParDoTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1529.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1529


commit 794b54ec6b7284ffabd0a117263a9cf0143c0b72
Author: Kenneth Knowles 
Date:   2016-08-05T22:21:23Z

Add OldDoFn -> DoFn adapter and expose raw DoFn in ParDo

Previously, ParDo.of(DoFn) would immediately wrap it into
an OldDoFn so the new DoFn could not be recovered, since

After this change, the new DoFn is temporarily available
via #getNewFn(), which will eventually overwrite #getFn().
When ParDo is instantiated via ParDo.of(OldDoFn), a wrapper
in the reverse direction will create a new DoFn that can
be accessed via the same method. So a ParDo transform
temporarily always carries both.

commit 06e4df97210b44081aba5288beb25446a8802191
Author: Kenneth Knowles 
Date:   2016-08-08T20:27:00Z

Deprecate more uses of OldDoFn




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727629#comment-15727629
 ] 

ASF GitHub Bot commented on BEAM-27:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1528

[BEAM-27] Add DoFn.OnTimerContext and support as a DoFn parameter

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam OnTimerContext

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1528.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1528


commit 44a7b915d502c318ffabaa6fb808207bb3ea15e8
Author: Kenneth Knowles 
Date:   2016-12-07T04:10:06Z

Add DoFn.OnTimerContext

commit 1934b704411fed76a58cbc657c7dc8be666c3885
Author: Kenneth Knowles 
Date:   2016-12-07T04:01:18Z

Add support for OnTimerContext parameter in DoFnSignature

commit 14d9a3794c4f8c68626075343a1dec1d4c017686
Author: Kenneth Knowles 
Date:   2016-12-07T04:10:21Z

Access to OnTimerContext via DoFnInvokers.ArgumentProvider




> Add user-ready API for interacting with timers
> --
>
> Key: BEAM-27
> URL: https://issues.apache.org/jira/browse/BEAM-27
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> Pipeline authors will benefit from a different factorization of interaction 
> with underlying timers. The current APIs are targeted at runner implementers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1528: [BEAM-27] Add DoFn.OnTimerContext and sup...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1528

[BEAM-27] Add DoFn.OnTimerContext and support as a DoFn parameter

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam OnTimerContext

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1528.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1528


commit 44a7b915d502c318ffabaa6fb808207bb3ea15e8
Author: Kenneth Knowles 
Date:   2016-12-07T04:10:06Z

Add DoFn.OnTimerContext

commit 1934b704411fed76a58cbc657c7dc8be666c3885
Author: Kenneth Knowles 
Date:   2016-12-07T04:01:18Z

Add support for OnTimerContext parameter in DoFnSignature

commit 14d9a3794c4f8c68626075343a1dec1d4c017686
Author: Kenneth Knowles 
Date:   2016-12-07T04:10:21Z

Access to OnTimerContext via DoFnInvokers.ArgumentProvider




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727614#comment-15727614
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1527

[BEAM-498] Port most of DoFnRunner Javadoc to new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @tgroh 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam DoFnRunner-javadoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1527.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1527


commit d26d9bbce073715d3c6b644b286574ce8c782597
Author: Kenneth Knowles 
Date:   2016-12-06T23:20:28Z

Port most of DoFnRunner Javadoc to new DoFn




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1527: [BEAM-498] Port most of DoFnRunner Javado...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1527

[BEAM-498] Port most of DoFnRunner Javadoc to new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @tgroh 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam DoFnRunner-javadoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1527.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1527


commit d26d9bbce073715d3c6b644b286574ce8c782597
Author: Kenneth Knowles 
Date:   2016-12-06T23:20:28Z

Port most of DoFnRunner Javadoc to new DoFn




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Issue Comment Deleted] (BEAM-1092) Shade commonly used libraries (e.g. Guava) to avoid class conflicts

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-1092:
---
Comment: was deleted

(was: Is 1) necessary if we do 2).

I think shading is very necessary (unfortunately) so I like this issue but I 
would also like to keep the example/archetype POMs as simple as possible.)

> Shade commonly used libraries (e.g. Guava) to avoid class conflicts
> ---
>
> Key: BEAM-1092
> URL: https://issues.apache.org/jira/browse/BEAM-1092
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, sdk-java-extensions
>Affects Versions: 0.3.0-incubating
>Reporter: Maximilian Michels
>Assignee: Frances Perry
> Fix For: 0.4.0-incubating
>
>
> Beam shades away some of its dependencies like Guava to avoid user classes 
> from clashing with these dependencies. Some of the artifacts, e.g. KafkaIO, 
> do not shade any classes and directly depend on potentially conflicting 
> libraries (e.g. Guava). Also, users might manually add such libraries as 
> dependencies.
> Runners who add classes to the classpath (e.g. Hadoop) can run into conflict 
> with multiple versions of the same class. To prevent that, we should adjust 
> the Maven archetypes pom files used for the Quickstart to perform shading of 
> commonly used libraries (again, Guava is often the culprit).
> To prevent the problem in the first place, we should expand the shading of 
> Guava and other libraries to all modules which make use of these. 
> To solve both dimensions of the issue, we need to address:
> 1. Adding shading of commonly used libraries to the archetypes poms
> 2. Properly shade all commonly used libraries in the SDK modules
> 2) seems to be of highest priority since it affects users who simply use the 
> provided IO modules.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1092) Shade commonly used libraries (e.g. Guava) to avoid class conflicts

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727530#comment-15727530
 ] 

Aljoscha Krettek commented on BEAM-1092:


Is 1) necessary if we do 2)?

I think shading is very necessary (unfortunately) so I like this issue but I 
would also like to keep the example/archetype POMs as simple as possible.

> Shade commonly used libraries (e.g. Guava) to avoid class conflicts
> ---
>
> Key: BEAM-1092
> URL: https://issues.apache.org/jira/browse/BEAM-1092
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, sdk-java-extensions
>Affects Versions: 0.3.0-incubating
>Reporter: Maximilian Michels
>Assignee: Frances Perry
> Fix For: 0.4.0-incubating
>
>
> Beam shades away some of its dependencies like Guava to avoid user classes 
> from clashing with these dependencies. Some of the artifacts, e.g. KafkaIO, 
> do not shade any classes and directly depend on potentially conflicting 
> libraries (e.g. Guava). Also, users might manually add such libraries as 
> dependencies.
> Runners who add classes to the classpath (e.g. Hadoop) can run into conflict 
> with multiple versions of the same class. To prevent that, we should adjust 
> the Maven archetypes pom files used for the Quickstart to perform shading of 
> commonly used libraries (again, Guava is often the culprit).
> To prevent the problem in the first place, we should expand the shading of 
> Guava and other libraries to all modules which make use of these. 
> To solve both dimensions of the issue, we need to address:
> 1. Adding shading of commonly used libraries to the archetypes poms
> 2. Properly shade all commonly used libraries in the SDK modules
> 2) seems to be of highest priority since it affects users who simply use the 
> provided IO modules.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1092) Shade commonly used libraries (e.g. Guava) to avoid class conflicts

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727529#comment-15727529
 ] 

Aljoscha Krettek commented on BEAM-1092:


Is 1) necessary if we do 2).

I think shading is very necessary (unfortunately) so I like this issue but I 
would also like to keep the example/archetype POMs as simple as possible.

> Shade commonly used libraries (e.g. Guava) to avoid class conflicts
> ---
>
> Key: BEAM-1092
> URL: https://issues.apache.org/jira/browse/BEAM-1092
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, sdk-java-extensions
>Affects Versions: 0.3.0-incubating
>Reporter: Maximilian Michels
>Assignee: Frances Perry
> Fix For: 0.4.0-incubating
>
>
> Beam shades away some of its dependencies like Guava to avoid user classes 
> from clashing with these dependencies. Some of the artifacts, e.g. KafkaIO, 
> do not shade any classes and directly depend on potentially conflicting 
> libraries (e.g. Guava). Also, users might manually add such libraries as 
> dependencies.
> Runners who add classes to the classpath (e.g. Hadoop) can run into conflict 
> with multiple versions of the same class. To prevent that, we should adjust 
> the Maven archetypes pom files used for the Quickstart to perform shading of 
> commonly used libraries (again, Guava is often the culprit).
> To prevent the problem in the first place, we should expand the shading of 
> Guava and other libraries to all modules which make use of these. 
> To solve both dimensions of the issue, we need to address:
> 1. Adding shading of commonly used libraries to the archetypes poms
> 2. Properly shade all commonly used libraries in the SDK modules
> 2) seems to be of highest priority since it affects users who simply use the 
> provided IO modules.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1744

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1744

[Apache Jenkins Server]

See 

[1/2] incubator-beam git commit: Display data keys in Python should be snake_case

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk f19f767b0 -> d5e8c79a3


Display data keys in Python should be snake_case

This fixes existing mis-cased display data keys.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0dcc9bc0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0dcc9bc0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0dcc9bc0

Branch: refs/heads/python-sdk
Commit: 0dcc9bc03772d8bd5ab1e0d138c445e9418ebf66
Parents: f19f767
Author: Pablo 
Authored: Mon Nov 28 13:05:47 2016 -0800
Committer: bchambers 
Committed: Tue Dec 6 17:17:16 2016 -0800

--
 sdks/python/apache_beam/io/avroio.py|  8 +++-
 sdks/python/apache_beam/io/avroio_test.py   |  8 
 sdks/python/apache_beam/io/filebasedsource.py   |  3 ++-
 .../apache_beam/io/filebasedsource_test.py  |  4 ++--
 sdks/python/apache_beam/io/fileio.py|  8 
 sdks/python/apache_beam/io/fileio_test.py   | 12 ++--
 sdks/python/apache_beam/io/pubsub.py|  2 +-
 sdks/python/apache_beam/io/pubsub_test.py   |  2 +-
 sdks/python/apache_beam/io/textio.py| 20 ++--
 sdks/python/apache_beam/io/textio_test.py   |  8 +---
 .../apache_beam/transforms/combiners_test.py|  8 
 sdks/python/apache_beam/transforms/core.py  |  8 
 12 files changed, 46 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0dcc9bc0/sdks/python/apache_beam/io/avroio.py
--
diff --git a/sdks/python/apache_beam/io/avroio.py 
b/sdks/python/apache_beam/io/avroio.py
index 6cba12d..35d0e94 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -75,8 +75,7 @@ class ReadFromAvro(PTransform):
   **kwargs: Additional keyword arguments to be passed to the base class.
 """
 super(ReadFromAvro, self).__init__()
-self._args = (file_pattern, min_bundle_size)
-self._source = _AvroSource(*self._args, validate=validate)
+self._source = _AvroSource(file_pattern, min_bundle_size, 
validate=validate)
 
   def apply(self, pvalue):
 return pvalue.pipeline | Read(self._source)
@@ -292,9 +291,8 @@ class WriteToAvro(beam.transforms.PTransform):
 Returns:
   A WriteToAvro transform usable for writing.
 """
-self._args = (file_path_prefix, schema, codec, file_name_suffix, 
num_shards,
-  shard_name_template, mime_type)
-self._sink = _AvroSink(*self._args)
+self._sink = _AvroSink(file_path_prefix, schema, codec, file_name_suffix,
+   num_shards, shard_name_template, mime_type)
 
   def apply(self, pcoll):
 return pcoll | beam.io.iobase.Write(self._sink)

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0dcc9bc0/sdks/python/apache_beam/io/avroio_test.py
--
diff --git a/sdks/python/apache_beam/io/avroio_test.py 
b/sdks/python/apache_beam/io/avroio_test.py
index e8fb12b..e5db196 100644
--- a/sdks/python/apache_beam/io/avroio_test.py
+++ b/sdks/python/apache_beam/io/avroio_test.py
@@ -157,7 +157,7 @@ class TestAvro(unittest.TestCase):
 # No extra avro parameters for AvroSource.
 expected_items = [
 DisplayDataItemMatcher('compression', 'auto'),
-DisplayDataItemMatcher('filePattern', file_name)]
+DisplayDataItemMatcher('file_pattern', file_name)]
 hc.assert_that(dd.items, hc.contains_inanyorder(*expected_items))
 
   def test_read_display_data(self):
@@ -168,7 +168,7 @@ class TestAvro(unittest.TestCase):
 # No extra avro parameters for AvroSource.
 expected_items = [
 DisplayDataItemMatcher('compression', 'auto'),
-DisplayDataItemMatcher('filePattern', file_name)]
+DisplayDataItemMatcher('file_pattern', file_name)]
 hc.assert_that(dd.items, hc.contains_inanyorder(*expected_items))
 
   def test_sink_display_data(self):
@@ -186,7 +186,7 @@ class TestAvro(unittest.TestCase):
 'schema',
 str(self.SCHEMA)),
 DisplayDataItemMatcher(
-'filePattern',
+'file_pattern',
 'some_avro_sink-%(shard_num)05d-of-%(num_shards)05d.end'),
 DisplayDataItemMatcher(
 'shards',
@@ -209,7 +209,7 @@ class TestAvro(unittest.TestCase):
 'schema',
 str(self.SCHEMA)),
 DisplayDataItemMatcher(
-'filePattern',
+'file_pattern',
 'some_avro_sink-%(shard_num)05d-of-%(num_shards)05d'),
 DisplayDataItemMatcher(
 'shards',

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0dcc9bc0/sdks/p

[2/2] incubator-beam git commit: Closes #1443

[bchambers]

Closes #1443


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d5e8c79a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d5e8c79a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d5e8c79a

Branch: refs/heads/python-sdk
Commit: d5e8c79a3f9f4ffabaff06629a8763e74254688c
Parents: f19f767 0dcc9bc
Author: bchambers 
Authored: Tue Dec 6 17:17:50 2016 -0800
Committer: bchambers 
Committed: Tue Dec 6 17:17:50 2016 -0800

--
 sdks/python/apache_beam/io/avroio.py|  8 +++-
 sdks/python/apache_beam/io/avroio_test.py   |  8 
 sdks/python/apache_beam/io/filebasedsource.py   |  3 ++-
 .../apache_beam/io/filebasedsource_test.py  |  4 ++--
 sdks/python/apache_beam/io/fileio.py|  8 
 sdks/python/apache_beam/io/fileio_test.py   | 12 ++--
 sdks/python/apache_beam/io/pubsub.py|  2 +-
 sdks/python/apache_beam/io/pubsub_test.py   |  2 +-
 sdks/python/apache_beam/io/textio.py| 20 ++--
 sdks/python/apache_beam/io/textio_test.py   |  8 +---
 .../apache_beam/transforms/combiners_test.py|  8 
 sdks/python/apache_beam/transforms/core.py  |  8 
 12 files changed, 46 insertions(+), 45 deletions(-)
--

[jira] [Commented] (BEAM-1101) Remove inconsistencies in Python PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727416#comment-15727416
 ] 

ASF GitHub Bot commented on BEAM-1101:
--

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1526

[BEAM-1101] Remove inconsistencies in Python PipelineOptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam 
poptions-inconsistencies

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1526.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1526


commit c6362aed50eae9cf8b9e5ec6d0885132f3278e32
Author: Pablo 
Date:   2016-12-07T02:01:54Z

Fixing inconsistencies in PipelineOptions




> Remove inconsistencies in Python PipelineOptions
> 
>
> Key: BEAM-1101
> URL: https://issues.apache.org/jira/browse/BEAM-1101
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Frances Perry
>
> Some options have been removed from Java, and some have different default 
> behavior in Java. Gotta make this consistent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1526: [BEAM-1101] Remove inconsistencies in Pyt...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1526

[BEAM-1101] Remove inconsistencies in Python PipelineOptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam 
poptions-inconsistencies

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1526.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1526


commit c6362aed50eae9cf8b9e5ec6d0885132f3278e32
Author: Pablo 
Date:   2016-12-07T02:01:54Z

Fixing inconsistencies in PipelineOptions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1101) Remove inconsistencies in Python PipelineOptions

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-1101:
---

 Summary: Remove inconsistencies in Python PipelineOptions
 Key: BEAM-1101
 URL: https://issues.apache.org/jira/browse/BEAM-1101
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Pablo Estrada
Assignee: Frances Perry


Some options have been removed from Java, and some have different default 
behavior in Java. Gotta make this consistent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-beam git commit: BigQueryIO.Read: support runtime options

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master b2b570f27 -> 0a2ed832c


BigQueryIO.Read: support runtime options


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0a2ed832
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0a2ed832
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0a2ed832

Branch: refs/heads/master
Commit: 0a2ed832ce5af7556db605e99b985ed4ffc1b152
Parents: b2b570f
Author: Sam McVeety 
Authored: Sun Oct 30 11:58:44 2016 -0700
Committer: Dan Halperin 
Committed: Tue Dec 6 18:05:42 2016 -0800

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 208 ++-
 .../apache/beam/sdk/io/gcp/ApiSurfaceTest.java  |   2 +
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java |  33 +--
 3 files changed, 176 insertions(+), 67 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0a2ed832/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index c00c19d..8bfbd53 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -89,6 +89,9 @@ import 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.JobService;
 import org.apache.beam.sdk.options.BigQueryOptions;
 import org.apache.beam.sdk.options.GcpOptions;
 import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.Create;
@@ -318,6 +321,38 @@ public class BigQueryIO {
 return sb.toString();
   }
 
+  private static class JsonTableRefToTableRef
+  implements SerializableFunction {
+@Override
+public TableReference apply(String from) {
+  return fromJsonString(from, TableReference.class);
+}
+  }
+
+  private static class TableRefToJson
+  implements SerializableFunction {
+@Override
+public String apply(TableReference from) {
+  return toJsonString(from);
+}
+  }
+
+  private static class TableRefToProjectId
+  implements SerializableFunction {
+@Override
+public String apply(TableReference from) {
+  return from.getProjectId();
+}
+  }
+
+  private static class TableSpecToTableRef
+  implements SerializableFunction {
+@Override
+public TableReference apply(String from) {
+  return parseTableSpec(from);
+}
+  }
+
   /**
* A {@link PTransform} that reads from a BigQuery table and returns a
* {@link PCollection} of {@link TableRow TableRows} containing each of the 
rows of the table.
@@ -345,6 +380,13 @@ public class BigQueryIO {
  * {@code "[dataset_id].[table_id]"} for tables within the current project.
  */
 public static Bound from(String tableSpec) {
+  return new Bound().from(StaticValueProvider.of(tableSpec));
+}
+
+/**
+ * Same as {@code from(String)}, but with a {@link ValueProvider}.
+ */
+public static Bound from(ValueProvider tableSpec) {
   return new Bound().from(tableSpec);
 }
 
@@ -352,6 +394,13 @@ public class BigQueryIO {
  * Reads results received after executing the given query.
  */
 public static Bound fromQuery(String query) {
+  return new Bound().fromQuery(StaticValueProvider.of(query));
+}
+
+/**
+ * Same as {@code from(String)}, but with a {@link ValueProvider}.
+ */
+public static Bound fromQuery(ValueProvider query) {
   return new Bound().fromQuery(query);
 }
 
@@ -374,8 +423,8 @@ public class BigQueryIO {
  * {@link PCollection} of {@link TableRow TableRows}.
  */
 public static class Bound extends PTransform> {
-  @Nullable final String jsonTableRef;
-  @Nullable final String query;
+  @Nullable final ValueProvider jsonTableRef;
+  @Nullable final ValueProvider query;
 
   /**
* Disable validation that the table exists or the query succeeds prior 
to pipeline
@@ -403,7 +452,8 @@ public class BigQueryIO {
   }
 
   private Bound(
-  String name, @Nullable String query, @Nullable String jsonTableRef, 
boolean validate,
+  String name, @Nullable ValueProvider query,
+  

[GitHub] incubator-beam pull request #1525: Fixes a bug in SDF checkpointing.

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1525

Fixes a bug in SDF checkpointing.

It would call checkpoint() multiple times if the SDF emits output several 
times per claim call.

(also slightly improves an error message when using SDF in an unsupported 
runner - spotted cause I was running the tests without having configured them 
for direct runner)

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam sdf-multiple-checkpoint

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1525.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1525


commit 64c7481780e039b235fa687b25c839c871e99b23
Author: Eugene Kirpichov 
Date:   2016-12-07T02:00:03Z

Fixes a bug in SDF checkpointing.

It would call checkpoint() multiple times if the SDF emits output
several times per claim call.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1100) Consider removing pylint from mvn verify

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1100:
-

 Summary: Consider removing pylint from mvn verify
 Key: BEAM-1100
 URL: https://issues.apache.org/jira/browse/BEAM-1100
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Ahmet Altay
Priority: Minor


pylint depends on git commands for getting a list of changed files to run the 
linter on. This does not make sense in some cases (for example when in the case 
code was copied/packaged out of git structure). A few possible remedies could 
be considered:

- Remove pylint step from mvn verify
- Add an option to pylint run on whole code base and use this in mvn verify
- Change pylint to run on the whole code base (instead of just changed files)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #2010

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Spark #330

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Spark #330

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-1085) Crunch Runner for Beam

[Neelesh Srinivas Salian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neelesh Srinivas Salian updated BEAM-1085:
--
Component/s: runner-ideas

> Crunch Runner for Beam
> --
>
> Key: BEAM-1085
> URL: https://issues.apache.org/jira/browse/BEAM-1085
> Project: Beam
>  Issue Type: Wish
>  Components: runner-ideas
>Reporter: Neelesh Srinivas Salian
>
> It came up during the BoF Beam talk earlier last month; opening this JIRA as 
> a placeholder for if there is interest/ desire to add this feature. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1743

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1743

[Apache Jenkins Server]

See 

[2/4] incubator-beam git commit: [BEAM-1047] Add DataflowClient wrapper on top of JSON library.

[bchambers]

[BEAM-1047] Add DataflowClient wrapper on top of JSON library.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e8c9686a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e8c9686a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e8c9686a

Branch: refs/heads/master
Commit: e8c9686a2e898d38afd692328eb171c542084747
Parents: ded5883
Author: Pei He 
Authored: Wed Nov 23 15:59:56 2016 -0800
Committer: bchambers 
Committed: Tue Dec 6 17:08:12 2016 -0800

--
 .../beam/runners/dataflow/DataflowClient.java   | 130 +++
 1 file changed, 130 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e8c9686a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
new file mode 100644
index 000..f2081db
--- /dev/null
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.api.services.dataflow.Dataflow;
+import com.google.api.services.dataflow.Dataflow.Projects.Jobs;
+import com.google.api.services.dataflow.model.Job;
+import com.google.api.services.dataflow.model.JobMetrics;
+import com.google.api.services.dataflow.model.LeaseWorkItemRequest;
+import com.google.api.services.dataflow.model.LeaseWorkItemResponse;
+import com.google.api.services.dataflow.model.ListJobMessagesResponse;
+import com.google.api.services.dataflow.model.ListJobsResponse;
+import com.google.api.services.dataflow.model.ReportWorkItemStatusRequest;
+import com.google.api.services.dataflow.model.ReportWorkItemStatusResponse;
+import java.io.IOException;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
+
+/**
+ * Client library for {@link Dataflow}.
+ */
+public class DataflowClient {
+
+  public static DataflowClient create(DataflowPipelineOptions options) {
+return new DataflowClient(options.getDataflowClient(), options);
+  }
+
+  private final Dataflow dataflow;
+  private final DataflowPipelineOptions options;
+
+  private DataflowClient(Dataflow dataflow, DataflowPipelineOptions options) {
+this.dataflow = checkNotNull(dataflow, "dataflow");
+this.options = checkNotNull(options, "options");
+  }
+
+  /**
+   * Creates the Dataflow {@link Job}.
+   */
+  public Job createJob(@Nonnull Job job) throws IOException {
+Jobs.Create jobsCreate = 
dataflow.projects().jobs().create(options.getProject(), job);
+return jobsCreate.execute();
+  }
+
+  /**
+   * Lists Dataflow {@link Job Jobs} in the project associated with
+   * the {@link DataflowPipelineOptions}.
+   */
+  public ListJobsResponse listJobs(@Nullable String pageToken) throws 
IOException {
+Jobs.List jobsList = dataflow.projects().jobs()
+.list(options.getProject())
+.setPageToken(pageToken);
+return jobsList.execute();
+  }
+
+  /**
+   * Updates the Dataflow {@link Job} with the given {@code jobId}.
+   */
+  public Job updateJob(@Nonnull String jobId, @Nonnull Job content) throws 
IOException {
+Jobs.Update jobsUpdate = dataflow.projects().jobs()
+.update(options.getProject(), jobId, content);
+return jobsUpdate.execute();
+  }
+
+  /**
+   * Gets the Dataflow {@link Job} with the given {@code jobId}.
+   */
+  public Job getJob(@Nonnull String jobId) throws IOException {
+Jobs.Get jobsGet = dataflow.projects().jobs()
+.get(options.getProject(), jobId);
+return jobsGet.execute();
+  }
+
+  /

[3/4] incubator-beam git commit: [Code Health] Remove redundant projectId from DataflowPipelineJob.

[bchambers]

[Code Health] Remove redundant projectId from DataflowPipelineJob.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ded58832
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ded58832
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ded58832

Branch: refs/heads/master
Commit: ded58832ceaef487f4590d9396f09744288c955d
Parents: afedd68
Author: Pei He 
Authored: Wed Nov 23 16:14:27 2016 -0800
Committer: bchambers 
Committed: Tue Dec 6 17:08:12 2016 -0800

--
 .../runners/dataflow/DataflowPipelineJob.java   | 22 +++--
 .../beam/runners/dataflow/DataflowRunner.java   |  4 +-
 .../dataflow/util/DataflowTemplateJob.java  |  2 +-
 .../dataflow/DataflowPipelineJobTest.java   | 48 ++--
 .../testing/TestDataflowRunnerTest.java | 36 +--
 5 files changed, 45 insertions(+), 67 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ded58832/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
index a2b632f..58e85e0 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
@@ -62,11 +62,6 @@ public class DataflowPipelineJob implements PipelineResult {
   private String jobId;
 
   /**
-   * Google cloud project to associate this pipeline with.
-   */
-  private String projectId;
-
-  /**
* Client for the Dataflow service. This can be used to query the service
* for information about the job.
*/
@@ -119,17 +114,14 @@ public class DataflowPipelineJob implements 
PipelineResult {
   /**
* Constructs the job.
*
-   * @param projectId the project id
* @param jobId the job id
* @param dataflowOptions used to configure the client for the Dataflow 
Service
* @param aggregatorTransforms a mapping from aggregators to PTransforms
*/
   public DataflowPipelineJob(
-  String projectId,
   String jobId,
   DataflowPipelineOptions dataflowOptions,
   DataflowAggregatorTransforms aggregatorTransforms) {
-this.projectId = projectId;
 this.jobId = jobId;
 this.dataflowOptions = dataflowOptions;
 this.aggregatorTransforms = aggregatorTransforms;
@@ -146,7 +138,7 @@ public class DataflowPipelineJob implements PipelineResult {
* Get the project this job exists in.
*/
   public String getProjectId() {
-return projectId;
+return dataflowOptions.getProject();
   }
 
   /**
@@ -249,7 +241,7 @@ public class DataflowPipelineJob implements PipelineResult {
   MonitoringUtil.JobMessagesHandler messageHandler,
   Sleeper sleeper,
   NanoClock nanoClock) throws IOException, InterruptedException {
-MonitoringUtil monitor = new MonitoringUtil(projectId, 
dataflowOptions.getDataflowClient());
+MonitoringUtil monitor = new MonitoringUtil(getProjectId(), 
dataflowOptions.getDataflowClient());
 
 long lastTimestamp = 0;
 BackOff backoff;
@@ -338,12 +330,12 @@ public class DataflowPipelineJob implements 
PipelineResult {
   @Override
   public State cancel() throws IOException {
 Job content = new Job();
-content.setProjectId(projectId);
+content.setProjectId(getProjectId());
 content.setId(jobId);
 content.setRequestedState("JOB_STATE_CANCELLED");
 try {
   dataflowOptions.getDataflowClient().projects().jobs()
-  .update(projectId, jobId, content)
+  .update(getProjectId(), jobId, content)
   .execute();
   return State.CANCELLED;
 } catch (IOException e) {
@@ -412,13 +404,13 @@ public class DataflowPipelineJob implements 
PipelineResult {
 Job job = dataflowOptions.getDataflowClient()
 .projects()
 .jobs()
-.get(projectId, jobId)
+.get(getProjectId(), jobId)
 .execute();
 State currentState = MonitoringUtil.toState(job.getCurrentState());
 if (currentState.isTerminal()) {
   terminalState = currentState;
   replacedByJob = new DataflowPipelineJob(
-  getProjectId(), job.getReplacedByJobId(), dataflowOptions, 
aggregatorTransforms);
+  job.getReplacedByJobId(), dataflowOptions, aggregatorTransforms);
 }
 return job;
   } catch (IOException exn) {
@@ -485,7 +477,7 @@ public class DataflowPipelineJob implem

[jira] [Commented] (BEAM-1047) DataflowRunner: support regionalization.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727302#comment-15727302
 ] 

ASF GitHub Bot commented on BEAM-1047:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1434


> DataflowRunner: support regionalization.
> 
>
> Key: BEAM-1047
> URL: https://issues.apache.org/jira/browse/BEAM-1047
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Pei He
>Assignee: Pei He
>
> Tracking bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/4] incubator-beam git commit: Closes #1434

[bchambers]

Closes #1434


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b2b570f2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b2b570f2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b2b570f2

Branch: refs/heads/master
Commit: b2b570f27808b1671bf6cd0fc60f874da671d4ca
Parents: afedd68 ce03f30
Author: bchambers 
Authored: Tue Dec 6 17:08:13 2016 -0800
Committer: bchambers 
Committed: Tue Dec 6 17:08:13 2016 -0800

--
 .../beam/runners/dataflow/DataflowClient.java   | 140 +++
 .../runners/dataflow/DataflowPipelineJob.java   |  31 ++--
 .../beam/runners/dataflow/DataflowRunner.java   |  20 +--
 .../dataflow/testing/TestDataflowRunner.java|   6 +-
 .../dataflow/util/DataflowTemplateJob.java  |   2 +-
 .../runners/dataflow/util/MonitoringUtil.java   |  22 +--
 .../dataflow/DataflowPipelineJobTest.java   |  49 ---
 .../testing/TestDataflowRunnerTest.java |  36 ++---
 .../transforms/DataflowGroupByKeyTest.java  |  14 +-
 .../dataflow/transforms/DataflowViewTest.java   |  16 ++-
 .../dataflow/util/MonitoringUtilTest.java   |  21 +--
 11 files changed, 235 insertions(+), 122 deletions(-)
--

[1/4] incubator-beam git commit: [BEAM-1047] Update dataflow runner code to use DataflowClient wrapper.

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/master afedd68e8 -> b2b570f27


[BEAM-1047] Update dataflow runner code to use DataflowClient wrapper.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ce03f30c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ce03f30c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ce03f30c

Branch: refs/heads/master
Commit: ce03f30c1ee0b84ad2e7f10a6272ffb25548244a
Parents: e8c9686
Author: Pei He 
Authored: Mon Nov 28 11:47:42 2016 -0800
Committer: bchambers 
Committed: Tue Dec 6 17:08:12 2016 -0800

--
 .../beam/runners/dataflow/DataflowClient.java   | 36 +---
 .../runners/dataflow/DataflowPipelineJob.java   | 23 ++---
 .../beam/runners/dataflow/DataflowRunner.java   | 16 +++--
 .../dataflow/testing/TestDataflowRunner.java|  6 ++--
 .../runners/dataflow/util/MonitoringUtil.java   | 22 +++-
 .../dataflow/DataflowPipelineJobTest.java   |  1 +
 .../transforms/DataflowGroupByKeyTest.java  | 14 +++-
 .../dataflow/transforms/DataflowViewTest.java   | 16 +++--
 .../dataflow/util/MonitoringUtilTest.java   | 21 +++-
 9 files changed, 80 insertions(+), 75 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ce03f30c/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
index f2081db..3536d72 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
@@ -35,27 +35,28 @@ import javax.annotation.Nullable;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
 
 /**
- * Client library for {@link Dataflow}.
+ * Wrapper around the generated {@link Dataflow} client to provide common 
functionality.
  */
 public class DataflowClient {
 
   public static DataflowClient create(DataflowPipelineOptions options) {
-return new DataflowClient(options.getDataflowClient(), options);
+return new DataflowClient(options.getDataflowClient(), 
options.getProject());
   }
 
   private final Dataflow dataflow;
-  private final DataflowPipelineOptions options;
+  private final String projectId;
 
-  private DataflowClient(Dataflow dataflow, DataflowPipelineOptions options) {
+  private DataflowClient(Dataflow dataflow, String projectId) {
 this.dataflow = checkNotNull(dataflow, "dataflow");
-this.options = checkNotNull(options, "options");
+this.projectId = checkNotNull(projectId, "options");
   }
 
   /**
* Creates the Dataflow {@link Job}.
*/
   public Job createJob(@Nonnull Job job) throws IOException {
-Jobs.Create jobsCreate = 
dataflow.projects().jobs().create(options.getProject(), job);
+checkNotNull(job, "job");
+Jobs.Create jobsCreate = dataflow.projects().jobs().create(projectId, job);
 return jobsCreate.execute();
   }
 
@@ -65,7 +66,7 @@ public class DataflowClient {
*/
   public ListJobsResponse listJobs(@Nullable String pageToken) throws 
IOException {
 Jobs.List jobsList = dataflow.projects().jobs()
-.list(options.getProject())
+.list(projectId)
 .setPageToken(pageToken);
 return jobsList.execute();
   }
@@ -74,8 +75,10 @@ public class DataflowClient {
* Updates the Dataflow {@link Job} with the given {@code jobId}.
*/
   public Job updateJob(@Nonnull String jobId, @Nonnull Job content) throws 
IOException {
+checkNotNull(jobId, "jobId");
+checkNotNull(content, "content");
 Jobs.Update jobsUpdate = dataflow.projects().jobs()
-.update(options.getProject(), jobId, content);
+.update(projectId, jobId, content);
 return jobsUpdate.execute();
   }
 
@@ -83,8 +86,9 @@ public class DataflowClient {
* Gets the Dataflow {@link Job} with the given {@code jobId}.
*/
   public Job getJob(@Nonnull String jobId) throws IOException {
+checkNotNull(jobId, "jobId");
 Jobs.Get jobsGet = dataflow.projects().jobs()
-.get(options.getProject(), jobId);
+.get(projectId, jobId);
 return jobsGet.execute();
   }
 
@@ -92,8 +96,9 @@ public class DataflowClient {
* Gets the {@link JobMetrics} with the given {@code jobId}.
*/
   public JobMetrics getJobMetrics(@Nonnull String jobId) throws IOException {
+checkNotNull(jobId, "jobId");
 Jobs.GetMetrics jobsGetMetrics = dataflow.projec

[GitHub] incubator-beam pull request #1434: [BEAM-1047] Update dataflow runner code t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1434


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Spark #329

[Apache Jenkins Server]

See 


Changes:

[davor] [BEAM-551] Add support for native PubSubIO

--
[...truncated 2438 lines...]
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.0.6/maven-settings-2.0.6.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.6/maven-plugin-parameter-documenter-2.0.6.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.6/maven-reporting-api-2.0.6.jar
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.6/maven-plugin-api-2.0.6.jar
 (13 KB at 199.3 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.0-alpha-7/doxia-sink-api-1.0-alpha-7.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.0.6/maven-settings-2.0.6.jar
 (48 KB at 978.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.6/maven-profile-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.0-alpha-7/doxia-sink-api-1.0-alpha-7.jar
 (6 KB at 101.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.6/maven-model-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.6/maven-profile-2.0.6.jar
 (35 KB at 537.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.6/maven-repository-metadata-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.6/maven-repository-metadata-2.0.6.jar
 (24 KB at 259.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.6/maven-error-diagnostics-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.6/maven-model-2.0.6.jar
 (85 KB at 852.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.6/maven-plugin-descriptor-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.6/maven-reporting-api-2.0.6.jar
 (10 KB at 92.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.6/maven-artifact-manager-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.6/maven-error-diagnostics-2.0.6.jar
 (14 KB at 110.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.6/maven-monitor-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.6/maven-plugin-descriptor-2.0.6.jar
 (37 KB at 277.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.6/maven-plugin-parameter-documenter-2.0.6.jar
 (21 KB at 137.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.6/maven-plugin-registry-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.6/maven-artifact-manager-2.0.6.jar
 (56 KB at 388.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.0.6/maven-artifact-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.6/maven-monitor-2.0.6.jar
 (11 KB at 68.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.15/plexus-interpolation-1.15.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.6/maven-plugin-registry-2.0.6.jar
 (29 KB at 170.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/2.4.4/plexus-archiver-2.4.4.jar
[INFO]

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2009

[Apache Jenkins Server]

See 


Changes:

[davor] [BEAM-551] Add support for native PubSubIO

--
[...truncated 4497 lines...]
[INFO] 
[INFO] --- maven-source-plugin:2.4:test-jar-no-fork (attach-test-sources) @ 
beam-sdks-java-core ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-jar-plugin:2.5:test-jar (default-test-jar) @ 
beam-sdks-java-core ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-shade-plugin:2.4.3:shade (bundle-and-repackage) @ 
beam-sdks-java-core ---
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-compat/3.0/maven-compat-3.0.pom
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jdom/jdom/1.1/jdom-1.1.pom
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jdom/jdom/1.1/jdom-1.1.pom (3 KB at 
17.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/vafer/jdependency/1.1/jdependency-1.1.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/vafer/jdependency/1.1/jdependency-1.1.pom
 (11 KB at 245.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/5.0.4/asm-5.0.4.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/5.0.4/asm-5.0.4.pom (2 KB 
at 51.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-parent/5.0.4/asm-parent-5.0.4.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-parent/5.0.4/asm-parent-5.0.4.pom
 (6 KB at 134.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-analysis/5.0.4/asm-analysis-5.0.4.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-analysis/5.0.4/asm-analysis-5.0.4.pom
 (3 KB at 53.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-tree/5.0.4/asm-tree-5.0.4.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-tree/5.0.4/asm-tree-5.0.4.pom
 (3 KB at 53.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-commons/5.0.4/asm-commons-5.0.4.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-commons/5.0.4/asm-commons-5.0.4.pom
 (3 KB at 53.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-util/5.0.4/asm-util-5.0.4.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-util/5.0.4/asm-util-5.0.4.pom
 (3 KB at 52.3 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/11.0.2/guava-11.0.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/11.0.2/guava-11.0.2.pom
 (6 KB at 130.3 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/11.0.2/guava-parent-11.0.2.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/11.0.2/guava-parent-11.0.2.pom
 (2 KB at 49.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.pom
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.pom
 (965 B at 23.6 KB/sec)
[WARNING] Failed to getClass for org.apache.maven.plugins.shade.mojo.ShadeMojo
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [ 25.091 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  3.872 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  7.165 s]
[INFO] Apache Beam :: SDKs :: Java  SUCCESS [  1.795 s]
[INFO] Apache Beam :: SDKs :: Java :: Core  FAILURE [03:20 min]
[INFO] Apache Beam :: Runners ..

[jira] [Commented] (BEAM-1099) Minor typos in KafkaIO

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727280#comment-15727280
 ] 

Davor Bonaci commented on BEAM-1099:


Thanks [~jghoman], much appreciated.

> Minor typos in KafkaIO
> --
>
> Key: BEAM-1099
> URL: https://issues.apache.org/jira/browse/BEAM-1099
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Jakob Homan
>Assignee: Davor Bonaci
>Priority: Trivial
> Fix For: 0.4.0-incubating
>
>
> While familiarizing myself with the KafkaIO support, I found and fixed a few 
> typos in the comments for that class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Spark #329

[Apache Jenkins Server]

See 


Changes:

[davor] [BEAM-551] Add support for native PubSubIO

--
[...truncated 2439 lines...]
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.6/maven-plugin-parameter-documenter-2.0.6.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.6/maven-reporting-api-2.0.6.jar
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.6/maven-plugin-api-2.0.6.jar
 (13 KB at 199.3 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.0-alpha-7/doxia-sink-api-1.0-alpha-7.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.0.6/maven-settings-2.0.6.jar
 (48 KB at 978.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.6/maven-profile-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.0-alpha-7/doxia-sink-api-1.0-alpha-7.jar
 (6 KB at 101.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.6/maven-model-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.0.6/maven-profile-2.0.6.jar
 (35 KB at 537.7 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.6/maven-repository-metadata-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.6/maven-repository-metadata-2.0.6.jar
 (24 KB at 259.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.6/maven-error-diagnostics-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.6/maven-model-2.0.6.jar
 (85 KB at 852.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.6/maven-plugin-descriptor-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.0.6/maven-reporting-api-2.0.6.jar
 (10 KB at 92.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.6/maven-artifact-manager-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.0.6/maven-error-diagnostics-2.0.6.jar
 (14 KB at 110.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.6/maven-monitor-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0.6/maven-plugin-descriptor-2.0.6.jar
 (37 KB at 277.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.0.6/maven-plugin-parameter-documenter-2.0.6.jar
 (21 KB at 137.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.6/maven-plugin-registry-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.6/maven-artifact-manager-2.0.6.jar
 (56 KB at 388.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.0.6/maven-artifact-2.0.6.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-monitor/2.0.6/maven-monitor-2.0.6.jar
 (11 KB at 68.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.15/plexus-interpolation-1.15.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.0.6/maven-plugin-registry-2.0.6.jar
 (29 KB at 170.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/2.4.4/plexus-archiver-2.4.4.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.6/maven-project-2.0.6.jar
 (114 KB 

[jira] [Commented] (BEAM-1099) Minor typos in KafkaIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727277#comment-15727277
 ] 

ASF GitHub Bot commented on BEAM-1099:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1524


> Minor typos in KafkaIO
> --
>
> Key: BEAM-1099
> URL: https://issues.apache.org/jira/browse/BEAM-1099
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Jakob Homan
>Assignee: Davor Bonaci
>Priority: Trivial
> Fix For: 0.4.0-incubating
>
>
> While familiarizing myself with the KafkaIO support, I found and fixed a few 
> typos in the comments for that class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1524: [BEAM-1099] Minor typos in KafkaIO

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1524


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1099) Minor typos in KafkaIO

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-1099.

   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> Minor typos in KafkaIO
> --
>
> Key: BEAM-1099
> URL: https://issues.apache.org/jira/browse/BEAM-1099
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Jakob Homan
>Assignee: Davor Bonaci
>Priority: Trivial
> Fix For: 0.4.0-incubating
>
>
> While familiarizing myself with the KafkaIO support, I found and fixed a few 
> typos in the comments for that class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-1099] Minor typos in KafkaIO

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 0af145bba -> afedd68e8


[BEAM-1099] Minor typos in KafkaIO


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e9f254ef
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e9f254ef
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e9f254ef

Branch: refs/heads/master
Commit: e9f254ef2769a082c7fbb500c1c28c6224ac5a7f
Parents: 0af145b
Author: Jakob Homan 
Authored: Tue Dec 6 16:59:50 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Dec 6 17:03:42 2016 -0800

--
 .../main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java   | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e9f254ef/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index 4212d59..c87d12b 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -106,7 +106,7 @@ import org.slf4j.LoggerFactory;
  * metadata like topic-partition and offset, along with key and value 
associated with a Kafka
  * record.
  *
- * Although most applications consumer single topic, the source can be 
configured to consume
+ * Although most applications consume a single topic, the source can be 
configured to consume
  * multiple topics or even a specific set of {@link TopicPartition}s.
  *
  * To configure a Kafka source, you must specify at the minimum Kafka 
bootstrapServers
@@ -250,8 +250,8 @@ public class KafkaIO {
 }
 
 /**
- * Returns a new {@link Read} that reads from the topics. All the 
partitions are from each
- * of the topics is read.
+ * Returns a new {@link Read} that reads from the topics. All the 
partitions from each
+ * of the topics are read.
  * See {@link UnboundedKafkaSource#generateInitialSplits(int, 
PipelineOptions)} for description
  * of how the partitions are distributed among the splits.
  */
@@ -1134,7 +1134,7 @@ public class KafkaIO {
 
   boolean isShutdown = false;
 
-  // Wait for threads to shutdown. Trying this a loop to handle a tiny 
race where poll thread
+  // Wait for threads to shutdown. Trying this as a loop to handle a tiny 
race where poll thread
   // might block to enqueue right after availableRecordsQueue.poll() below.
   while (!isShutdown) {
 
@@ -1178,7 +1178,7 @@ public class KafkaIO {
 }
 
 /**
- * Returns a new {@link Write} transform that write to given topic.
+ * Returns a new {@link Write} transform that writes to given topic.
  */
 public Write withTopic(String topic) {
   return new Write(topic, keyCoder, valueCoder, producerConfig);

[2/2] incubator-beam git commit: This closes #1524

[davor]

This closes #1524


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/afedd68e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/afedd68e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/afedd68e

Branch: refs/heads/master
Commit: afedd68e806830549724dfc0f2565d756cc6b46d
Parents: 0af145b e9f254e
Author: Davor Bonaci 
Authored: Tue Dec 6 17:03:54 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Dec 6 17:03:54 2016 -0800

--
 .../main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java   | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)
--

[jira] [Commented] (BEAM-1099) Minor typos in KafkaIO

[Jakob Homan (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727272#comment-15727272
 ] 

Jakob Homan commented on BEAM-1099:
---

https://github.com/apache/incubator-beam/pull/1524

> Minor typos in KafkaIO
> --
>
> Key: BEAM-1099
> URL: https://issues.apache.org/jira/browse/BEAM-1099
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Jakob Homan
>Assignee: Davor Bonaci
>Priority: Trivial
>
> While familiarizing myself with the KafkaIO support, I found and fixed a few 
> typos in the comments for that class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1099) Minor typos in KafkaIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727271#comment-15727271
 ] 

ASF GitHub Bot commented on BEAM-1099:
--

GitHub user jghoman opened a pull request:

https://github.com/apache/incubator-beam/pull/1524

[BEAM-1099] Minor typos in KafkaIO

Fix various distracting typos in KafkaIO


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jghoman/incubator-beam BEAM-1099

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1524.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1524


commit a71e9e4b99c14fbe453fa3dca486fa268c9782f5
Author: Jakob Homan 
Date:   2016-12-07T00:59:50Z

[BEAM-1099] Minor typos in KafkaIO




> Minor typos in KafkaIO
> --
>
> Key: BEAM-1099
> URL: https://issues.apache.org/jira/browse/BEAM-1099
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Jakob Homan
>Assignee: Davor Bonaci
>Priority: Trivial
>
> While familiarizing myself with the KafkaIO support, I found and fixed a few 
> typos in the comments for that class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727268#comment-15727268
 ] 

ASF GitHub Bot commented on BEAM-551:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1504


> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-1098) Minor typos in KafkaIO

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-1098.
--
   Resolution: Duplicate
Fix Version/s: Not applicable

> Minor typos in KafkaIO
> --
>
> Key: BEAM-1098
> URL: https://issues.apache.org/jira/browse/BEAM-1098
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Jakob Homan
>Assignee: Davor Bonaci
>Priority: Trivial
> Fix For: Not applicable
>
>
> While familiarizing myself with the KafkaIO support, I found and fixed a few 
> typos in the comments for that class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1524: [BEAM-1099] Minor typos in KafkaIO

[jghoman]

GitHub user jghoman opened a pull request:

https://github.com/apache/incubator-beam/pull/1524

[BEAM-1099] Minor typos in KafkaIO

Fix various distracting typos in KafkaIO


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jghoman/incubator-beam BEAM-1099

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1524.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1524


commit a71e9e4b99c14fbe453fa3dca486fa268c9782f5
Author: Jakob Homan 
Date:   2016-12-07T00:59:50Z

[BEAM-1099] Minor typos in KafkaIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1504

[davor]

This closes #1504


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0af145bb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0af145bb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0af145bb

Branch: refs/heads/master
Commit: 0af145bba721f7cdf3fbcd15330df433d5a8431d
Parents: b733113 5c4415e
Author: Davor Bonaci 
Authored: Tue Dec 6 17:00:31 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Dec 6 17:00:31 2016 -0800

--
 .../beam/runners/dataflow/DataflowRunner.java   | 37 +++-
 .../apache/beam/sdk/io/PubsubUnboundedSink.java |  4 +++
 .../beam/sdk/io/PubsubUnboundedSource.java  | 10 ++
 .../apache/beam/sdk/options/ValueProvider.java  | 14 
 .../org/apache/beam/sdk/util/PropertyNames.java |  2 ++
 .../beam/sdk/options/ValueProviderTest.java |  9 +
 6 files changed, 68 insertions(+), 8 deletions(-)
--

[GitHub] incubator-beam pull request #1504: [BEAM-551] Add support for native PubSubI...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1504


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-551] Add support for native PubSubIO

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master b73311342 -> 0af145bba


[BEAM-551] Add support for native PubSubIO


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5c4415e2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5c4415e2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5c4415e2

Branch: refs/heads/master
Commit: 5c4415e2f0ac0ad3b12b6c78c4d41160b10acc64
Parents: b733113
Author: Sam McVeety 
Authored: Sat Dec 3 13:40:38 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Dec 6 17:00:15 2016 -0800

--
 .../beam/runners/dataflow/DataflowRunner.java   | 37 +++-
 .../apache/beam/sdk/io/PubsubUnboundedSink.java |  4 +++
 .../beam/sdk/io/PubsubUnboundedSource.java  | 10 ++
 .../apache/beam/sdk/options/ValueProvider.java  | 14 
 .../org/apache/beam/sdk/util/PropertyNames.java |  2 ++
 .../beam/sdk/options/ValueProviderTest.java |  9 +
 6 files changed, 68 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5c4415e2/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index f485cb8..339771b 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -118,6 +118,7 @@ import org.apache.beam.sdk.io.Write;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsValidator;
 import org.apache.beam.sdk.options.StreamingOptions;
+import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.runners.TransformHierarchy;
 import org.apache.beam.sdk.transforms.Aggregator;
@@ -2125,14 +2126,27 @@ public class DataflowRunner extends 
PipelineRunner {
   PubsubUnboundedSource overriddenTransform = 
transform.getOverriddenTransform();
   context.addStep(transform, "ParallelRead");
   context.addInput(PropertyNames.FORMAT, "pubsub");
-  if (overriddenTransform.getTopic() != null) {
-context.addInput(PropertyNames.PUBSUB_TOPIC,
- overriddenTransform.getTopic().getV1Beta1Path());
+  if (overriddenTransform.getTopicProvider() != null) {
+if (overriddenTransform.getTopicProvider().isAccessible()) {
+  context.addInput(
+  PropertyNames.PUBSUB_TOPIC, 
overriddenTransform.getTopic().getV1Beta1Path());
+} else {
+  context.addInput(
+  PropertyNames.PUBSUB_TOPIC_OVERRIDE,
+  ((NestedValueProvider) 
overriddenTransform.getTopicProvider()).propertyName());
+}
   }
-  if (overriddenTransform.getSubscription() != null) {
-context.addInput(
-PropertyNames.PUBSUB_SUBSCRIPTION,
-overriddenTransform.getSubscription().getV1Beta1Path());
+  if (overriddenTransform.getSubscriptionProvider() != null) {
+if (overriddenTransform.getSubscriptionProvider().isAccessible()) {
+  context.addInput(
+  PropertyNames.PUBSUB_SUBSCRIPTION,
+  overriddenTransform.getSubscription().getV1Beta1Path());
+} else {
+  context.addInput(
+  PropertyNames.PUBSUB_SUBSCRIPTION_OVERRIDE,
+  ((NestedValueProvider) 
overriddenTransform.getSubscriptionProvider())
+  .propertyName());
+}
   }
   if (overriddenTransform.getTimestampLabel() != null) {
 context.addInput(PropertyNames.PUBSUB_TIMESTAMP_LABEL,
@@ -2195,7 +2209,14 @@ public class DataflowRunner extends 
PipelineRunner {
   PubsubUnboundedSink overriddenTransform = 
transform.getOverriddenTransform();
   context.addStep(transform, "ParallelWrite");
   context.addInput(PropertyNames.FORMAT, "pubsub");
-  context.addInput(PropertyNames.PUBSUB_TOPIC, 
overriddenTransform.getTopic().getV1Beta1Path());
+  if (overriddenTransform.getTopicProvider().isAccessible()) {
+context.addInput(
+PropertyNames.PUBSUB_TOPIC, 
overriddenTransform.getTopic().getV1Beta1Path());
+  } else {
+context.addInput(
+PropertyNames.PUBSUB_TOPIC_OVERRIDE,
+((NestedValueProvider) 
overriddenTransform.getTopicProvider()).propertyName());
+  }
   if (overriddenTransform.getTimestampLabel() != nul

[jira] [Created] (BEAM-1099) Minor typos in KafkaIO

[Jakob Homan (JIRA)]

Jakob Homan created BEAM-1099:
-

 Summary: Minor typos in KafkaIO
 Key: BEAM-1099
 URL: https://issues.apache.org/jira/browse/BEAM-1099
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Jakob Homan
Assignee: Davor Bonaci
Priority: Trivial


While familiarizing myself with the KafkaIO support, I found and fixed a few 
typos in the comments for that class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1098) Minor typos in KafkaIO

[Jakob Homan (JIRA)]

Jakob Homan created BEAM-1098:
-

 Summary: Minor typos in KafkaIO
 Key: BEAM-1098
 URL: https://issues.apache.org/jira/browse/BEAM-1098
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Jakob Homan
Assignee: Davor Bonaci
Priority: Trivial


While familiarizing myself with the KafkaIO support, I found and fixed a few 
typos in the comments for that class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-9) Storm Runner

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727260#comment-15727260
 ] 

Davor Bonaci commented on BEAM-9:
-

Having a Storm runner would be great! Awesome!

At some point (in the future), I think it is worth clarifying the pros and cons 
of where the runner lives. On the technical side, I think it comes down to the 
API stability between different pairs of components (but, of course, there are 
other considerations as well).

> Storm Runner
> 
>
> Key: BEAM-9
> URL: https://issues.apache.org/jira/browse/BEAM-9
> Project: Beam
>  Issue Type: Wish
>  Components: runner-ideas
>Reporter: Frances Perry
>Assignee: Sriharsha Chintalapani
>
> Gathering place for interest in a Storm runner for Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1496: Add labels to lambdas in write finalizati...

[charlesccychen]

Github user charlesccychen closed the pull request at:

https://github.com/apache/incubator-beam/pull/1496


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1380: Fix issue where batch GCS renames were no...

[charlesccychen]

Github user charlesccychen closed the pull request at:

https://github.com/apache/incubator-beam/pull/1380


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1379: Improve GcsIO throughput by 10x

[charlesccychen]

Github user charlesccychen closed the pull request at:

https://github.com/apache/incubator-beam/pull/1379


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2007

[Apache Jenkins Server]

See 

--
[...truncated 12185 lines...]
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at 
org.jvnet.hudson.maven3.launcher.Maven32Launcher.main(Maven32Launcher.java:132)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
at jenkins.maven3.agent.Maven32Main.launch(Maven32Main.java:186)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:136)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:71)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not 
resolve dependencies for project 
org.apache.beam:beam-examples-java8:jar:0.4.0-incubating-SNAPSHOT: Failed to 
collect dependencies at org.apache.spark:spark-core_2.10:jar:1.6.2
at 
org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:180)
at 
org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
... 34 more
Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed 
to collect dependencies at org.apache.spark:spark-core_2.10:jar:1.6.2
at 
org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:291)
at 
org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:316)
at 
org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:172)
... 35 more
Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to 
read artifact descriptor for org.apache.spark:spark-core_2.10:jar:1.6.2
at 
org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:302)
at 
org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:218)
at 
org.eclipse.aether.internal.impl.DefaultDependencyCollector.resolveCachedArtifactDescriptor(DefaultDependencyCollector.java:535)
at 
org.eclipse.aether.internal.impl.DefaultDependencyCollector.getArtifactDescriptorResult(DefaultDependencyCollector.java:519)
at 
org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:409)
at 
org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:363)
at 
org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:351)
at 
org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependen

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2008

[Apache Jenkins Server]

See 


Changes:

[davor] [BEAM-551] Add DisplayData handling of ValueProvider

--
[...truncated 13106 lines...]
:45:
 error: reference not found
 * A {@link SourceDStream} is an {@link InputDStream} of {@link 
SourceRDD.Unbounded}s.
^
:54:
 error: reference not found
 * Records bound is controlled by the {@link RateController} mechanism.
 ^
:45:
 error: reference not found
 * Classes implementing Beam {@link Source} {@link RDD}s.
   ^
:51:
 error: reference not found
   * and creates a Spark {@link RDD}.
^
:51:
 error: reference not found
 * Read from within a stateful operation {@link 
JavaPairInputDStream#mapWithState(StateSpec)}
^
:23:
 error: reference not found
 * {@link org.apache.hadoop.mapreduce.lib.output.FileOutputFormat} implement to 
indicate
  ^
:26:
 error: reference not found
 * An adapter between the {@link NamedAggregators} and codahale's {@link Metric}
 ^
:27:
 error: reference not found
 * A Spark {@link Source} that is tailored to expose an {@link 
AggregatorMetric},
  ^
:48:
 error: reference not found
 * A {@link MetricRegistry} decorator-like* that supports {@link 
AggregatorMetric} by exposing
^
:50:
 error: reference not found
 * aggregators as {@link Gauge}s.
 ^
:52:
 error: reference not found
 * *{@link MetricRegistry} is not an interface, so this is not a by-the-book 
decorator.
   ^
:30:
 error: reference not found
 * A Spark {@link Sink} that is tailored to report {@link AggregatorMetric} 
metrics
  ^
:30:
 error: reference not found
 * A Spark {@link Sink} that is tailored to report {@link AggregatorMetric} 
metrics
  ^
:32:
 error: reference not found
 * Custom {@link com.esotericsoftware.kryo.Serializer}s for Beam's Spark runner 
needs.
 ^
:38:
 error: reference not found
 * A {@code WritableCoder} is a {@link Coder} for a Java class that implements 
{@link Writable}.

  ^

[1/2] incubator-beam git commit: [BEAM-551] Add DisplayData handling of ValueProvider

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5274dce5b -> b73311342


[BEAM-551] Add DisplayData handling of ValueProvider


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2c8311f0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2c8311f0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2c8311f0

Branch: refs/heads/master
Commit: 2c8311f0eaabc130f09cb8566717ce8ea885ba97
Parents: 5274dce
Author: Sam McVeety 
Authored: Sat Dec 3 13:20:24 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Dec 6 15:14:14 2016 -0800

--
 .../sdk/transforms/display/DisplayData.java | 16 
 .../sdk/transforms/display/DisplayDataTest.java | 86 
 2 files changed, 102 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2c8311f0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
index 1a2de97..f0040f7 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
@@ -37,6 +37,7 @@ import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
 import javax.annotation.Nullable;
+import org.apache.beam.sdk.options.ValueProvider;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.joda.time.Duration;
 import org.joda.time.Instant;
@@ -863,6 +864,21 @@ public class DisplayData implements Serializable {
   }
 
   /**
+   * Create a display item for the specified key and {@link ValueProvider}.
+   */
+  public static ItemSpec item(String key, ValueProvider value) {
+if (value.isAccessible()) {
+  Object got = value.get();
+  Type type = inferType(got);
+  if (type == null) {
+throw new RuntimeException(String.format("Unknown value type: %s", 
got));
+  }
+  return item(key, type, got);
+}
+return item(key, Type.STRING, value.toString());
+  }
+
+  /**
* Create a display item for the specified key and integer value.
*/
   public static ItemSpec item(String key, @Nullable Integer value) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2c8311f0/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
index 770b836..fccd031 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
@@ -53,6 +53,8 @@ import java.io.Serializable;
 import java.util.Collection;
 import java.util.Map;
 import java.util.regex.Pattern;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.display.DisplayData.Builder;
 import org.apache.beam.sdk.transforms.display.DisplayData.Item;
@@ -165,6 +167,90 @@ public class DisplayDataTest implements Serializable {
   }
 
   @Test
+  public void testStaticValueProviderDate() {
+final Instant value = Instant.now();
+DisplayData data =
+DisplayData.from(new HasDisplayData() {
+  @Override
+  public void populateDisplayData(DisplayData.Builder builder) {
+builder.add(DisplayData.item(
+"foo", StaticValueProvider.of(value)));
+  }
+});
+
+@SuppressWarnings("unchecked")
+DisplayData.Item item = (DisplayData.Item) data.items().toArray()[0];
+
+@SuppressWarnings("unchecked")
+Matcher matchesAllOf = Matchers.allOf(
+hasKey("foo"),
+hasType(DisplayData.Type.TIMESTAMP),
+hasValue(ISO_FORMATTER.print(value)));
+
+assertThat(item, matchesAllOf);
+  }
+
+  @Test
+  public void testStaticValueProviderString() {
+DisplayData data =
+DisplayData.from(new HasDisplayData() {
+  @Override
+  public void populateDisplayData(DisplayData.Builder builder) {
+builder.add(DisplayData.item(
+"foo", StaticValueProvider.of("bar")));
+  }
+});
+
+assertThat(data.items(), hasSize(1));
+assertThat(data, hasDispl

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727142#comment-15727142
 ] 

ASF GitHub Bot commented on BEAM-551:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1503


> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1503: [BEAM-551] Add DisplayData handling of Va...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1503


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1503

[davor]

This closes #1503


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b7331134
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b7331134
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b7331134

Branch: refs/heads/master
Commit: b733113426975acca7fb5c782ed92d1ecce9481e
Parents: 5274dce 2c8311f
Author: Davor Bonaci 
Authored: Tue Dec 6 15:14:34 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Dec 6 15:14:34 2016 -0800

--
 .../sdk/transforms/display/DisplayData.java | 16 
 .../sdk/transforms/display/DisplayDataTest.java | 86 
 2 files changed, 102 insertions(+)
--

[GitHub] incubator-beam pull request #1523: [BEAM-1038] Allow stateful DoFn in Datafl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1523


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1038) Support for new State API in DataflowRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727019#comment-15727019
 ] 

ASF GitHub Bot commented on BEAM-1038:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1523


> Support for new State API in DataflowRunner
> ---
>
> Key: BEAM-1038
> URL: https://issues.apache.org/jira/browse/BEAM-1038
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Allow stateful DoFn in DataflowRunner

[kenn]

Allow stateful DoFn in DataflowRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/42bb15d2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/42bb15d2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/42bb15d2

Branch: refs/heads/master
Commit: 42bb15d2df28b99b6788010450f41f2932095771
Parents: c72708c
Author: Kenneth Knowles 
Authored: Tue Dec 6 13:51:19 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Dec 6 15:05:53 2016 -0800

--
 runners/google-cloud-dataflow-java/pom.xml  |  1 -
 .../dataflow/DataflowPipelineTranslator.java| 22 +++-
 2 files changed, 8 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/42bb15d2/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index adebb2a..9ead74a 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -78,7 +78,6 @@
 runnable-on-service-tests
 
   
-org.apache.beam.sdk.testing.UsesStatefulParDo,
 org.apache.beam.sdk.testing.UsesSplittableParDo
   
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/42bb15d2/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 1cff42a..f43e176 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -77,6 +77,7 @@ import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.HasDisplayData;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
 import org.apache.beam.sdk.transforms.windowing.Window;
@@ -960,7 +961,6 @@ public class DataflowPipelineTranslator {
   private  void translateMultiHelper(
   ParDo.BoundMulti transform,
   TranslationContext context) {
-rejectStatefulDoFn(transform.getNewFn());
 
 context.addStep(transform, "ParallelDo");
 translateInputs(context.getInput(transform), 
transform.getSideInputs(), context);
@@ -990,7 +990,6 @@ public class DataflowPipelineTranslator {
   private  void translateSingleHelper(
   ParDo.Bound transform,
   TranslationContext context) {
-rejectStatefulDoFn(transform.getNewFn());
 
 context.addStep(transform, "ParallelDo");
 translateInputs(context.getInput(transform), 
transform.getSideInputs(), context);
@@ -1038,18 +1037,6 @@ public class DataflowPipelineTranslator {
 registerTransformTranslator(Read.Bounded.class, new ReadTranslator());
   }
 
-  private static void rejectStatefulDoFn(DoFn doFn) {
-if (DoFnSignatures.getSignature(doFn.getClass()).isStateful()) {
-throw new UnsupportedOperationException(
-String.format(
-"Found %s annotations on %s, but %s cannot yet be used with state 
in the %s.",
-DoFn.StateId.class.getSimpleName(),
-doFn.getClass().getName(),
-DoFn.class.getSimpleName(),
-DataflowRunner.class.getSimpleName()));
-}
-  }
-
   private static void translateInputs(
   PCollection input,
   List> sideInputs,
@@ -1081,6 +1068,9 @@ public class DataflowPipelineTranslator {
   TranslationContext context,
   long mainOutput,
   Map> outputMap) {
+
+DoFnSignature signature = DoFnSignatures.getSignature(fn.getClass());
+
 context.addInput(PropertyNames.USER_FN, fn.getClass().getName());
 context.addInput(
 PropertyNames.SERIALIZED_FN,
@@ -1088,6 +1078,10 @@ public class DataflowPipelineTranslator {
 serializeToByteArray(
 DoFnInfo.forFn(
 fn, windowingStrategy, sideInputs, inputCoder, mainOutput, 
outputMap;
+
+if (signature.isStateful()) {
+  context.addInput(PropertyNames.USES_KEY

[1/2] incubator-beam git commit: This closes #1523

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master c72708cd2 -> 5274dce5b


This closes #1523


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5274dce5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5274dce5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5274dce5

Branch: refs/heads/master
Commit: 5274dce5b7d4b0f84e5bd020f054bb4cc045154c
Parents: c72708c 42bb15d
Author: Kenneth Knowles 
Authored: Tue Dec 6 15:05:53 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Dec 6 15:05:53 2016 -0800

--
 runners/google-cloud-dataflow-java/pom.xml  |  1 -
 .../dataflow/DataflowPipelineTranslator.java| 22 +++-
 2 files changed, 8 insertions(+), 15 deletions(-)
--

[GitHub] incubator-beam pull request #1523: [BEAM-1038] Allow stateful DoFn in Datafl...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1523

[BEAM-1038] Allow stateful DoFn in DataflowRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Confirmed that the post commit should succeed, via the following command:

```
mvn verify \
--batch-mode \
--errors \
--projects runners/google-cloud-dataflow-java \
-DforkCount=0 \
-DfailIfNoTests=false \
-Dtest=org.apache.beam.sdk.transforms.ParDoTest \
-DrunnableOnServicePipelineOptions='[

"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner",
"--project=...",
"--tempRoot=..." ]'
```

with output ending in:

```
Running org.apache.beam.sdk.transforms.ParDoTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 273.736 
sec - in org.apache.beam.sdk.transforms.ParDoTest

```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
DataflowRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1523.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1523


commit 9e4d40731ff53b2cd0f2a57d18c6f560db441483
Author: Kenneth Knowles 
Date:   2016-12-06T21:51:19Z

Allow stateful DoFn in DataflowRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1038) Support for new State API in DataflowRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726929#comment-15726929
 ] 

ASF GitHub Bot commented on BEAM-1038:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1523

[BEAM-1038] Allow stateful DoFn in DataflowRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Confirmed that the post commit should succeed, via the following command:

```
mvn verify \
--batch-mode \
--errors \
--projects runners/google-cloud-dataflow-java \
-DforkCount=0 \
-DfailIfNoTests=false \
-Dtest=org.apache.beam.sdk.transforms.ParDoTest \
-DrunnableOnServicePipelineOptions='[

"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner",
"--project=...",
"--tempRoot=..." ]'
```

with output ending in:

```
Running org.apache.beam.sdk.transforms.ParDoTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 273.736 
sec - in org.apache.beam.sdk.transforms.ParDoTest

```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
DataflowRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1523.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1523


commit 9e4d40731ff53b2cd0f2a57d18c6f560db441483
Author: Kenneth Knowles 
Date:   2016-12-06T21:51:19Z

Allow stateful DoFn in DataflowRunner




> Support for new State API in DataflowRunner
> ---
>
> Key: BEAM-1038
> URL: https://issues.apache.org/jira/browse/BEAM-1038
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1097) Dataflow error message for non-existing gcpTempLocation is misleading

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726917#comment-15726917
 ] 

ASF GitHub Bot commented on BEAM-1097:
--

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1522

[BEAM-1097] Provide a better error message for non-existing gcpTempLocation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

gcpTempLocation will default to using the value for tmpLocation, as long
as the value is a valid GCP path. Non-valid GCP paths are silently
discarded.

This change removes existence validation from the default value logic
such that downstream validation can provide a better error message.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam gcp-temp-location-error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1522.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1522


commit 9d768df4323a246baa705fd5fb75d08c78abb7f0
Author: Scott Wegner 
Date:   2016-12-06T22:19:12Z

Provide a better error message for non-existing gcpTempLocation

gcpTempLocation will default to using the value for tmpLocation, as long
as the value is a valid GCP path. Non-valid GCP paths are silently
discarded.

This change removes existence validation from the default value logic
such that downstream validation can provide a better error message.




> Dataflow error message for non-existing gcpTempLocation is misleading
> -
>
> Key: BEAM-1097
> URL: https://issues.apache.org/jira/browse/BEAM-1097
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> The error message for specifying a GCP tempLocation which doesn't exist is 
> misleading. Rather than mentioning the given path doesn't exist, it says none 
> ways specified.
> This is particularly frustrating because it's one of the few configuration 
> values necessary to get started with an example or starter archetype, and 
> it's easy to introduce a typo as it's specified on the commandline. In my 
> case, I was specifying "gs://swegner-tmp" instead of "gs://swegner-test".
> Repro:
> 1. Clone the starter archetype: {noformat}mvn archetype:generate 
> -DarchetypeGroupId=org.apache.beam 
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-starter{noformat}
> 2. Add beam-runners-google-cloud-dataflow-java as a dependency in the 
> generated pom.xml
> 3. Build: {noformat}mvn install{noformat}
> 4. Run: {noformat}mvn exec:java -DmainClass=swegner.StarterPipeline 
> -Dexec.args="--runner=DataflowRunner 
> --tempLocation=gs://swegner-tmp"{noformat}
> Expected: An error message along the lines of: "The specified GCP temp 
> location 'gs://swegner-tmp' does not exist under project 'myGcpProject'"
> bq. [ERROR] Failed to execute goal 
> org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project 
> counter-names-test: An exception occured while executing the Java class. 
> null: InvocationTargetException: Failed to construct instance from factory 
> method DataflowRunner#fromOptions(interface 
> org.apache.beam.sdk.options.PipelineOptions): DataflowRunner requires 
> gcpTempLocation, and it is missing in PipelineOptions. -> [Help 1]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1522: [BEAM-1097] Provide a better error messag...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1522

[BEAM-1097] Provide a better error message for non-existing gcpTempLocation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

gcpTempLocation will default to using the value for tmpLocation, as long
as the value is a valid GCP path. Non-valid GCP paths are silently
discarded.

This change removes existence validation from the default value logic
such that downstream validation can provide a better error message.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam gcp-temp-location-error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1522.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1522


commit 9d768df4323a246baa705fd5fb75d08c78abb7f0
Author: Scott Wegner 
Date:   2016-12-06T22:19:12Z

Provide a better error message for non-existing gcpTempLocation

gcpTempLocation will default to using the value for tmpLocation, as long
as the value is a valid GCP path. Non-valid GCP paths are silently
discarded.

This change removes existence validation from the default value logic
such that downstream validation can provide a better error message.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1097) Dataflow error message for non-existing gcpTempLocation is misleading

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-1097:
--

 Summary: Dataflow error message for non-existing gcpTempLocation 
is misleading
 Key: BEAM-1097
 URL: https://issues.apache.org/jira/browse/BEAM-1097
 Project: Beam
  Issue Type: Bug
  Components: examples-java, runner-dataflow
Reporter: Scott Wegner
Assignee: Scott Wegner
Priority: Minor


The error message for specifying a GCP tempLocation which doesn't exist is 
misleading. Rather than mentioning the given path doesn't exist, it says none 
ways specified.

This is particularly frustrating because it's one of the few configuration 
values necessary to get started with an example or starter archetype, and it's 
easy to introduce a typo as it's specified on the commandline. In my case, I 
was specifying "gs://swegner-tmp" instead of "gs://swegner-test".

Repro:

1. Clone the starter archetype: {noformat}mvn archetype:generate 
-DarchetypeGroupId=org.apache.beam 
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-starter{noformat}
2. Add beam-runners-google-cloud-dataflow-java as a dependency in the generated 
pom.xml
3. Build: {noformat}mvn install{noformat}
4. Run: {noformat}mvn exec:java -DmainClass=swegner.StarterPipeline 
-Dexec.args="--runner=DataflowRunner --tempLocation=gs://swegner-tmp"{noformat}

Expected: An error message along the lines of: "The specified GCP temp location 
'gs://swegner-tmp' does not exist under project 'myGcpProject'"

bq. [ERROR] Failed to execute goal 
org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project 
counter-names-test: An exception occured while executing the Java class. null: 
InvocationTargetException: Failed to construct instance from factory method 
DataflowRunner#fromOptions(interface 
org.apache.beam.sdk.options.PipelineOptions): DataflowRunner requires 
gcpTempLocation, and it is missing in PipelineOptions. -> [Help 1]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-986) ReduceFnRunner doesn't batch prefetching pane firings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726841#comment-15726841
 ] 

ASF GitHub Bot commented on BEAM-986:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1519


> ReduceFnRunner doesn't batch prefetching pane firings
> -
>
> Key: BEAM-986
> URL: https://issues.apache.org/jira/browse/BEAM-986
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Sam Whittle
>Assignee: Sam Whittle
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Specifically
> - in ProcessElements, if there are multiple windows to consider each is 
> processed sequentially with sequential state fetches instead of a bulk 
> prefetch
> - onTimer method doesn't evaluate multiple timers at a time meaning that if 
> multiple timers are fired at once each is processed sequentially without 
> batched prefetching



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1519: [BEAM-986] Improvements to ReduceFnRunner...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1519


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1519

[kenn]

This closes #1519


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c72708cd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c72708cd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c72708cd

Branch: refs/heads/master
Commit: c72708cd2c5725a739a4198bcede1560812bb5af
Parents: 9d380de 2b044f3
Author: Kenneth Knowles 
Authored: Tue Dec 6 13:55:32 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Dec 6 13:55:32 2016 -0800

--
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  11 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |  48 +-
 .../beam/runners/core/PaneInfoTracker.java  |   4 +
 .../runners/core/ReduceFnContextFactory.java|   9 +-
 .../beam/runners/core/ReduceFnRunner.java   | 493 ---
 .../apache/beam/runners/core/WatermarkHold.java |   5 +
 .../triggers/TriggerStateMachineRunner.java |  14 +-
 .../beam/runners/core/ReduceFnTester.java   |  77 ++-
 .../triggers/TriggerStateMachineTester.java |  17 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   5 +-
 10 files changed, 440 insertions(+), 243 deletions(-)
--

[1/2] incubator-beam git commit: Improve ReduceFnRunner prefetching

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 9d380de96 -> c72708cd2


Improve ReduceFnRunner prefetching

- add prefetch* methods for prefetching state matching existing methods
- prefetch triggers in processElements
- replace onTimer with batched onTimers method to allow prefetching across
  timers

Additionally remove deprecated TimerCallback usage


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2b044f3f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2b044f3f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2b044f3f

Branch: refs/heads/master
Commit: 2b044f3f315655b863dbc7fd298f33c196fb8ef7
Parents: 9d380de
Author: Sam Whittle 
Authored: Thu Nov 10 12:59:49 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Dec 6 13:54:50 2016 -0800

--
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  11 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |  48 +-
 .../beam/runners/core/PaneInfoTracker.java  |   4 +
 .../runners/core/ReduceFnContextFactory.java|   9 +-
 .../beam/runners/core/ReduceFnRunner.java   | 493 ---
 .../apache/beam/runners/core/WatermarkHold.java |   5 +
 .../triggers/TriggerStateMachineRunner.java |  14 +-
 .../beam/runners/core/ReduceFnTester.java   |  77 ++-
 .../triggers/TriggerStateMachineTester.java |  17 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   5 +-
 10 files changed, 440 insertions(+), 243 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2b044f3f/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
index 2082269..14171b3 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
@@ -26,7 +26,6 @@ import org.apache.beam.sdk.transforms.Sum;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.SystemDoFnInternal;
 import org.apache.beam.sdk.util.TimerInternals;
-import org.apache.beam.sdk.util.TimerInternals.TimerData;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateInternalsFactory;
@@ -72,9 +71,9 @@ public class GroupAlsoByWindowViaWindowSetDoFn<
 
   @Override
   public void processElement(ProcessContext c) throws Exception {
-KeyedWorkItem element = c.element();
+KeyedWorkItem keyedWorkItem = c.element();
 
-K key = c.element().key();
+K key = keyedWorkItem.key();
 TimerInternals timerInternals = c.windowingInternals().timerInternals();
 StateInternals stateInternals = 
stateInternalsFactory.stateInternalsForKey(key);
 
@@ -92,10 +91,8 @@ public class GroupAlsoByWindowViaWindowSetDoFn<
 reduceFn,
 c.getPipelineOptions());
 
-reduceFnRunner.processElements(element.elementsIterable());
-for (TimerData timer : element.timersIterable()) {
-  reduceFnRunner.onTimer(timer);
-}
+reduceFnRunner.processElements(keyedWorkItem.elementsIterable());
+reduceFnRunner.onTimers(keyedWorkItem.timersIterable());
 reduceFnRunner.persist();
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2b044f3f/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
index b4b366c..9189191 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
@@ -18,17 +18,18 @@
 package org.apache.beam.runners.core;
 
 import com.google.common.collect.Iterables;
+import java.util.ArrayList;
 import java.util.List;
 import org.apache.beam.runners.core.triggers.ExecutableTriggerStateMachine;
 import org.apache.beam.runners.core.triggers.TriggerStateMachines;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.SystemDoFnInternal;
+import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.bea

[GitHub] incubator-beam pull request #1521: Add reference to the >> and | operators f...

[mariapython]

GitHub user mariapython opened a pull request:

https://github.com/apache/incubator-beam/pull/1521

Add reference to the >> and | operators for pipelines.

This comes after getting questions on what those operators represent.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariapython/incubator-beam python-sdk-dec6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1521.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1521


commit 9c1b43ba0692eedca3c5b65d6596d8afb56cec35
Author: Maria Garcia Herrero 
Date:   2016-12-06T20:29:03Z

Add reference to the >> and | operators for pipelines.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726722#comment-15726722
 ] 

ASF GitHub Bot commented on BEAM-551:
-

Github user sammcveety closed the pull request at:

https://github.com/apache/incubator-beam/pull/1238


> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726721#comment-15726721
 ] 

ASF GitHub Bot commented on BEAM-551:
-

GitHub user sammcveety reopened a pull request:

https://github.com/apache/incubator-beam/pull/1238

[BEAM-551] BigqueryIO.Read support for ValueProvider 

R: @dhalperi 

This is the serialization issue I was referencing, where I think the issue 
is that there is a deferred translation from String -> TableRef -> String, 
causing a serialization error.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/bq_vp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1238.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1238


commit ced1073597e3fbcc826f10a417e810803c7317ce
Author: Sam McVeety 
Date:   2016-10-30T18:58:44Z

Add BQ VP code

BQ VP

Fix most tests

Make serializable

Fix BQ tests

Fix tests

Update API

Fix validation case

Fix one more query reference

commit 2d05f107499ce94bb35f05a4c386625e66fe660c
Author: Sam McVeety 
Date:   2016-12-06T05:24:20Z

Minor fixes

commit 1ca49af0659fd32ffa06e4e5778e3e4f90d85be6
Author: Sam McVeety 
Date:   2016-12-06T05:31:34Z

Address serialization issue




> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Sam McVeety
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1238: [BEAM-551] BigqueryIO.Read support for Va...

[sammcveety]

GitHub user sammcveety reopened a pull request:

https://github.com/apache/incubator-beam/pull/1238

[BEAM-551] BigqueryIO.Read support for ValueProvider 

R: @dhalperi 

This is the serialization issue I was referencing, where I think the issue 
is that there is a deferred translation from String -> TableRef -> String, 
causing a serialization error.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/bq_vp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1238.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1238


commit ced1073597e3fbcc826f10a417e810803c7317ce
Author: Sam McVeety 
Date:   2016-10-30T18:58:44Z

Add BQ VP code

BQ VP

Fix most tests

Make serializable

Fix BQ tests

Fix tests

Update API

Fix validation case

Fix one more query reference

commit 2d05f107499ce94bb35f05a4c386625e66fe660c
Author: Sam McVeety 
Date:   2016-12-06T05:24:20Z

Minor fixes

commit 1ca49af0659fd32ffa06e4e5778e3e4f90d85be6
Author: Sam McVeety 
Date:   2016-12-06T05:31:34Z

Address serialization issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1238: [BEAM-551] BigqueryIO.Read support for Va...

[sammcveety]

Github user sammcveety closed the pull request at:

https://github.com/apache/incubator-beam/pull/1238


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-641) Need to test the generated archetypes projects

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci resolved BEAM-641.
---
   Resolution: Invalid
Fix Version/s: (was: 0.4.0-incubating)
   Not applicable

> Need to test the generated archetypes projects
> --
>
> Key: BEAM-641
> URL: https://issues.apache.org/jira/browse/BEAM-641
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Pei He
>Assignee: Pei He
> Fix For: Not applicable
>
>
> Travis and Jenkins pre-submits don't test building the generated archetypes 
> projects.
> Currently, changes to archetypes have to be manually verified by:
> mvn archetype:generate \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeVersion=0.3.0-incubating-SNAPSHOT \
> -DgroupId=com.example \
> -DartifactId=first-beam \
> -Dversion="0.3.0-incubating-SNAPSHOT" \
> -DinteractiveMode=false \
> -Dpackage=org.apache.beam.examples
> and did "mvn clean install" in first-beam project.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Improve BigQuery load error message

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8a1fb2164 -> 9d380de96


Improve BigQuery load error message


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f3edaeb4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f3edaeb4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f3edaeb4

Branch: refs/heads/master
Commit: f3edaeb4f07c66d5ef48afdf16ebd2226f67cefa
Parents: 8a1fb21
Author: Sam McVeety 
Authored: Sun Dec 4 14:28:39 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Dec 6 12:24:07 2016 -0800

--
 .../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java  | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3edaeb4/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 0e01246..c00c19d 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -2140,15 +2140,16 @@ public class BigQueryIO {
   .setProjectId(projectId)
   .setJobId(jobId);
   jobService.startLoadJob(jobRef, loadConfig);
-  Status jobStatus =
-  parseStatus(jobService.pollJob(jobRef, 
Bound.LOAD_JOB_POLL_MAX_RETRIES));
+  Job job = jobService.pollJob(jobRef, 
Bound.LOAD_JOB_POLL_MAX_RETRIES);
+  Status jobStatus = parseStatus(job);
   switch (jobStatus) {
 case SUCCEEDED:
   return;
 case UNKNOWN:
   throw new RuntimeException("Failed to poll the load job status 
of job " + jobId);
 case FAILED:
-  LOG.info("BigQuery load job failed: {}", jobId);
+  LOG.info("BigQuery load job failed. Status: {} Details: {}",
+  jobId, job.getStatus());
   continue;
 default:
   throw new IllegalStateException(String.format("Unexpected job 
status: %s of job %s",

[2/2] incubator-beam git commit: This closes #1506

[davor]

This closes #1506


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9d380de9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9d380de9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9d380de9

Branch: refs/heads/master
Commit: 9d380de965940d37cc1090caf615b9836c70cfc6
Parents: 8a1fb21 f3edaeb
Author: Davor Bonaci 
Authored: Tue Dec 6 12:24:10 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Dec 6 12:24:10 2016 -0800

--
 .../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java  | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)
--

[2/2] incubator-beam git commit: This closes #1348

[davor]

This closes #1348


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8a1fb216
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8a1fb216
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8a1fb216

Branch: refs/heads/master
Commit: 8a1fb2164e9d3172994a458198e858e8753b8081
Parents: ca6ab6c 6b00531
Author: Davor Bonaci 
Authored: Tue Dec 6 12:21:56 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Dec 6 12:21:56 2016 -0800

--
 .../dataflow/options/DataflowPipelineWorkerPoolOptions.java| 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--

[1/2] incubator-beam git commit: Update subnetwork documentation

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ca6ab6c68 -> 8a1fb2164


Update subnetwork documentation


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6b005311
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6b005311
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6b005311

Branch: refs/heads/master
Commit: 6b005311d4aace7112c62a3c1494035bcaaab4b4
Parents: ca6ab6c
Author: sammcveety 
Authored: Fri Nov 11 12:42:22 2016 -0600
Committer: Davor Bonaci 
Committed: Tue Dec 6 12:21:36 2016 -0800

--
 .../dataflow/options/DataflowPipelineWorkerPoolOptions.java| 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6b005311/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
index b6ec9c3..ffb5a3a 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
@@ -154,9 +154,9 @@ public interface DataflowPipelineWorkerPoolOptions extends 
PipelineOptions {
* workers.
*
* Default is up to the Dataflow service. Expected format is
-   * regions/REGION/subnetworks/SUBNETWORK.
-   *
-   * You may also need to specify network option.
+   * regions/REGION/subnetworks/SUBNETWORK or the fully qualified subnetwork 
name, beginning with
+   * https://..., e.g. 
https://www.googleapis.com/compute/alpha/projects/PROJECT/
+   *   regions/REGION/subnetworks/SUBNETWORK
*/
   @Description("GCE subnetwork for launching workers. For more information, 
see the reference "
   + "documentation https://cloud.google.com/compute/docs/networking. "

[jira] [Assigned] (BEAM-507) Fill in the documentation/runners/spark portion of the website

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela reassigned BEAM-507:
--

Assignee: Amit Sela  (was: James Malone)

> Fill in the documentation/runners/spark portion of the website
> --
>
> Key: BEAM-507
> URL: https://issues.apache.org/jira/browse/BEAM-507
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: Amit Sela
>
> As per 
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
> Should be a landing page for Spark-specific information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1096) flink streaming side output optimization using SplitStream

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726461#comment-15726461
 ] 

ASF GitHub Bot commented on BEAM-1096:
--

GitHub user xhumanoid opened a pull request:

https://github.com/apache/incubator-beam/pull/1520

[BEAM-1096] flink streaming side output optimization using SplitStream

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

@aljoscha check please

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xhumanoid/incubator-beam 
stream_side_output_optimization

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1520.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1520


commit 568d73f74219a42cac4028b198f53f22a832990f
Author: Alexey Diomin 
Date:   2016-12-06T19:30:54Z

[BEAM-1096] flink streaming side output optimization using SplitStream




> flink streaming side output optimization using SplitStream
> --
>
> Key: BEAM-1096
> URL: https://issues.apache.org/jira/browse/BEAM-1096
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Affects Versions: 0.4.0-incubating
>Reporter: Alexey Diomin
>Priority: Minor
>
> Current implementation:
> 1) send all events in all output streams
> 2) filtering streams for necessary tags
> Cons: increased cpu usage for serialization all events
> Proposed implementation:
> 1) route event in correct streams based on tag



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1520: [BEAM-1096] flink streaming side output o...

[xhumanoid]

GitHub user xhumanoid opened a pull request:

https://github.com/apache/incubator-beam/pull/1520

[BEAM-1096] flink streaming side output optimization using SplitStream

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

@aljoscha check please

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xhumanoid/incubator-beam 
stream_side_output_optimization

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1520.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1520


commit 568d73f74219a42cac4028b198f53f22a832990f
Author: Alexey Diomin 
Date:   2016-12-06T19:30:54Z

[BEAM-1096] flink streaming side output optimization using SplitStream




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-986) ReduceFnRunner doesn't batch prefetching pane firings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726441#comment-15726441
 ] 

ASF GitHub Bot commented on BEAM-986:
-

GitHub user scwhittle opened a pull request:

https://github.com/apache/incubator-beam/pull/1519

[BEAM-986] Improvements to ReduceFnRunner prefetching

- add prefetch* methods for prefetching state matching existing methods
- replace onTimer with batched onTimers method to allow prefetching across 
timers
- remove deprecated TimerCallback usage
- prefetch triggers in processElements


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scwhittle/incubator-beam 
reduce_fn_prefetching_no_sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1519.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1519


commit 5085c77736a314519c6dcdb4c4c2c21b425b9668
Author: Sam Whittle 
Date:   2016-11-10T20:59:49Z

Improvements to ReduceFnRunner prefetching:
- add prefetch* methods for prefetching state matching existing methods
- replace onTimer with batched onTimers method to allow prefetching
  across timers
- prefetch triggers in processElements
Additionally remove deprecated TimerCallback usage




> ReduceFnRunner doesn't batch prefetching pane firings
> -
>
> Key: BEAM-986
> URL: https://issues.apache.org/jira/browse/BEAM-986
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Sam Whittle
>Assignee: Sam Whittle
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Specifically
> - in ProcessElements, if there are multiple windows to consider each is 
> processed sequentially with sequential state fetches instead of a bulk 
> prefetch
> - onTimer method doesn't evaluate multiple timers at a time meaning that if 
> multiple timers are fired at once each is processed sequentially without 
> batched prefetching



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)