[GitHub] beam pull request #2402: Add example to BoundedCountingInput javadoc

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2402


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Add example to BoundedCountingInput javadoc

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 2dc61f637 -> de36e8398


Add example to BoundedCountingInput javadoc


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/65b03ebe
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/65b03ebe
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/65b03ebe

Branch: refs/heads/master
Commit: 65b03ebe4b9f197877af36395ed73ec2494251e6
Parents: 2dc61f6
Author: wtanaka.com 
Authored: Sat Apr 1 16:44:14 2017 -1000
Committer: wtanaka.com 
Committed: Mon Apr 3 18:39:33 2017 -1000

--
 .../core/src/main/java/org/apache/beam/sdk/io/CountingInput.java | 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/65b03ebe/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java
index ac70aca..72ebd97 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingInput.java
@@ -118,6 +118,10 @@ public class CountingInput {
   /**
* A {@link PTransform} that will produce a specified number of {@link Long 
Longs} starting from
* 0.
+   *
+   * {@code
+   * PCollection bounded = p.apply(CountingInput.upTo(10L));
+   * }
*/
   public static class BoundedCountingInput extends PTransform {
 private final long startIndex;

[2/2] beam git commit: This closes #2402

[dhalperi]

This closes #2402


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/de36e839
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/de36e839
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/de36e839

Branch: refs/heads/master
Commit: de36e8398ecb37b9826175c4723f94d127e63d03
Parents: 2dc61f6 65b03eb
Author: Dan Halperin 
Authored: Mon Apr 3 22:28:10 2017 -0700
Committer: Dan Halperin 
Committed: Mon Apr 3 22:28:10 2017 -0700

--
 .../core/src/main/java/org/apache/beam/sdk/io/CountingInput.java | 4 
 1 file changed, 4 insertions(+)
--

[jira] [Commented] (BEAM-1812) Allow configuring checkpoints in Flink Runner PipelineOptions

[Jins George (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954561#comment-15954561
 ] 

Jins George commented on BEAM-1812:
---

Hi [~aljoscha],
Can you assign this ticket to me. I have analyzed and added external 
check-pointing configurations to FlinkPipelineOptions in my forked repo. I was 
able to resume from the last kafka offset after a cancel and restart.  I can 
submit a pull request.

Thanks,
Jins

> Allow configuring checkpoints in Flink Runner PipelineOptions
> -
>
> Key: BEAM-1812
> URL: https://issues.apache.org/jira/browse/BEAM-1812
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> Flink allows fine grained configuration of checkpointing: 
> https://github.com/aljoscha/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/CheckpointConfig.java
> Among other things this allows to configure externalised checkpoints, which 
> is a valuable feature when running a job in production because it allows 
> restoring a job after a failure as from a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-564) Update source framework so that remaining and consumed number of split points can be reported

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-564.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Update source framework so that remaining and consumed number of split points 
> can be reported
> -
>
> Key: BEAM-564
> URL: https://issues.apache.org/jira/browse/BEAM-564
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> We have to update Python SDK source framework so that sources can report 
> consumed and remaining number of split points. Runners can use this 
> information to determine how many times a given source can be split into and 
> parallelize reading accordingly.
> Corresponding API for JAVA SDK is here:
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedSource.java#L258



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-564) Update source framework so that remaining and consumed number of split points can be reported

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954547#comment-15954547
 ] 

Ahmet Altay commented on BEAM-564:
--

Is it fixed?

> Update source framework so that remaining and consumed number of split points 
> can be reported
> -
>
> Key: BEAM-564
> URL: https://issues.apache.org/jira/browse/BEAM-564
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> We have to update Python SDK source framework so that sources can report 
> consumed and remaining number of split points. Runners can use this 
> information to determine how many times a given source can be split into and 
> parallelize reading accordingly.
> Corresponding API for JAVA SDK is here:
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedSource.java#L258



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-524) Description of "type" argument in Aggregator is incorrect

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954544#comment-15954544
 ] 

Ahmet Altay commented on BEAM-524:
--

Closing this since aggregator API is removed.

> Description of "type" argument in Aggregator is incorrect
> -
>
> Key: BEAM-524
> URL: https://issues.apache.org/jira/browse/BEAM-524
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Frank Yellin
>Priority: Minor
> Fix For: Not applicable
>
>
> Two problems with documentation for "type" argument.
> Trivial:  Remove "by default".  This phrase implies that there are other 
> alternatives besides what is listed.  There aren't.
> Non trivial.  The documentation says "types appropriate to the combine_fn" 
> are accepted.  I tried 
> Accumulator("foo", max, datetime.datetime)
> This failed even though "datetime.datetime" is a perfectly reasonable type to 
> want to take the max of.  (I wanted to know precisely when the last job 
> finished.)
> Either the documentation needs to be changed to specify that max/min only 
> apply to numeric types, or the code needs to be changed to allow other uses 
> of min and max.
> I suspect that the other operators also suffer similar limitations.  I doubt 
> that I'd be allowed to sum numpy arrays, even though they support +.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-524) Description of "type" argument in Aggregator is incorrect

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-524.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Description of "type" argument in Aggregator is incorrect
> -
>
> Key: BEAM-524
> URL: https://issues.apache.org/jira/browse/BEAM-524
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Frank Yellin
>Priority: Minor
> Fix For: Not applicable
>
>
> Two problems with documentation for "type" argument.
> Trivial:  Remove "by default".  This phrase implies that there are other 
> alternatives besides what is listed.  There aren't.
> Non trivial.  The documentation says "types appropriate to the combine_fn" 
> are accepted.  I tried 
> Accumulator("foo", max, datetime.datetime)
> This failed even though "datetime.datetime" is a perfectly reasonable type to 
> want to take the max of.  (I wanted to know precisely when the last job 
> finished.)
> Either the documentation needs to be changed to specify that max/min only 
> apply to numeric types, or the code needs to be changed to allow other uses 
> of min and max.
> I suspect that the other operators also suffer similar limitations.  I doubt 
> that I'd be allowed to sum numpy arrays, even though they support +.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-460) Implement Python support for size-estimation aggregators

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954543#comment-15954543
 ] 

Ahmet Altay commented on BEAM-460:
--

Can we close this?

> Implement Python support for size-estimation aggregators
> 
>
> Key: BEAM-460
> URL: https://issues.apache.org/jira/browse/BEAM-460
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Charles Chen
>Assignee: Charles Chen
>
> Size-estimation aggregators are provided by the execution of each step in a 
> Beam pipeline and help clarify the amount of data processed by each pipeline 
> step.  To ease implementation of this feature for runners, we should first 
> expose size-estimation support for Coder objects.  Runners can then use this 
> functionality to implement full support for size-estimation aggregators.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-437) Data-dependent BigQueryIO in batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954482#comment-15954482
 ] 

ASF GitHub Bot commented on BEAM-437:
-

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/2418

[BEAM-437]

Properly address new API for dynamic table writes.

R: @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam fix_bq_javadoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2418.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2418


commit a773a66f2b752f19585a0e435ee010a07e207fac
Author: Reuven Lax 
Date:   2016-11-14T04:29:31Z

Add PubSub attributes support to PubsubIO.

commit 246698f7fae72fe2519e82fb336530876f26bfe2
Author: Reuven Lax 
Date:   2017-04-04T02:23:09Z

Fix Javadoc for BigQueryIO




> Data-dependent BigQueryIO in batch
> --
>
> Key: BEAM-437
> URL: https://issues.apache.org/jira/browse/BEAM-437
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Reuven Lax
>Priority: Minor
>
> Blocked by [BEAM-92].
> Right now, we use BigQuery's streaming write API when using window-dependent 
> tables in BigQuery. We should
> 1. Support data-dependent tables as well.
> 2. Find a way to use the batch write API.
> 3. This requires careful design to be idempotent or, at least, as close to 
> idempotent as possible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2418: [BEAM-437]

[reuvenlax]

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/2418

[BEAM-437]

Properly address new API for dynamic table writes.

R: @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam fix_bq_javadoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2418.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2418


commit a773a66f2b752f19585a0e435ee010a07e207fac
Author: Reuven Lax 
Date:   2016-11-14T04:29:31Z

Add PubSub attributes support to PubsubIO.

commit 246698f7fae72fe2519e82fb336530876f26bfe2
Author: Reuven Lax 
Date:   2017-04-04T02:23:09Z

Fix Javadoc for BigQueryIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1873) Javadoc in BigQueryIO doesn't reflect recent changes

[Reuven Lax (JIRA)]

Reuven Lax created BEAM-1873:


 Summary: Javadoc in BigQueryIO doesn't reflect recent changes
 Key: BEAM-1873
 URL: https://issues.apache.org/jira/browse/BEAM-1873
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Reuven Lax
Assignee: Reuven Lax






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1872) implement Reshuffle transform

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954454#comment-15954454
 ] 

Ahmet Altay commented on BEAM-1872:
---

Similar to the 
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Reshuffle.java

> implement Reshuffle transform
> -
>
> Key: BEAM-1872
> URL: https://issues.apache.org/jira/browse/BEAM-1872
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>  Labels: newbie, sdk-consistency, starter
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1872) implement Reshuffle transform

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1872:
-

 Summary: implement Reshuffle transform
 Key: BEAM-1872
 URL: https://issues.apache.org/jira/browse/BEAM-1872
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Ahmet Altay






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #1726

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1534) Create a dockerized developer environment for Beam

[Ekrem Aksoy (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954448#comment-15954448
 ] 

Ekrem Aksoy commented on BEAM-1534:
---

I'll start with single Docker image for corresponding BEAM codebase. Focus on 
build aspect.

> Create a dockerized developer environment for Beam
> --
>
> Key: BEAM-1534
> URL: https://issues.apache.org/jira/browse/BEAM-1534
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-java
>Reporter: Nitin Lamba
>Assignee: Ekrem Aksoy
>
> This will help create a repeatable developer environment setup.
> Other Apache projects can be used as a reference:
> https://github.com/apache/ambari/tree/trunk/dev-support/docker
> https://github.com/apache/geode/blob/develop/docker/Dockerfile



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1494

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3150

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove GetProducingTransformInternal

--
[...truncated 718.69 KB...]
---
Running org.apache.beam.sdk.io.gcp.GcpApiSurfaceTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.62 sec - in 
org.apache.beam.sdk.io.gcp.GcpApiSurfaceTest
Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableRowIteratorTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.538 sec - in 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableRowIteratorTest
Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtilsTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.287 sec - in 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtilsTest
Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOTest
Tests run: 71, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 6.228 sec - in 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOTest
Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtilTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.839 sec - in 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtilTest
Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImplTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 sec - in 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImplTest
Running org.apache.beam.sdk.io.gcp.bigtable.BigtableIOTest
Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.925 sec - in 
org.apache.beam.sdk.io.gcp.bigtable.BigtableIOTest
Running org.apache.beam.sdk.io.gcp.datastore.DatastoreV1Test
Tests run: 56, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec - in 
org.apache.beam.sdk.io.gcp.datastore.DatastoreV1Test
Running org.apache.beam.sdk.io.gcp.storage.GcsFileSystemRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.io.gcp.storage.GcsFileSystemRegistrarTest
Running org.apache.beam.sdk.io.gcp.storage.GcsResourceIdTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.io.gcp.storage.GcsResourceIdTest
Running org.apache.beam.sdk.io.gcp.storage.GcsFileSystemTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 sec - in 
org.apache.beam.sdk.io.gcp.storage.GcsFileSystemTest

Results :

Tests run: 205, Failures: 0, Errors: 0, Skipped: 4

[JENKINS] Recording test results
2017-04-04T01:01:04.981 [INFO] 
2017-04-04T01:01:04.981 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-io-google-cloud-platform ---
2017-04-04T01:01:05.213 [INFO] 
2017-04-04T01:01:05.213 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-io-google-cloud-platform ---
2017-04-04T01:01:05.234 [INFO] Building jar: 

2017-04-04T01:01:05.300 [INFO] 
2017-04-04T01:01:05.300 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-io-google-cloud-platform ---
2017-04-04T01:01:05.350 [INFO] 
2017-04-04T01:01:05.350 [INFO] --- maven-javadoc-plugin:2.10.4:jar 
(attach-javadocs) @ beam-sdks-java-io-google-cloud-platform ---
2017-04-04T01:01:08.336 [INFO] Building jar: 

2017-04-04T01:01:08.400 [INFO] 
2017-04-04T01:01:08.400 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-io-google-cloud-platform ---
2017-04-04T01:01:08.404 [INFO] Building jar: 

2017-04-04T01:01:08.470 [INFO] 
2017-04-04T01:01:08.470 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-io-google-cloud-platform ---
2017-04-04T01:01:08.473 [INFO] Building jar: 

2017-04-04T01:01:08.587 [INFO] 
2017-04-04T01:01:08.587 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-io-google-cloud-platform ---
2017-04-04T01:01:08.626 [INFO] Building jar: 

2017-04-04T01:01:08.689 [INFO] 
2017-04-04T01:01:08.689 [INFO] --- 
maven-failsafe-plugin:2.19.1:integration-test (default) @ 

[jira] [Commented] (BEAM-1148) Port PAssert away from Aggregators

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954429#comment-15954429
 ] 

ASF GitHub Bot commented on BEAM-1148:
--

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2417

[BEAM-1148] Port PAssert away from Aggregators

Still running into trouble with the direct runner, but unit tests in other 
runners are passing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam play-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2417


commit a32d9ac4f2e8c3898ca9b7e3346be2d948bb4f9b
Author: Pablo 
Date:   2017-03-29T21:49:53Z

Playing with PAssert.java




> Port PAssert away from Aggregators
> --
>
> Key: BEAM-1148
> URL: https://issues.apache.org/jira/browse/BEAM-1148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> One step in the removal of Aggregators (in favor of Metrics) is to remove our 
> reliance on them for PAssert checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2417: [BEAM-1148] Port PAssert away from Aggregators

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2417

[BEAM-1148] Port PAssert away from Aggregators

Still running into trouble with the direct runner, but unit tests in other 
runners are passing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam play-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2417.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2417


commit a32d9ac4f2e8c3898ca9b7e3346be2d948bb4f9b
Author: Pablo 
Date:   2017-03-29T21:49:53Z

Playing with PAssert.java




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1871) Thin Java SDK Core

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1871:
-

 Summary: Thin Java SDK Core
 Key: BEAM-1871
 URL: https://issues.apache.org/jira/browse/BEAM-1871
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Daniel Halperin
 Fix For: First stable release


Before first stable release we need to thin out {{sdk-java-core}} module. Some 
candidates for removal, but not a non-exhaustive list:

{{sdk/io}}

* anything BigQuery related
* anything PubSub related
* everything Protobuf related
* TFRecordIO
* XMLSink

{{sdk/util}}

* Everything GCS related
* Everything Backoff related
* Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
* Everything CloudObject-related
* Pubsub stuff

{{sdk/coders}}

* JAXBCoder
* TableRowJsoNCoder





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1493

[Apache Jenkins Server]

See 

[2/2] beam git commit: This closes #2416: Remove getProducingTransformInternal

[kenn]

This closes #2416: Remove getProducingTransformInternal


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2dc61f63
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2dc61f63
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2dc61f63

Branch: refs/heads/master
Commit: 2dc61f637b76cd68c1ade644034d3ec346f5521c
Parents: 6970632 3d53b72
Author: Kenneth Knowles 
Authored: Mon Apr 3 17:26:25 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Apr 3 17:26:25 2017 -0700

--
 .../beam/sdk/values/POutputValueBase.java   | 41 +++-
 1 file changed, 6 insertions(+), 35 deletions(-)
--

[jira] [Commented] (BEAM-1869) getProducingTransformInternal should not be available on any PValue

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954416#comment-15954416
 ] 

ASF GitHub Bot commented on BEAM-1869:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2416


> getProducingTransformInternal should not be available on any PValue
> ---
>
> Key: BEAM-1869
> URL: https://issues.apache.org/jira/browse/BEAM-1869
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> This method is not guaranteed to return the correct value, as the 
> TransformHierarchy can be updated without calling back into this method.
> It also is not a responsibility of POutput/POutputValueBase, but is a 
> responsibility of the Transform Hierarchy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2416: [BEAM-1869] Remove GetProducingTransformInternal

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2416


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Remove GetProducingTransformInternal

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 697063297 -> 2dc61f637


Remove GetProducingTransformInternal

This isn't part of the PValue API, so remove from POutputValueBase.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3d53b72c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3d53b72c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3d53b72c

Branch: refs/heads/master
Commit: 3d53b72c9a23705e55ec0a802c81b756d113e601
Parents: 9beb04e
Author: Thomas Groh 
Authored: Mon Apr 3 16:26:10 2017 -0700
Committer: Thomas Groh 
Committed: Mon Apr 3 16:29:15 2017 -0700

--
 .../beam/sdk/values/POutputValueBase.java   | 41 +++-
 1 file changed, 6 insertions(+), 35 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3d53b72c/sdks/java/core/src/main/java/org/apache/beam/sdk/values/POutputValueBase.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/POutputValueBase.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/POutputValueBase.java
index cdef58c..5bd424d 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/POutputValueBase.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/POutputValueBase.java
@@ -53,40 +53,16 @@ public abstract class POutputValueBase implements POutput {
   }
 
   /**
-   * Returns the {@link AppliedPTransform} that this {@link POutputValueBase}
-   * is an output of.
+   * Records that this {@link POutputValueBase} is an output with the given 
name of the given {@link
+   * AppliedPTransform}.
*
-   * For internal use only.
-   */
-  public AppliedPTransform getProducingTransformInternal() {
-return producingTransform;
-  }
-
-  /**
-   * Records that this {@link POutputValueBase} is an output with the
-   * given name of the given {@link AppliedPTransform}.
+   * By default, does nothing.
*
-   * To be invoked only by {@link POutput#recordAsOutput}
-   * implementations.  Not to be invoked directly by user code.
+   * To be invoked only by {@link POutput#recordAsOutput} implementations. 
Not to be invoked
+   * directly by user code.
*/
   @Override
-  public void recordAsOutput(AppliedPTransform transform) {
-if (producingTransform != null) {
-  // Already used this POutput as a PTransform output.  This can
-  // happen if the POutput is an output of a transform within a
-  // composite transform, and is also the result of the composite.
-  // We want to record the "immediate" atomic transform producing
-  // this output, and ignore all later composite transforms that
-  // also produce this output.
-  //
-  // Pipeline.applyInternal() uses !hasProducingTransform() to
-  // avoid calling this operation redundantly, but
-  // hasProducingTransform() doesn't apply to POutputValueBases
-  // that aren't PValues or composites of PValues, e.g., PDone.
-  return;
-}
-producingTransform = transform;
-  }
+  public void recordAsOutput(AppliedPTransform transform) {}
 
   /**
* Default behavior for {@link #finishSpecifyingOutput(PInput, PTransform)}} 
is
@@ -95,9 +71,4 @@ public abstract class POutputValueBase implements POutput {
*/
   @Override
   public void finishSpecifyingOutput(PInput input, PTransform transform) 
{ }
-
-  /**
-   * The {@link PTransform} that produces this {@link POutputValueBase}.
-   */
-  private AppliedPTransform producingTransform;
 }

[jira] [Created] (BEAM-1870) ByteKey / ByteKeyRangeTracker should not use ByteString on public API surface

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1870:
-

 Summary: ByteKey / ByteKeyRangeTracker should not use ByteString 
on public API surface
 Key: BEAM-1870
 URL: https://issues.apache.org/jira/browse/BEAM-1870
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Daniel Halperin
Assignee: Ismaël Mejía
 Fix For: First stable release


We don't want these Google Protocol Buffer dependencies on the public API. We 
should replace the use of {{ByteString}} with something in the code Java 
libraries.

What's the open source standard here? I guess Avro uses {{ByteBuffer}} for 
wrapping {{byte[]}} ?

[~iemejia] -- tentatively assigned to you as you brought this up.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1870) ByteKey / ByteKeyRangeTracker should not use ByteString on public API surface

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1870:
--
Description: 
We don't want these Google Protocol Buffer dependencies on the public API. We 
should replace the use of {{ByteString}} with something in the core Java 
libraries.

What's the open source standard here? I guess Avro uses {{ByteBuffer}} for 
wrapping {{byte[]}} ?

[~iemejia] -- tentatively assigned to you as you brought this up.

  was:
We don't want these Google Protocol Buffer dependencies on the public API. We 
should replace the use of {{ByteString}} with something in the code Java 
libraries.

What's the open source standard here? I guess Avro uses {{ByteBuffer}} for 
wrapping {{byte[]}} ?

[~iemejia] -- tentatively assigned to you as you brought this up.


> ByteKey / ByteKeyRangeTracker should not use ByteString on public API surface
> -
>
> Key: BEAM-1870
> URL: https://issues.apache.org/jira/browse/BEAM-1870
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Ismaël Mejía
> Fix For: First stable release
>
>
> We don't want these Google Protocol Buffer dependencies on the public API. We 
> should replace the use of {{ByteString}} with something in the core Java 
> libraries.
> What's the open source standard here? I guess Avro uses {{ByteBuffer}} for 
> wrapping {{byte[]}} ?
> [~iemejia] -- tentatively assigned to you as you brought this up.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Python_Verify #1725

[Apache Jenkins Server]

See 


--
[...truncated 223.39 KB...]
 x [deleted] (none) -> origin/pr/942/head
 x [deleted] (none) -> origin/pr/942/merge
 x [deleted] (none) -> origin/pr/943/head
 x [deleted] (none) -> origin/pr/943/merge
 x [deleted] (none) -> origin/pr/944/head
 x [deleted] (none) -> origin/pr/945/head
 x [deleted] (none) -> origin/pr/945/merge
 x [deleted] (none) -> origin/pr/946/head
 x [deleted] (none) -> origin/pr/946/merge
 x [deleted] (none) -> origin/pr/947/head
 x [deleted] (none) -> origin/pr/947/merge
 x [deleted] (none) -> origin/pr/948/head
 x [deleted] (none) -> origin/pr/948/merge
 x [deleted] (none) -> origin/pr/949/head
 x [deleted] (none) -> origin/pr/949/merge
 x [deleted] (none) -> origin/pr/95/head
 x [deleted] (none) -> origin/pr/95/merge
 x [deleted] (none) -> origin/pr/950/head
 x [deleted] (none) -> origin/pr/951/head
 x [deleted] (none) -> origin/pr/951/merge
 x [deleted] (none) -> origin/pr/952/head
 x [deleted] (none) -> origin/pr/952/merge
 x [deleted] (none) -> origin/pr/953/head
 x [deleted] (none) -> origin/pr/954/head
 x [deleted] (none) -> origin/pr/954/merge
 x [deleted] (none) -> origin/pr/955/head
 x [deleted] (none) -> origin/pr/955/merge
 x [deleted] (none) -> origin/pr/956/head
 x [deleted] (none) -> origin/pr/957/head
 x [deleted] (none) -> origin/pr/958/head
 x [deleted] (none) -> origin/pr/959/head
 x [deleted] (none) -> origin/pr/959/merge
 x [deleted] (none) -> origin/pr/96/head
 x [deleted] (none) -> origin/pr/96/merge
 x [deleted] (none) -> origin/pr/960/head
 x [deleted] (none) -> origin/pr/960/merge
 x [deleted] (none) -> origin/pr/961/head
 x [deleted] (none) -> origin/pr/962/head
 x [deleted] (none) -> origin/pr/962/merge
 x [deleted] (none) -> origin/pr/963/head
 x [deleted] (none) -> origin/pr/963/merge
 x [deleted] (none) -> origin/pr/964/head
 x [deleted] (none) -> origin/pr/965/head
 x [deleted] (none) -> origin/pr/965/merge
 x [deleted] (none) -> origin/pr/966/head
 x [deleted] (none) -> origin/pr/967/head
 x [deleted] (none) -> origin/pr/967/merge
 x [deleted] (none) -> origin/pr/968/head
 x [deleted] (none) -> origin/pr/968/merge
 x [deleted] (none) -> origin/pr/969/head
 x [deleted] (none) -> origin/pr/969/merge
 x [deleted] (none) -> origin/pr/97/head
 x [deleted] (none) -> origin/pr/97/merge
 x [deleted] (none) -> origin/pr/970/head
 x [deleted] (none) -> origin/pr/970/merge
 x [deleted] (none) -> origin/pr/971/head
 x [deleted] (none) -> origin/pr/971/merge
 x [deleted] (none) -> origin/pr/972/head
 x [deleted] (none) -> origin/pr/973/head
 x [deleted] (none) -> origin/pr/974/head
 x [deleted] (none) -> origin/pr/974/merge
 x [deleted] (none) -> origin/pr/975/head
 x [deleted] (none) -> origin/pr/975/merge
 x [deleted] (none) -> origin/pr/976/head
 x [deleted] (none) -> origin/pr/976/merge
 x [deleted] (none) -> origin/pr/977/head
 x [deleted] (none) -> origin/pr/977/merge
 x [deleted] (none) -> origin/pr/978/head
 x [deleted] (none) -> origin/pr/978/merge
 x [deleted] (none) -> origin/pr/979/head
 x [deleted] (none) -> origin/pr/979/merge
 x [deleted] (none) -> origin/pr/98/head
 x [deleted] (none) -> origin/pr/980/head
 x [deleted] (none) -> origin/pr/980/merge
 x [deleted] (none) -> origin/pr/981/head
 x [deleted] (none) -> origin/pr/982/head
 x [deleted] (none) -> origin/pr/982/merge
 x [deleted] (none) -> origin/pr/983/head
 x [deleted] (none) -> origin/pr/983/merge
 x [deleted] (none) -> origin/pr/984/head
 x [deleted] (none) -> origin/pr/984/merge
 x [deleted] (none) -> origin/pr/985/head
 x [deleted] (none) -> origin/pr/985/merge
 x [deleted] (none) -> origin/pr/986/head
 x [deleted] (none) -> origin/pr/986/merge
 x [deleted] (none) -> origin/pr/987/head
 x [deleted] (none) -> origin/pr/988/head
 x [deleted] (none) -> origin/pr/988/merge
 x [deleted] (none) -> origin/pr/989/head
 

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954384#comment-15954384
 ] 

ASF GitHub Bot commented on BEAM-1866:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2412


> FnAPI support for Metrics
> -
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-fn-api
>Reporter: Daniel Halperin
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2412: [BEAM-1866] DataflowRunner: disable PAssert use of ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2412


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2412

[dhalperi]

This closes #2412


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/69706329
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/69706329
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/69706329

Branch: refs/heads/master
Commit: 69706329732b6b710994910016ab8776326ccee8
Parents: 844e53e ef75c07
Author: Dan Halperin 
Authored: Mon Apr 3 17:10:52 2017 -0700
Committer: Dan Halperin 
Committed: Mon Apr 3 17:10:52 2017 -0700

--
 .../beam/runners/dataflow/testing/TestDataflowRunner.java   | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)
--

Build failed in Jenkins: beam_PerformanceTests_Dataflow #266

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove Direct Runner "doneness" configuration

[tgroh] Use Batch Replacement in the Flink Runner

[klk] JavaDoc for function `PaneInfo.getTiming()`

[altay] [BEAM-1719] Remove test modules from PyDoc

[altay] Update PR template with correct ICLA link

[altay] Update integration tests in post commits

[dhalperi] DataflowRunner: automatically determine container image type

--
[...truncated 245.60 KB...]
error: unable to resolve reference refs/remotes/origin/pr/2346/merge: No such 
file or directory
 ! 63fb4e9...1afc8c3 refs/pull/2346/merge -> origin/pr/2346/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2347/merge: No such 
file or directory
 ! 20602a3...f071213 refs/pull/2347/merge -> origin/pr/2347/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2348/merge: No such 
file or directory
 ! d4bc004...820de53 refs/pull/2348/merge -> origin/pr/2348/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2350/merge: No such 
file or directory
 ! 01b49d6...894eb39 refs/pull/2350/merge -> origin/pr/2350/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2351/merge: No such 
file or directory
 ! 92c78c6...662028f refs/pull/2351/merge -> origin/pr/2351/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2352/merge: No such 
file or directory
 ! 1a629a1...7a8b036 refs/pull/2352/merge -> origin/pr/2352/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2353/merge: No such 
file or directory
 ! 9850fc7...d2f9da0 refs/pull/2353/merge -> origin/pr/2353/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2354/merge: No such 
file or directory
 ! 336d9b1...68e8856 refs/pull/2354/merge -> origin/pr/2354/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2355/merge: No such 
file or directory
 ! ab11836...db7d4c9 refs/pull/2355/merge -> origin/pr/2355/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2357/merge: No such 
file or directory
 ! 9757a3b...3fdd88c refs/pull/2357/merge -> origin/pr/2357/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2358/merge: No such 
file or directory
 ! c8e4eaf...bac2186 refs/pull/2358/merge -> origin/pr/2358/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2362/merge: No such 
file or directory
 ! 4be97f9...ccf1604 refs/pull/2362/merge -> origin/pr/2362/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2363/merge: No such 
file or directory
 ! bd21625...7cc3f08 refs/pull/2363/merge -> origin/pr/2363/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2364/merge: No such 
file or directory
 ! f2158bf...75c86de refs/pull/2364/merge -> origin/pr/2364/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2365/merge: No such 
file or directory
 ! f6e645a...5c0bca0 refs/pull/2365/merge -> origin/pr/2365/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2366/merge: No such 
file or directory
 ! cbe91db...d6cb85e refs/pull/2366/merge -> origin/pr/2366/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2367/merge: No such 
file or directory
 ! 6f12eb1...7518036 refs/pull/2367/merge -> origin/pr/2367/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2368/merge: No such 
file or directory
 ! 98a4c84...161dce1 refs/pull/2368/merge -> origin/pr/2368/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2369/merge: No such 
file or directory
 ! 1f792b0...a10c8c8 refs/pull/2369/merge -> origin/pr/2369/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2370/merge: No such 
file or directory
 ! e5062cd...2b99954 refs/pull/2370/merge -> origin/pr/2370/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2371/merge: No such 
file or directory
 ! 6f1bba9...026c423 refs/pull/2371/merge -> origin/pr/2371/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2372/merge: No such 
file or directory
 ! a2d2759...b7af200 refs/pull/2372/merge -> origin/pr/2372/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2373/merge: No such 
file or directory
 ! cc49cff...082e803 refs/pull/2373/merge -> origin/pr/2373/merge  

[GitHub] beam-site pull request #199: Docker server command should run non-incrementa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/199


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam-site git commit: This closes #199

[altay]

This closes #199


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/56d16cd7
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/56d16cd7
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/56d16cd7

Branch: refs/heads/asf-site
Commit: 56d16cd79b31c733c56ff919cd6b96b35f7951e6
Parents: 5359c32 e311e06
Author: Ahmet Altay 
Authored: Mon Apr 3 16:55:23 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 16:55:23 2017 -0700

--
 run_with_docker.sh | 3 +++
 1 file changed, 3 insertions(+)
--

[1/2] beam-site git commit: Docker server command should run non-incremental build

[altay]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 5359c3294 -> 56d16cd79


Docker server command should run non-incremental build


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/e311e06d
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/e311e06d
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/e311e06d

Branch: refs/heads/asf-site
Commit: e311e06db714a6f28156c86306650b0bd84cc3ea
Parents: 5359c32
Author: Sourabh Bajaj 
Authored: Mon Apr 3 11:48:31 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 16:55:14 2017 -0700

--
 run_with_docker.sh | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/e311e06d/run_with_docker.sh
--
diff --git a/run_with_docker.sh b/run_with_docker.sh
index 47ff9a1..5924146 100755
--- a/run_with_docker.sh
+++ b/run_with_docker.sh
@@ -39,6 +39,9 @@ fi
 
 case "$1" in
   server)
+_runner "bundle exec jekyll server --force_polling --watch -H 0.0.0.0 -P 
4000";
+;;
+  server_incremental)
 _runner "bundle exec jekyll server --force_polling --watch -H 0.0.0.0 -P 
4000 --incremental";
 ;;
   test)

[jira] [Commented] (BEAM-1823) TimedOutException in postcommit

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954347#comment-15954347
 ] 

Ahmet Altay commented on BEAM-1823:
---

That also works, we can just print job id from there.

(Although I think there is value in capturing output from stuck tests in 
general to debug where they got stuck. This does not need to be solved now.)

> TimedOutException in postcommit
> ---
>
> Key: BEAM-1823
> URL: https://issues.apache.org/jira/browse/BEAM-1823
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>
> Mark, do you know what this error means? Where is the timeout configured.
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1657/console
> I _think_ this is one of the underlying Dataflow executions and it completed 
> (although much slower than usual):
> https://pantheon.corp.google.com/dataflow/job/2017-03-28_14_25_21-13472017589125356257?project=apache-beam-testing=433637338589
> It makes sense to time out the test but I want to know how it is configured. 
> Also, is it possible to print out output logs for failed/timed out tests so 
> that we can clearly associate tests with job executions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1823) TimedOutException in postcommit

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954342#comment-15954342
 ] 

Mark Liu commented on BEAM-1823:


I'm okay with this trick before the issue is fixed in nosetest side, since 
general test run with nose framework will remain the same. 

One possible place I can think of to add is before 
[test_dataflow_runner.wait_until_finish()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py#L33].
 Since this is more Dataflow related issue, printing job_id in 
test_dataflow_runner will not affect logs of other runners.

> TimedOutException in postcommit
> ---
>
> Key: BEAM-1823
> URL: https://issues.apache.org/jira/browse/BEAM-1823
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>
> Mark, do you know what this error means? Where is the timeout configured.
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1657/console
> I _think_ this is one of the underlying Dataflow executions and it completed 
> (although much slower than usual):
> https://pantheon.corp.google.com/dataflow/job/2017-03-28_14_25_21-13472017589125356257?project=apache-beam-testing=433637338589
> It makes sense to time out the test but I want to know how it is configured. 
> Also, is it possible to print out output logs for failed/timed out tests so 
> that we can clearly associate tests with job executions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1869) getProducingTransformInternal should not be available on any PValue

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954329#comment-15954329
 ] 

ASF GitHub Bot commented on BEAM-1869:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2416

[BEAM-1869] Remove GetProducingTransformInternal

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This isn't part of the PValue API, so remove from POutputValueBase.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam 
remove_get_producing_transform_internal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2416.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2416


commit 09d8c76bb847d354363828efdc1f5f2958e9
Author: Thomas Groh 
Date:   2017-04-03T23:26:10Z

Remove GetProducingTransformInternal

This isn't part of the API surface, and is no longer part of
POutputValueBase.




> getProducingTransformInternal should not be available on any PValue
> ---
>
> Key: BEAM-1869
> URL: https://issues.apache.org/jira/browse/BEAM-1869
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> This method is not guaranteed to return the correct value, as the 
> TransformHierarchy can be updated without calling back into this method.
> It also is not a responsibility of POutput/POutputValueBase, but is a 
> responsibility of the Transform Hierarchy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2416: [BEAM-1869] Remove GetProducingTransformInternal

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2416

[BEAM-1869] Remove GetProducingTransformInternal

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This isn't part of the PValue API, so remove from POutputValueBase.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam 
remove_get_producing_transform_internal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2416.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2416


commit 09d8c76bb847d354363828efdc1f5f2958e9
Author: Thomas Groh 
Date:   2017-04-03T23:26:10Z

Remove GetProducingTransformInternal

This isn't part of the API surface, and is no longer part of
POutputValueBase.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1869) getProducingTransformInternal should not be available on any PValue

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1869:
-

 Summary: getProducingTransformInternal should not be available on 
any PValue
 Key: BEAM-1869
 URL: https://issues.apache.org/jira/browse/BEAM-1869
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh


This method is not guaranteed to return the correct value, as the 
TransformHierarchy can be updated without calling back into this method.

It also is not a responsibility of POutput/POutputValueBase, but is a 
responsibility of the Transform Hierarchy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1867) Element counts missing on Cloud Dataflow when PCollection has anything other than hardcoded name pattern

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954306#comment-15954306
 ] 

Kenneth Knowles commented on BEAM-1867:
---

[~dhalp...@google.com] and [~tgroh] there might be SDK-side mitigations but 
otherwise this is just an internal bug, it seems. I'll leave open for you to 
agree/disagree.

> Element counts missing on Cloud Dataflow when PCollection has anything other 
> than hardcoded name pattern
> 
>
> Key: BEAM-1867
> URL: https://issues.apache.org/jira/browse/BEAM-1867
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Priority: Blocker
> Fix For: First stable release
>
>
> In 0.6.0 and 0.7.0-SNAPSHOT (and possibly all past versions, these are just 
> those where it is confirmed) element count and byte metrics are not reported 
> correctly when the output PCollection for a primitive transform is not 
> {{transformname + ".out" + index}}.
> In 0.7.0-SNAPSHOT, the DataflowRunner uses pipeline surgery to replace the 
> composite {{ParDoSingle}} (that contains a {{ParDoMulti}}) with a 
> Dataflow-specific non-composite {{ParDoSingle}}. So metrics are reported for 
> names like {{"ParDoSingle(MyDoFn).out"}} when they should be reported for 
> {{"ParDoSingle/ParDoMulti(MyDoFn).out"}}. So all single-output ParDo 
> transforms lack these metrics on their outputs.
> In 0.6.0 the same problem occurs if the user ever uses 
> {{PCollection.setName}} to give their collection a meaningful name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-1863) Allow users to override the base container image but still choose image type

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-1863.
-
   Resolution: Fixed
Fix Version/s: First stable release

> Allow users to override the base container image but still choose image type
> 
>
> Key: BEAM-1863
> URL: https://issues.apache.org/jira/browse/BEAM-1863
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: First stable release
>
>
> Right now, the {{DataflowRunner}} will automatically determine the specific 
> URI of the container to be used fully automatically or completely 
> user-dependent, based on whether the user overrides the 
> {{--workerHarnessContainerImage}} flag.
> There's a third mode that would be nice: user sets the general URL and 
> version of the container, but the runner itself chooses the image type. A 
> simple refactoring will enable this to work.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2407: [BEAM-1863] DataflowRunner: automatically determine...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2407


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2407

[dhalperi]

This closes #2407


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/844e53e3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/844e53e3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/844e53e3

Branch: refs/heads/master
Commit: 844e53e341bf6d465e4185903a3a5047a3e68282
Parents: 3711c0c 6b7593b
Author: Dan Halperin 
Authored: Mon Apr 3 15:58:11 2017 -0700
Committer: Dan Halperin 
Committed: Mon Apr 3 15:58:11 2017 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   | 18 ++
 .../options/DataflowPipelineDebugOptions.java   |  2 +-
 .../DataflowPipelineWorkerPoolOptions.java  | 10 +---
 .../runners/dataflow/DataflowRunnerTest.java| 25 
 4 files changed, 41 insertions(+), 14 deletions(-)
--

[1/2] beam git commit: DataflowRunner: automatically determine container image type

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 3711c0caf -> 844e53e34


DataflowRunner: automatically determine container image type

Even if user supplies a base container image policy.

Of course, preserve the ability of the user to fully override the image.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6b7593b0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6b7593b0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6b7593b0

Branch: refs/heads/master
Commit: 6b7593b0a3cd0514145bb6002fff0958a6630303
Parents: 3711c0c
Author: Dan Halperin 
Authored: Mon Apr 3 09:25:57 2017 -0700
Committer: Dan Halperin 
Committed: Mon Apr 3 15:58:07 2017 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   | 18 ++
 .../options/DataflowPipelineDebugOptions.java   |  2 +-
 .../DataflowPipelineWorkerPoolOptions.java  | 10 +---
 .../runners/dataflow/DataflowRunnerTest.java| 25 
 4 files changed, 41 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6b7593b0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index ea96ae8..6eec8f8 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -68,7 +68,6 @@ import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.JobSpecificat
 import 
org.apache.beam.runners.dataflow.StreamingViewOverrides.StreamingCreatePCollectionViewFactory;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
-import 
org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions;
 import org.apache.beam.runners.dataflow.util.DataflowTemplateJob;
 import org.apache.beam.runners.dataflow.util.DataflowTransport;
 import org.apache.beam.runners.dataflow.util.MonitoringUtil;
@@ -543,9 +542,7 @@ public class DataflowRunner extends 
PipelineRunner {
 
 // Set the Docker container image that executes Dataflow worker harness, 
residing in Google
 // Container Registry. Translator is guaranteed to create a worker pool 
prior to this point.
-String workerHarnessContainerImage =
-options.as(DataflowPipelineWorkerPoolOptions.class)
-.getWorkerHarnessContainerImage();
+String workerHarnessContainerImage = getContainerImageForJob(options);
 for (WorkerPool workerPool : newJob.getEnvironment().getWorkerPools()) {
   workerPool.setWorkerHarnessContainerImage(workerHarnessContainerImage);
 }
@@ -1341,4 +1338,17 @@ public class DataflowRunner extends 
PipelineRunner {
 }
   }
 
+  @VisibleForTesting
+  static String getContainerImageForJob(DataflowPipelineOptions options) {
+String workerHarnessContainerImage = 
options.getWorkerHarnessContainerImage();
+if (!workerHarnessContainerImage.contains("IMAGE")) {
+  return workerHarnessContainerImage;
+} else if (hasExperiment(options, "beam_fn_api")) {
+  return workerHarnessContainerImage.replace("IMAGE", "java");
+} else if (options.isStreaming()) {
+  return workerHarnessContainerImage.replace("IMAGE", 
"beam-java-streaming");
+} else {
+  return workerHarnessContainerImage.replace("IMAGE", "beam-java-batch");
+}
+  }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/6b7593b0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java
index 729bca4..d0ea722 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java
@@ -56,7 +56,7 @@ public interface DataflowPipelineDebugOptions extends 
PipelineOptions {
   @Experimental
   @Nullable
   List getExperiments();
-  void setExperiments(List value);
+  void 

[jira] [Commented] (BEAM-1711) Document extra features on quick start guide

[Sourabh Bajaj (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954289#comment-15954289
 ] 

Sourabh Bajaj commented on BEAM-1711:
-

This can be closed now.

> Document extra features on quick start guide
> 
>
> Key: BEAM-1711
> URL: https://issues.apache.org/jira/browse/BEAM-1711
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>
> Add something like below to avoid confusion
> """
> You may need extra packages for some additional features and this is the list 
> of extra_features and what they do.
> feature1: required for x, y,  z
> """



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1711) Document extra features on quick start guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954287#comment-15954287
 ] 

ASF GitHub Bot commented on BEAM-1711:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/201


> Document extra features on quick start guide
> 
>
> Key: BEAM-1711
> URL: https://issues.apache.org/jira/browse/BEAM-1711
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>
> Add something like below to avoid confusion
> """
> You may need extra packages for some additional features and this is the list 
> of extra_features and what they do.
> feature1: required for x, y,  z
> """



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #201: [BEAM-1711] Add information about extra require...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/201


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] beam-site git commit: [BEAM-1711] Add information about extra requires in Python Quickstart

[altay]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 85103fb5f -> 5359c3294


[BEAM-1711] Add information about extra requires in Python Quickstart


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/0dd610f7
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/0dd610f7
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/0dd610f7

Branch: refs/heads/asf-site
Commit: 0dd610f73b6b6b673c3dec1b0633c2cf63ec5a60
Parents: 85103fb
Author: Sourabh Bajaj 
Authored: Mon Apr 3 13:40:04 2017 -0700
Committer: Sourabh Bajaj 
Committed: Mon Apr 3 15:23:37 2017 -0700

--
 src/get-started/quickstart-py.md | 18 ++
 1 file changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/0dd610f7/src/get-started/quickstart-py.md
--
diff --git a/src/get-started/quickstart-py.md b/src/get-started/quickstart-py.md
index 6d729bd..efaa9dc 100644
--- a/src/get-started/quickstart-py.md
+++ b/src/get-started/quickstart-py.md
@@ -68,6 +68,24 @@ Install the latest Python SDK from PyPI:
 pip install apache-beam
 ```
 
+ Extra Requirements
+
+The above installation will not install all the extra dependencies for using 
features like the Google Cloud Dataflow runner. Information on what extra 
packages are required for different features are highlighted below. It is 
possible to install multitple extra requirements using something like `pip 
install apache-beam[feature1, feature2]`.
+
+- **Google Cloud Platform**
+  - Installation Command: `pip install apache-beam[gcp]`
+  - Required for:
+- Google Cloud Dataflow Runner
+- GCS IO
+- Datastore IO
+- BigQuery IO
+- **Tests**
+  - Installation Command: `pip install apache-beam[test]`
+  - Required for developing on beam and running unittests
+- **Docs**
+  - Installation Command: `pip install apache-beam[docs]`
+  - Generating API documentation using Sphinx
+
 ## Execute a pipeline locally
 
 The Apache Beam 
[examples](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples)
 directory has many examples. All examples can be run locally by passing the 
required arguments described in the example script.

[2/3] beam-site git commit: Regenerate website

[altay]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/f011e303
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/f011e303
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/f011e303

Branch: refs/heads/asf-site
Commit: f011e303c71c81675ec6d7e535cd815fb5c32074
Parents: 0dd610f
Author: Ahmet Altay 
Authored: Mon Apr 3 15:52:46 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 15:52:46 2017 -0700

--
 content/get-started/quickstart-py/index.html | 37 ++-
 1 file changed, 36 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/f011e303/content/get-started/quickstart-py/index.html
--
diff --git a/content/get-started/quickstart-py/index.html 
b/content/get-started/quickstart-py/index.html
index ce774cb..7c8f2d3 100644
--- a/content/get-started/quickstart-py/index.html
+++ b/content/get-started/quickstart-py/index.html
@@ -163,7 +163,10 @@
   
   Get Apache 
Beam
   Create and activate 
a virtual environment
-  Download and install
+  Download and install
+  Extra Requirements
+
+  
 
   
   Execute a pipeline locally
@@ -227,6 +230,38 @@ environment’s directories.
 
 
 
+Extra Requirements
+
+The above installation will not install all the extra dependencies for 
using features like the Google Cloud Dataflow runner. Information on what extra 
packages are required for different features are highlighted below. It is 
possible to install multitple extra requirements using something like pip install apache-beam[feature1, 
feature2].
+
+
+  Google Cloud Platform
+
+  Installation Command: pip install 
apache-beam[gcp]
+  Required for:
+
+  Google Cloud Dataflow Runner
+  GCS IO
+  Datastore IO
+  BigQuery IO
+
+  
+
+  
+  Tests
+
+  Installation Command: pip install 
apache-beam[test]
+  Required for developing on beam and running unittests
+
+  
+  Docs
+
+  Installation Command: pip install 
apache-beam[docs]
+  Generating API documentation using Sphinx
+
+  
+
+
 Execute a pipeline locally
 
 The Apache Beam https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples;>examples
 directory has many examples. All examples can be run locally by passing the 
required arguments described in the example script.

[3/3] beam-site git commit: This closes #201

[altay]

This closes #201


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/5359c329
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/5359c329
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/5359c329

Branch: refs/heads/asf-site
Commit: 5359c329460ac91db8340befb1d55c8687196017
Parents: 85103fb f011e30
Author: Ahmet Altay 
Authored: Mon Apr 3 15:52:46 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 15:52:46 2017 -0700

--
 content/get-started/quickstart-py/index.html | 37 ++-
 src/get-started/quickstart-py.md | 18 +++
 2 files changed, 54 insertions(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-800) DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954276#comment-15954276
 ] 

ASF GitHub Bot commented on BEAM-800:
-

Github user manuzhang closed the pull request at:

https://github.com/apache/beam/pull/1232


> DoFnLifeCycleManager should hold DoFnInvoker, not DoFn/OldDoFn
> --
>
> Key: BEAM-800
> URL: https://issues.apache.org/jira/browse/BEAM-800
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Kenneth Knowles
>Assignee: Manu Zhang
>Priority: Minor
> Fix For: Not applicable
>
>
> The {{DirectRunner}}'s support class {{DoFnLifecycleManager}} holds a cache 
> of deserialized {{OldDoFn}} s, now being ported to {{DoFn}} s. But to execute 
> a {{DoFn}} there is another layer of indirection through a {{DoFnInvoker}}, 
> which is the best object to cache here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1867) Element counts missing on Cloud Dataflow when PCollection is renamed (by user or pipeline surgery)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1867:
--
Description: 
In 0.6.0 and 0.7.0-SNAPSHOT (and possibly all past versions, these are just 
those where it is confirmed) element count and byte metrics are not reported 
correctly when the output PCollection for a primitive transform is not 
{{transformname + ".out" + index}}.

In 0.7.0-SNAPSHOT, the DataflowRunner uses pipeline surgery to replace the 
composite {{ParDoSingle}} (that contains a {{ParDoMulti}}) with a 
Dataflow-specific non-composite {{ParDoSingle}}. So metrics are reported for 
names like {{"ParDoSingle(MyDoFn).out"}} when they should be reported for 
{{"ParDoSingle/ParDoMulti(MyDoFn).out"}}. So all single-output ParDo transforms 
lack these metrics on their outputs.

In 0.6.0 the same problem occurs if the user ever uses {{PCollection.setName}} 
to give their collection a meaningful name.

  was:It seems that most PCollections have lost their element counts. It seems 
likely that the problem exists in the translator. I'm investigating.


> Element counts missing on Cloud Dataflow when PCollection is renamed (by user 
> or pipeline surgery)
> --
>
> Key: BEAM-1867
> URL: https://issues.apache.org/jira/browse/BEAM-1867
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Priority: Blocker
> Fix For: First stable release
>
>
> In 0.6.0 and 0.7.0-SNAPSHOT (and possibly all past versions, these are just 
> those where it is confirmed) element count and byte metrics are not reported 
> correctly when the output PCollection for a primitive transform is not 
> {{transformname + ".out" + index}}.
> In 0.7.0-SNAPSHOT, the DataflowRunner uses pipeline surgery to replace the 
> composite {{ParDoSingle}} (that contains a {{ParDoMulti}}) with a 
> Dataflow-specific non-composite {{ParDoSingle}}. So metrics are reported for 
> names like {{"ParDoSingle(MyDoFn).out"}} when they should be reported for 
> {{"ParDoSingle/ParDoMulti(MyDoFn).out"}}. So all single-output ParDo 
> transforms lack these metrics on their outputs.
> In 0.6.0 the same problem occurs if the user ever uses 
> {{PCollection.setName}} to give their collection a meaningful name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1867) Element counts missing on Cloud Dataflow when PCollection is renamed (by user or pipeline surgery)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-1867:
-

Assignee: (was: Kenneth Knowles)

> Element counts missing on Cloud Dataflow when PCollection is renamed (by user 
> or pipeline surgery)
> --
>
> Key: BEAM-1867
> URL: https://issues.apache.org/jira/browse/BEAM-1867
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Priority: Blocker
> Fix For: First stable release
>
>
> It seems that most PCollections have lost their element counts. It seems 
> likely that the problem exists in the translator. I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2393: Update integration tests in post commits

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2393


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Update integration tests in post commits

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 570d0e2a1 -> 3711c0caf


Update integration tests in post commits


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ffe8355b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ffe8355b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ffe8355b

Branch: refs/heads/master
Commit: ffe8355b212e2d7f98d2dd152bbf7bc37448ad58
Parents: 570d0e2
Author: Ahmet Altay 
Authored: Fri Mar 31 15:11:43 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 15:14:28 2017 -0700

--
 .../cookbook/bigquery_tornadoes_it_test.py  |  3 +++
 .../apache_beam/examples/wordcount_it_test.py   |  6 +-
 sdks/python/run_postcommit.sh   | 20 +---
 3 files changed, 17 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ffe8355b/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py
--
diff --git 
a/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py 
b/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py
index 306a09e..709e3d5 100644
--- a/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py
@@ -32,6 +32,9 @@ from apache_beam.tests.pipeline_verifiers import 
PipelineStateMatcher
 
 class BigqueryTornadoesIT(unittest.TestCase):
 
+  # Enable nose tests running in parallel
+  _multiprocess_can_split_ = True
+
   # The default checksum is a SHA-1 hash generated from sorted rows reading
   # from expected Bigquery table.
   DEFAULT_CHECKSUM = '83789a7c1bca7959dcf23d3bc37e9204e594330f'

http://git-wip-us.apache.org/repos/asf/beam/blob/ffe8355b/sdks/python/apache_beam/examples/wordcount_it_test.py
--
diff --git a/sdks/python/apache_beam/examples/wordcount_it_test.py 
b/sdks/python/apache_beam/examples/wordcount_it_test.py
index 1c700b6..54e54e8 100644
--- a/sdks/python/apache_beam/examples/wordcount_it_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_it_test.py
@@ -18,6 +18,7 @@
 """End-to-end test for the wordcount example."""
 
 import logging
+import time
 import unittest
 
 from hamcrest.core.core.allof import all_of
@@ -31,6 +32,9 @@ from apache_beam.tests.pipeline_verifiers import 
FileChecksumMatcher
 
 class WordCountIT(unittest.TestCase):
 
+  # Enable nose tests running in parallel
+  _multiprocess_can_split_ = True
+
   # The default checksum is a SHA-1 hash generated from a sorted list of
   # lines read from expected output.
   DEFAULT_CHECKSUM = '33535a832b7db6d78389759577d4ff495980b9c0'
@@ -41,7 +45,7 @@ class WordCountIT(unittest.TestCase):
 
 # Set extra options to the pipeline for test purpose
 output = '/'.join([test_pipeline.get_option('output'),
-   test_pipeline.get_option('job_name'),
+   str(int(time.time())),
'results'])
 arg_sleep_secs = test_pipeline.get_option('sleep_secs')
 sleep_secs = int(arg_sleep_secs) if arg_sleep_secs is not None else None

http://git-wip-us.apache.org/repos/asf/beam/blob/ffe8355b/sdks/python/run_postcommit.sh
--
diff --git a/sdks/python/run_postcommit.sh b/sdks/python/run_postcommit.sh
index 50338e2..dd3182a 100755
--- a/sdks/python/run_postcommit.sh
+++ b/sdks/python/run_postcommit.sh
@@ -56,12 +56,9 @@ python -m apache_beam.examples.wordcount --output 
/tmp/py-wordcount-direct
 
 # Run tests on the service.
 
-# Where to store wordcount output.
+# Where to store integration test outputs.
 GCS_LOCATION=gs://temp-storage-for-end-to-end-tests
 
-# Job name needs to be unique
-JOBNAME_E2E_WC=py-wordcount-`date +%s`
-
 PROJECT=apache-beam-testing
 
 # Create a tarball
@@ -88,18 +85,19 @@ python setup.py nosetests \
 --requirements_file=postcommit_requirements.txt \
 --num_workers=1"
 
-# Run wordcount on the Google Cloud Dataflow service
-# and validate job that finishes successfully.
-echo ">>> RUNNING TEST DATAFLOW RUNNER py-wordcount"
+# Run integration tests on the Google Cloud Dataflow service
+# and validate that jobs finish successfully.
+echo ">>> RUNNING TEST DATAFLOW RUNNER it tests"
 python setup.py nosetests \
   -a IT \
+  --processes=4 \
+  --process-timeout=600 \
   --test-pipeline-options=" \
 --runner=TestDataflowRunner \
 --project=$PROJECT \
---staging_location=$GCS_LOCATION/staging-wordcount \
---temp_location=$GCS_LOCATION/temp-wordcount \
---output=$GCS_LOCATION/py-wordcount-cloud/output \
+--staging_location=$GCS_LOCATION/staging-it \

[2/2] beam git commit: This closes #2393

[altay]

This closes #2393


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3711c0ca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3711c0ca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3711c0ca

Branch: refs/heads/master
Commit: 3711c0caf91e1c4d32c055bdff098f81f56b49c1
Parents: 570d0e2 ffe8355
Author: Ahmet Altay 
Authored: Mon Apr 3 15:15:11 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 15:15:11 2017 -0700

--
 .../cookbook/bigquery_tornadoes_it_test.py  |  3 +++
 .../apache_beam/examples/wordcount_it_test.py   |  6 +-
 sdks/python/run_postcommit.sh   | 20 +---
 3 files changed, 17 insertions(+), 12 deletions(-)
--

[jira] [Assigned] (BEAM-1760) Potential null dereference in HDFSFileSink#doFinalize

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-1760:
-

Assignee: Ted Yu  (was: Davor Bonaci)

> Potential null dereference in HDFSFileSink#doFinalize
> -
>
> Key: BEAM-1760
> URL: https://issues.apache.org/jira/browse/BEAM-1760
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: Not applicable
>
>
> Here is related code:
> {code}
>   for (FileStatus s : statuses) {
> String name = s.getPath().getName();
> int pos = name.indexOf('.');
> String ext = pos > 0 ? name.substring(pos) : "";
> fs.rename(
> s.getPath(),
> new Path(s.getPath().getParent(), String.format("part-r-%05d%s", 
> i, ext)));
> i++;
>   }
> }
> {code}
> We should check whether s.getPath().getParent() is null.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-689) Remove getPCollection from (Un)CommittedBundle

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-689:
-
Labels: newbie starter  (was: )

> Remove getPCollection from (Un)CommittedBundle
> --
>
> Key: BEAM-689
> URL: https://issues.apache.org/jira/browse/BEAM-689
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>  Labels: newbie, starter
>
> After https://github.com/apache/incubator-beam/pull/1019 is merged, Root 
> bundles will no longer belong to a PCollection in all cases. Any (Bundle, 
> PCollection) pair that must be tracked should be done so outside of the 
> Bundle class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-689) Remove getPCollection from (Un)CommittedBundle

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954230#comment-15954230
 ] 

Thomas Groh commented on BEAM-689:
--

Right now we're leaving them nullable. There are a few places where we expect 
to be able to get a PCollection so we can find the producer of the bundle, but 
that could be part of the constructor, or so we can get a Coder (which should 
be explicit).

> Remove getPCollection from (Un)CommittedBundle
> --
>
> Key: BEAM-689
> URL: https://issues.apache.org/jira/browse/BEAM-689
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>  Labels: newbie, starter
>
> After https://github.com/apache/incubator-beam/pull/1019 is merged, Root 
> bundles will no longer belong to a PCollection in all cases. Any (Bundle, 
> PCollection) pair that must be tracked should be done so outside of the 
> Bundle class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-689) Remove getPCollection from (Un)CommittedBundle

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-689:


Assignee: (was: Thomas Groh)

> Remove getPCollection from (Un)CommittedBundle
> --
>
> Key: BEAM-689
> URL: https://issues.apache.org/jira/browse/BEAM-689
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Priority: Minor
>  Labels: newbie, starter
>
> After https://github.com/apache/incubator-beam/pull/1019 is merged, Root 
> bundles will no longer belong to a PCollection in all cases. Any (Bundle, 
> PCollection) pair that must be tracked should be done so outside of the 
> Bundle class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1642) Combine transformation evaluation fails on direct runner with Avro as a fallback coder

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954228#comment-15954228
 ] 

Thomas Groh commented on BEAM-1642:
---

When the AvroCoder provider is set, the output of the {{Values}} transform is 
set to {{AvroCoder}}. When the provider is not set, its output is (correctly) 
set to {{VarLongCoder}}. This means that there's a bug somewhere in our coder 
propagation and inference.

> Combine transformation evaluation fails on direct runner with Avro as a 
> fallback coder
> --
>
> Key: BEAM-1642
> URL: https://issues.apache.org/jira/browse/BEAM-1642
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 0.4.0, 0.5.0, 0.6.0
>Reporter: Sathya Hariesh
>Priority: Trivial
> Fix For: First stable release
>
>
> This does not happen in 0.3.0. Would probably need to investigate the root 
> cause.
> Steps to reproduce:
> public class WordCount {
> public static void main(String[] args) {
> DirectOptions directPipelineOptions = 
> PipelineOptionsFactory.as(DirectOptions.class);
> directPipelineOptions.setRunner(DirectRunner.class);
> Pipeline pipeline = Pipeline.create(directPipelineOptions);
> 
> pipeline.getCoderRegistry().setFallbackCoderProvider(AvroCoder.PROVIDER);
> pipeline.apply(Create.of(Arrays.asList(100, 
> 200))).apply(Count.globally());
> pipeline.run().waitUntilFinish();
> }
> }
> Stack trace:
> Exception in thread "main" java.lang.ClassCastException: java.lang.Object 
> cannot be cast to java.lang.Iterable
>   at 
> org.apache.beam.runners.direct.ViewEvaluatorFactory$1.processElement(ViewEvaluatorFactory.java:81)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1642) Combine transformation evaluation fails on direct runner with Avro as a fallback coder

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-1642:
-

Assignee: (was: Thomas Groh)

> Combine transformation evaluation fails on direct runner with Avro as a 
> fallback coder
> --
>
> Key: BEAM-1642
> URL: https://issues.apache.org/jira/browse/BEAM-1642
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 0.4.0, 0.5.0, 0.6.0
>Reporter: Sathya Hariesh
>Priority: Trivial
> Fix For: First stable release
>
>
> This does not happen in 0.3.0. Would probably need to investigate the root 
> cause.
> Steps to reproduce:
> public class WordCount {
> public static void main(String[] args) {
> DirectOptions directPipelineOptions = 
> PipelineOptionsFactory.as(DirectOptions.class);
> directPipelineOptions.setRunner(DirectRunner.class);
> Pipeline pipeline = Pipeline.create(directPipelineOptions);
> 
> pipeline.getCoderRegistry().setFallbackCoderProvider(AvroCoder.PROVIDER);
> pipeline.apply(Create.of(Arrays.asList(100, 
> 200))).apply(Count.globally());
> pipeline.run().waitUntilFinish();
> }
> }
> Stack trace:
> Exception in thread "main" java.lang.ClassCastException: java.lang.Object 
> cannot be cast to java.lang.Iterable
>   at 
> org.apache.beam.runners.direct.ViewEvaluatorFactory$1.processElement(ViewEvaluatorFactory.java:81)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-987) TestStream.advanceWatermarkToInfinity should perhaps also advance processing time

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-987:
-
Labels: starter  (was: )

> TestStream.advanceWatermarkToInfinity should perhaps also advance processing 
> time
> -
>
> Key: BEAM-987
> URL: https://issues.apache.org/jira/browse/BEAM-987
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct, testing
>Reporter: Eugene Kirpichov
>  Labels: starter
>
> I ran into this when writing a test for Splittable DoFn whose input was a 
> TestStream. I constructed a TestStream that didn't call 
> advanceProcessingTime, and as a result, the SDF's timers didn't fire and the 
> test got stuck.
> I think the meaning of "advanceWatermarkToInfinity" is "don't add any more 
> elements to the stream and see what happens eventually", and "eventually" 
> includes "eventually in processing time domain", not just in event-time 
> domain (watermark).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2414: Update PR template with correct ICLA link

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2414


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2414

[altay]

This closes #2414


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/570d0e2a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/570d0e2a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/570d0e2a

Branch: refs/heads/master
Commit: 570d0e2a119716dfdfc8b902e07ee3963561a419
Parents: 6d223bb 2a95b8f
Author: Ahmet Altay 
Authored: Mon Apr 3 14:34:59 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 14:34:59 2017 -0700

--
 .github/PULL_REQUEST_TEMPLATE.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] beam git commit: Update PR template with correct ICLA link

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 6d223bb13 -> 570d0e2a1


Update PR template with correct ICLA link


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2a95b8fc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2a95b8fc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2a95b8fc

Branch: refs/heads/master
Commit: 2a95b8fcb142c392fa3e48c4e80c7abc3b96a500
Parents: 6d223bb
Author: Sourabh Bajaj 
Authored: Mon Apr 3 13:21:00 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 14:34:52 2017 -0700

--
 .github/PULL_REQUEST_TEMPLATE.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2a95b8fc/.github/PULL_REQUEST_TEMPLATE.md
--
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index b17f620..9bbc9f7 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -8,6 +8,6 @@ quickly and easily:
  - [ ] Replace `` in the title with the actual Jira issue
number, if there is one.
  - [ ] If this contribution is large, please file an Apache
-   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).
+   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).
 
 ---

[GitHub] beam-site pull request #200: Fix the link to the apache ICLA

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/200


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam-site git commit: Regenerate website

[altay]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/3ee76398
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/3ee76398
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/3ee76398

Branch: refs/heads/asf-site
Commit: 3ee763988c610d145b6de6a6b532510aa3ba27e3
Parents: 30d2dc7
Author: Ahmet Altay 
Authored: Mon Apr 3 14:33:08 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 14:33:08 2017 -0700

--
 content/contribute/contribution-guide/index.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/3ee76398/content/contribute/contribution-guide/index.html
--
diff --git a/content/contribute/contribution-guide/index.html 
b/content/contribute/contribution-guide/index.html
index f5fefc1..b029d4c 100644
--- a/content/contribute/contribution-guide/index.html
+++ b/content/contribute/contribution-guide/index.html
@@ -283,7 +283,7 @@
 One-time Setup
 
 [Potentially] Submit 
Contributor License Agreement
-Apache Software Foundation (ASF) desires that all contributors of ideas, 
code, or documentation to the Apache projects complete, sign, and submit an https://www.apache.org/licenses/icla.txt;>Individual Contributor License 
Agreement (ICLA). The purpose of this agreement is to clearly define the 
terms under which intellectual property has been contributed to the ASF and 
thereby allow us to defend the project should there be a legal dispute 
regarding the software at some future time.
+Apache Software Foundation (ASF) desires that all contributors of ideas, 
code, or documentation to the Apache projects complete, sign, and submit an https://www.apache.org/licenses/icla.pdf;>Individual Contributor License 
Agreement (ICLA). The purpose of this agreement is to clearly define the 
terms under which intellectual property has been contributed to the ASF and 
thereby allow us to defend the project should there be a legal dispute 
regarding the software at some future time.
 
 We require you to have an ICLA on file with the Apache Secretary for larger 
contributions only. For smaller ones, however, we rely on http://www.apache.org/licenses/LICENSE-2.0#contributions;>clause five 
of the Apache License, Version 2.0, describing licensing of intentionally 
submitted contributions and do not require an ICLA in that case.
 

[3/3] beam-site git commit: This closes #200

[altay]

This closes #200


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/85103fb5
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/85103fb5
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/85103fb5

Branch: refs/heads/asf-site
Commit: 85103fb5fcff145509beac2a6757e3ffa90accb3
Parents: 787d570 3ee7639
Author: Ahmet Altay 
Authored: Mon Apr 3 14:33:08 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 14:33:08 2017 -0700

--
 content/contribute/contribution-guide/index.html | 2 +-
 src/contribute/contribution-guide.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--

[1/3] beam-site git commit: Fix the link to the apache ICLA

[altay]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 787d5706d -> 85103fb5f


Fix the link to the apache ICLA


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/30d2dc7f
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/30d2dc7f
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/30d2dc7f

Branch: refs/heads/asf-site
Commit: 30d2dc7fa86d2b4e0edff128c0c465ba1a793bff
Parents: 787d570
Author: Sourabh Bajaj 
Authored: Mon Apr 3 13:18:09 2017 -0700
Committer: Sourabh Bajaj 
Committed: Mon Apr 3 13:18:09 2017 -0700

--
 src/contribute/contribution-guide.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/30d2dc7f/src/contribute/contribution-guide.md
--
diff --git a/src/contribute/contribution-guide.md 
b/src/contribute/contribution-guide.md
index 74ee879..d55dd25 100644
--- a/src/contribute/contribution-guide.md
+++ b/src/contribute/contribution-guide.md
@@ -68,7 +68,7 @@ When developing a new `PTransform`, consult the [PTransform 
Style Guide]({{ site
 ### One-time Setup
 
  [Potentially] Submit Contributor License Agreement
-Apache Software Foundation (ASF) desires that all contributors of ideas, code, 
or documentation to the Apache projects complete, sign, and submit an 
[Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt) (ICLA). The purpose of 
this agreement is to clearly define the terms under which intellectual property 
has been contributed to the ASF and thereby allow us to defend the project 
should there be a legal dispute regarding the software at some future time.
+Apache Software Foundation (ASF) desires that all contributors of ideas, code, 
or documentation to the Apache projects complete, sign, and submit an 
[Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf) (ICLA). The purpose of 
this agreement is to clearly define the terms under which intellectual property 
has been contributed to the ASF and thereby allow us to defend the project 
should there be a legal dispute regarding the software at some future time.
 
 We require you to have an ICLA on file with the Apache Secretary for larger 
contributions only. For smaller ones, however, we rely on [clause 
five](http://www.apache.org/licenses/LICENSE-2.0#contributions) of the Apache 
License, Version 2.0, describing licensing of intentionally submitted 
contributions and do not require an ICLA in that case.
 

[jira] [Commented] (BEAM-437) Data-dependent BigQueryIO in batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954162#comment-15954162
 ] 

ASF GitHub Bot commented on BEAM-437:
-

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/2415

[BEAM-437] Support data-dependent writes using BigQuery batch load jobs

This pull request adds support for data-dependent writes when using batch 
load jobs. This is accomplished via refactoring BigQueryIO into separate 
transforms, with the first being a common PrepareWrite transform that 
determines which tables records should go to, followed by transforms that know 
how to interpret this.

One side benefit of this refactoring is that the different components can 
be used on their own. For example, one request has been to allow dynamic 
creation of datasets in BigQueryIO. A user can now accomplish this by running 
PrepareWrite themselves, followed by their own custom transform to create 
datasets, and then the remaining transform.

In order to test this, BigQueryIOTest was modified to use a proper fake 
service, removing the dependency on mockito.

R: @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam 
dynamic_writes_in_batch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2415.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2415


commit 6088bf19dc03bb5ca0ccb760c52793ae27dfc06b
Author: Reuven Lax 
Date:   2017-03-28T18:21:59Z

Use tableRefFunction throughout BigQueryIO. Constant table writes use 
ConstantTableSpecFunction.

commit 73fa547e4ca2b44c4f11d7c7ed4d7ac77a701ad5
Author: Reuven Lax 
Date:   2017-03-28T19:53:27Z

Add PrepareWrite transform.

commit 60040c4991ee2fe5572d3dd7e2dfd381e21cead8
Author: Reuven Lax 
Date:   2017-03-29T02:34:56Z

Refactor streaming write branch into separate reusable components.

commit 359685ab997c934837c601610fec471b3da1dcbd
Author: Reuven Lax 
Date:   2017-03-29T14:34:10Z

Refactor batch load job path, and add support for data-dependent tables.

commit c9a1f2916af5cd2837d4d73887005e3b2ceff401
Author: Reuven Lax 
Date:   2017-03-31T18:19:25Z

Refactor batch loads, and add support for windowed writes.

commit 477b14f4952881d965f22b7591da1032dcfd0495
Author: Reuven Lax 
Date:   2017-03-31T21:16:48Z

Update tests

commit a6fb0292879b7ff9a68de2884417a4efd21f6479
Author: Reuven Lax 
Date:   2017-04-01T01:53:04Z

testing changes

commit 5a2a2dc55bb7339a5c17280ed6ad66cb13eef54d
Author: Reuven Lax 
Date:   2017-04-02T18:32:37Z

Fix more tests

commit cc146874470b51b0295a02cdcb81effda03372af
Author: Reuven Lax 
Date:   2017-04-02T18:37:06Z

Fix CheckStyle issues

commit 89f2dc88431e71f8d11cd9942c2ef653bfc1a2c1
Author: Reuven Lax 
Date:   2017-04-03T02:47:03Z

Final tests all work now

commit 6662121da44f16d79718c68dccf6eb6a86329268
Author: Reuven Lax 
Date:   2017-04-03T02:57:50Z

Some cleanups and comments

commit 257ccc06f10cd048b8190e124b241f3bd98c647b
Author: Reuven Lax 
Date:   2017-04-03T03:27:16Z

Remove ReturnT

commit 1ad3720c0273a808cafc2dd4d6e096b4f492c42b
Author: Reuven Lax 
Date:   2017-04-03T04:39:50Z

Separate streaming writes into two pluggable components - CreateTables, and 
StreamingWriteTables.

commit a111b148a2bf8bbb5f1119c0bff922c0801d0582
Author: Reuven Lax 
Date:   2017-04-03T04:43:16Z

Checkstyle fixes




> Data-dependent BigQueryIO in batch
> --
>
> Key: BEAM-437
> URL: https://issues.apache.org/jira/browse/BEAM-437
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Reuven Lax
>Priority: Minor
>
> Blocked by [BEAM-92].
> Right now, we use BigQuery's streaming write API when using window-dependent 
> tables in BigQuery. We should
> 1. Support data-dependent tables as well.
> 2. Find a way to use the batch write API.
> 3. This requires careful design to be idempotent or, at least, as close to 
> idempotent as possible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2415: [BEAM-437] Support data-dependent writes using BigQ...

[reuvenlax]

GitHub user reuvenlax opened a pull request:

https://github.com/apache/beam/pull/2415

[BEAM-437] Support data-dependent writes using BigQuery batch load jobs

This pull request adds support for data-dependent writes when using batch 
load jobs. This is accomplished via refactoring BigQueryIO into separate 
transforms, with the first being a common PrepareWrite transform that 
determines which tables records should go to, followed by transforms that know 
how to interpret this.

One side benefit of this refactoring is that the different components can 
be used on their own. For example, one request has been to allow dynamic 
creation of datasets in BigQueryIO. A user can now accomplish this by running 
PrepareWrite themselves, followed by their own custom transform to create 
datasets, and then the remaining transform.

In order to test this, BigQueryIOTest was modified to use a proper fake 
service, removing the dependency on mockito.

R: @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam 
dynamic_writes_in_batch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2415.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2415


commit 6088bf19dc03bb5ca0ccb760c52793ae27dfc06b
Author: Reuven Lax 
Date:   2017-03-28T18:21:59Z

Use tableRefFunction throughout BigQueryIO. Constant table writes use 
ConstantTableSpecFunction.

commit 73fa547e4ca2b44c4f11d7c7ed4d7ac77a701ad5
Author: Reuven Lax 
Date:   2017-03-28T19:53:27Z

Add PrepareWrite transform.

commit 60040c4991ee2fe5572d3dd7e2dfd381e21cead8
Author: Reuven Lax 
Date:   2017-03-29T02:34:56Z

Refactor streaming write branch into separate reusable components.

commit 359685ab997c934837c601610fec471b3da1dcbd
Author: Reuven Lax 
Date:   2017-03-29T14:34:10Z

Refactor batch load job path, and add support for data-dependent tables.

commit c9a1f2916af5cd2837d4d73887005e3b2ceff401
Author: Reuven Lax 
Date:   2017-03-31T18:19:25Z

Refactor batch loads, and add support for windowed writes.

commit 477b14f4952881d965f22b7591da1032dcfd0495
Author: Reuven Lax 
Date:   2017-03-31T21:16:48Z

Update tests

commit a6fb0292879b7ff9a68de2884417a4efd21f6479
Author: Reuven Lax 
Date:   2017-04-01T01:53:04Z

testing changes

commit 5a2a2dc55bb7339a5c17280ed6ad66cb13eef54d
Author: Reuven Lax 
Date:   2017-04-02T18:32:37Z

Fix more tests

commit cc146874470b51b0295a02cdcb81effda03372af
Author: Reuven Lax 
Date:   2017-04-02T18:37:06Z

Fix CheckStyle issues

commit 89f2dc88431e71f8d11cd9942c2ef653bfc1a2c1
Author: Reuven Lax 
Date:   2017-04-03T02:47:03Z

Final tests all work now

commit 6662121da44f16d79718c68dccf6eb6a86329268
Author: Reuven Lax 
Date:   2017-04-03T02:57:50Z

Some cleanups and comments

commit 257ccc06f10cd048b8190e124b241f3bd98c647b
Author: Reuven Lax 
Date:   2017-04-03T03:27:16Z

Remove ReturnT

commit 1ad3720c0273a808cafc2dd4d6e096b4f492c42b
Author: Reuven Lax 
Date:   2017-04-03T04:39:50Z

Separate streaming writes into two pluggable components - CreateTables, and 
StreamingWriteTables.

commit a111b148a2bf8bbb5f1119c0bff922c0801d0582
Author: Reuven Lax 
Date:   2017-04-03T04:43:16Z

Checkstyle fixes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1823) TimedOutException in postcommit

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954156#comment-15954156
 ] 

Ahmet Altay commented on BEAM-1823:
---

Should we do the `-s` trick in our tests, does it have any other implications 
for us?

> TimedOutException in postcommit
> ---
>
> Key: BEAM-1823
> URL: https://issues.apache.org/jira/browse/BEAM-1823
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>
> Mark, do you know what this error means? Where is the timeout configured.
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1657/console
> I _think_ this is one of the underlying Dataflow executions and it completed 
> (although much slower than usual):
> https://pantheon.corp.google.com/dataflow/job/2017-03-28_14_25_21-13472017589125356257?project=apache-beam-testing=433637338589
> It makes sense to time out the test but I want to know how it is configured. 
> Also, is it possible to print out output logs for failed/timed out tests so 
> that we can clearly associate tests with job executions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-503) FileBasedSource should take a list of files/globs

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954154#comment-15954154
 ] 

Chamikara Jayalath commented on BEAM-503:
-

We currently don't have any plans to do this.

This use-case should be covered by SplittableDoFn: 
https://issues.apache.org/jira/browse/BEAM-1630

> FileBasedSource should take a list of files/globs
> -
>
> Key: BEAM-503
> URL: https://issues.apache.org/jira/browse/BEAM-503
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> This is the Python SDK version of following issue.
> https://issues.apache.org/jira/browse/BEAM-76



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-503) FileBasedSource should take a list of files/globs

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath closed BEAM-503.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> FileBasedSource should take a list of files/globs
> -
>
> Key: BEAM-503
> URL: https://issues.apache.org/jira/browse/BEAM-503
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> This is the Python SDK version of following issue.
> https://issues.apache.org/jira/browse/BEAM-76



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #201: [BEAM-1711] Add information about extra require...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam-site/pull/201

[BEAM-1711] Add information about extra requires in Python Quickstart

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam-site 
BEAM-1711-extra-requires-documentation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/201.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #201


commit 0287102828a73cf0b4e0ef08c50a6439dd1849cc
Author: Sourabh Bajaj 
Date:   2017-04-03T20:40:04Z

[BEAM-1711] Add information about extra requires in Python Quickstart




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3143

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2414: Update PR template with correct ICLA link

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2414

Update PR template with correct ICLA link

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aaltay @davorbonaci PTAL

Same as https://github.com/apache/beam-site/pull/200 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-fix-ICLA

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2414.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2414


commit 4768e9b3ef48e1167b96b3bd324a22f7c1e133ad
Author: Sourabh Bajaj 
Date:   2017-04-03T20:21:00Z

Update PR template with correct ICLA link




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #200: Fix the link to the apache ICLA

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam-site/pull/200

Fix the link to the apache ICLA

R: @davorbonaci @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam-site BEAM-fix-icla-link

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/200.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #200


commit 30d2dc7fa86d2b4e0edff128c0c465ba1a793bff
Author: Sourabh Bajaj 
Date:   2017-04-03T20:18:09Z

Fix the link to the apache ICLA




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #2727

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-1264) Python ChannelFactory Raise Inconsistent Error for Local FS and GCS

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu closed BEAM-1264.
--

> Python ChannelFactory Raise Inconsistent Error for Local FS and GCS
> ---
>
> Key: BEAM-1264
> URL: https://issues.apache.org/jira/browse/BEAM-1264
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Mark Liu
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>
> ChannelFactory raises different errors for local fs (RuntimeError) and GCS 
> (IOError) when reading failed. 
> We want to return consistent error for both.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1868) CreateStreamTest is flaky

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1868:
-

 Summary: CreateStreamTest is flaky
 Key: BEAM-1868
 URL: https://issues.apache.org/jira/browse/BEAM-1868
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Thomas Groh


Example failure:

https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/3142/testReport/junit/org.apache.beam.runners.spark.translation.streaming/CreateStreamTest/testLateDataAccumulating/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1823) TimedOutException in postcommit

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954083#comment-15954083
 ] 

Mark Liu commented on BEAM-1823:


nose capture logs but not print them when using mutiprocess and interrupted by 
TimedOutException. Created an issue on nose side: 
https://github.com/nose-devs/nose/issues/1044

A work around solution is to print job id/metrics to std out with "-s" flag 
(http://nose.readthedocs.io/en/latest/man.html#cmdoption-s).

> TimedOutException in postcommit
> ---
>
> Key: BEAM-1823
> URL: https://issues.apache.org/jira/browse/BEAM-1823
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Mark Liu
>
> Mark, do you know what this error means? Where is the timeout configured.
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1657/console
> I _think_ this is one of the underlying Dataflow executions and it completed 
> (although much slower than usual):
> https://pantheon.corp.google.com/dataflow/job/2017-03-28_14_25_21-13472017589125356257?project=apache-beam-testing=433637338589
> It makes sense to time out the test but I want to know how it is configured. 
> Also, is it possible to print out output logs for failed/timed out tests so 
> that we can clearly associate tests with job executions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1400) Transient Apex runner postcommit failure in WindowTest.testOutputTimeFnDefault

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-1400:
--
Labels: flake  (was: )

> Transient Apex runner postcommit failure in WindowTest.testOutputTimeFnDefault
> --
>
> Key: BEAM-1400
> URL: https://issues.apache.org/jira/browse/BEAM-1400
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Kenneth Knowles
>  Labels: flake
>
> The output timestamp appears to have come back as {{now()}} instead of the 
> end of the window.
> https://builds.apache.org/job/beam_PostCommit_Java_RunnableOnService_Apex/411/org.apache.beam$beam-runners-apex/testReport/junit/org.apache.beam.sdk.transforms.windowing/WindowTest/testOutputTimeFnDefault/
> It is possible that this is a flake in some runners/core-java code. Keeping 
> it here for now until we see it happen somewhere else.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-688) Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency error

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-688:
-
Labels: flake  (was: )

> Failure of beam-sdks-java-maven-archetypes-starter with undeclared dependency 
> error
> ---
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>  Labels: flake
>
> The starter archetype has flaky dependencies. It is reported to fail reliably 
> on repeatedly install.
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-1582:
--
Labels: flake  (was: )

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>  Labels: flake
> Fix For: First stable release
>
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1719) Test modules are included in generated documentation

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1719.
---
   Resolution: Fixed
Fix Version/s: First stable release

> Test modules are included in generated documentation
> 
>
> Key: BEAM-1719
> URL: https://issues.apache.org/jira/browse/BEAM-1719
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Tibor Kiss
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: First stable release
>
>
> Python-SDK uses Sphinx to generate documentation.
> As of now the test modules are also included in the documentation generation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1719) Test modules are included in generated documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954081#comment-15954081
 ] 

ASF GitHub Bot commented on BEAM-1719:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2410


> Test modules are included in generated documentation
> 
>
> Key: BEAM-1719
> URL: https://issues.apache.org/jira/browse/BEAM-1719
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Tibor Kiss
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> Python-SDK uses Sphinx to generate documentation.
> As of now the test modules are also included in the documentation generation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2410: [BEAM-1719] Remove test modules from PyDoc

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2410


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1864) Shorten combining state names: "CombiningValue" and "AccumulatorCombiningState" to Combining (as appropriate)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954079#comment-15954079
 ] 

ASF GitHub Bot commented on BEAM-1864:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2413

[BEAM-1864] Rename CombiningState and friends to be more user-friendly

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam combining-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2413.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2413


commit d723ccfce33a5ae2969e44ad20692c152c54ce78
Author: Kenneth Knowles 
Date:   2017-04-03T18:26:30Z

Rename CombiningState to GroupingState

commit 1aeb69d04a42d9e7d37c65f3c7db9ab33cf5dfd0
Author: Kenneth Knowles 
Date:   2017-04-03T18:27:26Z

Rename AccumulatorCombiningState to CombiningState

commit a272ef908dc8c123894949f85bf041b34096482b
Author: Kenneth Knowles 
Date:   2017-04-03T18:37:50Z

Rename combiningValue to combining in StateSpecs




> Shorten combining state names: "CombiningValue" and 
> "AccumulatorCombiningState" to Combining (as appropriate)
> -
>
> Key: BEAM-1864
> URL: https://issues.apache.org/jira/browse/BEAM-1864
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: First stable release
>
>
> This will clean up potential confusion around different kinds of state, 
> making it very clear which type is actually analogous to the {{Combine}} 
> transform and takes a {{CombineFn}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2410

[altay]

This closes #2410


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6d223bb1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6d223bb1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6d223bb1

Branch: refs/heads/master
Commit: 6d223bb13596998a227e055335c8b4e3e91020ec
Parents: 8a8dbcf f3385b2
Author: Ahmet Altay 
Authored: Mon Apr 3 13:04:14 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 13:04:14 2017 -0700

--
 sdks/python/generate_pydoc.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] beam git commit: [BEAM-1719] Remove test modules from PyDoc

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 8a8dbcfc5 -> 6d223bb13


[BEAM-1719] Remove test modules from PyDoc


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f3385b25
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f3385b25
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f3385b25

Branch: refs/heads/master
Commit: f3385b25a8af6169c81fb2d7f4ad60b132ea0150
Parents: 8a8dbcf
Author: Sourabh Bajaj 
Authored: Mon Apr 3 12:06:26 2017 -0700
Committer: Ahmet Altay 
Committed: Mon Apr 3 13:04:00 2017 -0700

--
 sdks/python/generate_pydoc.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f3385b25/sdks/python/generate_pydoc.sh
--
diff --git a/sdks/python/generate_pydoc.sh b/sdks/python/generate_pydoc.sh
index 1fe7ddf..b04e27a 100755
--- a/sdks/python/generate_pydoc.sh
+++ b/sdks/python/generate_pydoc.sh
@@ -39,7 +39,7 @@ excluded_internal_clients=(
 apache_beam/examples/complete/juliaset/setup.py)
 
 python $(type -p sphinx-apidoc) -f -o target/docs/source apache_beam \
-"${excluded_internal_clients[@]}"
+"${excluded_internal_clients[@]}" "*_test.py"
 
 # Remove Cython modules from doc template; they won't load
 sed -i -e '/.. automodule:: apache_beam.coders.stream/d' \

[GitHub] beam pull request #2413: [BEAM-1864] Rename CombiningState and friends to be...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2413

[BEAM-1864] Rename CombiningState and friends to be more user-friendly

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam combining-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2413.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2413


commit d723ccfce33a5ae2969e44ad20692c152c54ce78
Author: Kenneth Knowles 
Date:   2017-04-03T18:26:30Z

Rename CombiningState to GroupingState

commit 1aeb69d04a42d9e7d37c65f3c7db9ab33cf5dfd0
Author: Kenneth Knowles 
Date:   2017-04-03T18:27:26Z

Rename AccumulatorCombiningState to CombiningState

commit a272ef908dc8c123894949f85bf041b34096482b
Author: Kenneth Knowles 
Date:   2017-04-03T18:37:50Z

Rename combiningValue to combining in StateSpecs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1030) DoFnSignatures should validate StateSpec against key type

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1030:
--
Fix Version/s: First stable release

> DoFnSignatures should validate StateSpec against key type
> -
>
> Key: BEAM-1030
> URL: https://issues.apache.org/jira/browse/BEAM-1030
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1867) Element counts missing on Cloud Dataflow with Beam 0.7.0-SNAPSHOT

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1867:
-

 Summary: Element counts missing on Cloud Dataflow with Beam 
0.7.0-SNAPSHOT
 Key: BEAM-1867
 URL: https://issues.apache.org/jira/browse/BEAM-1867
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Priority: Blocker
 Fix For: First stable release


It seems that most PCollections have lost their element counts. It seems likely 
that the problem exists in the translator. I'm investigating.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954063#comment-15954063
 ] 

ASF GitHub Bot commented on BEAM-1866:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2412

[BEAM-1866] DataflowRunner: disable PAssert use of metrics when FnAPI is 
enabled

FnApi does not yet support metrics

R: @tgroh 

CC: @pabloem Pablo, I think the work you're doing to improve `PAssert` 
might be useful here.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam fnapi-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2412.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2412


commit 0a3a8b918ce66f11c610ce1948ce290e5f699520
Author: Dan Halperin 
Date:   2017-04-03T19:46:33Z

DataflowRunner: disable PAssert use of metrics when FnAPI is enabled

FnApi does not yet support metrics




> FnAPI support for Metrics
> -
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-fn-api
>Reporter: Daniel Halperin
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3141

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-1866) FnAPI support for Metrics

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1866:
--
Issue Type: New Feature  (was: Improvement)

> FnAPI support for Metrics
> -
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-fn-api
>Reporter: Daniel Halperin
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)