[jira] [Commented] (BEAM-1569) HDFSFileSource: Unable to read from filePattern with spaces in path

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887422#comment-15887422
 ] 

Jean-Baptiste Onofré commented on BEAM-1569:


Yes, I agree. A PR would be helpful. Ready to review !

> HDFSFileSource: Unable to read from filePattern with spaces in path
> ---
>
> Key: BEAM-1569
> URL: https://issues.apache.org/jira/browse/BEAM-1569
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>
> After the merge of the changes introduced with 
> https://issues.apache.org/jira/browse/BEAM-1497 we are unable to read from 
> files containing spaces in path. We encounter following stack trace
> {noformat}
> java.lang.reflect.UndeclaredThrowableException
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1713)
>   at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource.validate(HDFSFileSource.java:337)
>   at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource.createReader(HDFSFileSource.java:329)
>   at 
> org.apache.beam.sdk.testing.SourceTestUtils.readFromSource(SourceTestUtils.java:138)
> Caused by: java.net.URISyntaxException: Illegal character in path at index 
> 77: 
> /var/folders/1t/s9pcmfj50nxbt68h3_2z_5wcgn/T/junit6887354597440386901/tmp 
> data.seq
>   at java.net.URI$Parser.fail(URI.java:2848)
>   at java.net.URI$Parser.checkChars(URI.java:3021)
>   at java.net.URI$Parser.parseHierarchical(URI.java:3105)
>   at java.net.URI$Parser.parse(URI.java:3063)
>   at java.net.URI.(URI.java:588)
>   at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource$7.run(HDFSFileSource.java:340)
>   at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource$7.run(HDFSFileSource.java:337)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>   ... 40 more
> {noformat}
> This can be reproduced for instance by 
> {noformat}
>   // shameless copy of existing test case
>   @Test
>   public void testFullyReadSingleFileWithSpaces() throws Exception {
> PipelineOptions options = PipelineOptionsFactory.create();
> List> expectedResults = createRandomRecords(3, 10, 
> 0);
> File file = createFileWithData("tmp data.seq", expectedResults);
> HDFSFileSource, IntWritable, Text> source =
> HDFSFileSource.from(
> file.toString(), SequenceFileInputFormat.class, 
> IntWritable.class, Text.class);
> assertEquals(file.length(), source.getEstimatedSizeBytes(null));
> assertThat(expectedResults, containsInAnyOrder(readFromSource(source, 
> options).toArray()));
>   }
> {noformat}
> Changing the implementation slightly to
> {noformat}
> diff --git 
> a/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java
>  
> b/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java
> index 2a731fb..df72643 100644
> --- 
> a/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java
> +++ 
> b/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java
> @@ -30,7 +30,6 @@ import java.io.ObjectInput;
>  import java.io.ObjectOutput;
>  import java.lang.reflect.InvocationTargetException;
>  import java.lang.reflect.Method;
> -import java.net.URI;
>  import java.security.PrivilegedExceptionAction;
>  import java.util.List;
>  import java.util.ListIterator;
> @@ -337,9 +336,10 @@ public abstract class HDFSFileSource extends 
> BoundedSource {
>  UGIHelper.getBestUGI(username()).doAs(new 
> PrivilegedExceptionAction() {
>@Override
>public Void run() throws Exception {
> -FileSystem fs = FileSystem.get(new URI(filepattern()),
> +final Path pathPattern = new Path(filepattern());
> +FileSystem fs = FileSystem.get(pathPattern.toUri(),
>  
> SerializableConfiguration.newConfiguration(serializableConfiguration()));
> -FileStatus[] fileStatuses = fs.globStatus(new 
> Path(filepattern()));
> +FileStatus[] fileStatuses = fs.globStatus(pathPattern);
>  checkState(
>  fileStatuses != null && fileStatuses.length > 0,
>  "Unable to find any files matching %s", filepattern());
> {noformat}
> seems to be fixing the issue for us.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1569) HDFSFileSource: Unable to read from filePattern with spaces in path

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887423#comment-15887423
 ] 

Jean-Baptiste Onofré commented on BEAM-1569:


By the way, the refactoring of the HDFS IO broke bunch of things in our code 
too.

> HDFSFileSource: Unable to read from filePattern with spaces in path
> ---
>
> Key: BEAM-1569
> URL: https://issues.apache.org/jira/browse/BEAM-1569
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>
> After the merge of the changes introduced with 
> https://issues.apache.org/jira/browse/BEAM-1497 we are unable to read from 
> files containing spaces in path. We encounter following stack trace
> {noformat}
> java.lang.reflect.UndeclaredThrowableException
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1713)
>   at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource.validate(HDFSFileSource.java:337)
>   at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource.createReader(HDFSFileSource.java:329)
>   at 
> org.apache.beam.sdk.testing.SourceTestUtils.readFromSource(SourceTestUtils.java:138)
> Caused by: java.net.URISyntaxException: Illegal character in path at index 
> 77: 
> /var/folders/1t/s9pcmfj50nxbt68h3_2z_5wcgn/T/junit6887354597440386901/tmp 
> data.seq
>   at java.net.URI$Parser.fail(URI.java:2848)
>   at java.net.URI$Parser.checkChars(URI.java:3021)
>   at java.net.URI$Parser.parseHierarchical(URI.java:3105)
>   at java.net.URI$Parser.parse(URI.java:3063)
>   at java.net.URI.(URI.java:588)
>   at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource$7.run(HDFSFileSource.java:340)
>   at 
> org.apache.beam.sdk.io.hdfs.HDFSFileSource$7.run(HDFSFileSource.java:337)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>   ... 40 more
> {noformat}
> This can be reproduced for instance by 
> {noformat}
>   // shameless copy of existing test case
>   @Test
>   public void testFullyReadSingleFileWithSpaces() throws Exception {
> PipelineOptions options = PipelineOptionsFactory.create();
> List> expectedResults = createRandomRecords(3, 10, 
> 0);
> File file = createFileWithData("tmp data.seq", expectedResults);
> HDFSFileSource, IntWritable, Text> source =
> HDFSFileSource.from(
> file.toString(), SequenceFileInputFormat.class, 
> IntWritable.class, Text.class);
> assertEquals(file.length(), source.getEstimatedSizeBytes(null));
> assertThat(expectedResults, containsInAnyOrder(readFromSource(source, 
> options).toArray()));
>   }
> {noformat}
> Changing the implementation slightly to
> {noformat}
> diff --git 
> a/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java
>  
> b/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java
> index 2a731fb..df72643 100644
> --- 
> a/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java
> +++ 
> b/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java
> @@ -30,7 +30,6 @@ import java.io.ObjectInput;
>  import java.io.ObjectOutput;
>  import java.lang.reflect.InvocationTargetException;
>  import java.lang.reflect.Method;
> -import java.net.URI;
>  import java.security.PrivilegedExceptionAction;
>  import java.util.List;
>  import java.util.ListIterator;
> @@ -337,9 +336,10 @@ public abstract class HDFSFileSource extends 
> BoundedSource {
>  UGIHelper.getBestUGI(username()).doAs(new 
> PrivilegedExceptionAction() {
>@Override
>public Void run() throws Exception {
> -FileSystem fs = FileSystem.get(new URI(filepattern()),
> +final Path pathPattern = new Path(filepattern());
> +FileSystem fs = FileSystem.get(pathPattern.toUri(),
>  
> SerializableConfiguration.newConfiguration(serializableConfiguration()));
> -FileStatus[] fileStatuses = fs.globStatus(new 
> Path(filepattern()));
> +FileStatus[] fileStatuses = fs.globStatus(pathPattern);
>  checkState(
>  fileStatuses != null && fileStatuses.length > 0,
>  "Unable to find any files matching %s", filepattern());
> {noformat}
> seems to be fixing the issue for us.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1571) Flatten on a single input PCollection should have a test associated with it

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887408#comment-15887408
 ] 

ASF GitHub Bot commented on BEAM-1571:
--

GitHub user tweise opened a pull request:

https://github.com/apache/beam/pull/2125

[BEAM-1571] fix FlattenPCollectionList with single input translation in 
ApexRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
R: @tgroh 
fix for #2120


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tweise/beam BEAM-1571-singleton-flatten

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2125.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2125


commit 0166c996039042452b17cee51af20b2d2f32f8a4
Author: Thomas Weise 
Date:   2017-02-28T06:32:28Z

BEAM-1571 fix FlattenPCollectionList with single input translation in 
ApexRunner




> Flatten on a single input PCollection should have a test associated with it
> ---
>
> Key: BEAM-1571
> URL: https://issues.apache.org/jira/browse/BEAM-1571
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The expected behavior is that an output PCollection containing all of the 
> elements of the input PCollection is produced.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2125: [BEAM-1571] fix FlattenPCollectionList with single ...

[tweise]

GitHub user tweise opened a pull request:

https://github.com/apache/beam/pull/2125

[BEAM-1571] fix FlattenPCollectionList with single input translation in 
ApexRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
R: @tgroh 
fix for #2120


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tweise/beam BEAM-1571-singleton-flatten

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2125.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2125


commit 0166c996039042452b17cee51af20b2d2f32f8a4
Author: Thomas Weise 
Date:   2017-02-28T06:32:28Z

BEAM-1571 fix FlattenPCollectionList with single input translation in 
ApexRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1567) hashStream should be closed in PackageUtil#createPackageAttributes()

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887307#comment-15887307
 ] 

Ismaël Mejía commented on BEAM-1567:


Welcome [~tedyu] ! Nice to see you here, and even better with a contribution, 
awesome!
If you want to work on any of the HBase JIRAs just tell me and I will free it 
for you.
(sorry guys to jump in the thread, just wanted to say Hi to Ted since we had a 
previous conversation about existing HBaseIO in the hbase mailing list).

> hashStream should be closed in PackageUtil#createPackageAttributes()
> 
>
> Key: BEAM-1567
> URL: https://issues.apache.org/jira/browse/BEAM-1567
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Ted Yu
>Priority: Minor
>  Labels: newbie, starter
>
> Here is related code:
> {code}
>   OutputStream hashStream = Funnels.asOutputStream(hasher);
> {code}
> hashStream should be closed upon return from the method



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Apex #628

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1572) Add per-stage matching of scope in metrics for the DirectRunner

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887219#comment-15887219
 ] 

Aviem Zur commented on BEAM-1572:
-

Please note that this relates to https://issues.apache.org/jira/browse/BEAM-1344
The ultimate goal is to have a uniform querying logic across all runners.

> Add per-stage matching of scope in metrics for the DirectRunner
> ---
>
> Key: BEAM-1572
> URL: https://issues.apache.org/jira/browse/BEAM-1572
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>
> e.g. Metrics with scope "Top/Outer/Inner" should be matched by queries with 
> "Top/Outer" scope.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1563) Flatten Spark runner libraries.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1563:

Description: 
Flatten Spark runner libraries into a single package (or two) so that 
everything is private.
See [~kenn] comment: 
https://github.com/apache/beam/pull/2050#discussion_r103136216

  was:
Flatten Spark runner libraries into a single package (or two) so that 
everything is private.
See [~kenn] comment: https://github.com/apache/beam/pull/2050


> Flatten Spark runner libraries.
> ---
>
> Key: BEAM-1563
> URL: https://issues.apache.org/jira/browse/BEAM-1563
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> Flatten Spark runner libraries into a single package (or two) so that 
> everything is private.
> See [~kenn] comment: 
> https://github.com/apache/beam/pull/2050#discussion_r103136216



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1076) DatastoreIO template Options

[Vikas Kedigehalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Kedigehalli updated BEAM-1076:

Summary: DatastoreIO template Options  (was: Datastore Delete template)

> DatastoreIO template Options
> 
>
> Key: BEAM-1076
> URL: https://issues.apache.org/jira/browse/BEAM-1076
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Apex #627

[Apache Jenkins Server]

See 

[1/3] beam-site git commit: Comment Ostatic link as it has been broken for a few weeks

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 70682771d -> b647ca562


Comment Ostatic link as it has been broken for a few weeks


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/f1f7063b
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/f1f7063b
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/f1f7063b

Branch: refs/heads/asf-site
Commit: f1f7063b7e63d04df5f4cf264ef9df1d85cd02a2
Parents: 7068277
Author: Sourabh Bajaj 
Authored: Mon Feb 27 17:24:58 2017 -0800
Committer: Davor Bonaci 
Committed: Mon Feb 27 18:10:01 2017 -0800

--
 src/_posts/2017-02-01-graduation-media-recap.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/f1f7063b/src/_posts/2017-02-01-graduation-media-recap.md
--
diff --git a/src/_posts/2017-02-01-graduation-media-recap.md 
b/src/_posts/2017-02-01-graduation-media-recap.md
index a6c54a6..ef40916 100644
--- a/src/_posts/2017-02-01-graduation-media-recap.md
+++ b/src/_posts/2017-02-01-graduation-media-recap.md
@@ -33,7 +33,7 @@ include:
 * Datanami: "[Google Lauds Outside Influence on Apache 
Beam](https://www.datanami.com/2017/01/10/google-lauds-outside-influence-apache-beam/)"
 by Alex Woodie.
 * InfoWorld / JavaWorld: "[Apache Beam unifies batch and streaming for big 
data](http://www.infoworld.com/article/3156598/big-data/apache-beam-unifies-batch-and-streaming-for-big-data.html)"
 by Serdar Yegulalp, and republished in 
[JavaWorld](http://www.javaworld.com/article/3156598/big-data/apache-beam-unifies-batch-and-streaming-for-big-data.html).
 * JAXenter: "[In a way, Apache Beam is the glue that connects many big data 
systems together](https://jaxenter.com/apache-beam-interview-131314.html)" by 
Kypriani Sinaris.
-* OStatic: "[Apache Beam Unifies Batch and Streaming Data 
Processing](http://ostatic.com/blog/apache-beam-unifies-batch-and-streaming-data-processing)"
 by Sam Dean.
+* OStatic: "Apache Beam Unifies Batch and Streaming Data Processing" by Sam 
Dean. 
 * Enterprise Apps Today: "[Apache Beam Graduates to Help Define Streaming Data 
Processing](http://www.enterpriseappstoday.com/business-intelligence/data-analytics/apache-beam-graduates-to-help-define-streaming-data-processing.html)"
 by Sean Michael Kerner.
 * The Register: "[Google must be Beaming as Apache announces its new top-level 
projects](http://www.theregister.co.uk/2017/01/10/google_must_be_ibeamiing_as_apache_announces_its_new_top_level_projects/)"
 by Alexander J. Martin.
 * SiliconANGLE: "[Apache Software Foundation announces two more top-level open 
source 
projects](http://siliconangle.com/blog/2017/01/11/apache-software-foundation-announces-2-top-level-projects/)"
 by Mike Wheatley.

[3/3] beam-site git commit: This closes #164

[davor]

This closes #164


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/b647ca56
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/b647ca56
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/b647ca56

Branch: refs/heads/asf-site
Commit: b647ca5622a5aadc06d754ef8ae78ad91565b559
Parents: 7068277 1b1757b
Author: Davor Bonaci 
Authored: Mon Feb 27 18:10:22 2017 -0800
Committer: Davor Bonaci 
Committed: Mon Feb 27 18:10:22 2017 -0800

--
 content/blog/2017/02/01/graduation-media-recap.html | 2 +-
 content/feed.xml| 2 +-
 src/_posts/2017-02-01-graduation-media-recap.md | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
--

[GitHub] beam-site pull request #164: Remove Ostatic link as it has been broken for a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/164


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/1b1757bb
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/1b1757bb
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/1b1757bb

Branch: refs/heads/asf-site
Commit: 1b1757bb9157e733d3aea7c7367f21a8eca10755
Parents: f1f7063
Author: Davor Bonaci 
Authored: Mon Feb 27 18:10:21 2017 -0800
Committer: Davor Bonaci 
Committed: Mon Feb 27 18:10:21 2017 -0800

--
 content/blog/2017/02/01/graduation-media-recap.html | 2 +-
 content/feed.xml| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/1b1757bb/content/blog/2017/02/01/graduation-media-recap.html
--
diff --git a/content/blog/2017/02/01/graduation-media-recap.html 
b/content/blog/2017/02/01/graduation-media-recap.html
index 70e317f..81409a7 100644
--- a/content/blog/2017/02/01/graduation-media-recap.html
+++ b/content/blog/2017/02/01/graduation-media-recap.html
@@ -188,7 +188,7 @@ include:
   Datanami: “https://www.datanami.com/2017/01/10/google-lauds-outside-influence-apache-beam/";>Google
 Lauds Outside Influence on Apache Beam” by Alex Woodie.
   InfoWorld / JavaWorld: “http://www.infoworld.com/article/3156598/big-data/apache-beam-unifies-batch-and-streaming-for-big-data.html";>Apache
 Beam unifies batch and streaming for big data” by Serdar Yegulalp, and 
republished in http://www.javaworld.com/article/3156598/big-data/apache-beam-unifies-batch-and-streaming-for-big-data.html";>JavaWorld.
   JAXenter: “https://jaxenter.com/apache-beam-interview-131314.html";>In a way, Apache 
Beam is the glue that connects many big data systems together” by 
Kypriani Sinaris.
-  OStatic: “http://ostatic.com/blog/apache-beam-unifies-batch-and-streaming-data-processing";>Apache
 Beam Unifies Batch and Streaming Data Processing” by Sam Dean.
+  OStatic: “Apache Beam Unifies Batch and Streaming Data Processing” 
by Sam Dean. 
   Enterprise Apps Today: “http://www.enterpriseappstoday.com/business-intelligence/data-analytics/apache-beam-graduates-to-help-define-streaming-data-processing.html";>Apache
 Beam Graduates to Help Define Streaming Data Processing” by Sean Michael 
Kerner.
   The Register: “http://www.theregister.co.uk/2017/01/10/google_must_be_ibeamiing_as_apache_announces_its_new_top_level_projects/";>Google
 must be Beaming as Apache announces its new top-level projects” by 
Alexander J. Martin.
   SiliconANGLE: “http://siliconangle.com/blog/2017/01/11/apache-software-foundation-announces-2-top-level-projects/";>Apache
 Software Foundation announces two more top-level open source projects” 
by Mike Wheatley.

http://git-wip-us.apache.org/repos/asf/beam-site/blob/1b1757bb/content/feed.xml
--
diff --git a/content/feed.xml b/content/feed.xml
index 5c0cd90..63a513b 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -619,7 +619,7 @@ include:

   
Datanami: “Google
 Lauds Outside Influence on Apache Beam” by Alex Woodie.
   
InfoWorld / JavaWorld: “Apache
 Beam unifies batch and streaming for big data” by Serdar Yegulalp, 
and republished in JavaWorld.
;
   
JAXenter: “In a 
way, Apache Beam is the glue that connects many big data systems 
together” by Kypriani Sinaris.
-  
OStatic: “Apache
 Beam Unifies Batch and Streaming Data Processing” by Sam 
Dean.
+  
OStatic: “Apache Beam Unifies Batch and Streaming Data 
Processing” by Sam Dean. 
   
Enterprise Apps Today: “Apache
 Beam Graduates to Help Define Streaming Data Processing” by Sean 
Michael Kerner.
   
The Register: “Google
 must be Beaming as Apache announces its new top-level projects” by 
Alexander J. Martin.

[GitHub] beam pull request #2124: Make side inputs a map, rather than embedding the n...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/2124

Make side inputs a map, rather than embedding the name in the message.

The "local" name only make sense in context.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam runner-protos

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2124.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2124


commit b2c0f93d0f4d63836a79956d75b6dd100e9138ba
Author: Robert Bradshaw 
Date:   2017-02-28T01:59:57Z

Make side inputs a map, rather than embedding the name in the message.

The "local" name only make sense in context.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2123: (Early Draft) Provide implementation of BQIO.Read t...

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/beam/pull/2123

(Early Draft) Provide implementation of BQIO.Read that does not use Source 
API

@tgroh (and other interested parties) does this match your understanding of 
what needs to be built, with respect to structure?  Is the code reuse of 
*Source reasonable?

@dhalperi FYI

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam 
add_side_input_bigquery_read

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2123.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2123


commit 234aca37f5e8fb7d87fce7087e39956ee0320ed5
Author: Sam McVeety 
Date:   2017-02-27T03:18:12Z

Provide implementation of BQIO.Read that does not use Source API




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2106: [BEAM-115] More Runner API refinements.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2106


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887038#comment-15887038
 ] 

ASF GitHub Bot commented on BEAM-115:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2106


> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/3] beam git commit: Make access_pattern a URN with params.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master c8dfb851c -> 16736a6b6


Make access_pattern a URN with params.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a0dcd3f9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a0dcd3f9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a0dcd3f9

Branch: refs/heads/master
Commit: a0dcd3f97f48cdca02bae4c371e4e143d00fdba1
Parents: c8dfb85
Author: Robert Bradshaw 
Authored: Fri Feb 24 15:53:39 2017 -0800
Committer: Robert Bradshaw 
Committed: Mon Feb 27 17:46:00 2017 -0800

--
 sdks/common/runner-api/src/main/proto/beam_runner_api.proto | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a0dcd3f9/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
--
diff --git a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto 
b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
index 989e4bb..ad6d0cb 100644
--- a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
+++ b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
@@ -591,7 +591,7 @@ message SideInput {
   // The only access pattern intended for Beam, because of its superior
   // performance possibilities, is "urn:beam:sideinput:multimap" (or some such
   // URN)
-  string access_pattern = 3;
+  UrnWithParameter access_pattern = 3;
 
   // (Required) The pipeline-scoped id for the FunctionSpec of the UDF that
   // adapts a particular access_pattern to a user-facing view type.

[3/3] beam git commit: Closes #2106

[robertwb]

Closes #2106


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/16736a6b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/16736a6b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/16736a6b

Branch: refs/heads/master
Commit: 16736a6b612ba38b19694ae2a23dea7f5fab1563
Parents: c8dfb85 c04e8ab
Author: Robert Bradshaw 
Authored: Mon Feb 27 17:46:02 2017 -0800
Committer: Robert Bradshaw 
Committed: Mon Feb 27 17:46:02 2017 -0800

--
 sdks/common/runner-api/src/main/proto/beam_runner_api.proto | 8 +++-
 1 file changed, 3 insertions(+), 5 deletions(-)
--

[2/3] beam git commit: Clarify side input name and remove redundant pcollection reference.

[robertwb]

Clarify side input name and remove redundant pcollection reference.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c04e8abd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c04e8abd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c04e8abd

Branch: refs/heads/master
Commit: c04e8abd41575a4e819c3f18dabff921338f55d8
Parents: a0dcd3f
Author: Robert Bradshaw 
Authored: Fri Feb 24 16:31:05 2017 -0800
Committer: Robert Bradshaw 
Committed: Mon Feb 27 17:46:01 2017 -0800

--
 sdks/common/runner-api/src/main/proto/beam_runner_api.proto | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c04e8abd/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
--
diff --git a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto 
b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
index ad6d0cb..258c278 100644
--- a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
+++ b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
@@ -576,12 +576,10 @@ message TimestampTransform {
 // A specification for how to "side input" a PCollection.
 message SideInput {
 
-  // (Required) A local name for this side input, as embedded in a serialized 
UDF.
+  // (Required) A local name for this side input, as interpreted by its
+  // parent PTransform (and/or its PTransform's UDFs).
   string name = 1;
 
-  // (Required) The pipeline-scoped unique id of the PCollection to be side 
input.
-  string pcollection_id = 2;
-
   // (Required) URN of the access pattern required by the `view_fn` to present
   // the desired SDK-specific interface to a UDF.
   //

[jira] [Commented] (BEAM-1572) Add per-stage matching of scope in metrics for the DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887030#comment-15887030
 ] 

ASF GitHub Bot commented on BEAM-1572:
--

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2122

[BEAM-1572] Adding per-stage level matching of metrics filters.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Also fixed a bug in Python metrics.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam metrics-filter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2122.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2122


commit bc2a99354406981f75e60b86fd9776d572cb0159
Author: Pablo 
Date:   2017-02-28T01:28:26Z

Adding per-stage level matching of metrics filters.




> Add per-stage matching of scope in metrics for the DirectRunner
> ---
>
> Key: BEAM-1572
> URL: https://issues.apache.org/jira/browse/BEAM-1572
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>
> e.g. Metrics with scope "Top/Outer/Inner" should be matched by queries with 
> "Top/Outer" scope.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2122: [BEAM-1572] Adding per-stage level matching of metr...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2122

[BEAM-1572] Adding per-stage level matching of metrics filters.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Also fixed a bug in Python metrics.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam metrics-filter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2122.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2122


commit bc2a99354406981f75e60b86fd9776d572cb0159
Author: Pablo 
Date:   2017-02-28T01:28:26Z

Adding per-stage level matching of metrics filters.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1538) Add a fast version of BufferedElementCountingOutputStream

[Vikas Kedigehalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Kedigehalli resolved BEAM-1538.
-
   Resolution: Fixed
Fix Version/s: Not applicable

No longer need this as we embed this functionality into the IterableCoder 
directly. 

> Add a fast version of BufferedElementCountingOutputStream
> -
>
> Key: BEAM-1538
> URL: https://issues.apache.org/jira/browse/BEAM-1538
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Vikas Kedigehalli
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: Not applicable
>
>
> We are currently using python version of the stream which is slow. We need to 
> implement a Cython version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1556) Spark executors need to register IO factories

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886971#comment-15886971
 ] 

Davor Bonaci edited comment on BEAM-1556 at 2/28/17 1:05 AM:
-

It certainly is an issue in the other runners as well.

I wouldn't do in the context of a {{FileBasedSource}}. Users should be able to 
call the {{FileSystem}} API from, say, {{@ProcessElement}} method of a 
{{DoFn}}. So, I think the registration should be done before any "user code" is 
invoked.

Doing it in worker startup might not be ideal -- the constructor takes 
{{PipelineOptions}} as an argument. Since jobs could have different options, it 
probably needs to happen on a per-task basis, likely at the point the worker 
receives the task from the master and deserializes {{PipelineOptions}}.


was (Author: davor):
It certainly is an issue in other runners as well.

I wouldn't do in the context of a {{FileBasedSource}}. Users should be able to 
call the {{FileSystem}} API from, say, {{@ProcessElement}} method of a 
{{DoFn}}. So, I think the registration should be done before any "user code" is 
invoked.

Doing it in worker startup might not be ideal -- the constructor takes 
{{PipelineOptions}} as an argument. Since jobs could have different options, it 
probably needs to happen on a per-task basis, likely at the point the worker 
receives the task from the master and deserializes {{PipelineOptions}}.

> Spark executors need to register IO factories
> -
>
> Key: BEAM-1556
> URL: https://issues.apache.org/jira/browse/BEAM-1556
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Frances Perry
>Assignee: Jean-Baptiste Onofré
>
> The Spark executors need to call IOChannelUtils.registerIOFactories(options) 
> in order to support GCS file and make the default WordCount example work.
> Context in this thread: 
> https://lists.apache.org/thread.html/469a139c9eb07e64e514cdea42ab8000678ab743794a090c365205d7@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1556) Spark executors need to register IO factories

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886971#comment-15886971
 ] 

Davor Bonaci commented on BEAM-1556:


It certainly is an issue in other runners as well.

I wouldn't do in the context of a {{FileBasedSource}}. Users should be able to 
call the {{FileSystem}} API from, say, {{@ProcessElement}} method of a 
{{DoFn}}. So, I think the registration should be done before any "user code" is 
invoked.

Doing it in worker startup might not be ideal -- the constructor takes 
{{PipelineOptions}} as an argument. Since jobs could have different options, it 
probably needs to happen on a per-task basis, likely at the point the worker 
receives the task from the master and deserializes {{PipelineOptions}}.

> Spark executors need to register IO factories
> -
>
> Key: BEAM-1556
> URL: https://issues.apache.org/jira/browse/BEAM-1556
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Frances Perry
>Assignee: Jean-Baptiste Onofré
>
> The Spark executors need to call IOChannelUtils.registerIOFactories(options) 
> in order to support GCS file and make the default WordCount example work.
> Context in this thread: 
> https://lists.apache.org/thread.html/469a139c9eb07e64e514cdea42ab8000678ab743794a090c365205d7@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Dataflow #2422

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1567) hashStream should be closed in PackageUtil#createPackageAttributes()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886905#comment-15886905
 ] 

ASF GitHub Bot commented on BEAM-1567:
--

GitHub user tedyu opened a pull request:

https://github.com/apache/beam/pull/2121

BEAM-1567 hashStream should be closed in 
PackageUtil#createPackageAttributes()

Here is related code:
```
  OutputStream hashStream = Funnels.asOutputStream(hasher);
```
hashStream should be closed upon return from the method

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tedyu/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2121






> hashStream should be closed in PackageUtil#createPackageAttributes()
> 
>
> Key: BEAM-1567
> URL: https://issues.apache.org/jira/browse/BEAM-1567
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Ted Yu
>Priority: Minor
>  Labels: newbie, starter
>
> Here is related code:
> {code}
>   OutputStream hashStream = Funnels.asOutputStream(hasher);
> {code}
> hashStream should be closed upon return from the method



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2121: BEAM-1567 hashStream should be closed in PackageUti...

[tedyu]

GitHub user tedyu opened a pull request:

https://github.com/apache/beam/pull/2121

BEAM-1567 hashStream should be closed in 
PackageUtil#createPackageAttributes()

Here is related code:
```
  OutputStream hashStream = Funnels.asOutputStream(hasher);
```
hashStream should be closed upon return from the method

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tedyu/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2121






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1572) Add per-stage matching of scope in metrics for the DirectRunner

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1572:
--

Assignee: Pablo Estrada  (was: Davor Bonaci)

> Add per-stage matching of scope in metrics for the DirectRunner
> ---
>
> Key: BEAM-1572
> URL: https://issues.apache.org/jira/browse/BEAM-1572
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>
> e.g. Metrics with scope "Top/Outer/Inner" should be matched by queries with 
> "Top/Outer" scope.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1572) Add per-stage matching of scope in metrics for the DirectRunner

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886885#comment-15886885
 ] 

Davor Bonaci commented on BEAM-1572:


Thanks [~pabloem].

> Add per-stage matching of scope in metrics for the DirectRunner
> ---
>
> Key: BEAM-1572
> URL: https://issues.apache.org/jira/browse/BEAM-1572
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>
> e.g. Metrics with scope "Top/Outer/Inner" should be matched by queries with 
> "Top/Outer" scope.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1572) Add per-stage matching of scope in metrics for the DirectRunner

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-1572:
---

 Summary: Add per-stage matching of scope in metrics for the 
DirectRunner
 Key: BEAM-1572
 URL: https://issues.apache.org/jira/browse/BEAM-1572
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Pablo Estrada
Assignee: Davor Bonaci


e.g. Metrics with scope "Top/Outer/Inner" should be matched by queries with 
"Top/Outer" scope.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1571) Flatten on a single input PCollection should have a test associated with it

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886842#comment-15886842
 ] 

ASF GitHub Bot commented on BEAM-1571:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2120

[BEAM-1571] Add a test for FlattenPCollectionList on a singleton list

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
This test validates that the output PCollection of a Flatten with a
single input PCollection contains the contents of the input PCollection.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam flatten_singleton_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2120.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2120






> Flatten on a single input PCollection should have a test associated with it
> ---
>
> Key: BEAM-1571
> URL: https://issues.apache.org/jira/browse/BEAM-1571
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The expected behavior is that an output PCollection containing all of the 
> elements of the input PCollection is produced.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1572) Add per-stage matching of scope in metrics for the DirectRunner

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886850#comment-15886850
 ] 

Pablo Estrada commented on BEAM-1572:
-

I'm working on this for both Python and Java.

> Add per-stage matching of scope in metrics for the DirectRunner
> ---
>
> Key: BEAM-1572
> URL: https://issues.apache.org/jira/browse/BEAM-1572
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Pablo Estrada
>Assignee: Davor Bonaci
>
> e.g. Metrics with scope "Top/Outer/Inner" should be matched by queries with 
> "Top/Outer" scope.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1571) Flatten on a single input PCollection should have a test associated with it

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1571:
-

 Summary: Flatten on a single input PCollection should have a test 
associated with it
 Key: BEAM-1571
 URL: https://issues.apache.org/jira/browse/BEAM-1571
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh


The expected behavior is that an output PCollection containing all of the 
elements of the input PCollection is produced.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2120: [BEAM-1571] Add a test for FlattenPCollectionList o...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2120

[BEAM-1571] Add a test for FlattenPCollectionList on a singleton list

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
This test validates that the output PCollection of a Flatten with a
single input PCollection contains the contents of the input PCollection.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam flatten_singleton_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2120.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2120






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1570) broken link on website (ostatic.com)

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1570:
--

Assignee: Sourabh Bajaj  (was: Davor Bonaci)

> broken link on website (ostatic.com)
> 
>
> Key: BEAM-1570
> URL: https://issues.apache.org/jira/browse/BEAM-1570
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Stephen Sisk
>Assignee: Sourabh Bajaj
>
> I see the following error when running the website test suite locally and on 
> jenkins: 
>   *  External link 
> http://ostatic.com/blog/apache-beam-unifies-batch-and-streaming-data-processing
>  failed: got a time out (response code 0)
> It looks like ostatic's website is down. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #166: [BEAM-1552] Add contrib guide comments about us...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/166


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1567) hashStream should be closed in PackageUtil#createPackageAttributes()

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1567:
--
Labels: newbie starter  (was: )

> hashStream should be closed in PackageUtil#createPackageAttributes()
> 
>
> Key: BEAM-1567
> URL: https://issues.apache.org/jira/browse/BEAM-1567
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Ted Yu
>Priority: Minor
>  Labels: newbie, starter
>
> Here is related code:
> {code}
>   OutputStream hashStream = Funnels.asOutputStream(hasher);
> {code}
> hashStream should be closed upon return from the method



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/3] beam-site git commit: This closes #166

[davor]

This closes #166


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/70682771
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/70682771
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/70682771

Branch: refs/heads/asf-site
Commit: 70682771d6bb8ca590e4b04b9943c3b50348496a
Parents: 2bb1902 582a7f4
Author: Davor Bonaci 
Authored: Mon Feb 27 16:18:18 2017 -0800
Committer: Davor Bonaci 
Committed: Mon Feb 27 16:18:18 2017 -0800

--
 content/contribute/contribution-guide/index.html | 8 +---
 src/contribute/contribution-guide.md | 8 +---
 2 files changed, 10 insertions(+), 6 deletions(-)
--

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/582a7f45
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/582a7f45
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/582a7f45

Branch: refs/heads/asf-site
Commit: 582a7f452eed30d815cfddc6af331812e63ccf61
Parents: 2f35897
Author: Davor Bonaci 
Authored: Mon Feb 27 16:18:18 2017 -0800
Committer: Davor Bonaci 
Committed: Mon Feb 27 16:18:18 2017 -0800

--
 content/contribute/contribution-guide/index.html | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/582a7f45/content/contribute/contribution-guide/index.html
--
diff --git a/content/contribute/contribution-guide/index.html 
b/content/contribute/contribution-guide/index.html
index 580b7e6..5f8ec98 100644
--- a/content/contribute/contribution-guide/index.html
+++ b/content/contribute/contribution-guide/index.html
@@ -419,9 +419,9 @@ $ git checkout -b  origin/master
 
 Remember to always use --rebase 
parameter to avoid extraneous merge commits.
 
-To push your local, committed changes to your (forked) repository on 
GitHub, run:
+Then you can push your local, committed changes to your (forked) repository 
on GitHub. Since rebase may change that branch’s history, you may need to 
force push. You’ll run:
 
-$ git push 
 
+$ git push 
  --force
 
 
 
@@ -438,12 +438,14 @@ $ git checkout -b  origin/master
 Once the initial code is complete and the tests pass, it’s time to start 
the code review process. We review and discuss all code, no matter who authors 
it. It’s a great way to build community, since you can learn from other 
developers, and they become familiar with your contribution. It also builds a 
strong project by encouraging a high quality bar and keeping code consistent 
throughout the project.
 
 Create a pull request
-Organize your commits to make your reviewer’s job easier. Use the 
following command to re-order, squash, edit, or change description of 
individual commits.
+Organize your commits to make your reviewer’s job easier. Reviewers 
normally prefer multiple small pull requests, instead of a single large pull 
request. Within a pull request, a relatively small number of commits that break 
the problem into logical steps is preferred. For most pull requests, you’ll 
squash your changes down to 1 commit. You can use the following command to 
re-order, squash, edit, or change description of individual commits.
 
 $ git rebase -i 
origin/master
 
 
 
+You’ll then push to your branch on GitHub. Note: when updating your 
commit after pull request feedback and use squash to get back to one commit, 
you will need to do a force submit to the branch on your repo.
+
 Navigate to the https://github.com/apache/beam";>Beam GitHub 
mirror to create a pull request. The title of the pull request should be 
strictly in the following format:
 
 [BEAM-]