date:20170305

[jira] [Assigned] (BEAM-1556) Spark executors need to register IO factories

2017-03-05 Thread Amit Sela (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela reassigned BEAM-1556:
---

Assignee: Amit Sela  (was: Jean-Baptiste Onofré)

> Spark executors need to register IO factories
> -
>
> Key: BEAM-1556
> URL: https://issues.apache.org/jira/browse/BEAM-1556
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Frances Perry
>Assignee: Amit Sela
>
> The Spark executors need to call IOChannelUtils.registerIOFactories(options) 
> in order to support GCS file and make the default WordCount example work.
> Context in this thread: 
> https://lists.apache.org/thread.html/469a139c9eb07e64e514cdea42ab8000678ab743794a090c365205d7@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

2017-03-05 Thread Guy Molinari (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896785#comment-15896785
 ] 

Guy Molinari commented on BEAM-1542:


I'm wondering if you guys could help me out.   I'm getting a weird generics 
related compilation error and I'm just plain stumped.   I created a pull 
request for the work so that you could get access to my work.

Thanks in advance.

> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Guy Molinari
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

2017-03-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896782#comment-15896782
 ] 

ASF GitHub Bot commented on BEAM-1542:
--

GitHub user NextDevBoard opened a pull request:

https://github.com/apache/beam/pull/2166

BEAM-1542 : Refactored SpannerIO class.   Getting weird compilation error.  
 Need help.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NextDevBoard/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2166.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2166


commit 86ad4e2d633d5306fe326a5af27e5ac176695d41
Author: MOLIG004 
Date:   2017-03-03T21:19:36Z

Added very raw basic version of SpannerIO.   No unit tests.   Must skip 
checkstyle to compile

commit 564e71405ece4d016a16a3501d26b4346aa64602
Author: MOLIG004 
Date:   2017-03-06T06:11:36Z

Simplified SpannerIO getting compilation error on OutputT




> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Guy Molinari
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2166: BEAM-1542 : Refactored SpannerIO class. Getting wei...

2017-03-05 Thread NextDevBoard

GitHub user NextDevBoard opened a pull request:

https://github.com/apache/beam/pull/2166

BEAM-1542 : Refactored SpannerIO class.   Getting weird compilation error.  
 Need help.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NextDevBoard/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2166.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2166


commit 86ad4e2d633d5306fe326a5af27e5ac176695d41
Author: MOLIG004 
Date:   2017-03-03T21:19:36Z

Added very raw basic version of SpannerIO.   No unit tests.   Must skip 
checkstyle to compile

commit 564e71405ece4d016a16a3501d26b4346aa64602
Author: MOLIG004 
Date:   2017-03-06T06:11:36Z

Simplified SpannerIO getting compilation error on OutputT




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-147) Introduce an easy API for pipeline metrics

2017-03-05 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-147:
---
Issue Type: New Feature  (was: Bug)

> Introduce an easy API for pipeline metrics
> --
>
> Key: BEAM-147
> URL: https://issues.apache.org/jira/browse/BEAM-147
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Robert Bradshaw
>Assignee: Ben Chambers
>
> The existing Aggregators are confusing both because of their name and because 
> they serve multiple purposes.
> Previous discussions around Aggregators/metrics/etc:
> [Discussion on user 
> list|http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser]
>  
> [Discussion on dev 
> list|http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser]
>  
> Exact name still being bikeshedded.
> [Design document|http://s.apache.org/beam-metrics-api]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1616) Gauge Metric type

2017-03-05 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1616:

Issue Type: New Feature  (was: Bug)

> Gauge Metric type
> -
>
> Key: BEAM-1616
> URL: https://issues.apache.org/jira/browse/BEAM-1616
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Aviem Zur
>Assignee: Ben Chambers
>
> Add support for Gauge metric type to the SDK.
> This will serve to get the last value reported.
> Interface should be along the lines of:
> {code}
> void set(long value);
> {code}
> Compare to 
> http://metrics.dropwizard.io/3.1.0/apidocs/com/codahale/metrics/Gauge.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Spark #1140

2017-03-05 Thread Apache Jenkins Server

See

[4/4] beam git commit: This closes #2165

2017-03-05 Thread davor

This closes #2165


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2f96bc32
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2f96bc32
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2f96bc32

Branch: refs/heads/master
Commit: 2f96bc3260b98a97d532f8451d8fa63809f1ae51
Parents: 9541f8f 0360b1b
Author: Davor Bonaci 
Authored: Sun Mar 5 18:01:46 2017 -0800
Committer: Davor Bonaci 
Committed: Sun Mar 5 18:01:46 2017 -0800

--
 .../main/resources/archetype-resources/pom.xml  | 46 ++
 .../main/resources/archetype-resources/pom.xml  | 64 ++--
 2 files changed, 78 insertions(+), 32 deletions(-)
--

[3/4] beam git commit: Java 8 examples: add SNAPSHOT repository to the pom.xml file

2017-03-05 Thread davor

Java 8 examples: add SNAPSHOT repository to the pom.xml file

This enables users of the archetype to reference nightly builds if they so 
choose.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/630df6f0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/630df6f0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/630df6f0

Branch: refs/heads/master
Commit: 630df6f0e328d52d0892a56ae8493946b78dd6b9
Parents: 9541f8f
Author: Davor Bonaci 
Authored: Sun Mar 5 15:21:22 2017 -0800
Committer: Davor Bonaci 
Committed: Sun Mar 5 18:01:32 2017 -0800

--
 .../src/main/resources/archetype-resources/pom.xml| 14 ++
 1 file changed, 14 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/630df6f0/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
index e025f9e..f30a35f 100644
--- 
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
@@ -30,6 +30,20 @@
 0.6.0-SNAPSHOT
   
 
+  
+
+  apache.snapshots
+  Apache Development Snapshot Repository
+  https://repository.apache.org/content/repositories/snapshots/
+  
+false
+  
+  
+true
+  
+
+  
+

[GitHub] beam pull request #2165: Java 8 examples fixup: add SNAPSHOT repository & ad...

2017-03-05 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2165


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/4] beam git commit: Java examples: move shade plugin into default lifecycle

2017-03-05 Thread davor

Repository: beam
Updated Branches:
  refs/heads/master 9541f8fcc -> 2f96bc326


Java examples: move shade plugin into default lifecycle

This helps users running the examples on runner that require the fat/bundled 
jar.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0360b1b3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0360b1b3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0360b1b3

Branch: refs/heads/master
Commit: 0360b1b3be6190d36e1af1d2d077251ef60106d8
Parents: 90c8094
Author: Davor Bonaci 
Authored: Sun Mar 5 16:43:48 2017 -0800
Committer: Davor Bonaci 
Committed: Sun Mar 5 18:01:32 2017 -0800

--
 .../main/resources/archetype-resources/pom.xml  | 64 ++--
 1 file changed, 32 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0360b1b3/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 6a12324..0ed23fd 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -73,6 +73,38 @@
   
 
   
+
+  
+  
+org.apache.maven.plugins
+maven-shade-plugin
+2.4.1
+
+  
+package
+
+  shade
+
+
+  
${project.artifactId}-bundled-${project.version}
+  
+
+  *:*
+  
+META-INF/LICENSE
+META-INF/*.SF
+META-INF/*.DSA
+META-INF/*.RSA
+  
+
+  
+
+  
+
+  
 
 
 
@@ -85,38 +117,6 @@
 false
   
 
-
-
-
-  org.apache.maven.plugins
-  maven-shade-plugin
-  2.4.1
-  
-
-  package
-  
-shade
-  
-  
-
${project.artifactId}-bundled-${project.version}
-
-  
-*:*
-
-  META-INF/LICENSE
-  META-INF/*.SF
-  META-INF/*.DSA
-  META-INF/*.RSA
-
-  
-
-  
-
-  
-

[2/4] beam git commit: Java 8 examples: add shade configuration for producing bundled/fat jar

2017-03-05 Thread davor

Java 8 examples: add shade configuration for producing bundled/fat jar

This helps users who may need the fat jar to submit it to the cluster.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/90c8094d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/90c8094d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/90c8094d

Branch: refs/heads/master
Commit: 90c8094d883983f0006065169fd3ef230963e083
Parents: 630df6f
Author: Davor Bonaci 
Authored: Sun Mar 5 15:23:07 2017 -0800
Committer: Davor Bonaci 
Committed: Sun Mar 5 18:01:32 2017 -0800

--
 .../main/resources/archetype-resources/pom.xml  | 32 
 1 file changed, 32 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/90c8094d/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
index f30a35f..3907fd5 100644
--- 
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
@@ -73,6 +73,38 @@
   
 
   
+
+  
+  
+org.apache.maven.plugins
+maven-shade-plugin
+2.4.1
+
+  
+package
+
+  shade
+
+
+  
${project.artifactId}-bundled-${project.version}
+  
+
+  *:*
+  
+META-INF/LICENSE
+META-INF/*.SF
+META-INF/*.DSA
+META-INF/*.RSA
+  
+
+  
+
+  
+
+

[jira] [Created] (BEAM-1628) Flink runner: logic around --flinkMaster is error-prone

2017-03-05 Thread Davor Bonaci (JIRA)

Davor Bonaci created BEAM-1628:
--

 Summary: Flink runner: logic around --flinkMaster is error-prone
 Key: BEAM-1628
 URL: https://issues.apache.org/jira/browse/BEAM-1628
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Reporter: Davor Bonaci
Assignee: Aljoscha Krettek
Priority: Minor


The logic for handling {{--flinkMaster}} seems not particularly user-friendly.

https://github.com/apache/beam/blob/fbcde4cdc7d68de8734bf540c079b2747631a854/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java#L132

{code}
if (masterUrl.equals("[local]")) {
} else if (masterUrl.equals("[collection]")) {
} else if (masterUrl.equals("[auto]")) {
} else if (masterUrl.matches(".*:\\d*")) {
} else {
  // use auto.
}
{code}

The options are constructed with "auto" set as default.

I think we should do the following:
* I assume there's a default port for the Flink master. We should default to it.
* We should treat a string without a colon as a host name. (Not default to 
local execution.)

This is super easy fix, hopefully someone can pick it up quickly ;-)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1627) Composite/DisplayData structure changed

2017-03-05 Thread Frances Perry (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frances Perry updated BEAM-1627:

Attachment: ParseGame-0.5.png
ParseGame-snapshot-extraComposite.png
FixedWindows-0.5.png
FixedWindows-snapshot-extraComposite-noDisplayData.png

> Composite/DisplayData structure changed
> ---
>
> Key: BEAM-1627
> URL: https://issues.apache.org/jira/browse/BEAM-1627
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Frances Perry
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 0.6.0
>
> Attachments: FixedWindows-0.5.png, 
> FixedWindows-snapshot-extraComposite-noDisplayData.png, ParseGame-0.5.png, 
> ParseGame-snapshot-extraComposite.png
>
>
> When running at head, pipeline composite structure has changed. My guess is 
> this is related to pull/2145. 
> (1) Steps that used to be leaf notes are now expandable composites with a 
> ParMultiDo inside them.
> (2) For some (but not all) display data appears to be lost
> This can be seen pretty clearly in the Dataflow monitoring UI. Attached 
> screenshots showing
> -- ParseGameEvent transform leaks an extra level of composite.
> -- FixedWindows transform leaks an extra composite and loses display data.
> [~tgroh] can you triage?
> [~altay] FYI potential 0.6 release blocker



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1627) Composite/DisplayData structure changed

2017-03-05 Thread Frances Perry (JIRA)

Frances Perry created BEAM-1627:
---

 Summary: Composite/DisplayData structure changed
 Key: BEAM-1627
 URL: https://issues.apache.org/jira/browse/BEAM-1627
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Frances Perry
Assignee: Thomas Groh
Priority: Blocker
 Fix For: 0.6.0


When running at head, pipeline composite structure has changed. My guess is 
this is related to pull/2145. 

(1) Steps that used to be leaf notes are now expandable composites with a 
ParMultiDo inside them.

(2) For some (but not all) display data appears to be lost

This can be seen pretty clearly in the Dataflow monitoring UI. Attached 
screenshots showing
-- ParseGameEvent transform leaks an extra level of composite.
-- FixedWindows transform leaks an extra composite and loses display data.

[~tgroh] can you triage?
[~altay] FYI potential 0.6 release blocker



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #2819

2017-03-05 Thread Apache Jenkins Server

See

[GitHub] beam pull request #2164: Update Guava version from 19.0 to 20.0 in the examp...

2017-03-05 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2164


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Update Guava version from 19.0 to 20.0 in example projects

2017-03-05 Thread davor

Repository: beam
Updated Branches:
  refs/heads/master 59451bca6 -> 9541f8fcc


Update Guava version from 19.0 to 20.0 in example projects

The Guava version was updated in the main project on January 19.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e11d6d27
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e11d6d27
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e11d6d27

Branch: refs/heads/master
Commit: e11d6d27174f866371a253449908c9a6033114d5
Parents: 59451bc
Author: Davor Bonaci 
Authored: Sun Mar 5 15:10:49 2017 -0800
Committer: Davor Bonaci 
Committed: Sun Mar 5 15:10:49 2017 -0800

--
 .../examples-java8/src/main/resources/archetype-resources/pom.xml  | 2 +-
 .../examples/src/main/resources/archetype-resources/pom.xml| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e11d6d27/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
index 55211ed..e025f9e 100644
--- 
a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
@@ -270,7 +270,7 @@
 
   com.google.guava
   guava
-  19.0
+  20.0
 
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/e11d6d27/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 654973c..6a12324 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -316,7 +316,7 @@
 
   com.google.guava
   guava
-  19.0
+  20.0

[2/2] beam git commit: This closes #2164

2017-03-05 Thread davor

This closes #2164


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9541f8fc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9541f8fc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9541f8fc

Branch: refs/heads/master
Commit: 9541f8fcc505830a24decb8a69b82532e9a8f69c
Parents: 59451bc e11d6d2
Author: Davor Bonaci 
Authored: Sun Mar 5 15:57:57 2017 -0800
Committer: Davor Bonaci 
Committed: Sun Mar 5 15:57:57 2017 -0800

--
 .../examples-java8/src/main/resources/archetype-resources/pom.xml  | 2 +-
 .../examples/src/main/resources/archetype-resources/pom.xml| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--

[GitHub] beam pull request #2165: Java 8 examples fixup: add SNAPSHOT repository & ad...

2017-03-05 Thread davorbonaci

GitHub user davorbonaci opened a pull request:

https://github.com/apache/beam/pull/2165

Java 8 examples fixup: add SNAPSHOT repository & add bundling configuration

R: @francesperry 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam examples

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2165.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2165


commit 799fca69a8868aaafe9a789c3d8296028e53c256
Author: Davor Bonaci 
Date:   2017-03-05T23:21:22Z

Java 8 examples: add SNAPSHOT repository to the pom.xml file

This enables users of the archetype to reference nightly builds if they so 
choose.

commit 97adef2e7b749623624bd8a94a0eed45d5d6396f
Author: Davor Bonaci 
Date:   2017-03-05T23:23:07Z

Java 8 examples: add shade configuration for producing bundled/fat jar

This helps users who may need the fat jar to submit it to the cluster.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2164: Update Guava version from 19.0 to 20.0 in the examp...

2017-03-05 Thread davorbonaci

GitHub user davorbonaci opened a pull request:

https://github.com/apache/beam/pull/2164

Update Guava version from 19.0 to 20.0 in the example projects

The Guava version was updated in the main project on January 19.

Before this change, generated examples would downgrade Guava back to 19.0, 
possibly causing execution issues if the underlying code actually needs 
functionality added in Guava 20. In fact, last week Flink runner added such 
functionality, breaking Quickstart and WordCount on the Flink runner.

R: @francesperry 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam guava-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2164.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2164


commit e11d6d27174f866371a253449908c9a6033114d5
Author: Davor Bonaci 
Date:   2017-03-05T23:10:49Z

Update Guava version from 19.0 to 20.0 in example projects

The Guava version was updated in the main project on January 19.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-68) Support for limiting parallelism of a step

2017-03-05 Thread Xu Mingmin (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896589#comment-15896589
 ] 

Xu Mingmin commented on BEAM-68:


Notice this task when tuning a Beam job on Flink.
Would like to bring another perspective, that users want to have more control 
on the parallelism of a data pipeline, to allocate more resource for the busy 
steps, and less for the costless. A fixed parallelism could have performance 
bottleneck, several use cases like:
1. source from a Kafka topic, the parallelism could not be larger then topic 
partition number; similar for other splittable IOs?
2. fewer grouped keys than parallelism;
3. process on a small portion from large input;
4. +1 for case2, to address quota limitation on external dependencies;
  

> Support for limiting parallelism of a step
> --
>
> Key: BEAM-68
> URL: https://issues.apache.org/jira/browse/BEAM-68
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>
> Users may want to limit the parallelism of a step. Two classic uses cases are:
> - User wants to produce at most k files, so sets 
> TextIO.Write.withNumShards(k).
> - External API only supports k QPS, so user sets a limit of k/(expected 
> QPS/step) on the ParDo that makes the API call.
> Unfortunately, there is no way to do this effectively within the Beam model. 
> A GroupByKey with exactly k keys will guarantee that only k elements are 
> produced, but runners are free to break fusion in ways that each element may 
> be processed in parallel later.
> To implement this functionaltiy, I believe we need to add this support to the 
> Beam Model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1626) Remove caching of read MapWithStateDStream.

2017-03-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896563#comment-15896563
 ] 

ASF GitHub Bot commented on BEAM-1626:
--

GitHub user amitsela opened a pull request:

https://github.com/apache/beam/pull/2163

[BEAM-1626] Remove cache of MapWithStateDStream on read.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/beam BEAM-1626

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2163.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2163


commit 08cae491497f1a81c89ad662bb6442bc43d8d6d7
Author: Sela 
Date:   2017-03-05T21:37:08Z

Remove cache of MapWithStateDStream on read.




> Remove caching of read MapWithStateDStream.
> ---
>
> Key: BEAM-1626
> URL: https://issues.apache.org/jira/browse/BEAM-1626
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> There's no real need for it since checkpointing caches as well, and from my 
> experiments I think it also has something to do with some of the flakes in 
> streaming tests.
> Anyway, I don't see a good reason to call {{cache()}} there, so let's remove 
> it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2163: [BEAM-1626] Remove cache of MapWithStateDStream on ...

2017-03-05 Thread amitsela

GitHub user amitsela opened a pull request:

https://github.com/apache/beam/pull/2163

[BEAM-1626] Remove cache of MapWithStateDStream on read.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/beam BEAM-1626

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2163.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2163


commit 08cae491497f1a81c89ad662bb6442bc43d8d6d7
Author: Sela 
Date:   2017-03-05T21:37:08Z

Remove cache of MapWithStateDStream on read.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1626) Remove caching of read MapWithStateDStream.

2017-03-05 Thread Amit Sela (JIRA)

Amit Sela created BEAM-1626:
---

 Summary: Remove caching of read MapWithStateDStream.
 Key: BEAM-1626
 URL: https://issues.apache.org/jira/browse/BEAM-1626
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Amit Sela
Assignee: Amit Sela


There's no real need for it since checkpointing caches as well, and from my 
experiments I think it also has something to do with some of the flakes in 
streaming tests.
Anyway, I don't see a good reason to call {{cache()}} there, so let's remove it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1625) BoundedDataset action() does not materialize RDD

2017-03-05 Thread Amit Sela (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1625.
-
   Resolution: Fixed
Fix Version/s: 0.6.0

> BoundedDataset action() does not materialize RDD
> 
>
> Key: BEAM-1625
> URL: https://issues.apache.org/jira/browse/BEAM-1625
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Aviem Zur
>Assignee: Aviem Zur
> Fix For: 0.6.0
>
>
> {{BoundedDataset#action}} does not materialize RDD since it uses 
> {{foreachPartition}} which is not an action, instead use {{foreach}} which is.
> See: http://spark.apache.org/docs/latest/programming-guide.html#actions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1397) Introduce IO metrics

2017-03-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896505#comment-15896505
 ] 

ASF GitHub Bot commented on BEAM-1397:
--

Github user aviemzur closed the pull request at:

https://github.com/apache/beam/pull/2082


> Introduce IO metrics
> 
>
> Key: BEAM-1397
> URL: https://issues.apache.org/jira/browse/BEAM-1397
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Introduce the usage of metrics API in IOs.
> POC using {{CountingInput}}:
> * Add metrics to {{CountingInput}}
> * {{RunnableOnService}} test which creates a pipeline which asserts these 
> metrics.
> * Close any gaps in Direct runner and Spark runner to support these metrics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1397) Introduce IO metrics

2017-03-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896509#comment-15896509
 ] 

ASF GitHub Bot commented on BEAM-1397:
--

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam/pull/2162

[BEAM-1397] Introduce IO metrics

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam introduce-io-metrics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2162.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2162


commit 62c699d84d37ddbfbd40fe4e970890b8cdb9f0a9
Author: Aviem Zur 
Date:   2017-03-05T19:37:05Z

[BEAM-1397] Introduce IO metrics




> Introduce IO metrics
> 
>
> Key: BEAM-1397
> URL: https://issues.apache.org/jira/browse/BEAM-1397
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Introduce the usage of metrics API in IOs.
> POC using {{CountingInput}}:
> * Add metrics to {{CountingInput}}
> * {{RunnableOnService}} test which creates a pipeline which asserts these 
> metrics.
> * Close any gaps in Direct runner and Spark runner to support these metrics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1397) Introduce IO metrics

2017-03-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896506#comment-15896506
 ] 

ASF GitHub Bot commented on BEAM-1397:
--

GitHub user aviemzur reopened a pull request:

https://github.com/apache/beam/pull/2082

[BEAM-1397] [BEAM-1398] Introduce IO metrics. Add KafkaIO metrics.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam io-metrics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2082.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2082


commit 31d94d4eecf492357c41424f65e1037fc3976d09
Author: Aviem Zur 
Date:   2017-02-22T14:18:13Z

[BEAM-1397] Introduce IO metrics

commit 8192724e65e65092117fee0a78408b476adf0245
Author: Aviem Zur 
Date:   2017-02-22T21:26:45Z

[BEAM-1398] KafkaIO metrics

commit 62d0ac450ff4631ddfd057a5caa785dae305065b
Author: Aviem Zur 
Date:   2017-02-23T04:56:43Z

Test Spark runner streaming IO metrics




> Introduce IO metrics
> 
>
> Key: BEAM-1397
> URL: https://issues.apache.org/jira/browse/BEAM-1397
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Introduce the usage of metrics API in IOs.
> POC using {{CountingInput}}:
> * Add metrics to {{CountingInput}}
> * {{RunnableOnService}} test which creates a pipeline which asserts these 
> metrics.
> * Close any gaps in Direct runner and Spark runner to support these metrics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2082: [BEAM-1397] [BEAM-1398] Introduce IO metrics. Add K...

2017-03-05 Thread aviemzur

Github user aviemzur closed the pull request at:

https://github.com/apache/beam/pull/2082


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2082: [BEAM-1397] [BEAM-1398] Introduce IO metrics. Add K...

2017-03-05 Thread aviemzur

Github user aviemzur closed the pull request at:

https://github.com/apache/beam/pull/2082


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2082: [BEAM-1397] [BEAM-1398] Introduce IO metrics. Add K...

2017-03-05 Thread aviemzur

GitHub user aviemzur reopened a pull request:

https://github.com/apache/beam/pull/2082

[BEAM-1397] [BEAM-1398] Introduce IO metrics. Add KafkaIO metrics.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam io-metrics

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2082.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2082


commit 31d94d4eecf492357c41424f65e1037fc3976d09
Author: Aviem Zur 
Date:   2017-02-22T14:18:13Z

[BEAM-1397] Introduce IO metrics

commit 8192724e65e65092117fee0a78408b476adf0245
Author: Aviem Zur 
Date:   2017-02-22T21:26:45Z

[BEAM-1398] KafkaIO metrics

commit 62d0ac450ff4631ddfd057a5caa785dae305065b
Author: Aviem Zur 
Date:   2017-02-23T04:56:43Z

Test Spark runner streaming IO metrics




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2139: Update Dataflow container version for release 0.6.0

2017-03-05 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2139


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2161: [BEAM-1625] BoundedDataset action() does not materi...

2017-03-05 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2161


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2161

2017-03-05 Thread staslevin

This closes #2161


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/59451bca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/59451bca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/59451bca

Branch: refs/heads/master
Commit: 59451bca60ef45b1daeb14b8194bac0f8bdcc98b
Parents: 11a381b 4febd95
Author: Stas Levin 
Authored: Sun Mar 5 17:51:59 2017 +0200
Committer: Stas Levin 
Committed: Sun Mar 5 17:51:59 2017 +0200

--
 .../apache/beam/runners/spark/translation/BoundedDataset.java  | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: [BEAM-1625] BoundedDataset action() does not materialize RDD

2017-03-05 Thread staslevin

Repository: beam
Updated Branches:
  refs/heads/master 11a381b23 -> 59451bca6


[BEAM-1625] BoundedDataset action() does not materialize RDD


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4febd954
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4febd954
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4febd954

Branch: refs/heads/master
Commit: 4febd954af00458032efbee45b7f9724fe0ea9ed
Parents: 11a381b
Author: Aviem Zur 
Authored: Sun Mar 5 16:17:35 2017 +0200
Committer: Aviem Zur 
Committed: Sun Mar 5 16:17:35 2017 +0200

--
 .../apache/beam/runners/spark/translation/BoundedDataset.java  | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4febd954/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/BoundedDataset.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/BoundedDataset.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/BoundedDataset.java
index 5e19846..7db04a8 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/BoundedDataset.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/BoundedDataset.java
@@ -20,7 +20,6 @@ package org.apache.beam.runners.spark.translation;
 
 import com.google.common.base.Function;
 import com.google.common.collect.Iterables;
-import java.util.Iterator;
 import java.util.List;
 import javax.annotation.Nullable;
 import org.apache.beam.runners.spark.coders.CoderHelpers;
@@ -107,9 +106,8 @@ public class BoundedDataset implements Dataset {
   @Override
   public void action() {
 // Empty function to force computation of RDD.
-rdd.foreachPartition(new VoidFunction>() {
-  @Override
-  public void call(Iterator windowedValueIterator) 
throws Exception {
+rdd.foreach(new VoidFunction() {
+  @Override public void call(WindowedValue tWindowedValue) throws 
Exception {
 // Empty implementation.
   }
 });

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Spark #1134

2017-03-05 Thread Apache Jenkins Server

See

[jira] [Updated] (BEAM-1625) BoundedDataset action() does not materialize RDD

2017-03-05 Thread Aviem Zur (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1625:

Description: 
{{BoundedDataset#action}} does not materialize RDD since it uses 
{{foreachPartition}} which is not an action, instead use {{foreach}} which is.

See: http://spark.apache.org/docs/latest/programming-guide.html#actions

  was:
BoundedDataset action() does not materialize RDD since it uses 
{{foreachPartition}} which is not an action, instead use {{foreach}} which is.

See: http://spark.apache.org/docs/latest/programming-guide.html#actions


> BoundedDataset action() does not materialize RDD
> 
>
> Key: BEAM-1625
> URL: https://issues.apache.org/jira/browse/BEAM-1625
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> {{BoundedDataset#action}} does not materialize RDD since it uses 
> {{foreachPartition}} which is not an action, instead use {{foreach}} which is.
> See: http://spark.apache.org/docs/latest/programming-guide.html#actions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1625) BoundedDataset action() does not materialize RDD

2017-03-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896275#comment-15896275
 ] 

ASF GitHub Bot commented on BEAM-1625:
--

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam/pull/2161

[BEAM-1625] BoundedDataset action() does not materialize RDD

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam spark-materialize-bug

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2161.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2161


commit f0cb2b77545680b44a6e424cea7f9738dafddbb2
Author: Aviem Zur 
Date:   2017-03-05T14:01:44Z

[BEAM-1625] BoundedDataset action() does not materialize RDD




> BoundedDataset action() does not materialize RDD
> 
>
> Key: BEAM-1625
> URL: https://issues.apache.org/jira/browse/BEAM-1625
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> BoundedDataset action() does not materialize RDD since it uses 
> {{foreachPartition}} which is not an action, instead use {{foreach}} which is.
> See: http://spark.apache.org/docs/latest/programming-guide.html#actions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2161: [BEAM-1625] BoundedDataset action() does not materi...

2017-03-05 Thread aviemzur

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam/pull/2161

[BEAM-1625] BoundedDataset action() does not materialize RDD

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam spark-materialize-bug

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2161.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2161


commit f0cb2b77545680b44a6e424cea7f9738dafddbb2
Author: Aviem Zur 
Date:   2017-03-05T14:01:44Z

[BEAM-1625] BoundedDataset action() does not materialize RDD




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1625) BoundedDataset action() does not materialize RDD

2017-03-05 Thread Aviem Zur (JIRA)

Aviem Zur created BEAM-1625:
---

 Summary: BoundedDataset action() does not materialize RDD
 Key: BEAM-1625
 URL: https://issues.apache.org/jira/browse/BEAM-1625
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Aviem Zur
Assignee: Aviem Zur


BoundedDataset action() does not materialize RDD since it uses 
{{foreachPartition}} which is not an action, instead use {{foreach}} which is.

See: http://spark.apache.org/docs/latest/programming-guide.html#actions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Spark #1133

2017-03-05 Thread Apache Jenkins Server

See

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Spark #1132

2017-03-05 Thread Apache Jenkins Server

See

43 matches

Mail list logo