[GitHub] incubator-gearpump pull request #128: [GEARPUMP-254] fix gearpump on Yarn
Github user asfgit closed the pull request at: https://github.com/apache/incubator-gearpump/pull/128 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump issue #128: [GEARPUMP-254] fix gearpump on Yarn
Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/128 +1 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (GEARPUMP-254) gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump on Yarn
[
https://issues.apache.org/jira/browse/GEARPUMP-254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang resolved GEARPUMP-254.
-
Resolution: Fixed
Fix Version/s: 0.8.3
Issue resolved by pull request 128
[https://github.com/apache/incubator-gearpump/pull/128]
> gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump
> on Yarn
> ---
>
> Key: GEARPUMP-254
> URL: https://issues.apache.org/jira/browse/GEARPUMP-254
> Project: Apache Gearpump
> Issue Type: Bug
>Reporter: Piotr Grabuszynski
>Priority: Trivial
> Fix For: 0.8.3
>
>
> This option should probably be optional. When I try to create instance on
> yarn w/o changing gear.conf file, such error occurs: {code}
> [ClusterConfig$] loading config file application.conf...
> Help:
> -package (required:false, default:) path on HDFS. If not specified, we will use default value
> /user/gearpump/gearpump.zip>
> -name (required:false, default:Gearpump)
> -verbose (required:false, default:false)
> -output (required:false, default:)
> Exception in thread "main" com.typesafe.config.ConfigException$Missing: No
> configuration setting found for key 'gearpump.yarn.user'
> at
> com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:170)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
> at
> com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:246)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:188)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at scala.util.Try$.apply(Try.scala:192)
> at org.apache.gearpump.util.AkkaApp$class.main(AkkaApp.scala:41)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:160)
> at
> org.apache.gearpump.experiments.yarn.client.Client$.main(Client.scala:49)
> at
> org.apache.gearpump.experiments.yarn.client.Client.main(Client.scala)
> {code}
> When value for key gearpump.yarn.user is set, I get another error:
> GEARPUMP-253
> commit used: 1bf6a9ba8fa774b985fcd90a5d6bef129a4fb9a5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-254) gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump on Yarn
[
https://issues.apache.org/jira/browse/GEARPUMP-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797260#comment-15797260
]
ASF GitHub Bot commented on GEARPUMP-254:
-
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-gearpump/pull/128
> gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump
> on Yarn
> ---
>
> Key: GEARPUMP-254
> URL: https://issues.apache.org/jira/browse/GEARPUMP-254
> Project: Apache Gearpump
> Issue Type: Bug
>Reporter: Piotr Grabuszynski
>Priority: Trivial
>
> This option should probably be optional. When I try to create instance on
> yarn w/o changing gear.conf file, such error occurs: {code}
> [ClusterConfig$] loading config file application.conf...
> Help:
> -package (required:false, default:) path on HDFS. If not specified, we will use default value
> /user/gearpump/gearpump.zip>
> -name (required:false, default:Gearpump)
> -verbose (required:false, default:false)
> -output (required:false, default:)
> Exception in thread "main" com.typesafe.config.ConfigException$Missing: No
> configuration setting found for key 'gearpump.yarn.user'
> at
> com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:170)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
> at
> com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:246)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:188)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at scala.util.Try$.apply(Try.scala:192)
> at org.apache.gearpump.util.AkkaApp$class.main(AkkaApp.scala:41)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:160)
> at
> org.apache.gearpump.experiments.yarn.client.Client$.main(Client.scala:49)
> at
> org.apache.gearpump.experiments.yarn.client.Client.main(Client.scala)
> {code}
> When value for key gearpump.yarn.user is set, I get another error:
> GEARPUMP-253
> commit used: 1bf6a9ba8fa774b985fcd90a5d6bef129a4fb9a5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-254) gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump on Yarn
[
https://issues.apache.org/jira/browse/GEARPUMP-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797251#comment-15797251
]
ASF GitHub Bot commented on GEARPUMP-254:
-
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/128
+1 LGTM
> gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump
> on Yarn
> ---
>
> Key: GEARPUMP-254
> URL: https://issues.apache.org/jira/browse/GEARPUMP-254
> Project: Apache Gearpump
> Issue Type: Bug
>Reporter: Piotr Grabuszynski
>Priority: Trivial
>
> This option should probably be optional. When I try to create instance on
> yarn w/o changing gear.conf file, such error occurs: {code}
> [ClusterConfig$] loading config file application.conf...
> Help:
> -package (required:false, default:) path on HDFS. If not specified, we will use default value
> /user/gearpump/gearpump.zip>
> -name (required:false, default:Gearpump)
> -verbose (required:false, default:false)
> -output (required:false, default:)
> Exception in thread "main" com.typesafe.config.ConfigException$Missing: No
> configuration setting found for key 'gearpump.yarn.user'
> at
> com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:170)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
> at
> com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:246)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:188)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at scala.util.Try$.apply(Try.scala:192)
> at org.apache.gearpump.util.AkkaApp$class.main(AkkaApp.scala:41)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:160)
> at
> org.apache.gearpump.experiments.yarn.client.Client$.main(Client.scala:49)
> at
> org.apache.gearpump.experiments.yarn.client.Client.main(Client.scala)
> {code}
> When value for key gearpump.yarn.user is set, I get another error:
> GEARPUMP-253
> commit used: 1bf6a9ba8fa774b985fcd90a5d6bef129a4fb9a5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-gearpump issue #128: [GEARPUMP-254] fix gearpump on Yarn
Github user codecov-io commented on the issue: https://github.com/apache/incubator-gearpump/pull/128 ## [Current coverage](https://codecov.io/gh/apache/incubator-gearpump/pull/128?src=pr) is 70.79% (diff: 100%) > Merging [#128](https://codecov.io/gh/apache/incubator-gearpump/pull/128?src=pr) into [master](https://codecov.io/gh/apache/incubator-gearpump/branch/master?src=pr) will increase coverage by **0.41%** ```diff @@ master #128 diff @@ == Files 186186 Lines 5988 5988 Methods5469 5469 Messages 0 0 Branches517517 == + Hits 4214 4239+25 + Misses 1774 1749-25 Partials 0 0 ```  > Powered by [Codecov](https://codecov.io?src=pr). Last update [2783559...4e14131](https://codecov.io/gh/apache/incubator-gearpump/compare/2783559dbf1fe0dd5f090cece152bef65251d5b8...4e14131227396cb2b980376fdcb66d6a0314?src=pr) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump pull request #127: [GEARPUMP-249] Force eager evaluation ...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-gearpump/pull/127 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump issue #127: [GEARPUMP-249] Force eager evaluation on chai...
Github user huafengw commented on the issue: https://github.com/apache/incubator-gearpump/pull/127 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump issue #127: [GEARPUMP-249] Force eager evaluation on chai...
Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/127 R: @huafengw --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump issue #128: [GEARPUMP-254] fix gearpump on Yarn
Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/128 UT failed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (GEARPUMP-254) gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump on Yarn
[
https://issues.apache.org/jira/browse/GEARPUMP-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797227#comment-15797227
]
ASF GitHub Bot commented on GEARPUMP-254:
-
Github user codecov-io commented on the issue:
https://github.com/apache/incubator-gearpump/pull/128
## [Current
coverage](https://codecov.io/gh/apache/incubator-gearpump/pull/128?src=pr) is
70.79% (diff: 100%)
> Merging
[#128](https://codecov.io/gh/apache/incubator-gearpump/pull/128?src=pr) into
[master](https://codecov.io/gh/apache/incubator-gearpump/branch/master?src=pr)
will increase coverage by **0.41%**
```diff
@@ master #128 diff @@
==
Files 186186
Lines 5988 5988
Methods5469 5469
Messages 0 0
Branches517517
==
+ Hits 4214 4239+25
+ Misses 1774 1749-25
Partials 0 0
```
> Powered by [Codecov](https://codecov.io?src=pr). Last update
[2783559...4e14131](https://codecov.io/gh/apache/incubator-gearpump/compare/2783559dbf1fe0dd5f090cece152bef65251d5b8...4e14131227396cb2b980376fdcb66d6a0314?src=pr)
> gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump
> on Yarn
> ---
>
> Key: GEARPUMP-254
> URL: https://issues.apache.org/jira/browse/GEARPUMP-254
> Project: Apache Gearpump
> Issue Type: Bug
>Reporter: Piotr Grabuszynski
>Priority: Trivial
>
> This option should probably be optional. When I try to create instance on
> yarn w/o changing gear.conf file, such error occurs: {code}
> [ClusterConfig$] loading config file application.conf...
> Help:
> -package (required:false, default:) path on HDFS. If not specified, we will use default value
> /user/gearpump/gearpump.zip>
> -name (required:false, default:Gearpump)
> -verbose (required:false, default:false)
> -output (required:false, default:)
> Exception in thread "main" com.typesafe.config.ConfigException$Missing: No
> configuration setting found for key 'gearpump.yarn.user'
> at
> com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:170)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
> at
> com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:246)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:188)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at scala.util.Try$.apply(Try.scala:192)
> at org.apache.gearpump.util.AkkaApp$class.main(AkkaApp.scala:41)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:160)
> at
> org.apache.gearpump.experiments.yarn.client.Client$.main(Client.scala:49)
> at
> org.apache.gearpump.experiments.yarn.client.Client.main(Client.scala)
> {code}
> When value for key gearpump.yarn.user is set, I get another error:
> GEARPUMP-253
> commit used: 1bf6a9ba8fa774b985fcd90a5d6bef129a4fb9a5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (GEARPUMP-258) Add conferences, meetups and slides to gearpump website
Manu Zhang created GEARPUMP-258: --- Summary: Add conferences, meetups and slides to gearpump website Key: GEARPUMP-258 URL: https://issues.apache.org/jira/browse/GEARPUMP-258 Project: Apache Gearpump Issue Type: Improvement Components: website Reporter: Manu Zhang Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-253) Yarn-client fail while lunching
[
https://issues.apache.org/jira/browse/GEARPUMP-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797204#comment-15797204
]
Huafeng Wang commented on GEARPUMP-253:
---
[~pgrabuszynski] Sorry for the late response. Hopefully this issue will also be
fixed in pr https://github.com/apache/incubator-gearpump/pull/128
> Yarn-client fail while lunching
> ---
>
> Key: GEARPUMP-253
> URL: https://issues.apache.org/jira/browse/GEARPUMP-253
> Project: Apache Gearpump
> Issue Type: Bug
>Reporter: Piotr Grabuszynski
>Priority: Minor
>
> I tried to spawn Gearpump on Yarn. First, all yarn-config files has been
> copied to conf/yarnconf and Gearpump has been stored in HDFS under path
> /usr/lib/gearpump/gearpump-2.11-0.8.3-SNAPSHOT.zip. Then, I called command:
> bin/yarnclient launch -package
> /usr/lib/gearpump/gearpump-2.11-0.8.3-SNAPSHOT.zip and got following error:
> {code}
> [ClusterConfig$] loading config file application.conf...
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/common/base/Preconditions
> at
> org.apache.hadoop.conf.Configuration$DeprecationDelta.(Configuration.java:318)
> at
> org.apache.hadoop.conf.Configuration$DeprecationDelta.(Configuration.java:331)
> at
> org.apache.hadoop.conf.Configuration.(Configuration.java:413)
> at
> org.apache.gearpump.experiments.yarn.glue.YarnConfig$.$lessinit$greater$default$1(YarnConfig.scala:27)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:193)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at scala.util.Try$.apply(Try.scala:192)
> at org.apache.gearpump.util.AkkaApp$class.main(AkkaApp.scala:41)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:160)
> at
> org.apache.gearpump.experiments.yarn.client.Client$.main(Client.scala:49)
> at
> org.apache.gearpump.experiments.yarn.client.Client.main(Client.scala)
> Caused by: java.lang.ClassNotFoundException:
> com.google.common.base.Preconditions
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 12 more
> {code}
> commit used to build package: 1bf6a9ba8fa774b985fcd90a5d6bef129a4fb9a5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-249) reduce is not executed when following flatMap in Stream DSL
[
https://issues.apache.org/jira/browse/GEARPUMP-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797172#comment-15797172
]
ASF GitHub Bot commented on GEARPUMP-249:
-
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-gearpump/pull/127
> reduce is not executed when following flatMap in Stream DSL
> ---
>
> Key: GEARPUMP-249
> URL: https://issues.apache.org/jira/browse/GEARPUMP-249
> Project: Apache Gearpump
> Issue Type: Bug
> Components: streaming
>Affects Versions: 0.8.2
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 0.8.3
>
>
> The issue can be reproduced if a redundant {{map}} is added between
> {{groupBy}} and {{sum}}
> {code}
> app.source(data.lines.toList, 1, "source").
> // word => (word, count)
> flatMap(line => line.split("[\\s]+")).map((_, 1)).
> // (word, count1), (word, count2) => (word, count1 + count2)
> groupByKey().sum.log
> {code}
> The problem is with {{AndThen#process}} method, where {{flatMap}} is lazily/
> never executed.
> {code}
> override def process(value: IN): TraversableOnce[OUT] = {
> first.process(value).flatMap(second.process)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-249) reduce is not executed when following flatMap in Stream DSL
[
https://issues.apache.org/jira/browse/GEARPUMP-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797168#comment-15797168
]
ASF GitHub Bot commented on GEARPUMP-249:
-
Github user huafengw commented on the issue:
https://github.com/apache/incubator-gearpump/pull/127
+1
> reduce is not executed when following flatMap in Stream DSL
> ---
>
> Key: GEARPUMP-249
> URL: https://issues.apache.org/jira/browse/GEARPUMP-249
> Project: Apache Gearpump
> Issue Type: Bug
> Components: streaming
>Affects Versions: 0.8.2
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 0.8.3
>
>
> The issue can be reproduced if a redundant {{map}} is added between
> {{groupBy}} and {{sum}}
> {code}
> app.source(data.lines.toList, 1, "source").
> // word => (word, count)
> flatMap(line => line.split("[\\s]+")).map((_, 1)).
> // (word, count1), (word, count2) => (word, count1 + count2)
> groupByKey().sum.log
> {code}
> The problem is with {{AndThen#process}} method, where {{flatMap}} is lazily/
> never executed.
> {code}
> override def process(value: IN): TraversableOnce[OUT] = {
> first.process(value).flatMap(second.process)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-249) reduce is not executed when following flatMap in Stream DSL
[
https://issues.apache.org/jira/browse/GEARPUMP-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797144#comment-15797144
]
ASF GitHub Bot commented on GEARPUMP-249:
-
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/127
R: @huafengw
> reduce is not executed when following flatMap in Stream DSL
> ---
>
> Key: GEARPUMP-249
> URL: https://issues.apache.org/jira/browse/GEARPUMP-249
> Project: Apache Gearpump
> Issue Type: Bug
> Components: streaming
>Affects Versions: 0.8.2
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 0.8.3
>
>
> The issue can be reproduced if a redundant {{map}} is added between
> {{groupBy}} and {{sum}}
> {code}
> app.source(data.lines.toList, 1, "source").
> // word => (word, count)
> flatMap(line => line.split("[\\s]+")).map((_, 1)).
> // (word, count1), (word, count2) => (word, count1 + count2)
> groupByKey().sum.log
> {code}
> The problem is with {{AndThen#process}} method, where {{flatMap}} is lazily/
> never executed.
> {code}
> override def process(value: IN): TraversableOnce[OUT] = {
> first.process(value).flatMap(second.process)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-254) gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump on Yarn
[
https://issues.apache.org/jira/browse/GEARPUMP-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797142#comment-15797142
]
ASF GitHub Bot commented on GEARPUMP-254:
-
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/128
UT failed
> gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump
> on Yarn
> ---
>
> Key: GEARPUMP-254
> URL: https://issues.apache.org/jira/browse/GEARPUMP-254
> Project: Apache Gearpump
> Issue Type: Bug
>Reporter: Piotr Grabuszynski
>Priority: Trivial
>
> This option should probably be optional. When I try to create instance on
> yarn w/o changing gear.conf file, such error occurs: {code}
> [ClusterConfig$] loading config file application.conf...
> Help:
> -package (required:false, default:) path on HDFS. If not specified, we will use default value
> /user/gearpump/gearpump.zip>
> -name (required:false, default:Gearpump)
> -verbose (required:false, default:false)
> -output (required:false, default:)
> Exception in thread "main" com.typesafe.config.ConfigException$Missing: No
> configuration setting found for key 'gearpump.yarn.user'
> at
> com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:170)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
> at
> com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:246)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:188)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at scala.util.Try$.apply(Try.scala:192)
> at org.apache.gearpump.util.AkkaApp$class.main(AkkaApp.scala:41)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:160)
> at
> org.apache.gearpump.experiments.yarn.client.Client$.main(Client.scala:49)
> at
> org.apache.gearpump.experiments.yarn.client.Client.main(Client.scala)
> {code}
> When value for key gearpump.yarn.user is set, I get another error:
> GEARPUMP-253
> commit used: 1bf6a9ba8fa774b985fcd90a5d6bef129a4fb9a5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-gearpump pull request #128: [GEARPUMP-254] fix gearpump on Yarn
GitHub user huafengw opened a pull request: https://github.com/apache/incubator-gearpump/pull/128 [GEARPUMP-254] fix gearpump on Yarn Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the commit message is formatted like: `[GEARPUMP-] Meaningful description of pull request` - [ ] Make sure tests pass via `sbt clean test`. - [ ] Make sure old documentation affected by the pull request has been updated and new documentation added for new functionality. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huafengw/incubator-gearpump fixYarn Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-gearpump/pull/128.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #128 commit f40bedf916864c4ac5b573287c6429a7b2ce8892 Author: huafengw Date: 2017-01-04T03:27:16Z [GEARPUMP-254] fix gearpump on Yarn --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump pull request #127: [GEARPUMP-249] Force eager evaluation ...
GitHub user manuzhang opened a pull request: https://github.com/apache/incubator-gearpump/pull/127 [GEARPUMP-249] Force eager evaluation on chained operations Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the commit message is formatted like: `[GEARPUMP-] Meaningful description of pull request` - [ ] Make sure tests pass via `sbt clean test`. - [ ] Make sure old documentation affected by the pull request has been updated and new documentation added for new functionality. You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-gearpump eager-eval Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-gearpump/pull/127.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #127 commit ddc1e98877326740481d76bbfb5c1646bd79fd4f Author: manuzhang Date: 2017-01-04T03:18:40Z [GEARPUMP-249] Force eager evaluation on chained operations --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump issue #127: [GEARPUMP-249] Force eager evaluation on chai...
Github user codecov-io commented on the issue: https://github.com/apache/incubator-gearpump/pull/127 ## [Current coverage](https://codecov.io/gh/apache/incubator-gearpump/pull/127?src=pr) is 70.39% (diff: 100%) > Merging [#127](https://codecov.io/gh/apache/incubator-gearpump/pull/127?src=pr) into [master](https://codecov.io/gh/apache/incubator-gearpump/branch/master?src=pr) will increase coverage by **0.01%** ```diff @@ master #127 diff @@ == Files 186186 Lines 5988 5988 Methods5469 5469 Messages 0 0 Branches517517 == + Hits 4214 4215 +1 + Misses 1774 1773 -1 Partials 0 0 ```  > Powered by [Codecov](https://codecov.io?src=pr). Last update [2783559...ddc1e98](https://codecov.io/gh/apache/incubator-gearpump/compare/2783559dbf1fe0dd5f090cece152bef65251d5b8...ddc1e98877326740481d76bbfb5c1646bd79fd4f?src=pr) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (GEARPUMP-249) reduce is not executed when following flatMap in Stream DSL
[
https://issues.apache.org/jira/browse/GEARPUMP-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797109#comment-15797109
]
ASF GitHub Bot commented on GEARPUMP-249:
-
Github user codecov-io commented on the issue:
https://github.com/apache/incubator-gearpump/pull/127
## [Current
coverage](https://codecov.io/gh/apache/incubator-gearpump/pull/127?src=pr) is
70.39% (diff: 100%)
> Merging
[#127](https://codecov.io/gh/apache/incubator-gearpump/pull/127?src=pr) into
[master](https://codecov.io/gh/apache/incubator-gearpump/branch/master?src=pr)
will increase coverage by **0.01%**
```diff
@@ master #127 diff @@
==
Files 186186
Lines 5988 5988
Methods5469 5469
Messages 0 0
Branches517517
==
+ Hits 4214 4215 +1
+ Misses 1774 1773 -1
Partials 0 0
```
> Powered by [Codecov](https://codecov.io?src=pr). Last update
[2783559...ddc1e98](https://codecov.io/gh/apache/incubator-gearpump/compare/2783559dbf1fe0dd5f090cece152bef65251d5b8...ddc1e98877326740481d76bbfb5c1646bd79fd4f?src=pr)
> reduce is not executed when following flatMap in Stream DSL
> ---
>
> Key: GEARPUMP-249
> URL: https://issues.apache.org/jira/browse/GEARPUMP-249
> Project: Apache Gearpump
> Issue Type: Bug
> Components: streaming
>Affects Versions: 0.8.2
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 0.8.3
>
>
> The issue can be reproduced if a redundant {{map}} is added between
> {{groupBy}} and {{sum}}
> {code}
> app.source(data.lines.toList, 1, "source").
> // word => (word, count)
> flatMap(line => line.split("[\\s]+")).map((_, 1)).
> // (word, count1), (word, count2) => (word, count1 + count2)
> groupByKey().sum.log
> {code}
> The problem is with {{AndThen#process}} method, where {{flatMap}} is lazily/
> never executed.
> {code}
> override def process(value: IN): TraversableOnce[OUT] = {
> first.process(value).flatMap(second.process)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-254) gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump on Yarn
[
https://issues.apache.org/jira/browse/GEARPUMP-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797066#comment-15797066
]
ASF GitHub Bot commented on GEARPUMP-254:
-
GitHub user huafengw opened a pull request:
https://github.com/apache/incubator-gearpump/pull/128
[GEARPUMP-254] fix gearpump on Yarn
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the commit message is formatted like:
`[GEARPUMP-] Meaningful description of pull request`
- [ ] Make sure tests pass via `sbt clean test`.
- [ ] Make sure old documentation affected by the pull request has been
updated and new documentation added for new functionality.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/huafengw/incubator-gearpump fixYarn
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-gearpump/pull/128.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #128
commit f40bedf916864c4ac5b573287c6429a7b2ce8892
Author: huafengw
Date: 2017-01-04T03:27:16Z
[GEARPUMP-254] fix gearpump on Yarn
> gearpump.yarn.user field in gear.conf is obligatory while spawning Gearpump
> on Yarn
> ---
>
> Key: GEARPUMP-254
> URL: https://issues.apache.org/jira/browse/GEARPUMP-254
> Project: Apache Gearpump
> Issue Type: Bug
>Reporter: Piotr Grabuszynski
>Priority: Trivial
>
> This option should probably be optional. When I try to create instance on
> yarn w/o changing gear.conf file, such error occurs: {code}
> [ClusterConfig$] loading config file application.conf...
> Help:
> -package (required:false, default:) path on HDFS. If not specified, we will use default value
> /user/gearpump/gearpump.zip>
> -name (required:false, default:Gearpump)
> -verbose (required:false, default:false)
> -output (required:false, default:)
> Exception in thread "main" com.typesafe.config.ConfigException$Missing: No
> configuration setting found for key 'gearpump.yarn.user'
> at
> com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:170)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at
> com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
> at
> com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:246)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:188)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at
> org.apache.gearpump.util.AkkaApp$$anonfun$main$1.apply(AkkaApp.scala:42)
> at scala.util.Try$.apply(Try.scala:192)
> at org.apache.gearpump.util.AkkaApp$class.main(AkkaApp.scala:41)
> at
> org.apache.gearpump.experiments.yarn.client.LaunchCluster$.main(LaunchCluster.scala:160)
> at
> org.apache.gearpump.experiments.yarn.client.Client$.main(Client.scala:49)
> at
> org.apache.gearpump.experiments.yarn.client.Client.main(Client.scala)
> {code}
> When value for key gearpump.yarn.user is set, I get another error:
> GEARPUMP-253
> commit used: 1bf6a9ba8fa774b985fcd90a5d6bef129a4fb9a5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-249) reduce is not executed when following flatMap in Stream DSL
[
https://issues.apache.org/jira/browse/GEARPUMP-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797050#comment-15797050
]
ASF GitHub Bot commented on GEARPUMP-249:
-
GitHub user manuzhang opened a pull request:
https://github.com/apache/incubator-gearpump/pull/127
[GEARPUMP-249] Force eager evaluation on chained operations
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the commit message is formatted like:
`[GEARPUMP-] Meaningful description of pull request`
- [ ] Make sure tests pass via `sbt clean test`.
- [ ] Make sure old documentation affected by the pull request has been
updated and new documentation added for new functionality.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/manuzhang/incubator-gearpump eager-eval
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-gearpump/pull/127.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #127
commit ddc1e98877326740481d76bbfb5c1646bd79fd4f
Author: manuzhang
Date: 2017-01-04T03:18:40Z
[GEARPUMP-249] Force eager evaluation on chained operations
> reduce is not executed when following flatMap in Stream DSL
> ---
>
> Key: GEARPUMP-249
> URL: https://issues.apache.org/jira/browse/GEARPUMP-249
> Project: Apache Gearpump
> Issue Type: Bug
> Components: streaming
>Affects Versions: 0.8.2
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 0.8.3
>
>
> The issue can be reproduced if a redundant {{map}} is added between
> {{groupBy}} and {{sum}}
> {code}
> app.source(data.lines.toList, 1, "source").
> // word => (word, count)
> flatMap(line => line.split("[\\s]+")).map((_, 1)).
> // (word, count1), (word, count2) => (word, count1 + count2)
> groupByKey().sum.log
> {code}
> The problem is with {{AndThen#process}} method, where {{flatMap}} is lazily/
> never executed.
> {code}
> override def process(value: IN): TraversableOnce[OUT] = {
> first.process(value).flatMap(second.process)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (GEARPUMP-249) reduce is not executed when following flatMap in Stream DSL
[
https://issues.apache.org/jira/browse/GEARPUMP-249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang reopened GEARPUMP-249:
-
> reduce is not executed when following flatMap in Stream DSL
> ---
>
> Key: GEARPUMP-249
> URL: https://issues.apache.org/jira/browse/GEARPUMP-249
> Project: Apache Gearpump
> Issue Type: Bug
> Components: streaming
>Affects Versions: 0.8.2
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 0.8.3
>
>
> The issue can be reproduced if a redundant {{map}} is added between
> {{groupBy}} and {{sum}}
> {code}
> app.source(data.lines.toList, 1, "source").
> // word => (word, count)
> flatMap(line => line.split("[\\s]+")).map((_, 1)).
> // (word, count1), (word, count2) => (word, count1 + count2)
> groupByKey().sum.log
> {code}
> The problem is with {{AndThen#process}} method, where {{flatMap}} is lazily/
> never executed.
> {code}
> override def process(value: IN): TraversableOnce[OUT] = {
> first.process(value).flatMap(second.process)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
