[jira] [Created] (GEARPUMP-250) add DSL integration, functional tests
Kam Kasravi created GEARPUMP-250: Summary: add DSL integration, functional tests Key: GEARPUMP-250 URL: https://issues.apache.org/jira/browse/GEARPUMP-250 Project: Apache Gearpump Issue Type: Bug Affects Versions: 0.8.2 Reporter: Kam Kasravi Fix For: 0.8.3 akka-streams has a great testkit - perhaps we should borrow from that -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-242) Update website for 0.8.2-incubating release
[ https://issues.apache.org/jira/browse/GEARPUMP-242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-242: Assignee: Kam Kasravi > Update website for 0.8.2-incubating release > --- > > Key: GEARPUMP-242 > URL: https://issues.apache.org/jira/browse/GEARPUMP-242 > Project: Apache Gearpump > Issue Type: Task >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > Update the site and then announce the release on gene...@incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-242) Update website for 0.8.2-incubating release
Kam Kasravi created GEARPUMP-242: Summary: Update website for 0.8.2-incubating release Key: GEARPUMP-242 URL: https://issues.apache.org/jira/browse/GEARPUMP-242 Project: Apache Gearpump Issue Type: Task Affects Versions: 0.8.1 Reporter: Kam Kasravi Fix For: 0.8.2 Update the site and then announce the release on gene...@incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-241) Update CHANGELOG.md to reflect all JIRA's
Kam Kasravi created GEARPUMP-241: Summary: Update CHANGELOG.md to reflect all JIRA's Key: GEARPUMP-241 URL: https://issues.apache.org/jira/browse/GEARPUMP-241 Project: Apache Gearpump Issue Type: Bug Affects Versions: 0.8.1 Reporter: Kam Kasravi Assignee: Kam Kasravi Fix For: 0.8.2 Add GEARPUMP-240, and this JIRA -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-239) Correct binary and source licenses subdirectory
Kam Kasravi created GEARPUMP-239: Summary: Correct binary and source licenses subdirectory Key: GEARPUMP-239 URL: https://issues.apache.org/jira/browse/GEARPUMP-239 Project: Apache Gearpump Issue Type: Bug Affects Versions: 0.8.1 Reporter: Kam Kasravi Assignee: Kam Kasravi Fix For: 0.8.2 The source release should only include 2 license files under the licenses/ subdirectory. The binary release did not include the licenses/ subdirectory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-238) Correct release naming and packaging
Kam Kasravi created GEARPUMP-238: Summary: Correct release naming and packaging Key: GEARPUMP-238 URL: https://issues.apache.org/jira/browse/GEARPUMP-238 Project: Apache Gearpump Issue Type: Bug Affects Versions: 0.8.1 Reporter: Kam Kasravi Assignee: Kam Kasravi Fix For: 0.8.2 Source and binary releases should include the scala version. The packaging inadvertently included a nested gearpump-2.11-0.8.2 folder -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-234) Update LICENSE and related files for 0.8.2-incubating binary release
[ https://issues.apache.org/jira/browse/GEARPUMP-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-234: Assignee: Kam Kasravi > Update LICENSE and related files for 0.8.2-incubating binary release > > > Key: GEARPUMP-234 > URL: https://issues.apache.org/jira/browse/GEARPUMP-234 > Project: Apache Gearpump > Issue Type: Task >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-22) Support akka-streams Gearpump Materializer
[ https://issues.apache.org/jira/browse/GEARPUMP-22?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-22: Fix Version/s: (was: 0.8.2) 0.8.3 > Support akka-streams Gearpump Materializer > -- > > Key: GEARPUMP-22 > URL: https://issues.apache.org/jira/browse/GEARPUMP-22 > Project: Apache Gearpump > Issue Type: New Feature > Components: akkastream >Affects Versions: 0.8.0 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.3 > > > Reenable experiments/akkastream to work with akka 2.4.3 (latest on master) > Create PR's for akka team related to changes required for Gearpump > Materializer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-112) DAG chart in dashboard does not show data, when flow is small
[ https://issues.apache.org/jira/browse/GEARPUMP-112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-112: - Fix Version/s: (was: 0.8.2) 0.8.3 > DAG chart in dashboard does not show data, when flow is small > - > > Key: GEARPUMP-112 > URL: https://issues.apache.org/jira/browse/GEARPUMP-112 > Project: Apache Gearpump > Issue Type: Bug > Components: Dashboard >Affects Versions: 0.8.0 >Reporter: Piotr Grabuszynski >Assignee: Karol Brejna > Fix For: 0.8.3 > > Attachments: gp-flow.jpg, gp-screen-2.jpg, gp-screen-metrics.jpg, > gp-screen-overview.jpg > > > BUG: > After deploying application on Apache Gearpump, DAG chart and data near it, > shows always the same value, and lines and nodes on chart doesn't grow if the > flow is about 3 requests / second. > WAY TO REPRODUCE: > 1. Deploy application on Apache Gearpump. > 2. To input kafka topic send ~3 requests/sec. > 3. Go to dashboard, choose your app and go to DAG tab. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-193) Need ability to request upstream messages (pull)
[ https://issues.apache.org/jira/browse/GEARPUMP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-193: - Fix Version/s: (was: 0.8.2) 0.8.3 > Need ability to request upstream messages (pull) > > > Key: GEARPUMP-193 > URL: https://issues.apache.org/jira/browse/GEARPUMP-193 > Project: Apache Gearpump > Issue Type: Sub-task >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.3 > > > akka-streams Interleave GraphStage will iterate through upstream processors > pulling a certain number of messages for each. We do not have this ability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-198) empty kafka topics show very high receive throughput
[ https://issues.apache.org/jira/browse/GEARPUMP-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-198: - Fix Version/s: (was: 0.8.2) 0.8.3 > empty kafka topics show very high receive throughput > > > Key: GEARPUMP-198 > URL: https://issues.apache.org/jira/browse/GEARPUMP-198 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Manu Zhang >Priority: Minor > Fix For: 0.8.3 > > > if the kafkasource reaches the end of the topic it shows very high message > throughput even though it's not really sending/receiving messages -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-144) Modify Gearpump's yarnclient so it can use provided user for HDFS and YARN
[ https://issues.apache.org/jira/browse/GEARPUMP-144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-144: - Fix Version/s: (was: 0.8.2) 0.8.3 > Modify Gearpump's yarnclient so it can use provided user for HDFS and YARN > -- > > Key: GEARPUMP-144 > URL: https://issues.apache.org/jira/browse/GEARPUMP-144 > Project: Apache Gearpump > Issue Type: Improvement > Components: yarn >Reporter: Karol Brejna >Assignee: Karol Brejna > Fix For: 0.8.3 > > > We have dedicated users for operations like deploying applications to yarn, > accessing hdfs etc. > Currently, there is no way to provide yarnclient (Gearpump component for > spawning Gearpump to yarn) the user to be used in the operation. > Gearpump code needs to be modified to overcome this shortcoming. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-199) high throughput results in high GC costs
[ https://issues.apache.org/jira/browse/GEARPUMP-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-199: - Fix Version/s: (was: 0.8.2) 0.8.3 > high throughput results in high GC costs > > > Key: GEARPUMP-199 > URL: https://issues.apache.org/jira/browse/GEARPUMP-199 > Project: Apache Gearpump > Issue Type: Improvement >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Manu Zhang > Fix For: 0.8.3 > > Attachments: Pasted image at 2016_08_31 01_50 AM.png > > > we should look to minimize GC costs - perhaps following what paypal squbs did > with persistentbuffer (also would be able to integrate with akka-streams) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-211) Look at merging or unifying akka-stream DSL and Gearpump DSL
[ https://issues.apache.org/jira/browse/GEARPUMP-211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-211: - Fix Version/s: (was: 0.8.2) 0.8.3 > Look at merging or unifying akka-stream DSL and Gearpump DSL > > > Key: GEARPUMP-211 > URL: https://issues.apache.org/jira/browse/GEARPUMP-211 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.3 > > > akka-streams expands on the functionality currently available within the > Gearpump DSL significantly bringing in many new concepts and operations. We > should either completely embrace this DSL or integrate the akka-stream > related tasks into Gearpump's DSL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-206) access to upstream and downstream processors
[ https://issues.apache.org/jira/browse/GEARPUMP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-206: - Fix Version/s: (was: 0.8.2) 0.8.3 > access to upstream and downstream processors > > > Key: GEARPUMP-206 > URL: https://issues.apache.org/jira/browse/GEARPUMP-206 > Project: Apache Gearpump > Issue Type: Sub-task >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.3 > > > akka-streams requires the ability to access both upstream and downstream > processors. For upstream akka-streams will pull - we need to emulate > something similar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-234) Update LICENSE and related files for 0.8.2-incubating binary release
Kam Kasravi created GEARPUMP-234: Summary: Update LICENSE and related files for 0.8.2-incubating binary release Key: GEARPUMP-234 URL: https://issues.apache.org/jira/browse/GEARPUMP-234 Project: Apache Gearpump Issue Type: Task Reporter: Kam Kasravi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-231) error when build package
[ https://issues.apache.org/jira/browse/GEARPUMP-231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633288#comment-15633288 ] Kam Kasravi commented on GEARPUMP-231: -- Thanks [~ffjl1985] - we'll update build instructions if necessary for 0.8.2-incubating > error when build package > > > Key: GEARPUMP-231 > URL: https://issues.apache.org/jira/browse/GEARPUMP-231 > Project: Apache Gearpump > Issue Type: Bug > Components: build >Affects Versions: 0.8.1 > Environment: win10 64 bit > cygwin > scala 2.11.8 > sbt >Reporter: Fei Feng >Priority: Blocker > > when building 0.8.1 with command sbt clean +assembly +packArchiveZip in > cygwin or idea terminal,error occured as followed: > [error] > E:\deveop\apache\incubator-gearpump\core\src\main\scala\org\apache\gearpump\metrics\Metrics.s > cala:26: object codahale is not a member of package org.apache.gearpump > [info] Resolving commons-codec#commons-codec;1.2 ... > [error] import org.apache.gearpump.codahale.metrics._ > [error]^ > [info] Resolving org.apache.gearpump#gearpump-streaming_2.11;0.8.2-SNAPSHOT > ... > [error] > E:\deveop\apache\incubator-gearpump\core\src\main\scala\org\apache\gearpump\cluster\UserConfi > g.scala:24: object google is not a member of package org.apache.gearpump > [error] import org.apache.gearpump.google.common.io.BaseEncoding > [error]^ > [info] Resolving io.netty#netty;3.8.0.Final ... > [error] > E:\deveop\apache\incubator-gearpump\core\src\main\scala\org\apache\gearpump\cluster\UserConfi > g.scala:94: not found: value BaseEncoding > [error] _config.get(key).map(BaseEncoding.base64().decode(_)) > [error] ^ > [info] Resolving com.typesafe.akka#akka-actor_2.11;2.4.3 ... > [error] > E:\deveop\apache\incubator-gearpump\core\src\main\scala\org\apache\gearpump\cluster\UserConfi > g.scala:101: not found: value BaseEncoding > [error] this.withString(key, BaseEncoding.base64().encode(value)) > [error]^ > [info] Resolving org.scala-lang.modules#scala-java8-compat_2.11;0.7.0 ... > [error] > E:\deveop\apache\incubator-gearpump\core\src\main\scala\org\apache\gearpump\cluster\UserConfi > g.scala:119: not found: value BaseEncoding > [error] _config.get(key).map(BaseEncoding.base64().decode(_)) > [error] ^ > [info] Resolving com.typesafe.akka#akka-protobuf_2.11;2.4.3 ... > [error] > E:\deveop\apache\incubator-gearpump\core\src\main\scala\org\apache\gearpump\cluster\UserConfi > g.scala:140: not found: value BaseEncoding > [info] Resolving org.uncommons.maths#uncommons-maths;1.2.2a ... > [error] val encoded = BaseEncoding.base64().encode(bytes) > [error] ^ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-22) Support akka-streams Gearpump Materializer
[ https://issues.apache.org/jira/browse/GEARPUMP-22?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-22: Fix Version/s: (was: 0.8.1) 0.8.2 > Support akka-streams Gearpump Materializer > -- > > Key: GEARPUMP-22 > URL: https://issues.apache.org/jira/browse/GEARPUMP-22 > Project: Apache Gearpump > Issue Type: New Feature > Components: akkastream >Affects Versions: 0.8.0 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > Reenable experiments/akkastream to work with akka 2.4.3 (latest on master) > Create PR's for akka team related to changes required for Gearpump > Materializer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-193) Need ability to request upstream messages (pull)
[ https://issues.apache.org/jira/browse/GEARPUMP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-193: Assignee: Kam Kasravi > Need ability to request upstream messages (pull) > > > Key: GEARPUMP-193 > URL: https://issues.apache.org/jira/browse/GEARPUMP-193 > Project: Apache Gearpump > Issue Type: Sub-task >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams Interleave GraphStage will iterate through upstream processors > pulling a certain number of messages for each. We do not have this ability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-196) expose physical DAG plan
[ https://issues.apache.org/jira/browse/GEARPUMP-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532009#comment-15532009 ] Kam Kasravi commented on GEARPUMP-196: -- This will likely need to be after 0.8.2 > expose physical DAG plan > > > Key: GEARPUMP-196 > URL: https://issues.apache.org/jira/browse/GEARPUMP-196 > Project: Apache Gearpump > Issue Type: New Feature >Affects Versions: 0.8.1 >Reporter: Kam Kasravi > > The DAG part of the UI only shows the logical DAG. If particular tasks have > parallelism > 1 they may exist across executors and their allocation may be > suboptimal. We should expose the physical DAG as well where the data flows > across executors can be traced. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-195) Single node DAG's should not show node as stalling
[ https://issues.apache.org/jira/browse/GEARPUMP-195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-195: - Fix Version/s: (was: 0.8.2) > Single node DAG's should not show node as stalling > -- > > Key: GEARPUMP-195 > URL: https://issues.apache.org/jira/browse/GEARPUMP-195 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > > DAG's with only one node should not show stalled. If a sink node is missing > we should either prevent the DAG from being run or not show the upstream node > status as stalled -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-196) expose physical DAG plan
[ https://issues.apache.org/jira/browse/GEARPUMP-196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-196: - Fix Version/s: (was: 0.8.2) > expose physical DAG plan > > > Key: GEARPUMP-196 > URL: https://issues.apache.org/jira/browse/GEARPUMP-196 > Project: Apache Gearpump > Issue Type: New Feature >Affects Versions: 0.8.1 >Reporter: Kam Kasravi > > The DAG part of the UI only shows the logical DAG. If particular tasks have > parallelism > 1 they may exist across executors and their allocation may be > suboptimal. We should expose the physical DAG as well where the data flows > across executors can be traced. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-211) Look at merging or unifying akka-stream DSL and Gearpump DSL
[ https://issues.apache.org/jira/browse/GEARPUMP-211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-211: Assignee: Kam Kasravi > Look at merging or unifying akka-stream DSL and Gearpump DSL > > > Key: GEARPUMP-211 > URL: https://issues.apache.org/jira/browse/GEARPUMP-211 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams expands on the functionality currently available within the > Gearpump DSL significantly bringing in many new concepts and operations. We > should either completely embrace this DSL or integrate the akka-stream > related tasks into Gearpump's DSL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-206) access to upstream and downstream processors
[ https://issues.apache.org/jira/browse/GEARPUMP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531760#comment-15531760 ] Kam Kasravi commented on GEARPUMP-206: -- It depends on the type of GraphStage in akka-streams. Upstreams could halt along batch boundaries. Downstreams could cache. Here are some GraphStage examples: Concat[T] * Takes multiple streams and outputs one stream formed from the input streams * by first emitting all of the elements from the first stream and then emitting * all of the elements from the second stream, etc. Interleave[T] * Interleave represents deterministic merge which takes N elements per input stream, * in-order of inputs, emits them downstream and then cycles/"wraps-around" the inputs. MergeSorted[T: Ordering] * Merge two pre-sorted streams such that the resulting stream is sorted. > access to upstream and downstream processors > > > Key: GEARPUMP-206 > URL: https://issues.apache.org/jira/browse/GEARPUMP-206 > Project: Apache Gearpump > Issue Type: Sub-task >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams requires the ability to access both upstream and downstream > processors. For upstream akka-streams will pull - we need to emulate > something similar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (GEARPUMP-206) access to upstream and downstream processors
[ https://issues.apache.org/jira/browse/GEARPUMP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494653#comment-15494653 ] Kam Kasravi edited comment on GEARPUMP-206 at 9/15/16 10:09 PM: For this JIRA, GEARPUMP-193 and GEARPUMP-191 a control message to halt one or more upstreams is I believe all we need. [~mauzhang] is there overlap with this functionality and the add watermark - there doesn't seem to be but thought I would check. I will implement this feature and see if it addresses current akka-streams GraphStages that require this functionality. was (Author: kam kasravi): For this JIRA and GEARPUMP-193 a control message to halt one or more upstreams is I believe all we need. [~mauzhang] is there overlap with this functionality and the add watermark - there doesn't seem to be but thought I would check. I will implement this feature and see if it addresses current akka-streams GraphStages that require this functionality. > access to upstream and downstream processors > > > Key: GEARPUMP-206 > URL: https://issues.apache.org/jira/browse/GEARPUMP-206 > Project: Apache Gearpump > Issue Type: Sub-task >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams requires the ability to access both upstream and downstream > processors. For upstream akka-streams will pull - we need to emulate > something similar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-206) access to upstream and downstream processors
[ https://issues.apache.org/jira/browse/GEARPUMP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494653#comment-15494653 ] Kam Kasravi commented on GEARPUMP-206: -- For this JIRA and GEARPUMP-193 a control message to halt one or more upstreams is I believe all we need. [~mauzhang] is there overlap with this functionality and the add watermark - there doesn't seem to be but thought I would check. I will implement this feature and see if it addresses current akka-streams GraphStages that require this functionality. > access to upstream and downstream processors > > > Key: GEARPUMP-206 > URL: https://issues.apache.org/jira/browse/GEARPUMP-206 > Project: Apache Gearpump > Issue Type: Sub-task >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams requires the ability to access both upstream and downstream > processors. For upstream akka-streams will pull - we need to emulate > something similar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-206) access to upstream and downstream processors
[ https://issues.apache.org/jira/browse/GEARPUMP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-206: - Assignee: Kam Kasravi > access to upstream and downstream processors > > > Key: GEARPUMP-206 > URL: https://issues.apache.org/jira/browse/GEARPUMP-206 > Project: Apache Gearpump > Issue Type: Sub-task >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams requires the ability to access both upstream and downstream > processors. For upstream akka-streams will pull - we need to emulate > something similar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-206) access to upstream and downstream processors
[ https://issues.apache.org/jira/browse/GEARPUMP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-206: - Assignee: (was: Kam Kasravi) > access to upstream and downstream processors > > > Key: GEARPUMP-206 > URL: https://issues.apache.org/jira/browse/GEARPUMP-206 > Project: Apache Gearpump > Issue Type: Sub-task >Affects Versions: 0.8.1 >Reporter: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams requires the ability to access both upstream and downstream > processors. For upstream akka-streams will pull - we need to emulate > something similar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-206) access to upstream and downstream processors
[ https://issues.apache.org/jira/browse/GEARPUMP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-206: Assignee: Kam Kasravi > access to upstream and downstream processors > > > Key: GEARPUMP-206 > URL: https://issues.apache.org/jira/browse/GEARPUMP-206 > Project: Apache Gearpump > Issue Type: Sub-task >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams requires the ability to access both upstream and downstream > processors. For upstream akka-streams will pull - we need to emulate > something similar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-211) Look at merging or unifying akka-stream DSL and Gearpump DSL
Kam Kasravi created GEARPUMP-211: Summary: Look at merging or unifying akka-stream DSL and Gearpump DSL Key: GEARPUMP-211 URL: https://issues.apache.org/jira/browse/GEARPUMP-211 Project: Apache Gearpump Issue Type: Bug Affects Versions: 0.8.1 Reporter: Kam Kasravi Fix For: 0.8.2 akka-streams expands on the functionality currently available within the Gearpump DSL significantly bringing in many new concepts and operations. We should either completely embrace this DSL or integrate the akka-stream related tasks into Gearpump's DSL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-208) Demo site goes down a lot. Also references gearpump.io when connection is refused
[ https://issues.apache.org/jira/browse/GEARPUMP-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488928#comment-15488928 ] Kam Kasravi commented on GEARPUMP-208: -- +1 - I think we need more formal support - maybe within TAP if they're doing a sandbox. > Demo site goes down a lot. Also references gearpump.io when connection is > refused > - > > Key: GEARPUMP-208 > URL: https://issues.apache.org/jira/browse/GEARPUMP-208 > Project: Apache Gearpump > Issue Type: Improvement >Affects Versions: 0.8.1 >Reporter: Kam Kasravi > Fix For: 0.8.2 > > > We should have a better ability to monitor this site and keep it up and also > correct the link from gearpump.io to be gearpump.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-191) add ability to control rate to TaskActor
[ https://issues.apache.org/jira/browse/GEARPUMP-191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488480#comment-15488480 ] Kam Kasravi commented on GEARPUMP-191: -- prerequisite for some of akka-streams GraphStages > add ability to control rate to TaskActor > > > Key: GEARPUMP-191 > URL: https://issues.apache.org/jira/browse/GEARPUMP-191 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams requires an ability to control the rate of input and output of > messages within a task. See Throttle > (http://doc.akka.io/docs/akka/current/scala/stream/stream-quickstart.html#Time-Based_Processing) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-206) access to upstream and downstream processors
[ https://issues.apache.org/jira/browse/GEARPUMP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488482#comment-15488482 ] Kam Kasravi commented on GEARPUMP-206: -- prerequisite for some of akka-streams GraphStages > access to upstream and downstream processors > > > Key: GEARPUMP-206 > URL: https://issues.apache.org/jira/browse/GEARPUMP-206 > Project: Apache Gearpump > Issue Type: New Feature >Affects Versions: 0.8.1 >Reporter: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams requires the ability to access both upstream and downstream > processors. For upstream akka-streams will pull - we need to emulate > something similar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-208) Demo site goes down a lot. Also references gearpump.io when connection is refused
Kam Kasravi created GEARPUMP-208: Summary: Demo site goes down a lot. Also references gearpump.io when connection is refused Key: GEARPUMP-208 URL: https://issues.apache.org/jira/browse/GEARPUMP-208 Project: Apache Gearpump Issue Type: Improvement Affects Versions: 0.8.1 Reporter: Kam Kasravi Fix For: 0.8.2 We should have a better ability to monitor this site and keep it up and also correct the link from gearpump.io to be gearpump.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-193) Need ability to request upstream messages (pull)
[ https://issues.apache.org/jira/browse/GEARPUMP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488481#comment-15488481 ] Kam Kasravi commented on GEARPUMP-193: -- prerequisite for some of akka-streams GraphStages > Need ability to request upstream messages (pull) > > > Key: GEARPUMP-193 > URL: https://issues.apache.org/jira/browse/GEARPUMP-193 > Project: Apache Gearpump > Issue Type: New Feature >Affects Versions: 0.8.1 >Reporter: Kam Kasravi > Fix For: 0.8.2 > > > akka-streams Interleave GraphStage will iterate through upstream processors > pulling a certain number of messages for each. We do not have this ability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-202) DSL improvement
[ https://issues.apache.org/jira/browse/GEARPUMP-202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15467971#comment-15467971 ] Kam Kasravi commented on GEARPUMP-202: -- Manu I think we need an ability to directly access both downstream and upstream tasks. We do have output(index, message) but I don't believe this is accessible from Task and the index is opaque. > DSL improvement > > > Key: GEARPUMP-202 > URL: https://issues.apache.org/jira/browse/GEARPUMP-202 > Project: Apache Gearpump > Issue Type: Improvement > Components: streaming >Affects Versions: 0.8.1 >Reporter: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-195) Single node DAG's should not show node as stalling
[ https://issues.apache.org/jira/browse/GEARPUMP-195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-195: - Description: DAG's with only one node should not show stalled. If a sink node is missing we should either prevent the DAG from being run or not show the upstream node status as stalled (was: DAG's with only one node should not show stalled. If a sink node is missing we should either prevent the DAG from being run or not show it's status as stalled) > Single node DAG's should not show node as stalling > -- > > Key: GEARPUMP-195 > URL: https://issues.apache.org/jira/browse/GEARPUMP-195 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.2 > > > DAG's with only one node should not show stalled. If a sink node is missing > we should either prevent the DAG from being run or not show the upstream node > status as stalled -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-198) empty kafka topics show very high receive throughput
[ https://issues.apache.org/jira/browse/GEARPUMP-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-198: - Summary: empty kafka topics show very high receive throughput (was: empty kafka topics show very high throughput) > empty kafka topics show very high receive throughput > > > Key: GEARPUMP-198 > URL: https://issues.apache.org/jira/browse/GEARPUMP-198 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Manu Zhang >Priority: Minor > Fix For: 0.8.2 > > > if the kafkasource reaches the end of the topic it shows very high message > throughput even though it's not really sending/receiving messages -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high
[
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kam Kasravi resolved GEARPUMP-185.
--
Resolution: Fixed
Fix Version/s: (was: 0.8.0)
0.8.2
> Yarn kills Apache Gearpump's worker if the memory usage is too high
> ---
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
> Issue Type: Bug
> Environment: yarn
>Reporter: Piotr Grabuszynski
>Assignee: Kam Kasravi
> Fix For: 0.8.2
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_03 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
> |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
> |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_03/stderr
>
> |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors'
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280 5.3 3.1 5406960 261356 pts/13 Sl+ 09:28 0:10
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=master
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417 7.7 1.8 5392212 154728 pts/27 Sl+ 09:29 0:05
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=worker
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0 3.1 4572132 255776 pts/27 Sl+ 09:33 0:11
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761 132 5.2 7268592 427200 pts/27 Sl+ 09:33 2:31
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 [...]
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 ...
> {code}
> h3. Steps to Reproduce
> To generate trafic I've used such application:
> {code}
> kafka
[jira] [Created] (GEARPUMP-200) A missing gear.conf in services and other gearpump commands should result in an ERROR to the log
Kam Kasravi created GEARPUMP-200: Summary: A missing gear.conf in services and other gearpump commands should result in an ERROR to the log Key: GEARPUMP-200 URL: https://issues.apache.org/jira/browse/GEARPUMP-200 Project: Apache Gearpump Issue Type: Bug Affects Versions: 0.8.1 Reporter: Kam Kasravi Assignee: Kam Kasravi Fix For: 0.8.2 Commands like services, master, local, etc are built with Config logic that uses gear.conf with fallback to geardefault.conf. If gear.conf is missing from the classpath of these commands we should issue a WARNING or ERROR and treat it not unlike a missing jar file since it is a critical dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high
[
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kam Kasravi updated GEARPUMP-185:
-
Comment: was deleted
(was: i'm able to confirm that the missing vmargs is related to the user
argument. Under the applications section of the dashboard UI - the vcap users
application's config had no vmargs. Launching a new application (which ran as
yarn) had a config that contained a valid vmargs for the executor)
> Yarn kills Apache Gearpump's worker if the memory usage is too high
> ---
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
> Issue Type: Bug
> Environment: yarn
>Reporter: Piotr Grabuszynski
>Assignee: Kam Kasravi
> Fix For: 0.8.0
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_03 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
> |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
> |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_03/stderr
>
> |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors'
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280 5.3 3.1 5406960 261356 pts/13 Sl+ 09:28 0:10
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=master
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417 7.7 1.8 5392212 154728 pts/27 Sl+ 09:29 0:05
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=worker
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0 3.1 4572132 255776 pts/27 Sl+ 09:33 0:11
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761 132 5.2 7268592 427200 pts/27 Sl+ 09:33 2:31
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
>
[jira] [Commented] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high
[
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452397#comment-15452397
]
Kam Kasravi commented on GEARPUMP-185:
--
The problem (and resolution) was that the services process CLASSPATH did not
include gear.conf (gear.conf was not copied into the conf subdirectory) when
this process was installed in TAP. Once gear.conf was copied into this
directory everything works as expected where the gear.conf used to build
gearpump commands (output/target/pack/bin) is used by all commands. I think a
reasonable improvement is a WARNING emitted to console if gear.conf is not
found.
> Yarn kills Apache Gearpump's worker if the memory usage is too high
> ---
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
> Issue Type: Bug
> Environment: yarn
>Reporter: Piotr Grabuszynski
>Assignee: Kam Kasravi
> Fix For: 0.8.0
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_03 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
> |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
> |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_03/stderr
>
> |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors'
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280 5.3 3.1 5406960 261356 pts/13 Sl+ 09:28 0:10
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=master
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417 7.7 1.8 5392212 154728 pts/27 Sl+ 09:29 0:05
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=worker
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0 3.1 4572132 255776 pts/27 Sl+ 09:33 0:11
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761 132 5.2 7268592 427200
[jira] [Updated] (GEARPUMP-199) high throughput results in high GC costs
[ https://issues.apache.org/jira/browse/GEARPUMP-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-199: - Attachment: Pasted image at 2016_08_31 01_50 AM.png this is a simple pipeline reading from kafkasource and converting the input > high throughput results in high GC costs > > > Key: GEARPUMP-199 > URL: https://issues.apache.org/jira/browse/GEARPUMP-199 > Project: Apache Gearpump > Issue Type: Improvement >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Manu Zhang > Fix For: 0.8.2 > > Attachments: Pasted image at 2016_08_31 01_50 AM.png > > > we should look to minimize GC costs - perhaps following what paypal squbs did > with persistentbuffer (also would be able to integrate with akka-streams) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-199) high throughput results in high GC costs
Kam Kasravi created GEARPUMP-199: Summary: high throughput results in high GC costs Key: GEARPUMP-199 URL: https://issues.apache.org/jira/browse/GEARPUMP-199 Project: Apache Gearpump Issue Type: Improvement Affects Versions: 0.8.1 Reporter: Kam Kasravi Assignee: Manu Zhang Fix For: 0.8.2 we should look to minimize GC costs - perhaps following what paypal squbs did with persistentbuffer (also would be able to integrate with akka-streams) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high
[
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kam Kasravi reassigned GEARPUMP-185:
Assignee: Kam Kasravi
> Yarn kills Apache Gearpump's worker if the memory usage is too high
> ---
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
> Issue Type: Bug
> Environment: yarn
>Reporter: Piotr Grabuszynski
>Assignee: Kam Kasravi
> Fix For: 0.8.0
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_03 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
> |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
> |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_03/stderr
>
> |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors'
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280 5.3 3.1 5406960 261356 pts/13 Sl+ 09:28 0:10
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=master
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417 7.7 1.8 5392212 154728 pts/27 Sl+ 09:29 0:05
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=worker
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0 3.1 4572132 255776 pts/27 Sl+ 09:33 0:11
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761 132 5.2 7268592 427200 pts/27 Sl+ 09:33 2:31
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 [...]
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 ...
> {code}
> h3. Steps to Reproduce
> To generate trafic I've used such application:
> {code}
> kafka source
> / \
>
[jira] [Commented] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high
[
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15447133#comment-15447133
]
Kam Kasravi commented on GEARPUMP-185:
--
for the yarn problem with the missing vmargs values … i looked at an executor
launched from yarn which is launched with a
-Dgearpump.config.file=/tmp/gearpump.conf argument. Looking in the tmp file
- there are no valid vmargs - it’s empty. Whereas standalone (also launched
with the /tmp/gearpump.conf file) the file has a valid vmargs. For yarn we
have a launch_container.sh for standalone we have a worker bash script. When
the /tmp/gearpump.conf file is created it uses
context.system.settings.config. In the case of yarn
context.system.settings.config does not contain what’s in gear.conf. I’m
thinking of an easy way to test this so launch_container.sh has the contents of
gear.conf.
> Yarn kills Apache Gearpump's worker if the memory usage is too high
> ---
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
> Issue Type: Bug
> Environment: yarn
>Reporter: Piotr Grabuszynski
> Fix For: 0.8.0
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_03 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
> |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
> |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_03/stderr
>
> |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors'
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280 5.3 3.1 5406960 261356 pts/13 Sl+ 09:28 0:10
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=master
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417 7.7 1.8 5392212 154728 pts/27 Sl+ 09:29 0:05
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=worker
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0 3.1 4572132 255776 pts/27 Sl+ 09:33 0:11
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80
[jira] [Updated] (GEARPUMP-198) empty kafka topics show very high throughput
[ https://issues.apache.org/jira/browse/GEARPUMP-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-198: - Assignee: Manu Zhang > empty kafka topics show very high throughput > > > Key: GEARPUMP-198 > URL: https://issues.apache.org/jira/browse/GEARPUMP-198 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.1 >Reporter: Kam Kasravi >Assignee: Manu Zhang >Priority: Minor > Fix For: 0.8.2 > > > if the kafkasource reaches the end of the topic it shows very high message > throughput even though it's not really sending/receiving messages -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-195) Single node DAG's should not show node as stalling
Kam Kasravi created GEARPUMP-195: Summary: Single node DAG's should not show node as stalling Key: GEARPUMP-195 URL: https://issues.apache.org/jira/browse/GEARPUMP-195 Project: Apache Gearpump Issue Type: Bug Affects Versions: 0.8.1 Reporter: Kam Kasravi Assignee: Kam Kasravi Fix For: 0.8.2 DAG's with only one node should not show stalled. If a sink node is missing we should either prevent the DAG from being run or not show it's status as stalled -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (GEARPUMP-187) update site to 0.8.1
[ https://issues.apache.org/jira/browse/GEARPUMP-187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi resolved GEARPUMP-187. -- Resolution: Fixed > update site to 0.8.1 > > > Key: GEARPUMP-187 > URL: https://issues.apache.org/jira/browse/GEARPUMP-187 > Project: Apache Gearpump > Issue Type: Task >Affects Versions: 0.8.0 >Reporter: Kam Kasravi > Fix For: 0.8.1 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (GEARPUMP-175) wrong download link (http://gearpump.apache.org/releases/latest/get-gearpump-distribution.html)
[
https://issues.apache.org/jira/browse/GEARPUMP-175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kam Kasravi resolved GEARPUMP-175.
--
Resolution: Fixed
Fix Version/s: 0.8.1
> wrong download link
> (http://gearpump.apache.org/releases/latest/get-gearpump-distribution.html)
> ---
>
> Key: GEARPUMP-175
> URL: https://issues.apache.org/jira/browse/GEARPUMP-175
> Project: Apache Gearpump
> Issue Type: Bug
>Reporter: Karol Brejna
>Assignee: Karol Brejna
>Priority: Minor
> Fix For: 0.8.1
>
>
> On page
> http://gearpump.apache.org/releases/latest/get-gearpump-distribution.html
> there is a fragment:
> {code}
> Download Release Binary
> If you choose to use pre-build package, then you don’t need to build from
> source code. The release package can be downloaded from:
> Download page
> {code}
> Download page link goes to http://gearpump.apache.org/download.html instead
> of http://gearpump.apache.org/downloads.html.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (GEARPUMP-189) Update project/Build.scala to enable publishing releases to https://repository.apache.org/content/repositories/releases/org/apache/gearpump/
Kam Kasravi created GEARPUMP-189: Summary: Update project/Build.scala to enable publishing releases to https://repository.apache.org/content/repositories/releases/org/apache/gearpump/ Key: GEARPUMP-189 URL: https://issues.apache.org/jira/browse/GEARPUMP-189 Project: Apache Gearpump Issue Type: Bug Affects Versions: 0.8.1 Reporter: Kam Kasravi Fix For: 0.8.2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-186) Create 0.8.1 Release
Kam Kasravi created GEARPUMP-186: Summary: Create 0.8.1 Release Key: GEARPUMP-186 URL: https://issues.apache.org/jira/browse/GEARPUMP-186 Project: Apache Gearpump Issue Type: Task Affects Versions: 0.8.0 Reporter: Kam Kasravi Assignee: Kam Kasravi Fix For: 0.8.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-182) Correct references required for source release in LICENSE file
Kam Kasravi created GEARPUMP-182: Summary: Correct references required for source release in LICENSE file Key: GEARPUMP-182 URL: https://issues.apache.org/jira/browse/GEARPUMP-182 Project: Apache Gearpump Issue Type: Task Affects Versions: 0.8.0 Reporter: Kam Kasravi Assignee: Kam Kasravi Fix For: 0.8.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-181) Create 0.8.1-RC4 release
Kam Kasravi created GEARPUMP-181: Summary: Create 0.8.1-RC4 release Key: GEARPUMP-181 URL: https://issues.apache.org/jira/browse/GEARPUMP-181 Project: Apache Gearpump Issue Type: Bug Reporter: Kam Kasravi We need to create this and revote on Sunday PST after this voting period is over. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-180) Add DISCLAIMER to release artifacts
[ https://issues.apache.org/jira/browse/GEARPUMP-180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-180: Assignee: Kam Kasravi > Add DISCLAIMER to release artifacts > --- > > Key: GEARPUMP-180 > URL: https://issues.apache.org/jira/browse/GEARPUMP-180 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.1 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-180) Add DISCLAIMER to release artifacts
Kam Kasravi created GEARPUMP-180: Summary: Add DISCLAIMER to release artifacts Key: GEARPUMP-180 URL: https://issues.apache.org/jira/browse/GEARPUMP-180 Project: Apache Gearpump Issue Type: Bug Affects Versions: 0.8.0 Reporter: Kam Kasravi Fix For: 0.8.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-177) Documentation build errors when trying to generate documentation
[
https://issues.apache.org/jira/browse/GEARPUMP-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369397#comment-15369397
]
Kam Kasravi commented on GEARPUMP-177:
--
Incubator disclaimer and logo will be handled within
https://github.com/apache/incubator-gearpump-site in a separate JIRA. It will
not prevent this release.
> Documentation build errors when trying to generate documentation
>
>
> Key: GEARPUMP-177
> URL: https://issues.apache.org/jira/browse/GEARPUMP-177
> Project: Apache Gearpump
> Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Kam Kasravi
>Assignee: Kam Kasravi
> Fix For: 0.8.1
>
>
> Running ./build_doc.sh 2.11 1 results in the following errors
> {code}
> - _site/basic-concepts.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/coding-style.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/commandline.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/deployment-configuration.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/deployment-docker.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/deployment-ha.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/deployment-local.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/deployment-msg-delivery.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/deployment-resource-isolation.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/deployment-security.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/deployment-standalone.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/deployment-ui-authentication.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/deployment-yarn.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/dev-connectors.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/dev-custom-serializer.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/dev-ide-setup.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/dev-non-streaming-example.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/dev-rest-api.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/dev-storm.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/dev-write-1st-app.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/faq.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/features.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/gearpump-internals.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/get-gearpump-distribution.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/hardware-requirement.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/index.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/maven-setting.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/message-delivery.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 116)
> - _site/performance-report.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> - _site/submit-your-1st-application.html
> * internally linking to how-to-contribute.html, which does not exist (line
> 119)
> htmlproof 2.5.1 | Error: HTML-Proofer found 30 failures!
> {code}
> and
> {code}
> [info] Resolving javax.servlet.jsp#jsp-api;2.1 ...
> [error]
> /Users/kamkasravi/release/incubator-gearpump/core/src/main/scala/org/apache/gearpump/metrics/Metrics.scala:26:
> object codahale is not a member of package org.apache.gearpump
> [error] import org.apache.gearpump.codahale.metrics._
> [error]^
> [info] Resolving
[jira] [Assigned] (GEARPUMP-163) CONTRIBUTING.md should be more "visible"
[ https://issues.apache.org/jira/browse/GEARPUMP-163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-163: Assignee: Kam Kasravi > CONTRIBUTING.md should be more "visible" > > > Key: GEARPUMP-163 > URL: https://issues.apache.org/jira/browse/GEARPUMP-163 > Project: Apache Gearpump > Issue Type: Improvement > Components: doc >Reporter: Karol Brejna >Assignee: Kam Kasravi > > Right now, going to gearpump.apache.org site, we can choose menu "more" and > then 'How to contribute'. > It presents some general information on contribution: > http://gearpump.apache.org/releases/latest/how-to-contribute.html > The doc that explains the procedure in details is here: > https://github.com/apache/incubator-gearpump/blob/master/CONTRIBUTING.md > It in the sources, but it is not visible in the docs. > I think we should at least link the docs in general contributing doc. Or make > it "first class citizen" in the docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-175) wrong download link (http://gearpump.apache.org/releases/latest/get-gearpump-distribution.html)
[
https://issues.apache.org/jira/browse/GEARPUMP-175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366380#comment-15366380
]
Kam Kasravi commented on GEARPUMP-175:
--
Thanks [~karol_brejna] - we'll need to merge the related PR into
incubator-gearpump-site once the release is approved
> wrong download link
> (http://gearpump.apache.org/releases/latest/get-gearpump-distribution.html)
> ---
>
> Key: GEARPUMP-175
> URL: https://issues.apache.org/jira/browse/GEARPUMP-175
> Project: Apache Gearpump
> Issue Type: Bug
>Reporter: Karol Brejna
>Assignee: Karol Brejna
>Priority: Minor
>
> On page
> http://gearpump.apache.org/releases/latest/get-gearpump-distribution.html
> there is a fragment:
> {code}
> Download Release Binary
> If you choose to use pre-build package, then you don’t need to build from
> source code. The release package can be downloaded from:
> Download page
> {code}
> Download page link goes to http://gearpump.apache.org/download.html instead
> of http://gearpump.apache.org/downloads.html.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (GEARPUMP-176) Create 0.8.1-RC2 release
Kam Kasravi created GEARPUMP-176: Summary: Create 0.8.1-RC2 release Key: GEARPUMP-176 URL: https://issues.apache.org/jira/browse/GEARPUMP-176 Project: Apache Gearpump Issue Type: Task Affects Versions: 0.8.0 Reporter: Kam Kasravi Assignee: Kam Kasravi Fix For: 0.8.1 identified akka version mismatch in kyro shaded library -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (GEARPUMP-159) Prep for first release
[ https://issues.apache.org/jira/browse/GEARPUMP-159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi resolved GEARPUMP-159. -- Resolution: Fixed > Prep for first release > -- > > Key: GEARPUMP-159 > URL: https://issues.apache.org/jira/browse/GEARPUMP-159 > Project: Apache Gearpump > Issue Type: Task >Affects Versions: 0.8.0 >Reporter: Kam Kasravi > Fix For: 0.8.1 > > > Per the release process: http://www.apache.org/dev/release-publishing.html we > need to generate a release key, create a KEYS file, probably redo the > ChangeLog, possibly restart the versioning numbers. We should document the > process here with intent to add update the existing release document > (https://github.com/apache/incubator-gearpump/blob/master/ReleaseProcess.md) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-170) Component dependencies should resolve to apache repo
[ https://issues.apache.org/jira/browse/GEARPUMP-170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-170: - Summary: Component dependencies should resolve to apache repo (was: Create 0.8.1 release) > Component dependencies should resolve to apache repo > > > Key: GEARPUMP-170 > URL: https://issues.apache.org/jira/browse/GEARPUMP-170 > Project: Apache Gearpump > Issue Type: Task > Components: all >Affects Versions: 0.8.0 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.1 > > > Commit any changes and version along with tagging. This will be closed when > the VOTE succeeds -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-170) Create 0.8.1 release
Kam Kasravi created GEARPUMP-170: Summary: Create 0.8.1 release Key: GEARPUMP-170 URL: https://issues.apache.org/jira/browse/GEARPUMP-170 Project: Apache Gearpump Issue Type: Task Components: all Affects Versions: 0.8.0 Reporter: Kam Kasravi Assignee: Kam Kasravi Fix For: 0.8.1 Commit any changes and version along with tagging. This will be closed when the VOTE succeeds -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (GEARPUMP-30) Track call for papers submissions, talks and upcoming CFP's somewhere
[ https://issues.apache.org/jira/browse/GEARPUMP-30?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi closed GEARPUMP-30. --- Resolution: Fixed Fix Version/s: 0.8.1 > Track call for papers submissions, talks and upcoming CFP's somewhere > -- > > Key: GEARPUMP-30 > URL: https://issues.apache.org/jira/browse/GEARPUMP-30 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Kam Kasravi >Assignee: Kam Kasravi >Priority: Minor > Fix For: 0.8.1 > > > We can probably put this on the website including slides that were used for > the talk (or a pointer to the talk) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-159) Prep for first release
[ https://issues.apache.org/jira/browse/GEARPUMP-159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334826#comment-15334826 ] Kam Kasravi commented on GEARPUMP-159: -- We should also include LICENSE and NOTICE files as noted in GEARPUMP-157 > Prep for first release > -- > > Key: GEARPUMP-159 > URL: https://issues.apache.org/jira/browse/GEARPUMP-159 > Project: Apache Gearpump > Issue Type: Task >Affects Versions: 0.8.0 >Reporter: Kam Kasravi > Fix For: 0.8.1 > > > Per the release process: http://www.apache.org/dev/release-publishing.html we > need to generate a release key, create a KEYS file, probably redo the > ChangeLog, possibly restart the versioning numbers. We should document the > process here with intent to add update the existing release document > (https://github.com/apache/incubator-gearpump/blob/master/ReleaseProcess.md) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-156) Replace logo in site, GitHub, Travis, etc
Kam Kasravi created GEARPUMP-156: Summary: Replace logo in site, GitHub, Travis, etc Key: GEARPUMP-156 URL: https://issues.apache.org/jira/browse/GEARPUMP-156 Project: Apache Gearpump Issue Type: Bug Reporter: Kam Kasravi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-95) Add parquet datasource and datasink connectors
[ https://issues.apache.org/jira/browse/GEARPUMP-95?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-95: Fix Version/s: (was: 0.8.1) > Add parquet datasource and datasink connectors > -- > > Key: GEARPUMP-95 > URL: https://issues.apache.org/jira/browse/GEARPUMP-95 > Project: Apache Gearpump > Issue Type: New Feature > Components: hadoop >Reporter: Manu Zhang >Assignee: Kam Kasravi > > imported from [https://github.com/gearpump/gearpump/issues/1279] on behalf of > [~kkasravi], > Define these connectors in external. These connectors should be able to read > from local or hdfs (any valid uri) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-42) Handle YARN Resource Manager restarts
[ https://issues.apache.org/jira/browse/GEARPUMP-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-42: Assignee: Tomasz Targonski > Handle YARN Resource Manager restarts > -- > > Key: GEARPUMP-42 > URL: https://issues.apache.org/jira/browse/GEARPUMP-42 > Project: Apache Gearpump > Issue Type: Improvement > Components: yarn >Affects Versions: 0.8.0 >Reporter: Kam Kasravi >Assignee: Tomasz Targonski >Priority: Minor > > YARN resource manager may restart in 2 ways: > * Non-work preserving > * Work preserving > In both causes the Application Master needs to handle specific coordination > messages sent from the Resource Manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-140) Provide DAG path latency statistics
[ https://issues.apache.org/jira/browse/GEARPUMP-140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-140: - Issue Type: Sub-task (was: New Feature) Parent: GEARPUMP-46 > Provide DAG path latency statistics > --- > > Key: GEARPUMP-140 > URL: https://issues.apache.org/jira/browse/GEARPUMP-140 > Project: Apache Gearpump > Issue Type: Sub-task > Components: Dashboard, restapi, streaming >Affects Versions: 0.8.0 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.1 > > > Related to GEARPUMP-46 where the frontend has been implemented. The backend > needs to provide this information via a REST API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-112) DAG chart in dashboard does not show data, when flow is small
[ https://issues.apache.org/jira/browse/GEARPUMP-112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-112: - Assignee: Karol Brejna (was: Kam Kasravi) > DAG chart in dashboard does not show data, when flow is small > - > > Key: GEARPUMP-112 > URL: https://issues.apache.org/jira/browse/GEARPUMP-112 > Project: Apache Gearpump > Issue Type: Bug > Components: Dashboard >Affects Versions: 0.8.0 >Reporter: Piotr Grabuszynski >Assignee: Karol Brejna > Attachments: gp-flow.jpg, gp-screen-2.jpg, gp-screen-metrics.jpg, > gp-screen-overview.jpg > > > BUG: > After deploying application on Apache Gearpump, DAG chart and data near it, > shows always the same value, and lines and nodes on chart doesn't grow if the > flow is about 3 requests / second. > WAY TO REPRODUCE: > 1. Deploy application on Apache Gearpump. > 2. To input kafka topic send ~3 requests/sec. > 3. Go to dashboard, choose your app and go to DAG tab. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-22) Support akka-streams Gearpump Materializer
[ https://issues.apache.org/jira/browse/GEARPUMP-22?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-22: Summary: Support akka-streams Gearpump Materializer (was: support akka-streams Gearpump Materializer) > Support akka-streams Gearpump Materializer > -- > > Key: GEARPUMP-22 > URL: https://issues.apache.org/jira/browse/GEARPUMP-22 > Project: Apache Gearpump > Issue Type: New Feature > Components: akkastream >Affects Versions: 0.8.0 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.1 > > > Reenable experiments/akkastream to work with akka 2.4.3 (latest on master) > Create PR's for akka team related to changes required for Gearpump > Materializer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-78) Support DAG with cycles
[ https://issues.apache.org/jira/browse/GEARPUMP-78?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-78: Summary: Support DAG with cycles (was: Support DAG with loops) > Support DAG with cycles > --- > > Key: GEARPUMP-78 > URL: https://issues.apache.org/jira/browse/GEARPUMP-78 > Project: Apache Gearpump > Issue Type: Improvement > Components: streaming >Reporter: Manu Zhang >Assignee: Manu Zhang > > imported from [https://github.com/gearpump/gearpump/issues/1231] on behalf of > [~clockfly], > DAG with loop should not be called DAG :) > We see use cases that require a feedback channel. > The challengues will be: > # how to support the low watermark clock? As the clock is defined recursively? > # How to support back-pressure. DAG with loop can possiblely create a > live-lock issue. With each processor wait for acks from other processor > before sending acks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-124) SinkTask fails to start after change parallelism
[ https://issues.apache.org/jira/browse/GEARPUMP-124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-124: - Summary: SinkTask fails to start after change parallelism (was: SinkTask fail to start after change parallelism) > SinkTask fails to start after change parallelism > > > Key: GEARPUMP-124 > URL: https://issues.apache.org/jira/browse/GEARPUMP-124 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Qi Shu >Assignee: Huafeng Wang > > App started, run for a while, everything is ok, change the parallelism of > sink(our own hdfs sink), then the sink got following exceptions: > [INFO] [05/11/2016 17:22:30.865] [Executor@app2exec23] Transit to state > Application Ready. This transition takes 99671 milliseconds > [ERROR] [05/11/2016 17:22:30.865] [Executor@app2exec23] We got > java.lang.reflect.InvocationTargetException from Some(TaskId(6,0)), we will > treat it as MessageLoss, so that the system will replay all lost message > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at io.gearpump.streaming.task.TaskWrapper.onStart(TaskWrapper.scala:89) > at io.gearpump.streaming.task.TaskActor.onStart(TaskActor.scala:102) > at > io.gearpump.streaming.task.TaskActor.io$gearpump$streaming$task$TaskActor$$onStartClock(TaskActor.scala:194) > at > io.gearpump.streaming.task.TaskActor$$anonfun$waitForStartClock$1.applyOrElse(TaskActor.scala:211) > at akka.actor.Actor$class.aroundReceive(Actor.scala:480) > at > io.gearpump.streaming.task.TaskActor.aroundReceive(TaskActor.scala:41) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) > at akka.actor.ActorCell.invoke(ActorCell.scala:495) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) > at akka.dispatch.Mailbox.run(Mailbox.scala:224) > at akka.dispatch.Mailbox.exec(Mailbox.scala:234) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:347) > at scala.None$.get(Option.scala:345) > at io.gearpump.streaming.sink.DataSinkTask.(DataSinkTask.scala:35) > ... 19 more > [WARN] [05/11/2016 17:22:30.867] [OneForOneStrategy] null > [INFO] [05/11/2016 17:22:30.877] [Executor@app2exec23] Executor received > restart tasks > [INFO] [05/11/2016 17:22:30.878] [Express$] RegisterLocalActor: 25769803776, > actor: processor_6_task_0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-140) Provide DAG path latency statistics
Kam Kasravi created GEARPUMP-140: Summary: Provide DAG path latency statistics Key: GEARPUMP-140 URL: https://issues.apache.org/jira/browse/GEARPUMP-140 Project: Apache Gearpump Issue Type: New Feature Components: Dashboard, restapi, streaming Affects Versions: 0.8.0 Reporter: Kam Kasravi Assignee: Kam Kasravi Fix For: 0.8.1 Related to GEARPUMP-46 where the frontend has been implemented. The backend needs to provide this information via a REST API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-78) Support DAG with loops
[ https://issues.apache.org/jira/browse/GEARPUMP-78?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-78: Assignee: Manu Zhang > Support DAG with loops > -- > > Key: GEARPUMP-78 > URL: https://issues.apache.org/jira/browse/GEARPUMP-78 > Project: Apache Gearpump > Issue Type: Improvement > Components: streaming >Reporter: Manu Zhang >Assignee: Manu Zhang > > imported from [https://github.com/gearpump/gearpump/issues/1231] on behalf of > [~clockfly], > DAG with loop should not be called DAG :) > We see use cases that require a feedback channel. > The challengues will be: > # how to support the low watermark clock? As the clock is defined recursively? > # How to support back-pressure. DAG with loop can possiblely create a > live-lock issue. With each processor wait for acks from other processor > before sending acks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-83) After killing all worker instances, application status should not be described as active
[
https://issues.apache.org/jira/browse/GEARPUMP-83?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kam Kasravi reassigned GEARPUMP-83:
---
Assignee: Kam Kasravi
> After killing all worker instances, application status should not be
> described as active
>
>
> Key: GEARPUMP-83
> URL: https://issues.apache.org/jira/browse/GEARPUMP-83
> Project: Apache Gearpump
> Issue Type: Bug
> Components: Dashboard
>Affects Versions: 0.8.0
>Reporter: Kam Kasravi
>Assignee: Kam Kasravi
>Priority: Minor
> Fix For: 0.8.1
>
>
> Step to reproduce:
> Start cluster with one worker
> Start a word count
> Kill the worker
> Expect /api/v1.0/master/applist actually returns app status as active, but
> application's detail page is not available. I think as there is no resource
> to run the application, the application is in some abnormal status. In order
> not to mislead user, I think we should invent a new status, might be
> recovering or something.
> Example output:
> {code}
> {"appMasters":[{"status":"active","appId":1,"appName":"dag","appMasterPath":"akka.tcp://app1-executor-1@127.0.0.1:46761/user/daemon/appdaemon1/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451894551477","startTime":"1451894553568","user":"qxu"},{"status":"active","appId":2,"appName":"wordCount","appMasterPath":"akka.tcp://app2-executor-1@127.0.0.1:49261/user/daemon/appdaemon2/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451898038991","startTime":"1451898040265","user":"qxu"}]}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-91) Apache Cassandra Integration
[ https://issues.apache.org/jira/browse/GEARPUMP-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282808#comment-15282808 ] Kam Kasravi commented on GEARPUMP-91: - Hi [~zapletal-martin] Were you still planning on implementing this? It could be a candidate for the next release (due in ~2 weeks). Let us know and we can reassign if needed. Thanks Kam > Apache Cassandra Integration > - > > Key: GEARPUMP-91 > URL: https://issues.apache.org/jira/browse/GEARPUMP-91 > Project: Apache Gearpump > Issue Type: New Feature >Reporter: Manu Zhang >Assignee: Martin Zapletal > > [Apache Cassandra|http://cassandra.apache.org/] is a scalable and highly > available distributed database. It is widely used as data stores and data > sinks in a data pipeline. > Imported from https://github.com/gearpump/gearpump/issues/1991 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-92) Able to see history job status
[ https://issues.apache.org/jira/browse/GEARPUMP-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282803#comment-15282803 ] Kam Kasravi commented on GEARPUMP-92: - [~clockfly] [~mauzhang] Don't we purge this from ApplicationMaster? I'm not sure we persist the codehale metrics anywhere. I can investigate. > Able to see history job status > -- > > Key: GEARPUMP-92 > URL: https://issues.apache.org/jira/browse/GEARPUMP-92 > Project: Apache Gearpump > Issue Type: New Feature > Components: Dashboard >Reporter: Manu Zhang >Priority: Minor > > Allow users to view history metrics data even an application has been killed, > which will be valuable to help users to do performance comparisons. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-92) Able to see history job status
[ https://issues.apache.org/jira/browse/GEARPUMP-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-92: --- Assignee: Kam Kasravi > Able to see history job status > -- > > Key: GEARPUMP-92 > URL: https://issues.apache.org/jira/browse/GEARPUMP-92 > Project: Apache Gearpump > Issue Type: New Feature > Components: Dashboard >Reporter: Manu Zhang >Assignee: Kam Kasravi >Priority: Minor > > Allow users to view history metrics data even an application has been killed, > which will be valuable to help users to do performance comparisons. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-94) pretty print configs on dashboard
[ https://issues.apache.org/jira/browse/GEARPUMP-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282799#comment-15282799 ] Kam Kasravi commented on GEARPUMP-94: - [~darion] Did you want to possibly include this. We would prefer angular components if possible. I can pick this up otherwise. Let me know. > pretty print configs on dashboard > - > > Key: GEARPUMP-94 > URL: https://issues.apache.org/jira/browse/GEARPUMP-94 > Project: Apache Gearpump > Issue Type: New Feature > Components: Dashboard >Reporter: Manu Zhang >Priority: Minor > > Config files are shown in raw format on dashboard. It will more user friendly > if we display configs with pretty print. > It will be even better if users could search/filter through configs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-95) Add parquet datasource and datasink connectors
[ https://issues.apache.org/jira/browse/GEARPUMP-95?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-95: --- Assignee: Kam Kasravi > Add parquet datasource and datasink connectors > -- > > Key: GEARPUMP-95 > URL: https://issues.apache.org/jira/browse/GEARPUMP-95 > Project: Apache Gearpump > Issue Type: New Feature > Components: hadoop >Reporter: Manu Zhang >Assignee: Kam Kasravi > > imported from [https://github.com/gearpump/gearpump/issues/1279] on behalf of > [~kkasravi], > Define these connectors in external. These connectors should be able to read > from local or hdfs (any valid uri) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-105) Provide non-persistent Sink Task so that examples like word count can materialize Sum results within the Client
[ https://issues.apache.org/jira/browse/GEARPUMP-105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-105: Assignee: Kam Kasravi > Provide non-persistent Sink Task so that examples like word count can > materialize Sum results within the Client > --- > > Key: GEARPUMP-105 > URL: https://issues.apache.org/jira/browse/GEARPUMP-105 > Project: Apache Gearpump > Issue Type: New Feature > Components: streaming >Affects Versions: 0.8.0 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.1 > > > Gearpump's application model is to submit a DAG to master for execution but > doesn't really provide a way to materialize any Sink nodes within the > application (client). We should provide this capability and add it to > wordcount so that it has functionality similar to flink and spark > (interactive mode). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-107) Stream DSL three-way merge not supported
[ https://issues.apache.org/jira/browse/GEARPUMP-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-107: - Assignee: Manu Zhang > Stream DSL three-way merge not supported > > > Key: GEARPUMP-107 > URL: https://issues.apache.org/jira/browse/GEARPUMP-107 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Manu Zhang >Assignee: Manu Zhang > > Three-way merge or more-than-three-way merge is decomposed into multiple > consecutive two-way merges. The current implementation assumes that a merge > will always be tailed by a SingleInputFunction which is not true for > consecutive merges. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-111) optimize Stream DSL broadcasting slave operations
[
https://issues.apache.org/jira/browse/GEARPUMP-111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kam Kasravi updated GEARPUMP-111:
-
Assignee: Manu Zhang
> optimize Stream DSL broadcasting slave operations
> -
>
> Key: GEARPUMP-111
> URL: https://issues.apache.org/jira/browse/GEARPUMP-111
> Project: Apache Gearpump
> Issue Type: Improvement
> Components: streaming
>Reporter: Manu Zhang
>Assignee: Manu Zhang
>Priority: Minor
>
> In Stream DSL, all broadcasting slave operations tailing a master operation
> will be translated to individual tasks.
> For example,
> {code}
> val stream = StreamApp.source(source, 1, "source")
> stream.filter(fn, "filter A")
> stream.filter(fn, "filter B")
> stream.filter(fn, "filter C")
> {code}
> This will be translated to 1 SourceTask and 3 FilterTasks although it's
> possible for them to be done in 1 Task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-112) DAG chart in dashboard does not show data, when flow is small
[ https://issues.apache.org/jira/browse/GEARPUMP-112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-112: Assignee: Kam Kasravi > DAG chart in dashboard does not show data, when flow is small > - > > Key: GEARPUMP-112 > URL: https://issues.apache.org/jira/browse/GEARPUMP-112 > Project: Apache Gearpump > Issue Type: Bug > Components: Dashboard >Affects Versions: 0.8.0 >Reporter: Piotr Grabuszynski >Assignee: Kam Kasravi > Attachments: gp-flow.jpg, gp-screen-2.jpg, gp-screen-metrics.jpg, > gp-screen-overview.jpg > > > BUG: > After deploying application on Apache Gearpump, DAG chart and data near it, > shows always the same value, and lines and nodes on chart doesn't grow if the > flow is about 3 requests / second. > WAY TO REPRODUCE: > 1. Deploy application on Apache Gearpump. > 2. To input kafka topic send ~3 requests/sec. > 3. Go to dashboard, choose your app and go to DAG tab. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-114) Dead loop in graph with circles
[ https://issues.apache.org/jira/browse/GEARPUMP-114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-114: - Assignee: Manu Zhang > Dead loop in graph with circles > --- > > Key: GEARPUMP-114 > URL: https://issues.apache.org/jira/browse/GEARPUMP-114 > Project: Apache Gearpump > Issue Type: Improvement > Components: core >Reporter: Yu Gong >Assignee: Manu Zhang >Priority: Minor > > Function `vertexHierarchyLevelMap()` in class `io.gearpump.util.Graph` may > cause dead loop when there are circles in graph. See > https://github.com/apache/incubator-gearpump/blob/master/core/src/main/scala/io/gearpump/util/Graph.scala#L380. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-123) Metrics about workers is not accurate in UI
[ https://issues.apache.org/jira/browse/GEARPUMP-123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-123: - Assignee: Manu Zhang > Metrics about workers is not accurate in UI > --- > > Key: GEARPUMP-123 > URL: https://issues.apache.org/jira/browse/GEARPUMP-123 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Qi Shu >Assignee: Manu Zhang > Attachments: screenshot-1.png > > > This issue is connected with > https://issues.apache.org/jira/browse/GEARPUMP-124. > After GEARPUMP-124 happened, kill the app through UI. After a while, all > executors are stopped running, but the metrics shows some executos are still > there, as "screenshot-1.png" shows, and I also checked the processes in > servers, there are no ActorSystemBooter running any more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-115) Create MQTT source/sink
[ https://issues.apache.org/jira/browse/GEARPUMP-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-115: Assignee: Kam Kasravi > Create MQTT source/sink > --- > > Key: GEARPUMP-115 > URL: https://issues.apache.org/jira/browse/GEARPUMP-115 > Project: Apache Gearpump > Issue Type: New Feature >Affects Versions: 0.8.1 >Reporter: Karol Brejna >Assignee: Kam Kasravi >Priority: Minor > > MQTT is a machine-to-machine (M2M)/Internet of Things connectivity protocol. > MQTT v3.1.1 has now become an OASIS Standard. > It is useful for connections with remote locations where a small code > footprint is required and/or network bandwidth is at a premium. > MQTT provides publish/subscribe mechanism. > There is a mature MQTT Java client from Eclipse Paho project > (http://www.eclipse.org/paho/): https://github.com/eclipse/paho.mqtt.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-93) Allow admin to add/remove a worker, add/remove a master
[ https://issues.apache.org/jira/browse/GEARPUMP-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282786#comment-15282786 ] Kam Kasravi commented on GEARPUMP-93: - [~HuafengWang] yes - though we should design it so it will work with other resource managers like mesos > Allow admin to add/remove a worker, add/remove a master > --- > > Key: GEARPUMP-93 > URL: https://issues.apache.org/jira/browse/GEARPUMP-93 > Project: Apache Gearpump > Issue Type: New Feature > Components: core, Dashboard >Reporter: Manu Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-126) Evaluate apache build services for CI
[ https://issues.apache.org/jira/browse/GEARPUMP-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-126: - Summary: Evaluate apache build services for CI (was: Evaluation apache build services for CI) > Evaluate apache build services for CI > - > > Key: GEARPUMP-126 > URL: https://issues.apache.org/jira/browse/GEARPUMP-126 > Project: Apache Gearpump > Issue Type: Bug > Components: all >Affects Versions: 0.8.0 >Reporter: Kam Kasravi >Assignee: Kam Kasravi >Priority: Minor > Fix For: 0.8.1 > > > https://ci.apache.org/ provides automated builds including jenkins. A number > of projects like beam, kafka are using this service. It may also have some > integration in automating staging and release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (GEARPUMP-26) Doc: 0.8.0 Release does not include scala 2.10 build
[
https://issues.apache.org/jira/browse/GEARPUMP-26?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kam Kasravi resolved GEARPUMP-26.
-
Resolution: Fixed
> Doc: 0.8.0 Release does not include scala 2.10 build
>
>
> Key: GEARPUMP-26
> URL: https://issues.apache.org/jira/browse/GEARPUMP-26
> Project: Apache Gearpump
> Issue Type: Bug
> Components: doc
>Affects Versions: 0.8.0
>Reporter: Sean Zhong
> Fix For: 0.8.1
>
>
> Reported by vishalkuo
> at: https://github.com/gearpump/gearpump/issues/2033
> {quote}
> https://github.com/gearpump/gearpump/releases/tag/0.8.0 does not contain a
> release for scala 2.10, as a result, following the download link from
> http://www.gearpump.io/download.html results in a 404. Not sure if this is
> the right place to be raising this, I just figured it ought to be documented
> somewhere.
> {quote}
> The release note of 0.8 says it drops supports for scala2.10
> https://github.com/gearpump/gearpump/releases/tag/0.8.0
> We should make sure the doc site gearpump.io is updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-26) Doc: 0.8.0 Release does not include scala 2.10 build
[
https://issues.apache.org/jira/browse/GEARPUMP-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282495#comment-15282495
]
Kam Kasravi commented on GEARPUMP-26:
-
documented now on website that we do not support 2.10 due to akka 2.4.+
> Doc: 0.8.0 Release does not include scala 2.10 build
>
>
> Key: GEARPUMP-26
> URL: https://issues.apache.org/jira/browse/GEARPUMP-26
> Project: Apache Gearpump
> Issue Type: Bug
> Components: doc
>Affects Versions: 0.8.0
>Reporter: Sean Zhong
> Fix For: 0.8.1
>
>
> Reported by vishalkuo
> at: https://github.com/gearpump/gearpump/issues/2033
> {quote}
> https://github.com/gearpump/gearpump/releases/tag/0.8.0 does not contain a
> release for scala 2.10, as a result, following the download link from
> http://www.gearpump.io/download.html results in a 404. Not sure if this is
> the right place to be raising this, I just figured it ought to be documented
> somewhere.
> {quote}
> The release note of 0.8 says it drops supports for scala2.10
> https://github.com/gearpump/gearpump/releases/tag/0.8.0
> We should make sure the doc site gearpump.io is updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-54) Composing dag from dashboard, user can not specify the application name
[ https://issues.apache.org/jira/browse/GEARPUMP-54?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-54: --- Assignee: Kam Kasravi > Composing dag from dashboard, user can not specify the application name > --- > > Key: GEARPUMP-54 > URL: https://issues.apache.org/jira/browse/GEARPUMP-54 > Project: Apache Gearpump > Issue Type: Bug > Components: Dashboard >Reporter: Huafeng Wang >Assignee: Kam Kasravi > > Since Gearpump will reject the application if the application name is already > existed, user can not compose two different dags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (GEARPUMP-64) Fail to submit an application from frontend UI when setting HDFS as jarStore
[ https://issues.apache.org/jira/browse/GEARPUMP-64?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi reassigned GEARPUMP-64: --- Assignee: Kam Kasravi > Fail to submit an application from frontend UI when setting HDFS as jarStore > > > Key: GEARPUMP-64 > URL: https://issues.apache.org/jira/browse/GEARPUMP-64 > Project: Apache Gearpump > Issue Type: Bug >Reporter: Huafeng Wang >Assignee: Kam Kasravi > > The cause is that the classpath of dashboard does not contains hadoop > libraries. > Migrated from https://github.com/gearpump/gearpump/issues/1346 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (GEARPUMP-82) Rename package from io.gearpump to org.apache.gearpump
[ https://issues.apache.org/jira/browse/GEARPUMP-82?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi closed GEARPUMP-82. --- Resolution: Fixed > Rename package from io.gearpump to org.apache.gearpump > -- > > Key: GEARPUMP-82 > URL: https://issues.apache.org/jira/browse/GEARPUMP-82 > Project: Apache Gearpump > Issue Type: Bug > Components: all >Affects Versions: 0.8.0 >Reporter: Kam Kasravi > Fix For: 0.8.1 > > > change code and update documentation if necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-126) Evaluation apache build services for CI
Kam Kasravi created GEARPUMP-126: Summary: Evaluation apache build services for CI Key: GEARPUMP-126 URL: https://issues.apache.org/jira/browse/GEARPUMP-126 Project: Apache Gearpump Issue Type: Bug Components: all Affects Versions: 0.8.0 Reporter: Kam Kasravi Assignee: Kam Kasravi Priority: Minor Fix For: 0.8.1 https://ci.apache.org/ provides automated builds including jenkins. A number of projects like beam, kafka are using this service. It may also have some integration in automating staging and release -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-123) Metrics about works is not accurate in UI
[ https://issues.apache.org/jira/browse/GEARPUMP-123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280238#comment-15280238 ] Kam Kasravi commented on GEARPUMP-123: -- Once you've submitted the application, the distributed tasks run forever in executors. Do you mean you've killed the application master or the client? > Metrics about works is not accurate in UI > - > > Key: GEARPUMP-123 > URL: https://issues.apache.org/jira/browse/GEARPUMP-123 > Project: Apache Gearpump > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Qi Shu > Attachments: screenshot-1.png > > > App started, change the parallelism of one task, then kill the app. After a > while, all executors are stopped running, but the metrics shows some executos > are still there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-30) Track call for papers submissions, talks and upcoming CFP's somewhere
[ https://issues.apache.org/jira/browse/GEARPUMP-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280010#comment-15280010 ] Kam Kasravi commented on GEARPUMP-30: - Strata Singapore Dec 6-8. Call for speakers ends 6/21 http://conferences.oreilly.com/strata/hadoop-big-data-sg/public/cfp/475?imm_mid=0e37bf=em-data-confpro-na-stsg16_em1_cfp > Track call for papers submissions, talks and upcoming CFP's somewhere > -- > > Key: GEARPUMP-30 > URL: https://issues.apache.org/jira/browse/GEARPUMP-30 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Kam Kasravi >Assignee: Kam Kasravi >Priority: Minor > > We can probably put this on the website including slides that were used for > the talk (or a pointer to the talk) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-18) Enable publishing gearpump artifacts to repository.apache.org
[ https://issues.apache.org/jira/browse/GEARPUMP-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277953#comment-15277953 ] Kam Kasravi commented on GEARPUMP-18: - The line i mentioned in .travis.yml is related to travis deployment using the github provider: releases. The secret is a github auth token so it doesn't need to be changed. I'm unsure how SONATYPE_USERNAME and SONATYPE_PASSWORD are set since it's not done in .travis.yml. We should try to build on travis and see if we're getting snapshots generated. > Enable publishing gearpump artifacts to repository.apache.org > - > > Key: GEARPUMP-18 > URL: https://issues.apache.org/jira/browse/GEARPUMP-18 > Project: Apache Gearpump > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Kam Kasravi >Assignee: Kam Kasravi > Fix For: 0.8.1 > > > Enable publishing gearpump artifacts to maven. > Process is described > [here|http://www.apache.org/dev/publishing-maven-artifacts.html] > Ticket has been opened > [here|https://issues.apache.org/jira/browse/INFRA-11607] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-105) Provide non-persistent Sink Task so that examples like word count can materialize Sum results within the Client
Kam Kasravi created GEARPUMP-105: Summary: Provide non-persistent Sink Task so that examples like word count can materialize Sum results within the Client Key: GEARPUMP-105 URL: https://issues.apache.org/jira/browse/GEARPUMP-105 Project: Apache Gearpump Issue Type: New Feature Components: streaming Affects Versions: 0.8.0 Reporter: Kam Kasravi Fix For: 0.8.1 Gearpump's application model is to submit a DAG to master for execution but doesn't really provide a way to materialize any Sink nodes within the application (client). We should provide this capability and add it to wordcount so that it has functionality similar to flink and spark (interactive mode). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
