[jira] [Updated] (GEARPUMP-139) Change package name to org.apache.gearpump of shaded libraries
[ https://issues.apache.org/jira/browse/GEARPUMP-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huafeng Wang updated GEARPUMP-139: -- Fix Version/s: 0.8.1 > Change package name to org.apache.gearpump of shaded libraries > -- > > Key: GEARPUMP-139 > URL: https://issues.apache.org/jira/browse/GEARPUMP-139 > Project: Apache Gearpump > Issue Type: Task >Reporter: Huafeng Wang >Assignee: Huafeng Wang >Priority: Minor > Fix For: 0.8.1 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GEARPUMP-95) Add parquet datasource and datasink connectors
[ https://issues.apache.org/jira/browse/GEARPUMP-95?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kam Kasravi updated GEARPUMP-95: Fix Version/s: (was: 0.8.1) > Add parquet datasource and datasink connectors > -- > > Key: GEARPUMP-95 > URL: https://issues.apache.org/jira/browse/GEARPUMP-95 > Project: Apache Gearpump > Issue Type: New Feature > Components: hadoop >Reporter: Manu Zhang >Assignee: Kam Kasravi > > imported from [https://github.com/gearpump/gearpump/issues/1279] on behalf of > [~kkasravi], > Define these connectors in external. These connectors should be able to read > from local or hdfs (any valid uri) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-40) KafkaSource poor performance
[ https://issues.apache.org/jira/browse/GEARPUMP-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313616#comment-15313616 ] Karol Brejna commented on GEARPUMP-40: -- [~mauzhang] Do you think https://github.com/apache/incubator-gearpump/pull/25 could improve the results? did you have a chance to reproduce the test after the changes? > KafkaSource poor performance > > > Key: GEARPUMP-40 > URL: https://issues.apache.org/jira/browse/GEARPUMP-40 > Project: Apache Gearpump > Issue Type: Improvement > Components: kafka >Affects Versions: 0.8.0 >Reporter: Manu Zhang >Assignee: Manu Zhang > > As per > [https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines], > a single thread Kafka consumer could consume at 89.7MB/s from 6x partition > 3x replica topic whose data are evenly distributed on a 3-node GbE cluster. I > carried out a similar experiment with KafkaSource and found that the > throughput is only at 10MB/s. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-gearpump pull request #33: fix GEARPUMP-32, introduce source water...
GitHub user manuzhang opened a pull request: https://github.com/apache/incubator-gearpump/pull/33 fix GEARPUMP-32, introduce source watermark This PR introduces source watermark such that 1. messages with timestamps earlier than watermark could be filtered out at source. 2. minclock of source processor will be bounded by watemark You can merge this pull request into a Git repository by running: $ git pull https://github.com/manuzhang/incubator-gearpump watermark Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-gearpump/pull/33.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #33 commit d3d6466a10a0475a9e83dd07d20e9b4d6c1bdb9e Author: manuzhang Date: 2016-05-31T05:10:24Z fix GEARPUMP-32, introduce source watermark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump issue #33: fix GEARPUMP-32, introduce source watermark
Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/33 The PR is not mature and opened early to gather thoughts from the community. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (GEARPUMP-32) Minimum clock of source Tasks maybe inaccurate
[
https://issues.apache.org/jira/browse/GEARPUMP-32?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313530#comment-15313530
]
ASF GitHub Bot commented on GEARPUMP-32:
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/33
The PR is not mature and opened early to gather thoughts from the
community.
> Minimum clock of source Tasks maybe inaccurate
> --
>
> Key: GEARPUMP-32
> URL: https://issues.apache.org/jira/browse/GEARPUMP-32
> Project: Apache Gearpump
> Issue Type: Bug
> Components: streaming
>Affects Versions: 0.8.0
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 0.8.1
>
>
> Moved from [https://github.com/gearpump/gearpump/issues/1835] and reported by
> [Zhu Yueqian|https://github.com/yueqianzhu]
> {quote}
> Source tasks have not any upstreamClocks. So, startClock is the minimum of
> pending clocks when recover happen.
> eg below:
> source task1: timeStamp:15,not ACK, minClockValue maybe is 15(<= 15).
> source task2: timeStamp:10,ACKed, minClockValue maybe is Long.MaxValue
> when recover happen,startClock maybe is 15. where is the data between 10 to
> 15 at task2?
> {quote}
> More context on this issue:
> In Gearpump, we maintain a global minimum clock tracked from a message's
> timestamp across all tasks. It means messages with timestamp before this
> clock have all been processed. An application will restart from this value on
> failure, and thus at-least-once message delivery could be guaranteed.
> The global minimum clock is the lower bound of all the Tasks' minimum clocks.
> For a task, the minimum clock is the lower of
> # upstream minimum clock
> # a. the minimum timestamp of unacked messages
>b. Long.MaxValue if all messages have been acked.
>
> Note that 2.b allows the global minimum clock to progress and it is almost
> safe since the clock is also bounded by the upstream minimum clock. I said
> "almost safe" because a source task has no upstream but we assume the
> upstream minimum clock is Long.MaxValue. Thus, the scenario described by Zhu
> Yueqian could happen and breaks at-least-once guarantee.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-32) Minimum clock of source Tasks maybe inaccurate
[
https://issues.apache.org/jira/browse/GEARPUMP-32?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313529#comment-15313529
]
ASF GitHub Bot commented on GEARPUMP-32:
GitHub user manuzhang opened a pull request:
https://github.com/apache/incubator-gearpump/pull/33
fix GEARPUMP-32, introduce source watermark
This PR introduces source watermark such that
1. messages with timestamps earlier than watermark could be filtered out at
source.
2. minclock of source processor will be bounded by watemark
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/manuzhang/incubator-gearpump watermark
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-gearpump/pull/33.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #33
commit d3d6466a10a0475a9e83dd07d20e9b4d6c1bdb9e
Author: manuzhang
Date: 2016-05-31T05:10:24Z
fix GEARPUMP-32, introduce source watermark
> Minimum clock of source Tasks maybe inaccurate
> --
>
> Key: GEARPUMP-32
> URL: https://issues.apache.org/jira/browse/GEARPUMP-32
> Project: Apache Gearpump
> Issue Type: Bug
> Components: streaming
>Affects Versions: 0.8.0
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 0.8.1
>
>
> Moved from [https://github.com/gearpump/gearpump/issues/1835] and reported by
> [Zhu Yueqian|https://github.com/yueqianzhu]
> {quote}
> Source tasks have not any upstreamClocks. So, startClock is the minimum of
> pending clocks when recover happen.
> eg below:
> source task1: timeStamp:15,not ACK, minClockValue maybe is 15(<= 15).
> source task2: timeStamp:10,ACKed, minClockValue maybe is Long.MaxValue
> when recover happen,startClock maybe is 15. where is the data between 10 to
> 15 at task2?
> {quote}
> More context on this issue:
> In Gearpump, we maintain a global minimum clock tracked from a message's
> timestamp across all tasks. It means messages with timestamp before this
> clock have all been processed. An application will restart from this value on
> failure, and thus at-least-once message delivery could be guaranteed.
> The global minimum clock is the lower bound of all the Tasks' minimum clocks.
> For a task, the minimum clock is the lower of
> # upstream minimum clock
> # a. the minimum timestamp of unacked messages
>b. Long.MaxValue if all messages have been acked.
>
> Note that 2.b allows the global minimum clock to progress and it is almost
> safe since the clock is also bounded by the upstream minimum clock. I said
> "almost safe" because a source task has no upstream but we assume the
> upstream minimum clock is Long.MaxValue. Thus, the scenario described by Zhu
> Yueqian could happen and breaks at-least-once guarantee.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-gearpump pull request #31: fix GEARPUMP-83, show application pendi...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-gearpump/pull/31 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (GEARPUMP-83) After killing all worker instances, application status should not be described as active
[
https://issues.apache.org/jira/browse/GEARPUMP-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313444#comment-15313444
]
ASF GitHub Bot commented on GEARPUMP-83:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-gearpump/pull/31
> After killing all worker instances, application status should not be
> described as active
>
>
> Key: GEARPUMP-83
> URL: https://issues.apache.org/jira/browse/GEARPUMP-83
> Project: Apache Gearpump
> Issue Type: Bug
> Components: Dashboard
>Affects Versions: 0.8.0
>Reporter: Kam Kasravi
>Assignee: Manu Zhang
>Priority: Minor
> Fix For: 0.8.1
>
>
> Step to reproduce:
> Start cluster with one worker
> Start a word count
> Kill the worker
> Expect /api/v1.0/master/applist actually returns app status as active, but
> application's detail page is not available. I think as there is no resource
> to run the application, the application is in some abnormal status. In order
> not to mislead user, I think we should invent a new status, might be
> recovering or something.
> Example output:
> {code}
> {"appMasters":[{"status":"active","appId":1,"appName":"dag","appMasterPath":"akka.tcp://app1-executor-1@127.0.0.1:46761/user/daemon/appdaemon1/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451894551477","startTime":"1451894553568","user":"qxu"},{"status":"active","appId":2,"appName":"wordCount","appMasterPath":"akka.tcp://app2-executor-1@127.0.0.1:49261/user/daemon/appdaemon2/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451898038991","startTime":"1451898040265","user":"qxu"}]}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (GEARPUMP-83) After killing all worker instances, application status should not be described as active
[
https://issues.apache.org/jira/browse/GEARPUMP-83?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manu Zhang resolved GEARPUMP-83.
Resolution: Fixed
Issue resolved by pull request 31
[https://github.com/apache/incubator-gearpump/pull/31]
> After killing all worker instances, application status should not be
> described as active
>
>
> Key: GEARPUMP-83
> URL: https://issues.apache.org/jira/browse/GEARPUMP-83
> Project: Apache Gearpump
> Issue Type: Bug
> Components: Dashboard
>Affects Versions: 0.8.0
>Reporter: Kam Kasravi
>Assignee: Manu Zhang
>Priority: Minor
> Fix For: 0.8.1
>
>
> Step to reproduce:
> Start cluster with one worker
> Start a word count
> Kill the worker
> Expect /api/v1.0/master/applist actually returns app status as active, but
> application's detail page is not available. I think as there is no resource
> to run the application, the application is in some abnormal status. In order
> not to mislead user, I think we should invent a new status, might be
> recovering or something.
> Example output:
> {code}
> {"appMasters":[{"status":"active","appId":1,"appName":"dag","appMasterPath":"akka.tcp://app1-executor-1@127.0.0.1:46761/user/daemon/appdaemon1/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451894551477","startTime":"1451894553568","user":"qxu"},{"status":"active","appId":2,"appName":"wordCount","appMasterPath":"akka.tcp://app2-executor-1@127.0.0.1:49261/user/daemon/appdaemon2/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451898038991","startTime":"1451898040265","user":"qxu"}]}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-gearpump issue #31: fix GEARPUMP-83, show application pending on w...
Github user huafengw commented on the issue: https://github.com/apache/incubator-gearpump/pull/31 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (GEARPUMP-83) After killing all worker instances, application status should not be described as active
[
https://issues.apache.org/jira/browse/GEARPUMP-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313412#comment-15313412
]
ASF GitHub Bot commented on GEARPUMP-83:
Github user huafengw commented on the issue:
https://github.com/apache/incubator-gearpump/pull/31
+1
> After killing all worker instances, application status should not be
> described as active
>
>
> Key: GEARPUMP-83
> URL: https://issues.apache.org/jira/browse/GEARPUMP-83
> Project: Apache Gearpump
> Issue Type: Bug
> Components: Dashboard
>Affects Versions: 0.8.0
>Reporter: Kam Kasravi
>Assignee: Manu Zhang
>Priority: Minor
> Fix For: 0.8.1
>
>
> Step to reproduce:
> Start cluster with one worker
> Start a word count
> Kill the worker
> Expect /api/v1.0/master/applist actually returns app status as active, but
> application's detail page is not available. I think as there is no resource
> to run the application, the application is in some abnormal status. In order
> not to mislead user, I think we should invent a new status, might be
> recovering or something.
> Example output:
> {code}
> {"appMasters":[{"status":"active","appId":1,"appName":"dag","appMasterPath":"akka.tcp://app1-executor-1@127.0.0.1:46761/user/daemon/appdaemon1/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451894551477","startTime":"1451894553568","user":"qxu"},{"status":"active","appId":2,"appName":"wordCount","appMasterPath":"akka.tcp://app2-executor-1@127.0.0.1:49261/user/daemon/appdaemon2/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451898038991","startTime":"1451898040265","user":"qxu"}]}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-gearpump issue #31: fix GEARPUMP-83, show application pending on w...
Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/31 @huafengw fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump pull request #11: fix #106 Gearpump Redis Integration
Github user darionyaphet commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/11#discussion_r65504318
--- Diff:
experiments/redis/src/main/scala/org/apache/gearpump/streaming/redis/RedisMessage.scala
---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gearpump.streaming.redis
+
+import java.nio.charset.Charset
+
+object RedisMessage {
+
+ private def toBytes(string: String,
+ charset: Charset = Charset.forName("UTF8")
+ ): Array[Byte] = string.getBytes(charset)
+
+ case class PublishMessage(message: Array[Byte]) {
+def this(message: String) = this(toBytes(message))
+ }
+
+ case class SetMessage(key: Array[Byte], value: Array[Byte]) {
+def this(key: String, value: String) = this(toBytes(key),
toBytes(value))
+ }
+
+ case class LPushMessage(key: Array[Byte], value: Array[Byte]) {
--- End diff --
`LPushMessage` and `RPushMessage ` are difference message to control redis
client add element at list's head or tail
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #31: fix GEARPUMP-83, show application pendi...
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/31#discussion_r65495393
--- Diff:
daemon/src/main/scala/org/apache/gearpump/cluster/master/AppManager.scala ---
@@ -310,22 +320,20 @@ private[cluster] class AppManager(kvService:
ActorRef, launcher: AppMasterLaunch
case class RecoverApplication(applicationStatus: ApplicationState)
private def cleanApplicationData(appId: Int): Unit = {
-// Add the dead app to dead appMaster
-appMasterRegistry.get(appId).foreach { pair =>
- val (appMasterActor, info) = pair
- deadAppMasters += appId -> (appMasterActor, info.copy(
-finishTime = System.currentTimeMillis()))
-}
-
-appMasterRegistry -= appId
+// Add the dead app to dead appMasters
+deadAppMasters += appId
--- End diff --
When the application is down, you don't update its finish time. You can see
the dashboard missed that info.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #11: fix #106 Gearpump Redis Integration
Github user darionyaphet commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/11#discussion_r65502913
--- Diff:
experiments/redis/src/main/scala/org/apache/gearpump/streaming/redis/RedisMessage.scala
---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gearpump.streaming.redis
+
+import java.nio.charset.Charset
+
+object RedisMessage {
+
+ private def toBytes(string: String,
+ charset: Charset = Charset.forName("UTF8")
+ ): Array[Byte] = string.getBytes(charset)
+
+ case class PublishMessage(message: Array[Byte]) {
--- End diff --
The messages are implemented in `storm-redis` I try to keep they are the
same .
BTW I will add some message both in gearpump and storm .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #31: fix GEARPUMP-83, show application pendi...
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/31#discussion_r65495140
--- Diff:
daemon/src/main/scala/org/apache/gearpump/cluster/master/AppManager.scala ---
@@ -268,9 +275,9 @@ private[cluster] class AppManager(kvService: ActorRef,
launcher: AppMasterLaunch
def terminationWatch: Receive = {
case terminate: Terminated =>
- terminate.getAddressTerminated()
- LOG.info(s"AppMaster(${terminate.actor.path}) is terminiated, " +
-s"network down: ${terminate.getAddressTerminated()}")
+ terminate.getAddressTerminated
--- End diff --
Is this line of code useless?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (GEARPUMP-83) After killing all worker instances, application status should not be described as active
[
https://issues.apache.org/jira/browse/GEARPUMP-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311968#comment-15311968
]
ASF GitHub Bot commented on GEARPUMP-83:
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/31
@huafengw fixed
> After killing all worker instances, application status should not be
> described as active
>
>
> Key: GEARPUMP-83
> URL: https://issues.apache.org/jira/browse/GEARPUMP-83
> Project: Apache Gearpump
> Issue Type: Bug
> Components: Dashboard
>Affects Versions: 0.8.0
>Reporter: Kam Kasravi
>Assignee: Manu Zhang
>Priority: Minor
> Fix For: 0.8.1
>
>
> Step to reproduce:
> Start cluster with one worker
> Start a word count
> Kill the worker
> Expect /api/v1.0/master/applist actually returns app status as active, but
> application's detail page is not available. I think as there is no resource
> to run the application, the application is in some abnormal status. In order
> not to mislead user, I think we should invent a new status, might be
> recovering or something.
> Example output:
> {code}
> {"appMasters":[{"status":"active","appId":1,"appName":"dag","appMasterPath":"akka.tcp://app1-executor-1@127.0.0.1:46761/user/daemon/appdaemon1/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451894551477","startTime":"1451894553568","user":"qxu"},{"status":"active","appId":2,"appName":"wordCount","appMasterPath":"akka.tcp://app2-executor-1@127.0.0.1:49261/user/daemon/appdaemon2/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451898038991","startTime":"1451898040265","user":"qxu"}]}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-152) upgrade Storm support to 1.0.x
[ https://issues.apache.org/jira/browse/GEARPUMP-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311944#comment-15311944 ] darion yaphet commented on GEARPUMP-152: Yes I try to upgrade to 1.0.X but I found package name and some interface have changed > upgrade Storm support to 1.0.x > -- > > Key: GEARPUMP-152 > URL: https://issues.apache.org/jira/browse/GEARPUMP-152 > Project: Apache Gearpump > Issue Type: Improvement > Components: storm >Reporter: Manu Zhang > > [Storm 1.0.0 | http://storm.apache.org/2016/04/12/storm100-released.html] has > been released with a lot of changes. We need to upgrade our support for Storm > to 1.0.x -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-83) After killing all worker instances, application status should not be described as active
[
https://issues.apache.org/jira/browse/GEARPUMP-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311879#comment-15311879
]
ASF GitHub Bot commented on GEARPUMP-83:
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/31#discussion_r65495393
--- Diff:
daemon/src/main/scala/org/apache/gearpump/cluster/master/AppManager.scala ---
@@ -310,22 +320,20 @@ private[cluster] class AppManager(kvService:
ActorRef, launcher: AppMasterLaunch
case class RecoverApplication(applicationStatus: ApplicationState)
private def cleanApplicationData(appId: Int): Unit = {
-// Add the dead app to dead appMaster
-appMasterRegistry.get(appId).foreach { pair =>
- val (appMasterActor, info) = pair
- deadAppMasters += appId -> (appMasterActor, info.copy(
-finishTime = System.currentTimeMillis()))
-}
-
-appMasterRegistry -= appId
+// Add the dead app to dead appMasters
+deadAppMasters += appId
--- End diff --
When the application is down, you don't update its finish time. You can see
the dashboard missed that info.
> After killing all worker instances, application status should not be
> described as active
>
>
> Key: GEARPUMP-83
> URL: https://issues.apache.org/jira/browse/GEARPUMP-83
> Project: Apache Gearpump
> Issue Type: Bug
> Components: Dashboard
>Affects Versions: 0.8.0
>Reporter: Kam Kasravi
>Assignee: Manu Zhang
>Priority: Minor
> Fix For: 0.8.1
>
>
> Step to reproduce:
> Start cluster with one worker
> Start a word count
> Kill the worker
> Expect /api/v1.0/master/applist actually returns app status as active, but
> application's detail page is not available. I think as there is no resource
> to run the application, the application is in some abnormal status. In order
> not to mislead user, I think we should invent a new status, might be
> recovering or something.
> Example output:
> {code}
> {"appMasters":[{"status":"active","appId":1,"appName":"dag","appMasterPath":"akka.tcp://app1-executor-1@127.0.0.1:46761/user/daemon/appdaemon1/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451894551477","startTime":"1451894553568","user":"qxu"},{"status":"active","appId":2,"appName":"wordCount","appMasterPath":"akka.tcp://app2-executor-1@127.0.0.1:49261/user/daemon/appdaemon2/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451898038991","startTime":"1451898040265","user":"qxu"}]}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-83) After killing all worker instances, application status should not be described as active
[
https://issues.apache.org/jira/browse/GEARPUMP-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311877#comment-15311877
]
ASF GitHub Bot commented on GEARPUMP-83:
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/31#discussion_r65495140
--- Diff:
daemon/src/main/scala/org/apache/gearpump/cluster/master/AppManager.scala ---
@@ -268,9 +275,9 @@ private[cluster] class AppManager(kvService: ActorRef,
launcher: AppMasterLaunch
def terminationWatch: Receive = {
case terminate: Terminated =>
- terminate.getAddressTerminated()
- LOG.info(s"AppMaster(${terminate.actor.path}) is terminiated, " +
-s"network down: ${terminate.getAddressTerminated()}")
+ terminate.getAddressTerminated
--- End diff --
Is this line of code useless?
> After killing all worker instances, application status should not be
> described as active
>
>
> Key: GEARPUMP-83
> URL: https://issues.apache.org/jira/browse/GEARPUMP-83
> Project: Apache Gearpump
> Issue Type: Bug
> Components: Dashboard
>Affects Versions: 0.8.0
>Reporter: Kam Kasravi
>Assignee: Manu Zhang
>Priority: Minor
> Fix For: 0.8.1
>
>
> Step to reproduce:
> Start cluster with one worker
> Start a word count
> Kill the worker
> Expect /api/v1.0/master/applist actually returns app status as active, but
> application's detail page is not available. I think as there is no resource
> to run the application, the application is in some abnormal status. In order
> not to mislead user, I think we should invent a new status, might be
> recovering or something.
> Example output:
> {code}
> {"appMasters":[{"status":"active","appId":1,"appName":"dag","appMasterPath":"akka.tcp://app1-executor-1@127.0.0.1:46761/user/daemon/appdaemon1/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451894551477","startTime":"1451894553568","user":"qxu"},{"status":"active","appId":2,"appName":"wordCount","appMasterPath":"akka.tcp://app2-executor-1@127.0.0.1:49261/user/daemon/appdaemon2/$c","workerPath":"akka.tcp://48a47aa6-81c0-493c-9948-9d7d4c946db6@127.0.0.1:59201/user/Worker48a47aa6-81c0-493c-9948-9d7d4c946db6","submissionTime":"1451898038991","startTime":"1451898040265","user":"qxu"}]}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
