[GitHub] storm pull request: STORM-1016: Generate trident bolt ids with sor...
Github user victor-wong commented on the pull request: https://github.com/apache/storm/pull/706#issuecomment-156359463 @revans2 sorry for replying so late, I just made some changes and it worked OK. Is that what you mean? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1016) Generate trident bolt ids with sorted group names
[ https://issues.apache.org/jira/browse/STORM-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003691#comment-15003691 ] ASF GitHub Bot commented on STORM-1016: --- Github user victor-wong commented on the pull request: https://github.com/apache/storm/pull/706#issuecomment-156359463 @revans2 sorry for replying so late, I just made some changes and it worked OK. Is that what you mean? > Generate trident bolt ids with sorted group names > - > > Key: STORM-1016 > URL: https://issues.apache.org/jira/browse/STORM-1016 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Jiasheng Wang > > In genBoltId() method of TridentTopology, groups are stored in HashSet. > So everytime I submit my trident topology, I get different id for my > component. > For example, this time my bolt ids are (b-0-abc, b-1-def), next time they are > (b-1-abc, b-0-def). > It makes harder for users to track their metrics, and the bolt names in UI > are changing too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1016: Generate trident bolt ids with sor...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/706#issuecomment-156442935 @victor-wong yes that is exactly what I was thinking, makes the code a lot smaller. I am +1 on this change, but it looks like there is a code conflict. Could you upmerge your changes? And don't worry about it taking a while. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1016) Generate trident bolt ids with sorted group names
[ https://issues.apache.org/jira/browse/STORM-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004021#comment-15004021 ] ASF GitHub Bot commented on STORM-1016: --- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/706#issuecomment-156442935 @victor-wong yes that is exactly what I was thinking, makes the code a lot smaller. I am +1 on this change, but it looks like there is a code conflict. Could you upmerge your changes? And don't worry about it taking a while. > Generate trident bolt ids with sorted group names > - > > Key: STORM-1016 > URL: https://issues.apache.org/jira/browse/STORM-1016 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Jiasheng Wang > > In genBoltId() method of TridentTopology, groups are stored in HashSet. > So everytime I submit my trident topology, I get different id for my > component. > For example, this time my bolt ids are (b-0-abc, b-1-def), next time they are > (b-1-abc, b-0-def). > It makes harder for users to track their metrics, and the bolt names in UI > are changing too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1203. worker metadata file creation does...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/878#issuecomment-156443611 @harshach great find. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1203) worker metadata file creation doesn't use storm.log.dir config
[ https://issues.apache.org/jira/browse/STORM-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004022#comment-15004022 ] ASF GitHub Bot commented on STORM-1203: --- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/878#issuecomment-156443611 @harshach great find. +1 > worker metadata file creation doesn't use storm.log.dir config > -- > > Key: STORM-1203 > URL: https://issues.apache.org/jira/browse/STORM-1203 > Project: Apache Storm > Issue Type: Bug >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani > Fix For: 0.11.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Added in feature diff
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/877#discussion_r44786316 --- Diff: STORM_JSTORM_FEATURE_DIFF.md --- @@ -0,0 +1,30 @@ + feature | Storm-0.11.0-SNAPSHOT (and expected pull requests) | JStorm-2.1.0 | Notes | JIRA | +--|---|--|---|| + Security | Basic Authentication/Authorization | N/A | The migration is not high risk. But JStorm daeons/connections need to be evaluated +Scheduler | Resource aware scheduling (Work In Progress) | Make the tasks of same component distributed evenly across the nodes in cluster.Make number of tasks in a worker is balanced.Try to assign two tasks which are transferring message directly into a same worker to reduce network cost.Support user defined assignment and to use the result of last assignment Different solution between Storm and JStorm. Actually, JStorm has ever used cpu/mem/disk/network demond for scheduling. But we faced the problem of low utilization of cluster. For example, because the configuration of topologys and resources of hardwares in a cluster are different. After running a period, it is possible that no topology will be assigned to the node with enough other resources but just lacking one resource. | Scheduler interface is pluggable so we should be able to support boht schedulers for the time being | +Nimbus HA | Support to configure more than one spare nimbus. When the master nimbus is down, a most appropriate spare nimbus(topologys in disk mostly matches the recoreds in zk) will be chosen to be promoted. | Different solution. | | +Topology structure | worker/executor/task | worker/task | It might be the biggest diff between Storm and JStorm. So, not sure if there are some features related to "executor" can not be merged. (Rebalancing allows us to combine multiple tasks on a single executor) This may go away in the future once RAS is stable | +Topology Master | (Heartbeat server is similar for scaling on a pull request) | New system bolt "topology master" was added, which is responsible for collecting task heartbeat info of all tasks and reports to nimbus. Besides task hb info, it also can be used to dispatch control messages in topology. Topology master significantly reduce the amout of read/write to zk. Before this change, zk is the bottleneck to depoly big cluster and topology. +Backpressure | Store backpressure status in zk which will trigger the source spout to start flow control. When flow control, relative souce spout will stop sending tuples. | Implement backpressure based on topology master. TM is responsible to process the trigger message and send flow control request to relative spouts. When flow control, spout will not stop to send tuples, but just slow down the sending.Enable user to update the configuration of backpressure dynamically without restarting topology, e.g. enable/disable backpressue, high/low water mark... | These two solution is totally different. We need to evaluate which one is better. | +Monitor of task execute thread | N/A | Monitor the status of the execute thread of task. It is effective to find the slow bolt in a topology, and potential dead lock. | +Message processing | Configurable multi receiving threads of worker (But likely to change to have deserialization happen in Netty) | Add receiving and transferring queue/thread for each task to make deserialization and serialization asynchronouslyRemove receiving and transferring thread on worker level to avoid unnecessary locks and to shorten the message processing phase | Low risk for merging to JStorm (Both implementations are similar) | +Batch tuples | Batching in DisruptorQueue | Do batch before sending tuple to transfer queue and support to adjust the batch size dynamically according to samples of actual batch size sent out for past intervals. | Should evaluate both implementations, and see which is better, or if we can pull in some of the dynamic batching to the disruptor level | +Grouping | Load aware balancing shuffle grouping | Add localfrist grouping that tuples are sent to the tasks in the same worker firstly. If the load of all local tasks is high, the tuples will be sent out to remote tasks.Improve localOrShuffle grouping that tuples are only sent to the tasks in the same worker or in same node.The shuffle of JStorm checks the load of network connection of local worker. | The risk for merging of checking remote load which was implemented in "Load aware balancing shuffle grouping" is low. +Rest API | YES | NA |The porting of rest API to JStorm is in progress +web-UI | | Redesign Web-UI, the new Web-UI code is clear and clean.Data display is much more beautiful and better user experience, especially topology graph and 30 minutes metric trend. url: http://storm.taobao.org/ +metric system | |
[GitHub] storm pull request: Added in feature diff
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/877#issuecomment-156448057 @wuchong and @bastiliu thanks for the feedback and comments. I updated the pull request based off of them. Any more feedback is definitely welcome. By the way great work on jstorm. My has spent way too much time playing around with http://storm.taobao.org/ being jealous of the features there. Very excited to see what all of us can accomplish together. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Added in feature diff
Github user bastiliu commented on a diff in the pull request: https://github.com/apache/storm/pull/877#discussion_r44792426 --- Diff: STORM_JSTORM_FEATURE_DIFF.md --- @@ -0,0 +1,30 @@ + feature | Storm-0.11.0-SNAPSHOT (and expected pull requests) | JStorm-2.1.0 | Notes | JIRA | +--|---|--|---|| + Security | Basic Authentication/Authorization | N/A | The migration is not high risk. But JStorm daeons/connections need to be evaluated +Scheduler | Resource aware scheduling (Work In Progress) | Make the tasks of same component distributed evenly across the nodes in cluster.Make number of tasks in a worker is balanced.Try to assign two tasks which are transferring message directly into a same worker to reduce network cost.Support user defined assignment and to use the result of last assignment Different solution between Storm and JStorm. Actually, JStorm has ever used cpu/mem/disk/network demond for scheduling. But we faced the problem of low utilization of cluster. For example, because the configuration of topologys and resources of hardwares in a cluster are different. After running a period, it is possible that no topology will be assigned to the node with enough other resources but just lacking one resource. | Scheduler interface is pluggable so we should be able to support boht schedulers for the time being | +Nimbus HA | Support to configure more than one spare nimbus. When the master nimbus is down, a most appropriate spare nimbus(topologys in disk mostly matches the recoreds in zk) will be chosen to be promoted. | Different solution. | | +Topology structure | worker/executor/task | worker/task | It might be the biggest diff between Storm and JStorm. So, not sure if there are some features related to "executor" can not be merged. (Rebalancing allows us to combine multiple tasks on a single executor) This may go away in the future once RAS is stable | +Topology Master | (Heartbeat server is similar for scaling on a pull request) | New system bolt "topology master" was added, which is responsible for collecting task heartbeat info of all tasks and reports to nimbus. Besides task hb info, it also can be used to dispatch control messages in topology. Topology master significantly reduce the amout of read/write to zk. Before this change, zk is the bottleneck to depoly big cluster and topology. +Backpressure | Store backpressure status in zk which will trigger the source spout to start flow control. When flow control, relative souce spout will stop sending tuples. | Implement backpressure based on topology master. TM is responsible to process the trigger message and send flow control request to relative spouts. When flow control, spout will not stop to send tuples, but just slow down the sending.Enable user to update the configuration of backpressure dynamically without restarting topology, e.g. enable/disable backpressue, high/low water mark... | These two solution is totally different. We need to evaluate which one is better. | +Monitor of task execute thread | N/A | Monitor the status of the execute thread of task. It is effective to find the slow bolt in a topology, and potential dead lock. | +Message processing | Configurable multi receiving threads of worker (But likely to change to have deserialization happen in Netty) | Add receiving and transferring queue/thread for each task to make deserialization and serialization asynchronouslyRemove receiving and transferring thread on worker level to avoid unnecessary locks and to shorten the message processing phase | Low risk for merging to JStorm (Both implementations are similar) | +Batch tuples | Batching in DisruptorQueue | Do batch before sending tuple to transfer queue and support to adjust the batch size dynamically according to samples of actual batch size sent out for past intervals. | Should evaluate both implementations, and see which is better, or if we can pull in some of the dynamic batching to the disruptor level | +Grouping | Load aware balancing shuffle grouping | Add localfrist grouping that tuples are sent to the tasks in the same worker firstly. If the load of all local tasks is high, the tuples will be sent out to remote tasks.Improve localOrShuffle grouping that tuples are only sent to the tasks in the same worker or in same node.The shuffle of JStorm checks the load of network connection of local worker. | The risk for merging of checking remote load which was implemented in "Load aware balancing shuffle grouping" is low. +Rest API | YES | NA |The porting of rest API to JStorm is in progress +web-UI | | Redesign Web-UI, the new Web-UI code is clear and clean.Data display is much more beautiful and better user experience, especially topology graph and 30 minutes metric trend. url: http://storm.taobao.org/ +metric system |
[jira] [Commented] (STORM-1202) Migrate APIs to org.apache.storm, but try to provide backwards compatability as a bridge
[ https://issues.apache.org/jira/browse/STORM-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004112#comment-15004112 ] Robert Joseph Evans commented on STORM-1202: OK so a shim layer is not going to work. To make it happen I would have to modify the generated thrift code, which really is a non-starter, but thinking about this I think I have a way to still do it. I should be able to borrow code from the maven shade plugin, and shade the submitted jar to translate backtype.storm -> org.apache.storm and storm.trident -> org.apache.storm.trident. This code would exist in a separate jar and would be a plugin that could be configured on for those that want it, and off for those that don't. That way we have a way to support backwards compatibility but can be turned off when you know everyone has upgraded. > Migrate APIs to org.apache.storm, but try to provide backwards compatability > as a bridge > > > Key: STORM-1202 > URL: https://issues.apache.org/jira/browse/STORM-1202 > Project: Apache Storm > Issue Type: New Feature >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > We want to move the storm classpaths to org.apache.storm wherever possible, > but we also want to provide backwards compatibility for user facing APIs > whenever possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Added in feature diff
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/877#discussion_r44794800 --- Diff: STORM_JSTORM_FEATURE_DIFF.md --- @@ -0,0 +1,30 @@ + feature | Storm-0.11.0-SNAPSHOT (and expected pull requests) | JStorm-2.1.0 | Notes | JIRA | +--|---|--|---|| + Security | Basic Authentication/Authorization | N/A | The migration is not high risk. But JStorm daeons/connections need to be evaluated +Scheduler | Resource aware scheduling (Work In Progress) | Make the tasks of same component distributed evenly across the nodes in cluster.Make number of tasks in a worker is balanced.Try to assign two tasks which are transferring message directly into a same worker to reduce network cost.Support user defined assignment and to use the result of last assignment Different solution between Storm and JStorm. Actually, JStorm has ever used cpu/mem/disk/network demond for scheduling. But we faced the problem of low utilization of cluster. For example, because the configuration of topologys and resources of hardwares in a cluster are different. After running a period, it is possible that no topology will be assigned to the node with enough other resources but just lacking one resource. | Scheduler interface is pluggable so we should be able to support boht schedulers for the time being | +Nimbus HA | Support to configure more than one spare nimbus. When the master nimbus is down, a most appropriate spare nimbus(topologys in disk mostly matches the recoreds in zk) will be chosen to be promoted. | Different solution. | | +Topology structure | worker/executor/task | worker/task | It might be the biggest diff between Storm and JStorm. So, not sure if there are some features related to "executor" can not be merged. (Rebalancing allows us to combine multiple tasks on a single executor) This may go away in the future once RAS is stable | +Topology Master | (Heartbeat server is similar for scaling on a pull request) | New system bolt "topology master" was added, which is responsible for collecting task heartbeat info of all tasks and reports to nimbus. Besides task hb info, it also can be used to dispatch control messages in topology. Topology master significantly reduce the amout of read/write to zk. Before this change, zk is the bottleneck to depoly big cluster and topology. +Backpressure | Store backpressure status in zk which will trigger the source spout to start flow control. When flow control, relative souce spout will stop sending tuples. | Implement backpressure based on topology master. TM is responsible to process the trigger message and send flow control request to relative spouts. When flow control, spout will not stop to send tuples, but just slow down the sending.Enable user to update the configuration of backpressure dynamically without restarting topology, e.g. enable/disable backpressue, high/low water mark... | These two solution is totally different. We need to evaluate which one is better. | +Monitor of task execute thread | N/A | Monitor the status of the execute thread of task. It is effective to find the slow bolt in a topology, and potential dead lock. | +Message processing | Configurable multi receiving threads of worker (But likely to change to have deserialization happen in Netty) | Add receiving and transferring queue/thread for each task to make deserialization and serialization asynchronouslyRemove receiving and transferring thread on worker level to avoid unnecessary locks and to shorten the message processing phase | Low risk for merging to JStorm (Both implementations are similar) | +Batch tuples | Batching in DisruptorQueue | Do batch before sending tuple to transfer queue and support to adjust the batch size dynamically according to samples of actual batch size sent out for past intervals. | Should evaluate both implementations, and see which is better, or if we can pull in some of the dynamic batching to the disruptor level | +Grouping | Load aware balancing shuffle grouping | Add localfrist grouping that tuples are sent to the tasks in the same worker firstly. If the load of all local tasks is high, the tuples will be sent out to remote tasks.Improve localOrShuffle grouping that tuples are only sent to the tasks in the same worker or in same node.The shuffle of JStorm checks the load of network connection of local worker. | The risk for merging of checking remote load which was implemented in "Load aware balancing shuffle grouping" is low. +Rest API | YES | NA |The porting of rest API to JStorm is in progress +web-UI | | Redesign Web-UI, the new Web-UI code is clear and clean.Data display is much more beautiful and better user experience, especially topology graph and 30 minutes metric trend. url: http://storm.taobao.org/ +metric system | |
[GitHub] storm pull request: Added in feature diff
Github user hustfxj commented on a diff in the pull request: https://github.com/apache/storm/pull/877#discussion_r44796241 --- Diff: STORM_JSTORM_FEATURE_DIFF.md --- @@ -0,0 +1,30 @@ + feature | Storm-0.11.0-SNAPSHOT (and expected pull requests) | JStorm-2.1.0 | Notes | JIRA | +--|---|--|---|| + Security | Basic Authentication/Authorization | N/A | The migration is not high risk. But JStorm daeons/connections need to be evaluated +Scheduler | Resource aware scheduling (Work In Progress) | Make the tasks of same component distributed evenly across the nodes in cluster.Make number of tasks in a worker is balanced.Try to assign two tasks which are transferring message directly into a same worker to reduce network cost.Support user defined assignment and to use the result of last assignment Different solution between Storm and JStorm. Actually, JStorm has ever used cpu/mem/disk/network demond for scheduling. But we faced the problem of low utilization of cluster. For example, because the configuration of topologys and resources of hardwares in a cluster are different. After running a period, it is possible that no topology will be assigned to the node with enough other resources but just lacking one resource. | Scheduler interface is pluggable so we should be able to support boht schedulers for the time being | +Nimbus HA | Support to configure more than one spare nimbus. When the master nimbus is down, a most appropriate spare nimbus(topologys in disk mostly matches the recoreds in zk) will be chosen to be promoted. | Different solution. | | +Topology structure | worker/executor/task | worker/task | It might be the biggest diff between Storm and JStorm. So, not sure if there are some features related to "executor" can not be merged. (Rebalancing allows us to combine multiple tasks on a single executor) This may go away in the future once RAS is stable | +Topology Master | (Heartbeat server is similar for scaling on a pull request) | New system bolt "topology master" was added, which is responsible for collecting task heartbeat info of all tasks and reports to nimbus. Besides task hb info, it also can be used to dispatch control messages in topology. Topology master significantly reduce the amout of read/write to zk. Before this change, zk is the bottleneck to depoly big cluster and topology. --- End diff -- TM is mainly used to manage topology laterï¼and we hope users can defines control stream in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/875#discussion_r44796485
--- Diff: storm-core/src/ui/public/index.html ---
@@ -156,6 +163,15 @@
]
});
$('#supervisor-summary [data-toggle="tooltip"]').tooltip();
+ $.getJSON("/api/v1/cluster/configuration", function(json){
+ var scheduler = json["storm.scheduler"];
+ if (scheduler !=
"backtype.storm.scheduler.resource.ResourceAwareScheduler"){
+ $('#supervisor-summary
td:nth-child(6),#supervisor-summary th:nth-child(6)').hide();
+ $('#supervisor-summary
td:nth-child(7),#supervisor-summary th:nth-child(7)').hide();
+ $('#supervisor-summary
td:nth-child(8),#supervisor-summary th:nth-child(8)').hide();
+ $('#supervisor-summary
td:nth-child(9),#supervisor-summary th:nth-child(9)').hide();
+ }
+ });
--- End diff --
Same here as above
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/875#discussion_r44796472
--- Diff: storm-core/src/ui/public/index.html ---
@@ -143,6 +143,13 @@
]
});
$('#topology-summary [data-toggle="tooltip"]').tooltip();
+ $.getJSON("/api/v1/cluster/configuration", function(json){
+ var scheduler = json["storm.scheduler"];
+ if (scheduler !=
"backtype.storm.scheduler.resource.ResourceAwareScheduler"){
+ $('#topology-summary td:nth-child(9),#topology-summary
th:nth-child(9)').hide();
+ $('#topology-summary td:nth-child(10),#topology-summary
th:nth-child(10)').hide();
+ }
+ });
--- End diff --
I don't really like having this be configured based off of a hard coded
scheduler. I would prefer to have a separate config, or even better to have
the rest API return a null for fields that it knows were not calculated
properly. Then have the template filter out those fields based off of those
values.
If we are going to go with a javascript hide, please add a class to all of
the fields you want to hide with something like RAS_ONLY and then just do
```$(".RAS_ONLY").hide();```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[
https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004148#comment-15004148
]
ASF GitHub Bot commented on STORM-1198:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/875#discussion_r44796485
--- Diff: storm-core/src/ui/public/index.html ---
@@ -156,6 +163,15 @@
]
});
$('#supervisor-summary [data-toggle="tooltip"]').tooltip();
+ $.getJSON("/api/v1/cluster/configuration", function(json){
+ var scheduler = json["storm.scheduler"];
+ if (scheduler !=
"backtype.storm.scheduler.resource.ResourceAwareScheduler"){
+ $('#supervisor-summary
td:nth-child(6),#supervisor-summary th:nth-child(6)').hide();
+ $('#supervisor-summary
td:nth-child(7),#supervisor-summary th:nth-child(7)').hide();
+ $('#supervisor-summary
td:nth-child(8),#supervisor-summary th:nth-child(8)').hide();
+ $('#supervisor-summary
td:nth-child(9),#supervisor-summary th:nth-child(9)').hide();
+ }
+ });
--- End diff --
Same here as above
> Web UI to show resource usages and Total Resources on all supervisors
> -
>
> Key: STORM-1198
> URL: https://issues.apache.org/jira/browse/STORM-1198
> Project: Apache Storm
> Issue Type: Story
> Components: storm-core
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Minor
> Attachments: supervisor-resources.png
>
>
> As we have resource aware scheduler (STORM-894), we want to be able to
> display resource capacity (CPU, memory; and network in future) and scheduled
> resource usage on each supervisor node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[
https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004147#comment-15004147
]
ASF GitHub Bot commented on STORM-1198:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/875#discussion_r44796472
--- Diff: storm-core/src/ui/public/index.html ---
@@ -143,6 +143,13 @@
]
});
$('#topology-summary [data-toggle="tooltip"]').tooltip();
+ $.getJSON("/api/v1/cluster/configuration", function(json){
+ var scheduler = json["storm.scheduler"];
+ if (scheduler !=
"backtype.storm.scheduler.resource.ResourceAwareScheduler"){
+ $('#topology-summary td:nth-child(9),#topology-summary
th:nth-child(9)').hide();
+ $('#topology-summary td:nth-child(10),#topology-summary
th:nth-child(10)').hide();
+ }
+ });
--- End diff --
I don't really like having this be configured based off of a hard coded
scheduler. I would prefer to have a separate config, or even better to have
the rest API return a null for fields that it knows were not calculated
properly. Then have the template filter out those fields based off of those
values.
If we are going to go with a javascript hide, please add a class to all of
the fields you want to hide with something like RAS_ONLY and then just do
```$(".RAS_ONLY").hide();```
> Web UI to show resource usages and Total Resources on all supervisors
> -
>
> Key: STORM-1198
> URL: https://issues.apache.org/jira/browse/STORM-1198
> Project: Apache Storm
> Issue Type: Story
> Components: storm-core
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Minor
> Attachments: supervisor-resources.png
>
>
> As we have resource aware scheduler (STORM-894), we want to be able to
> display resource capacity (CPU, memory; and network in future) and scheduled
> resource usage on each supervisor node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-156469658 It also looks like you need to upmerge and regenerate the thrift code with thrift-0.9.3. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1190: System Load too high after recent ...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/870 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004152#comment-15004152 ] ASF GitHub Bot commented on STORM-1198: --- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-156469658 It also looks like you need to upmerge and regenerate the thrift code with thrift-0.9.3. > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1190) System load spikes in recent snapshot
[ https://issues.apache.org/jira/browse/STORM-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004155#comment-15004155 ] ASF GitHub Bot commented on STORM-1190: --- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/870 > System load spikes in recent snapshot > - > > Key: STORM-1190 > URL: https://issues.apache.org/jira/browse/STORM-1190 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.11.0 > Environment: 10x (CoreOS stable (766.4.0) / k8s 1.0.1 / docker > running on Azure VMs) >Reporter: Michael Schonfeld >Priority: Critical > Attachments: Screenshot 2015-11-08 22.17.57.png, Screenshot > 2015-11-08 22.18.06.png > > > We've been running Storm's snapshots on our production cluster for a little > while now (that back pressure support really helped us), and we've noticed a > sudden spike in system load when going from > commit@ba1250993d10ffc523c9f5464371fbeb406d216f to the current latest > commit@c12e28c829fcfabc0a3a775fb9714968b7e3e349. Both versions were running > the exact same topologies, and there was no significant change in workload. > Not exactly sure how to even begin to debug this, so we ended up just rolling > back. Thoughts? > Stats screenshots attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: (STORM-956) When the execute() or nextTuple() ...
Github user hustfxj commented on the pull request: https://github.com/apache/storm/pull/647#issuecomment-156471890 Maybe it is not good ï¼the time-out check about execute()&nextTuple() should depends on the user . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-956) When the execute() or nextTuple() hang on external resources, stop the Worker's heartbeat
[ https://issues.apache.org/jira/browse/STORM-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004164#comment-15004164 ] ASF GitHub Bot commented on STORM-956: -- Github user hustfxj commented on the pull request: https://github.com/apache/storm/pull/647#issuecomment-156471890 Maybe it is not good ，the time-out check about execute()&nextTuple() should depends on the user . > When the execute() or nextTuple() hang on external resources, stop the > Worker's heartbeat > - > > Key: STORM-956 > URL: https://issues.apache.org/jira/browse/STORM-956 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Chuanlei Ni >Assignee: Chuanlei Ni >Priority: Minor > Original Estimate: 6h > Remaining Estimate: 6h > > Sometimes the work threads produced by mk-threads in executor.clj hang on > external resources or other unknown reasons. This makes the workers stop > processing the tuples. I think it is better to kill this worker to resolve > the "hang". I plan to : > 1. like `setup-ticks`, send a system-tick to receive-queue > 2. the tuple-action-fn deal with this system-tick and remember the time that > processes this tuple in the executor-data > 3. when worker do local heartbeat, check the time the executor writes to > executor-data. If the time is long from current (for example, 3 minutes), the > worker does not do the heartbeat. So the supervisor could deal with this > problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1190) System load spikes in recent snapshot
[ https://issues.apache.org/jira/browse/STORM-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-1190. Resolution: Fixed Assignee: Robert Joseph Evans Fix Version/s: 0.11.0 Thanks for all of the help from everyone on this. Especially Daniel Schonfeld who proposed the final solution. The new CPU utilization should be just slightly higher then it was without batching. On a happy side note this reduced the CPU utilization everywhere with batching so now with all of the performance changes that have gone in I can run a 2 worker ThroughputVsLatency topology at 43,000 sentences per second on my MBP, with a max spout pending of 100 and Automatic Back Pressure off. I don't have an apples to apples comparison with before this change. Because it looks like ABP was limiting things to about 35,000 sentences per second. Either way we are running a lot better then we were before where we would max out at about 6,500 sentences per second. > System load spikes in recent snapshot > - > > Key: STORM-1190 > URL: https://issues.apache.org/jira/browse/STORM-1190 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.11.0 > Environment: 10x (CoreOS stable (766.4.0) / k8s 1.0.1 / docker > running on Azure VMs) >Reporter: Michael Schonfeld >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.11.0 > > Attachments: Screenshot 2015-11-08 22.17.57.png, Screenshot > 2015-11-08 22.18.06.png > > > We've been running Storm's snapshots on our production cluster for a little > while now (that back pressure support really helped us), and we've noticed a > sudden spike in system load when going from > commit@ba1250993d10ffc523c9f5464371fbeb406d216f to the current latest > commit@c12e28c829fcfabc0a3a775fb9714968b7e3e349. Both versions were running > the exact same topologies, and there was no significant change in workload. > Not exactly sure how to even begin to debug this, so we ended up just rolling > back. Thoughts? > Stats screenshots attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-885) Heartbeat Server (Pacemaker)
[ https://issues.apache.org/jira/browse/STORM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004357#comment-15004357 ] ASF GitHub Bot commented on STORM-885: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/838#issuecomment-156498367 I'm +1 for adding this functionality, but I would say that vote is non-binding since this is new code/functionality I'm not that familiar with yet. The code looks good to me, but I would defer to those more familiar with it to determine if it's ready. I would like to see more documentation. Specifically: * Rationale * Benefits * Configuration/Usage guidelines I'm okay with handling that as a separate JIRA, but I think it would be needed before we include this in a release. > Heartbeat Server (Pacemaker) > > > Key: STORM-885 > URL: https://issues.apache.org/jira/browse/STORM-885 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Kyle Nusbaum > > Large highly connected topologies and large clusters write a lot of data into > ZooKeeper. The heartbeats, that make up the majority of this data, do not > need to be persisted to disk. Pacemaker is intended to be a secure > replacement for storing the heartbeats without changing anything within the > heartbeats. In the future as more metrics are added in, we may want to look > into switching it over to look more like Heron, where a metrics server is > running for each node/topology. And can be used to aggregate/per-aggregate > them in a more scalable manor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1203. worker metadata file creation does...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/878#issuecomment-156498567 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-885] Heartbeat Server (Pacemaker)
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/838#issuecomment-156498367 I'm +1 for adding this functionality, but I would say that vote is non-binding since this is new code/functionality I'm not that familiar with yet. The code looks good to me, but I would defer to those more familiar with it to determine if it's ready. I would like to see more documentation. Specifically: * Rationale * Benefits * Configuration/Usage guidelines I'm okay with handling that as a separate JIRA, but I think it would be needed before we include this in a release. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1203) worker metadata file creation doesn't use storm.log.dir config
[ https://issues.apache.org/jira/browse/STORM-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004358#comment-15004358 ] ASF GitHub Bot commented on STORM-1203: --- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/878#issuecomment-156498567 +1 > worker metadata file creation doesn't use storm.log.dir config > -- > > Key: STORM-1203 > URL: https://issues.apache.org/jira/browse/STORM-1203 > Project: Apache Storm > Issue Type: Bug >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani > Fix For: 0.11.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-885] Heartbeat Server (Pacemaker)
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/838#issuecomment-156504222 @ptgoetz Sounds good. The good news is that Pacemaker is still off by default, so aside from making ClusterState pluggable, there should be no changes to the existing code path. I'm currently working on documentation for it as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-885) Heartbeat Server (Pacemaker)
[ https://issues.apache.org/jira/browse/STORM-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004414#comment-15004414 ] ASF GitHub Bot commented on STORM-885: -- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/838#issuecomment-156504222 @ptgoetz Sounds good. The good news is that Pacemaker is still off by default, so aside from making ClusterState pluggable, there should be no changes to the existing code path. I'm currently working on documentation for it as well. > Heartbeat Server (Pacemaker) > > > Key: STORM-885 > URL: https://issues.apache.org/jira/browse/STORM-885 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Kyle Nusbaum > > Large highly connected topologies and large clusters write a lot of data into > ZooKeeper. The heartbeats, that make up the majority of this data, do not > need to be persisted to disk. Pacemaker is intended to be a secure > replacement for storing the heartbeats without changing anything within the > heartbeats. In the future as more metrics are added in, we may want to look > into switching it over to look more like Heron, where a metrics server is > running for each node/topology. And can be used to aggregate/per-aggregate > them in a more scalable manor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1167: Add windowing support for storm co...
Github user ptgoetz commented on a diff in the pull request:
https://github.com/apache/storm/pull/855#discussion_r44814116
--- Diff: storm-core/src/jvm/backtype/storm/topology/TopologyBuilder.java
---
@@ -177,6 +179,21 @@ public BoltDeclarer setBolt(String id, IBasicBolt
bolt, Number parallelism_hint)
}
/**
+ * Define a new bolt in this topology. This defines a windowed bolt,
intended
+ * for windowing operations. The {@link
IWindowedBolt#execute(TupleWindow)} method
+ * is triggered for each window interval with the list of current
events in the window.
+ *
+ * @param id the id of this component. This id is referenced by other
components that want to consume this bolt's outputs.
+ * @param bolt the windowed bolt
+ * @param parallelism_hint the number of tasks that should be assigned
to execute this bolt. Each task will run on a thread in a process somwehere
around the cluster.
+ * @return use the returned object to declare the inputs to this
component
+ * @throws IllegalArgumentException if {@code parallelism_hint} is not
positive
+ */
+public BoltDeclarer setBolt(String id, IWindowedBolt bolt, Number
parallelism_hint) throws IllegalArgumentException {
--- End diff --
This API addition will require changes in Flux in order for it to support
windowing.
We should file a JIRA to track that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1167) Add sliding & tumbling window support for core storm
[
https://issues.apache.org/jira/browse/STORM-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004416#comment-15004416
]
ASF GitHub Bot commented on STORM-1167:
---
Github user ptgoetz commented on a diff in the pull request:
https://github.com/apache/storm/pull/855#discussion_r44814116
--- Diff: storm-core/src/jvm/backtype/storm/topology/TopologyBuilder.java
---
@@ -177,6 +179,21 @@ public BoltDeclarer setBolt(String id, IBasicBolt
bolt, Number parallelism_hint)
}
/**
+ * Define a new bolt in this topology. This defines a windowed bolt,
intended
+ * for windowing operations. The {@link
IWindowedBolt#execute(TupleWindow)} method
+ * is triggered for each window interval with the list of current
events in the window.
+ *
+ * @param id the id of this component. This id is referenced by other
components that want to consume this bolt's outputs.
+ * @param bolt the windowed bolt
+ * @param parallelism_hint the number of tasks that should be assigned
to execute this bolt. Each task will run on a thread in a process somwehere
around the cluster.
+ * @return use the returned object to declare the inputs to this
component
+ * @throws IllegalArgumentException if {@code parallelism_hint} is not
positive
+ */
+public BoltDeclarer setBolt(String id, IWindowedBolt bolt, Number
parallelism_hint) throws IllegalArgumentException {
--- End diff --
This API addition will require changes in Flux in order for it to support
windowing.
We should file a JIRA to track that.
> Add sliding & tumbling window support for core storm
>
>
> Key: STORM-1167
> URL: https://issues.apache.org/jira/browse/STORM-1167
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> Currently, topologies that needs windowing support requires writing custom
> logic inside bolts making it tedious to handle the windowing and acking logic
> with custom logic.
> We can add framework level support to core storm bolts to process tuples in a
> time or a count based window. Sliding and tumbling windows can be supported.
> Later this can be extended to trident apis as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-1204) Logviewer should graceful report page-not-found instead of 500 for bad topo-id etc
Kishor Patil created STORM-1204: --- Summary: Logviewer should graceful report page-not-found instead of 500 for bad topo-id etc Key: STORM-1204 URL: https://issues.apache.org/jira/browse/STORM-1204 Project: Apache Storm Issue Type: Bug Components: storm-core Reporter: Kishor Patil Assignee: Kishor Patil Whenever topology-id or filename is wrong or ( in case of secure cluster if user is not authorized), the logviewer shows HTTP-500 exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1205) Make nimbus independent of sync thread to download blobs in nimbus ha and make the blobstore handle it
Sanket Reddy created STORM-1205: --- Summary: Make nimbus independent of sync thread to download blobs in nimbus ha and make the blobstore handle it Key: STORM-1205 URL: https://issues.apache.org/jira/browse/STORM-1205 Project: Apache Storm Issue Type: Story Reporter: Sanket Reddy Priority: Minor The nimbus handles the syncing of blobs logic. It will be nice in to move the sync thread in nimbus for ha from nimbus to blobstore, making the blobstore truly independent entity -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1206) Reduce logviewer memory usage
Zhuo Liu created STORM-1206: --- Summary: Reduce logviewer memory usage Key: STORM-1206 URL: https://issues.apache.org/jira/browse/STORM-1206 Project: Apache Storm Issue Type: Improvement Components: storm-core Reporter: Zhuo Liu Assignee: Zhuo Liu In production, we ran into an issue with logviewers bouncing with out of memory errors. Note that this happens very rarely, we met this in some extreme case when super frequently restarting of workers generates a huge number of gc files (~1M files). What was happening is that if there are lots of log files (~1 M files) for a particular headless user, we would have so many strings resident in memory that logviewer would run out of heap space. We were able to work around this by increasing the heap space, but we should consider putting some sort of an upper bound on the number of files so that we don't run in to this issue, even with the bigger heap. Using the java DirectoryStream can avoid holding all file names in memory during file listing. Also, a multi-round directory cleaner can be introduced to delete files while disk quota is exceeded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-831] Adding jira and central logging li...
Github user kishorvpatil commented on the pull request: https://github.com/apache/storm/pull/559#issuecomment-156562230 @ptgoetz and @revans2 Added MIT license to both LICENSE files. Thanks again for reviewing this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-831) Add Jira and Central Logging URL to UI
[ https://issues.apache.org/jira/browse/STORM-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004748#comment-15004748 ] ASF GitHub Bot commented on STORM-831: -- Github user kishorvpatil commented on the pull request: https://github.com/apache/storm/pull/559#issuecomment-156562230 @ptgoetz and @revans2 Added MIT license to both LICENSE files. Thanks again for reviewing this. > Add Jira and Central Logging URL to UI > -- > > Key: STORM-831 > URL: https://issues.apache.org/jira/browse/STORM-831 > Project: Apache Storm > Issue Type: Documentation > Components: documentation >Reporter: Kishor Patil >Assignee: Kishor Patil >Priority: Trivial > > As a user, I would like to see a link to take me to JIRA for reporting bug. > Also, optionally if link to splunk/logstash/kibana from UI would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-831] Adding jira and central logging li...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/559#issuecomment-156566901 +1 I think the copyright line should read: ``` Copyright (c) 2015 Github, Inc. ``` But that can be done at merge time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-831) Add Jira and Central Logging URL to UI
[ https://issues.apache.org/jira/browse/STORM-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004765#comment-15004765 ] ASF GitHub Bot commented on STORM-831: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/559#issuecomment-156566901 +1 I think the copyright line should read: ``` Copyright (c) 2015 Github, Inc. ``` But that can be done at merge time. > Add Jira and Central Logging URL to UI > -- > > Key: STORM-831 > URL: https://issues.apache.org/jira/browse/STORM-831 > Project: Apache Storm > Issue Type: Documentation > Components: documentation >Reporter: Kishor Patil >Assignee: Kishor Patil >Priority: Trivial > > As a user, I would like to see a link to take me to JIRA for reporting bug. > Also, optionally if link to splunk/logstash/kibana from UI would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1207) Add flux support for handling windowed bolts
Arun Mahadevan created STORM-1207: - Summary: Add flux support for handling windowed bolts Key: STORM-1207 URL: https://issues.apache.org/jira/browse/STORM-1207 Project: Apache Storm Issue Type: Sub-task Reporter: Arun Mahadevan Assignee: Arun Mahadevan Add support to flux for defining and configuring windowed bolts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1167: Add windowing support for storm co...
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/storm/pull/855#discussion_r44855135
--- Diff: storm-core/src/jvm/backtype/storm/topology/TopologyBuilder.java
---
@@ -177,6 +179,21 @@ public BoltDeclarer setBolt(String id, IBasicBolt
bolt, Number parallelism_hint)
}
/**
+ * Define a new bolt in this topology. This defines a windowed bolt,
intended
+ * for windowing operations. The {@link
IWindowedBolt#execute(TupleWindow)} method
+ * is triggered for each window interval with the list of current
events in the window.
+ *
+ * @param id the id of this component. This id is referenced by other
components that want to consume this bolt's outputs.
+ * @param bolt the windowed bolt
+ * @param parallelism_hint the number of tasks that should be assigned
to execute this bolt. Each task will run on a thread in a process somwehere
around the cluster.
+ * @return use the returned object to declare the inputs to this
component
+ * @throws IllegalArgumentException if {@code parallelism_hint} is not
positive
+ */
+public BoltDeclarer setBolt(String id, IWindowedBolt bolt, Number
parallelism_hint) throws IllegalArgumentException {
--- End diff --
Thanks. I 've added a tracking JIRA
https://issues.apache.org/jira/browse/STORM-1207
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1167) Add sliding & tumbling window support for core storm
[
https://issues.apache.org/jira/browse/STORM-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005165#comment-15005165
]
ASF GitHub Bot commented on STORM-1167:
---
Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/storm/pull/855#discussion_r44855135
--- Diff: storm-core/src/jvm/backtype/storm/topology/TopologyBuilder.java
---
@@ -177,6 +179,21 @@ public BoltDeclarer setBolt(String id, IBasicBolt
bolt, Number parallelism_hint)
}
/**
+ * Define a new bolt in this topology. This defines a windowed bolt,
intended
+ * for windowing operations. The {@link
IWindowedBolt#execute(TupleWindow)} method
+ * is triggered for each window interval with the list of current
events in the window.
+ *
+ * @param id the id of this component. This id is referenced by other
components that want to consume this bolt's outputs.
+ * @param bolt the windowed bolt
+ * @param parallelism_hint the number of tasks that should be assigned
to execute this bolt. Each task will run on a thread in a process somwehere
around the cluster.
+ * @return use the returned object to declare the inputs to this
component
+ * @throws IllegalArgumentException if {@code parallelism_hint} is not
positive
+ */
+public BoltDeclarer setBolt(String id, IWindowedBolt bolt, Number
parallelism_hint) throws IllegalArgumentException {
--- End diff --
Thanks. I 've added a tracking JIRA
https://issues.apache.org/jira/browse/STORM-1207
> Add sliding & tumbling window support for core storm
>
>
> Key: STORM-1167
> URL: https://issues.apache.org/jira/browse/STORM-1167
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> Currently, topologies that needs windowing support requires writing custom
> logic inside bolts making it tedious to handle the windowing and acking logic
> with custom logic.
> We can add framework level support to core storm bolts to process tuples in a
> time or a count based window. Sliding and tumbling windows can be supported.
> Later this can be extended to trident apis as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1075 add external module storm-cassandra
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/827#issuecomment-156642074 +1 for adding this. I've done a lot of work with Cassandra in the past (Back when the thrift API was the only game in town) and have always wanted to see first class support for it in Storm -- even if it is an optional component. Moving to the DS Java driver is clearly the only way to go. I'd be willing to act as a sponsor for this as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1075) Storm Cassandra connector
[ https://issues.apache.org/jira/browse/STORM-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005178#comment-15005178 ] ASF GitHub Bot commented on STORM-1075: --- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/827#issuecomment-156642074 +1 for adding this. I've done a lot of work with Cassandra in the past (Back when the thrift API was the only game in town) and have always wanted to see first class support for it in Storm -- even if it is an optional component. Moving to the DS Java driver is clearly the only way to go. I'd be willing to act as a sponsor for this as well. > Storm Cassandra connector > - > > Key: STORM-1075 > URL: https://issues.apache.org/jira/browse/STORM-1075 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Sriharsha Chintalapani >Assignee: Satish Duggana > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
