[jira] [Assigned] (STORM-1217) making small fixes in RAS
[ https://issues.apache.org/jira/browse/STORM-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng reassigned STORM-1217: Assignee: Boyang Jerry Peng > making small fixes in RAS > - > > Key: STORM-1217 > URL: https://issues.apache.org/jira/browse/STORM-1217 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1217) making small fixes in RAS
Boyang Jerry Peng created STORM-1217: Summary: making small fixes in RAS Key: STORM-1217 URL: https://issues.apache.org/jira/browse/STORM-1217 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler
[ https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-898: Attachment: (was: Resource Aware Scheduler for Storm.pdf) > Add priorities and per user resource guarantees to Resource Aware Scheduler > --- > > Key: STORM-898 > URL: https://issues.apache.org/jira/browse/STORM-898 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Attachments: Resource Aware Scheduler for Storm.pdf > > > In a multi-tenant environment we would like to be able to give individual > users a guarantee of how much CPU/Memory/Network they will be able to use in > a cluster. We would also like to know which topologies a user feels are the > most important to keep running if there are not enough resources to run all > of their topologies. > Each user should be able to specify if their topology is production, staging, > or development. Within each of those categories a user should be able to give > a topology a priority, 0 to 10 with 10 being the highest priority (or > something like this). > If there are not enough resources on a cluster to run a topology assume this > topology is running using resources and find the user that is most over their > guaranteed resources. Shoot the lowest priority topology for that user, and > repeat until, this topology is able to run, or this topology would be the one > shot. Ideally we don't actually shoot anything until we know that we would > have made enough room. > If the cluster is over-subscribed and everyone is under their guarantee, and > this topology would not put the user over their guarantee. Shoot the lowest > priority topology in this workers resource pool until there is enough room to > run the topology or this topology is the one that would be shot. We might > also want to think about what to do if we are going to shoot a production > topology in an oversubscribed case, and perhaps we can shoot a non-production > topology instead even if the other user is not over their guarantee. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler
[ https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-898: Attachment: Resource Aware Scheduler for Storm.pdf Documentation > Add priorities and per user resource guarantees to Resource Aware Scheduler > --- > > Key: STORM-898 > URL: https://issues.apache.org/jira/browse/STORM-898 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Attachments: Resource Aware Scheduler for Storm.pdf > > > In a multi-tenant environment we would like to be able to give individual > users a guarantee of how much CPU/Memory/Network they will be able to use in > a cluster. We would also like to know which topologies a user feels are the > most important to keep running if there are not enough resources to run all > of their topologies. > Each user should be able to specify if their topology is production, staging, > or development. Within each of those categories a user should be able to give > a topology a priority, 0 to 10 with 10 being the highest priority (or > something like this). > If there are not enough resources on a cluster to run a topology assume this > topology is running using resources and find the user that is most over their > guaranteed resources. Shoot the lowest priority topology for that user, and > repeat until, this topology is able to run, or this topology would be the one > shot. Ideally we don't actually shoot anything until we know that we would > have made enough room. > If the cluster is over-subscribed and everyone is under their guarantee, and > this topology would not put the user over their guarantee. Shoot the lowest > priority topology in this workers resource pool until there is enough room to > run the topology or this topology is the one that would be shot. We might > also want to think about what to do if we are going to shoot a production > topology in an oversubscribed case, and perhaps we can shoot a non-production > topology instead even if the other user is not over their guarantee. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1370) Nimbus crashes due to bug in MultitenantScheduler
[ https://issues.apache.org/jira/browse/STORM-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng reassigned STORM-1370: Assignee: Boyang Jerry Peng > Nimbus crashes due to bug in MultitenantScheduler > - > > Key: STORM-1370 > URL: https://issues.apache.org/jira/browse/STORM-1370 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > Nimbus crashes from an exception being thrown by the multitenant scheduler > trying to assign executors from an isolated topology to a node that is full. > Error in nimbus.log: > java.lang.IllegalStateException: Trying to assign to a full node > at backtype.storm.scheduler.multitenant.Node.assign(Node.java:232) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at > backtype.storm.scheduler.multitenant.NodePool$RoundRobinSlotScheduler.assignSlotTo(NodePool.java:171) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at > backtype.storm.scheduler.multitenant.IsolatedPool.scheduleAsNeeded(IsolatedPool.java:164) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at > backtype.storm.scheduler.multitenant.MultitenantScheduler.schedule(MultitenantScheduler.java:96) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) ~[?:?] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_40] > at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40] > at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) > ~[clojure-1.6.0.jar:?] > at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) > ~[clojure-1.6.0.jar:?] > at > backtype.storm.daemon.nimbus$compute_new_scheduler_assignments.invoke(nimbus.clj:750) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:806) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.6.0.jar:?] > at > backtype.storm.daemon.nimbus$fn_6009$exec_fn1502auto__6010$fn6020$fn_6021.invoke(nimbus.clj:1245) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at > backtype.storm.daemon.nimbus$fn_6009$exec_fn1502auto__6010$fn_6020.invoke(nimbus.clj:1244) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at backtype.storm.timer$schedule_recurring$this__4635.invoke(timer.clj:105) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at backtype.storm.timer$mk_timer$fn_4618$fn_4619.invoke(timer.clj:50) > [storm-core-0.10.1.y.jar:0.10.1.y] > at backtype.storm.timer$mk_timer$fn__4618.invoke(timer.clj:42) > [storm-core-0.10.1.y.jar:0.10.1.y] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1370) Nimbus crashes due to bug in MultitenantScheduler
Boyang Jerry Peng created STORM-1370: Summary: Nimbus crashes due to bug in MultitenantScheduler Key: STORM-1370 URL: https://issues.apache.org/jira/browse/STORM-1370 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng Nimbus crashes from an exception being thrown by the multitenant scheduler trying to assign executors from an isolated topology to a node that is full. Error in nimbus.log: java.lang.IllegalStateException: Trying to assign to a full node at backtype.storm.scheduler.multitenant.Node.assign(Node.java:232) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.scheduler.multitenant.NodePool$RoundRobinSlotScheduler.assignSlotTo(NodePool.java:171) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.scheduler.multitenant.IsolatedPool.scheduleAsNeeded(IsolatedPool.java:164) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.scheduler.multitenant.MultitenantScheduler.schedule(MultitenantScheduler.java:96) ~[storm-core-0.10.1.y.jar:0.10.1.y] at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_40] at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40] at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:?] at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.6.0.jar:?] at backtype.storm.daemon.nimbus$compute_new_scheduler_assignments.invoke(nimbus.clj:750) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:806) ~[storm-core-0.10.1.y.jar:0.10.1.y] at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.6.0.jar:?] at backtype.storm.daemon.nimbus$fn_6009$exec_fn1502auto__6010$fn6020$fn_6021.invoke(nimbus.clj:1245) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.daemon.nimbus$fn_6009$exec_fn1502auto__6010$fn_6020.invoke(nimbus.clj:1244) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.timer$schedule_recurring$this__4635.invoke(timer.clj:105) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.timer$mk_timer$fn_4618$fn_4619.invoke(timer.clj:50) [storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.timer$mk_timer$fn__4618.invoke(timer.clj:42) [storm-core-0.10.1.y.jar:0.10.1.y] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1370) Bug fixes for MultitenantScheduler
[ https://issues.apache.org/jira/browse/STORM-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1370: - Summary: Bug fixes for MultitenantScheduler (was: Nimbus crashes due to bug in MultitenantScheduler) > Bug fixes for MultitenantScheduler > -- > > Key: STORM-1370 > URL: https://issues.apache.org/jira/browse/STORM-1370 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > Nimbus crashes from an exception being thrown by the multitenant scheduler > trying to assign executors from an isolated topology to a node that is full. > Error in nimbus.log: > java.lang.IllegalStateException: Trying to assign to a full node > at backtype.storm.scheduler.multitenant.Node.assign(Node.java:232) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at > backtype.storm.scheduler.multitenant.NodePool$RoundRobinSlotScheduler.assignSlotTo(NodePool.java:171) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at > backtype.storm.scheduler.multitenant.IsolatedPool.scheduleAsNeeded(IsolatedPool.java:164) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at > backtype.storm.scheduler.multitenant.MultitenantScheduler.schedule(MultitenantScheduler.java:96) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) ~[?:?] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_40] > at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40] > at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) > ~[clojure-1.6.0.jar:?] > at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) > ~[clojure-1.6.0.jar:?] > at > backtype.storm.daemon.nimbus$compute_new_scheduler_assignments.invoke(nimbus.clj:750) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:806) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.6.0.jar:?] > at > backtype.storm.daemon.nimbus$fn_6009$exec_fn1502auto__6010$fn6020$fn_6021.invoke(nimbus.clj:1245) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at > backtype.storm.daemon.nimbus$fn_6009$exec_fn1502auto__6010$fn_6020.invoke(nimbus.clj:1244) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at backtype.storm.timer$schedule_recurring$this__4635.invoke(timer.clj:105) > ~[storm-core-0.10.1.y.jar:0.10.1.y] > at backtype.storm.timer$mk_timer$fn_4618$fn_4619.invoke(timer.clj:50) > [storm-core-0.10.1.y.jar:0.10.1.y] > at backtype.storm.timer$mk_timer$fn__4618.invoke(timer.clj:42) > [storm-core-0.10.1.y.jar:0.10.1.y] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1370) Bug fixes for MultitenantScheduler
[ https://issues.apache.org/jira/browse/STORM-1370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1370: - Description: Bug 1: Sort nodes by slots used when scheduing isolated Because nimbus removes "dead" slots (slots for which their workers have not yet sent a heartbeat) before schedule is called, we cannot rely on teh number of free slots on a node. This will break for clusters whose nodes have a heterogenious number of slots configured. Derive the effective number of hosts by taking the minimum of the config's value and the number of executors in the topology. If the user requests the topology be scheduled among a number of hosts, then retry scheduling when the effective number does not match the scheduled number. Bug 2: Nimbus crashes from an exception being thrown by the multitenant scheduler trying to assign executors from an isolated topology to a node that is full. Error in nimbus.log: java.lang.IllegalStateException: Trying to assign to a full node x at backtype.storm.scheduler.multitenant.Node.assign(Node.java:232) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.scheduler.multitenant.NodePool$RoundRobinSlotScheduler.assignSlotTo(NodePool.java:171) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.scheduler.multitenant.IsolatedPool.scheduleAsNeeded(IsolatedPool.java:164) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.scheduler.multitenant.MultitenantScheduler.schedule(MultitenantScheduler.java:96) ~[storm-core-0.10.1.y.jar:0.10.1.y] at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_40] at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40] at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:?] at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.6.0.jar:?] at backtype.storm.daemon.nimbus$compute_new_scheduler_assignments.invoke(nimbus.clj:750) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:806) ~[storm-core-0.10.1.y.jar:0.10.1.y] at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.6.0.jar:?] at backtype.storm.daemon.nimbus$fn_6009$exec_fn1502auto__6010$fn6020$fn_6021.invoke(nimbus.clj:1245) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.daemon.nimbus$fn_6009$exec_fn1502auto__6010$fn_6020.invoke(nimbus.clj:1244) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.timer$schedule_recurring$this__4635.invoke(timer.clj:105) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.timer$mk_timer$fn_4618$fn_4619.invoke(timer.clj:50) [storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.timer$mk_timer$fn__4618.invoke(timer.clj:42) [storm-core-0.10.1.y.jar:0.10.1.y] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40] was: Nimbus crashes from an exception being thrown by the multitenant scheduler trying to assign executors from an isolated topology to a node that is full. Error in nimbus.log: java.lang.IllegalStateException: Trying to assign to a full node at backtype.storm.scheduler.multitenant.Node.assign(Node.java:232) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.scheduler.multitenant.NodePool$RoundRobinSlotScheduler.assignSlotTo(NodePool.java:171) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.scheduler.multitenant.IsolatedPool.scheduleAsNeeded(IsolatedPool.java:164) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.scheduler.multitenant.MultitenantScheduler.schedule(MultitenantScheduler.java:96) ~[storm-core-0.10.1.y.jar:0.10.1.y] at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_40] at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40] at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:?] at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.6.0.jar:?] at backtype.storm.daemon.nimbus$compute_new_scheduler_assignments.invoke(nimbus.clj:750) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:806) ~[storm-core-0.10.1.y.jar:0.10.1.y] at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.6.0.jar:?] at backtype.storm.daemon.nimbus$fn_6009$exec_fn1502auto__6010$fn6020$fn_6021.invoke(nimbus.clj:1245) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.daemon.nimbus$fn_6009$exec_fn1502auto__6010$fn_6020.invoke(nimbus.clj:1244) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.timer$schedule_recurring$this__4635.invoke(timer.clj:105) ~[storm-core-0.10.1.y.jar:0.10.1.y] at backtype.storm.timer$mk_timer$fn_4618$fn_4619.invoke(timer.clj:50) [s
[jira] [Updated] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1450: - Summary: Fix bugs and refactor code in ResourceAwareScheduler (was: Refactor some code in ResourceAwareScheduler) > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1450: - Description: Bug fixes: Minor bug in displaying the assigned resources for a supervisor on the UI this line: _ (reset! (:node-id->resources nimbus) (.getSupervisorsResourcesMap cluster)) needs to be _ (reset! (:node-id->resources nimbus) (merge (deref (:node-id->resources nimbus)) (.getSupervisorsResourcesMap cluster))) > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Bug fixes: > Minor bug in displaying the assigned resources for a supervisor on the UI > this line: > _ (reset! (:node-id->resources nimbus) (.getSupervisorsResourcesMap cluster)) > needs to be > _ (reset! (:node-id->resources nimbus) (merge (deref (:node-id->resources > nimbus)) (.getSupervisorsResourcesMap cluster))) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1450: - Description: Code refactored: 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. Each RAS_Node will now be initialized with a map of all its assignments. Each RAS_Node will also figure out resources used and available. 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly modify it 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so that the semantics of what a scheduling strategy should do will be more clear. Each scheduling strategy shouldn't be actually assigning anything. The strategy should only calculate a scheduling. Bug fixes: 1. Minor bug in displaying the assigned resources for a supervisor on the UI. The function updateSupervisorResources was placed in the wrong place 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that was done. was: Bug fixes: Minor bug in displaying the assigned resources for a supervisor on the UI this line: _ (reset! (:node-id->resources nimbus) (.getSupervisorsResourcesMap cluster)) needs to be _ (reset! (:node-id->resources nimbus) (merge (deref (:node-id->resources nimbus)) (.getSupervisorsResourcesMap cluster))) > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1450: - Description: Code refactored: 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. Each RAS_Node will now be initialized with a map of all its assignments. Each RAS_Node will also figure out resources used and available. Removed unnecessary functions. 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly modify it 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so that the semantics of what a scheduling strategy should do will be more clear. Each scheduling strategy shouldn't be actually assigning anything. The strategy should only calculate a scheduling. Bug fixes: 1. Minor bug in displaying the assigned resources for a supervisor on the UI. The function updateSupervisorResources was placed in the wrong place 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that was done. was: Code refactored: 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. Each RAS_Node will now be initialized with a map of all its assignments. Each RAS_Node will also figure out resources used and available. 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly modify it 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so that the semantics of what a scheduling strategy should do will be more clear. Each scheduling strategy shouldn't be actually assigning anything. The strategy should only calculate a scheduling. Bug fixes: 1. Minor bug in displaying the assigned resources for a supervisor on the UI. The function updateSupervisorResources was placed in the wrong place 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that was done. > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler
[ https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng resolved STORM-1450. -- Resolution: Done > Fix bugs and refactor code in ResourceAwareScheduler > > > Key: STORM-1450 > URL: https://issues.apache.org/jira/browse/STORM-1450 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Code refactored: > 1. Refactor RAS_Nodes. Pushed some of the functionality in to RAS_Nodes. > Each RAS_Node will now be initialized with a map of all its assignments. > Each RAS_Node will also figure out resources used and available. Removed > unnecessary functions. > 2. Made WorkerSlot immutable so that a scheduling strategy won't mistakenly > modify it > 3. Added a wrapping layer for RAS_Node to feed into scheduling strategies so > that the semantics of what a scheduling strategy should do will be more > clear. Each scheduling strategy shouldn't be actually assigning anything. > The strategy should only calculate a scheduling. > Bug fixes: > 1. Minor bug in displaying the assigned resources for a supervisor on the UI. > The function updateSupervisorResources was placed in the wrong place > 2. Minor bug fix in freeing memory in RAS_Node there was some wrong math that > was done. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1336) Evalute/Port JStorm cgroup support
[ https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15107291#comment-15107291 ] Boyang Jerry Peng commented on STORM-1336: -- Hello Everyone, Currently at Yahoo, we want to enable the Resource Aware Scheduler we built to have cgroup support. The CGroup code that is part of JStorm looks good and perhaps we can modify it slightly so that the Resource Aware Scheduler can interact with it. What I would like to do is modify the CGroup code that already exists in JStorm to be able to start jvm workers that is limited to the amount of resources that the resource aware scheduler has allocated for that worker. I would like to have a discussion (especially with people that worked on JStorm) about how we can integrate support for the resource aware scheduler into the existing CGroups code. I know the folks at Alibaba is working on converting the supervisor.clj to java. What is the status of that? > Evalute/Port JStorm cgroup support > -- > > Key: STORM-1336 > URL: https://issues.apache.org/jira/browse/STORM-1336 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: 冯健 > Labels: jstorm-merger > > Supports controlling the upper limit of CPU core usage for a worker using > cgroups > Sounds like a good start, will be nice to integrate it with RAS requests too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (STORM-1336) Evalute/Port JStorm cgroup support
[ https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15107291#comment-15107291 ] Boyang Jerry Peng edited comment on STORM-1336 at 1/19/16 7:50 PM: --- Hello Everyone, Currently at Yahoo, we want to enable the Resource Aware Scheduler we built to have cgroup support. The CGroup code that is part of JStorm looks good and perhaps we can modify it slightly so that the Resource Aware Scheduler can interact with it. What I would like to do is modify the CGroup code that already exists in JStorm to be able to start jvm workers that is limited to the amount of resources that the resource aware scheduler has allocated for that worker and move it to Storm. I would like to have a discussion (especially with people that worked on JStorm) about how we can integrate support for the resource aware scheduler into the existing CGroups code. I know the folks at Alibaba is working on converting the supervisor.clj to java. What is the status of that? was (Author: jerrypeng): Hello Everyone, Currently at Yahoo, we want to enable the Resource Aware Scheduler we built to have cgroup support. The CGroup code that is part of JStorm looks good and perhaps we can modify it slightly so that the Resource Aware Scheduler can interact with it. What I would like to do is modify the CGroup code that already exists in JStorm to be able to start jvm workers that is limited to the amount of resources that the resource aware scheduler has allocated for that worker. I would like to have a discussion (especially with people that worked on JStorm) about how we can integrate support for the resource aware scheduler into the existing CGroups code. I know the folks at Alibaba is working on converting the supervisor.clj to java. What is the status of that? > Evalute/Port JStorm cgroup support > -- > > Key: STORM-1336 > URL: https://issues.apache.org/jira/browse/STORM-1336 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: 冯健 > Labels: jstorm-merger > > Supports controlling the upper limit of CPU core usage for a worker using > cgroups > Sounds like a good start, will be nice to integrate it with RAS requests too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1336) Evalute/Port JStorm cgroup support
[ https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15108916#comment-15108916 ] Boyang Jerry Peng commented on STORM-1336: -- [~basti.lj] I don't think it will be too hard to modify the existing cgroup code in JStorm to support RAS. I can come up with a implementation pretty quick I think. How about this. I'll work on implementing capability between the existing CGroups code and RAS. I'll create version of the CGroups code that can let RAS set the resource limits for the user or allow the user to set them manually. After I am done with my implementation, I will make a PR and we can pull in the CGroups code that way. How about that? Is [~fengjian] working on this jira or can I assign it to myself? > Evalute/Port JStorm cgroup support > -- > > Key: STORM-1336 > URL: https://issues.apache.org/jira/browse/STORM-1336 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: 冯健 > Labels: jstorm-merger > > Supports controlling the upper limit of CPU core usage for a worker using > cgroups > Sounds like a good start, will be nice to integrate it with RAS requests too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1336) Evalute/Port JStorm cgroup support
[ https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng reassigned STORM-1336: Assignee: Boyang Jerry Peng (was: 冯健) > Evalute/Port JStorm cgroup support > -- > > Key: STORM-1336 > URL: https://issues.apache.org/jira/browse/STORM-1336 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Labels: jstorm-merger > > Supports controlling the upper limit of CPU core usage for a worker using > cgroups > Sounds like a good start, will be nice to integrate it with RAS requests too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1336) Evalute/Port JStorm cgroup support
[ https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110852#comment-15110852 ] Boyang Jerry Peng commented on STORM-1336: -- Ok sounds good > Evalute/Port JStorm cgroup support > -- > > Key: STORM-1336 > URL: https://issues.apache.org/jira/browse/STORM-1336 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Labels: jstorm-merger > > Supports controlling the upper limit of CPU core usage for a worker using > cgroups > Sounds like a good start, will be nice to integrate it with RAS requests too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1507) Getting the parallelism of components via thrift API after a rebalance/change in parallelism is incorrect
Boyang Jerry Peng created STORM-1507: Summary: Getting the parallelism of components via thrift API after a rebalance/change in parallelism is incorrect Key: STORM-1507 URL: https://issues.apache.org/jira/browse/STORM-1507 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng When the parallelism of a component in a topology is changed, the thrift APIs to get the parallelism of components still return the original -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1507) Getting the parallelism of components via thrift API after a rebalance/change in parallelism is incorrect
[ https://issues.apache.org/jira/browse/STORM-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1507: - Description: When the parallelism of a component in a topology is changed, the thrift APIs to get the parallelism of components still return the original parallelism of the component and not the parallelism it changed to (was: When the parallelism of a component in a topology is changed, the thrift APIs to get the parallelism of components still return the original ) > Getting the parallelism of components via thrift API after a rebalance/change > in parallelism is incorrect > - > > Key: STORM-1507 > URL: https://issues.apache.org/jira/browse/STORM-1507 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng > > When the parallelism of a component in a topology is changed, the thrift APIs > to get the parallelism of components still return the original parallelism of > the component and not the parallelism it changed to -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1507) Getting the parallelism of components via thrift API after a rebalance/change in parallelism is incorrect
[ https://issues.apache.org/jira/browse/STORM-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1507: - Description: When the parallelism of a component in a topology is changed, the thrift APIs to get the parallelism of components still return the original parallelism of the component and not the parallelism it changed to. This due to the fact that the thrift object responsible for holding the information about a component only gets initialized once in nimbus.clj but never gets updated when a change in parallelism happens (was: When the parallelism of a component in a topology is changed, the thrift APIs to get the parallelism of components still return the original parallelism of the component and not the parallelism it changed to) > Getting the parallelism of components via thrift API after a rebalance/change > in parallelism is incorrect > - > > Key: STORM-1507 > URL: https://issues.apache.org/jira/browse/STORM-1507 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng > > When the parallelism of a component in a topology is changed, the thrift APIs > to get the parallelism of components still return the original parallelism of > the component and not the parallelism it changed to. This due to the fact > that the thrift object responsible for holding the information about a > component only gets initialized once in nimbus.clj but never gets updated > when a change in parallelism happens -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1507) Getting the parallelism of components via thrift API after a rebalance/change in parallelism is incorrect
[
https://issues.apache.org/jira/browse/STORM-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boyang Jerry Peng updated STORM-1507:
-
Description: When the parallelism of a component in a topology is changed,
the thrift APIs to get the parallelism ("get_parallelism_hint()") of components
still return the original parallelism of the component and not the parallelism
it changed to. This due to the fact that the thrift object responsible for
holding the information about a component only gets initialized once in
nimbus.clj but never gets updated when a change in parallelism happens (was:
When the parallelism of a component in a topology is changed, the thrift APIs
to get the parallelism of components still return the original parallelism of
the component and not the parallelism it changed to. This due to the fact that
the thrift object responsible for holding the information about a component
only gets initialized once in nimbus.clj but never gets updated when a change
in parallelism happens)
> Getting the parallelism of components via thrift API after a rebalance/change
> in parallelism is incorrect
> -
>
> Key: STORM-1507
> URL: https://issues.apache.org/jira/browse/STORM-1507
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Boyang Jerry Peng
>
> When the parallelism of a component in a topology is changed, the thrift APIs
> to get the parallelism ("get_parallelism_hint()") of components still return
> the original parallelism of the component and not the parallelism it changed
> to. This due to the fact that the thrift object responsible for holding the
> information about a component only gets initialized once in nimbus.clj but
> never gets updated when a change in parallelism happens
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-1507) Getting the parallelism of components via thrift API after a rebalance/change in parallelism is incorrect
[
https://issues.apache.org/jira/browse/STORM-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boyang Jerry Peng updated STORM-1507:
-
Description: When the parallelism of a component in a topology is changed,
the thrift APIs to get the parallelism ("get_parallelism_hint()") of components
still return the original parallelism of the component and not the parallelism
it changed to. This due to the fact that the thrift object responsible for
holding the information about a component only gets initialized once in
nimbus.clj but never gets updated when a change in parallelism happens. This
may also cause the UI to not display the correct parallelism as well. (was:
When the parallelism of a component in a topology is changed, the thrift APIs
to get the parallelism ("get_parallelism_hint()") of components still return
the original parallelism of the component and not the parallelism it changed
to. This due to the fact that the thrift object responsible for holding the
information about a component only gets initialized once in nimbus.clj but
never gets updated when a change in parallelism happens)
> Getting the parallelism of components via thrift API after a rebalance/change
> in parallelism is incorrect
> -
>
> Key: STORM-1507
> URL: https://issues.apache.org/jira/browse/STORM-1507
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Boyang Jerry Peng
>
> When the parallelism of a component in a topology is changed, the thrift APIs
> to get the parallelism ("get_parallelism_hint()") of components still return
> the original parallelism of the component and not the parallelism it changed
> to. This due to the fact that the thrift object responsible for holding the
> information about a component only gets initialized once in nimbus.clj but
> never gets updated when a change in parallelism happens. This may also cause
> the UI to not display the correct parallelism as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-1519) Storm syslog logging not confirming to RFC5426 3.1
Boyang Jerry Peng created STORM-1519: Summary: Storm syslog logging not confirming to RFC5426 3.1 Key: STORM-1519 URL: https://issues.apache.org/jira/browse/STORM-1519 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng AS per RFC 5426 section 3.1, there should be only one message per datagram: 3.1. One Message Per Datagram Each syslog UDP datagram MUST contain only one syslog message, which MAY be complete or truncated. The message MUST be formatted and truncated according to RFC 5424 [2]. Additional data MUST NOT be present in the datagram payload. For example UDP package: <174>1 2016-02-02T22:44:05.558Z aboderode.corp.ne1.yahoo.com [nimbus] - [nobody:S0] [mdc@18060 ClassName="?"] Using custom scheduler: backtype.storm.scheduler.bridge.MultitenantResourceAwareBridgeScheduler <174>1 2016-02-02T22:44:05.591Z aboderode.corp.ne1.yahoo.com [nimbus] - [nobody:S0] [mdc@18060 ClassName="?"] Creating new blob store based in /home/y/var/storm/blobs but the two log messages should be each in their own udp packet -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1519) Storm syslog logging not confirming to RFC5426 3.1
[ https://issues.apache.org/jira/browse/STORM-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1519: - Description: AS per RFC 5426 section 3.1, there should be only one message per datagram: 3.1. One Message Per Datagram Each syslog UDP datagram MUST contain only one syslog message, which MAY be complete or truncated. The message MUST be formatted and truncated according to RFC 5424 [2]. Additional data MUST NOT be present in the datagram payload. For example UDP package: <174>1 2016-02-02T22:44:05.558Z localhost [nimbus] - [nobody:S0] [mdc@18060 ClassName="?"] Using custom scheduler: backtype.storm.scheduler.bridge.MultitenantResourceAwareBridgeScheduler <174>1 2016-02-02T22:44:05.591Z localhost [nimbus] - [nobody:S0] [mdc@18060 ClassName="?"] Creating new blob store based in /home/y/var/storm/blobs but the two log messages should be each in their own udp packet was: AS per RFC 5426 section 3.1, there should be only one message per datagram: 3.1. One Message Per Datagram Each syslog UDP datagram MUST contain only one syslog message, which MAY be complete or truncated. The message MUST be formatted and truncated according to RFC 5424 [2]. Additional data MUST NOT be present in the datagram payload. For example UDP package: <174>1 2016-02-02T22:44:05.558Z aboderode.corp.ne1.yahoo.com [nimbus] - [nobody:S0] [mdc@18060 ClassName="?"] Using custom scheduler: backtype.storm.scheduler.bridge.MultitenantResourceAwareBridgeScheduler <174>1 2016-02-02T22:44:05.591Z aboderode.corp.ne1.yahoo.com [nimbus] - [nobody:S0] [mdc@18060 ClassName="?"] Creating new blob store based in /home/y/var/storm/blobs but the two log messages should be each in their own udp packet > Storm syslog logging not confirming to RFC5426 3.1 > -- > > Key: STORM-1519 > URL: https://issues.apache.org/jira/browse/STORM-1519 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng > > AS per RFC 5426 section 3.1, there should be only one message per datagram: > 3.1. One Message Per Datagram > Each syslog UDP datagram MUST contain only one syslog message, which > MAY be complete or truncated. The message MUST be formatted and > truncated according to RFC 5424 [2]. Additional data MUST NOT be > present in the datagram payload. > For example > UDP package: <174>1 2016-02-02T22:44:05.558Z localhost [nimbus] - [nobody:S0] > [mdc@18060 ClassName="?"] Using custom scheduler: > backtype.storm.scheduler.bridge.MultitenantResourceAwareBridgeScheduler > <174>1 2016-02-02T22:44:05.591Z localhost [nimbus] - [nobody:S0] [mdc@18060 > ClassName="?"] Creating new blob store based in /home/y/var/storm/blobs > but the two log messages should be each in their own udp packet -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1519) Storm syslog logging not confirming to RFC5426 3.1
[ https://issues.apache.org/jira/browse/STORM-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129257#comment-15129257 ] Boyang Jerry Peng commented on STORM-1519: -- though changing the syslog setting in cluster.xml immediateFlush="false" => "true" seems to fix the issue Description from log4j2 website: immediateFlush - When set to true - the default, each write will be followed by a flush. This will guarantee the data is written to disk but could impact performance. > Storm syslog logging not confirming to RFC5426 3.1 > -- > > Key: STORM-1519 > URL: https://issues.apache.org/jira/browse/STORM-1519 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng > > AS per RFC 5426 section 3.1, there should be only one message per datagram: > 3.1. One Message Per Datagram > Each syslog UDP datagram MUST contain only one syslog message, which > MAY be complete or truncated. The message MUST be formatted and > truncated according to RFC 5424 [2]. Additional data MUST NOT be > present in the datagram payload. > For example > UDP package: <174>1 2016-02-02T22:44:05.558Z localhost [nimbus] - [nobody:S0] > [mdc@18060 ClassName="?"] Using custom scheduler: > backtype.storm.scheduler.bridge.MultitenantResourceAwareBridgeScheduler > <174>1 2016-02-02T22:44:05.591Z localhost [nimbus] - [nobody:S0] [mdc@18060 > ClassName="?"] Creating new blob store based in /home/y/var/storm/blobs > but the two log messages should be each in their own udp packet -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1519) Storm syslog logging not confirming to RFC5426 3.1
[ https://issues.apache.org/jira/browse/STORM-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1519: - Priority: Minor (was: Major) > Storm syslog logging not confirming to RFC5426 3.1 > -- > > Key: STORM-1519 > URL: https://issues.apache.org/jira/browse/STORM-1519 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Priority: Minor > > AS per RFC 5426 section 3.1, there should be only one message per datagram: > 3.1. One Message Per Datagram > Each syslog UDP datagram MUST contain only one syslog message, which > MAY be complete or truncated. The message MUST be formatted and > truncated according to RFC 5424 [2]. Additional data MUST NOT be > present in the datagram payload. > For example > UDP package: <174>1 2016-02-02T22:44:05.558Z localhost [nimbus] - [nobody:S0] > [mdc@18060 ClassName="?"] Using custom scheduler: > backtype.storm.scheduler.bridge.MultitenantResourceAwareBridgeScheduler > <174>1 2016-02-02T22:44:05.591Z localhost [nimbus] - [nobody:S0] [mdc@18060 > ClassName="?"] Creating new blob store based in /home/y/var/storm/blobs > but the two log messages should be each in their own udp packet -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1253) port backtype.storm.timer to java
[ https://issues.apache.org/jira/browse/STORM-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng reassigned STORM-1253: Assignee: Boyang Jerry Peng > port backtype.storm.timer to java > -- > > Key: STORM-1253 > URL: https://issues.apache.org/jira/browse/STORM-1253 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Labels: java-migration, jstorm-merger > > Timer like class that uses simulated time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1253) port backtype.storm.timer to java
[ https://issues.apache.org/jira/browse/STORM-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143249#comment-15143249 ] Boyang Jerry Peng commented on STORM-1253: -- Timer :reset-log-levels-timer never gets cancelled in worker.clj. So I also added the code to cancel it along with the other timer events > port backtype.storm.timer to java > -- > > Key: STORM-1253 > URL: https://issues.apache.org/jira/browse/STORM-1253 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Labels: java-migration, jstorm-merger > > Timer like class that uses simulated time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1538) TTransportException thrown when trying to visit topology page
Boyang Jerry Peng created STORM-1538: Summary: TTransportException thrown when trying to visit topology page Key: STORM-1538 URL: https://issues.apache.org/jira/browse/STORM-1538 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng Priority: Critical When trying to access the topology page on the storm ui, exceptions are thrown. In nimbus log: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.core$_PLUS_ at clojure.lang.RT.seqFrom(RT.java:528) at clojure.lang.RT.seq(RT.java:509) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:26) at clojure.core.protocols$fn__6500.invoke(protocols.clj:83) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6515) at org.apache.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:564) at org.apache.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:744) at org.apache.storm.stats$fn__1257.invoke(stats.clj:792) at clojure.lang.MultiFn.invoke(MultiFn.java:243) at clojure.core$partial$fn__4529.invoke(core.clj:2501) at clojure.core.protocols$fn__6523.invoke(protocols.clj:167) at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6519) at org.apache.storm.stats$aggregate_topo_stats.invoke(stats.clj:874) at org.apache.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1037) at org.apache.storm.daemon.nimbus$fn__4817$exec_fn__2064__auto__$reify__4846.getTopologyPageInfo(nimbus.clj:2105) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3800) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3784) at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1538) TTransportException thrown when trying to visit topology page
[ https://issues.apache.org/jira/browse/STORM-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1538: - Description: When trying to access the topology page on the storm ui, exceptions are thrown. Visiting a topology page displays a TTransportException. The cause can be seen in nimbus.log: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.core$_PLUS_ at clojure.lang.RT.seqFrom(RT.java:528) at clojure.lang.RT.seq(RT.java:509) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:26) at clojure.core.protocols$fn__6500.invoke(protocols.clj:83) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6515) at org.apache.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:564) at org.apache.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:744) at org.apache.storm.stats$fn__1257.invoke(stats.clj:792) at clojure.lang.MultiFn.invoke(MultiFn.java:243) at clojure.core$partial$fn__4529.invoke(core.clj:2501) at clojure.core.protocols$fn__6523.invoke(protocols.clj:167) at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6519) at org.apache.storm.stats$aggregate_topo_stats.invoke(stats.clj:874) at org.apache.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1037) at org.apache.storm.daemon.nimbus$fn__4817$exec_fn__2064__auto__$reify__4846.getTopologyPageInfo(nimbus.clj:2105) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3800) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3784) at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) was: When trying to access the topology page on the storm ui, exceptions are thrown. In nimbus log: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.core$_PLUS_ at clojure.lang.RT.seqFrom(RT.java:528) at clojure.lang.RT.seq(RT.java:509) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:26) at clojure.core.protocols$fn__6500.invoke(protocols.clj:83) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6515) at org.apache.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:564) at org.apache.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:744) at org.apache.storm.stats$fn__1257.invoke(stats.clj:792) at clojure.lang.MultiFn.invoke(MultiFn.java:243) at clojure.core$partial$fn__4529.invoke(core.clj:2501) at clojure.core.protocols$fn__6523.invoke(protocols.clj:167) at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6519) at org.apache.storm.stats$aggregate_topo_stats.invoke(stats.clj:874) at org.apache.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1037) at org.apache.storm.daemon.nimbus$fn__4817$exec_fn__2064__auto__$reify__4846.getTopologyPageInfo(nimbus.clj:2105) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3800) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3784) at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.storm.security.
[jira] [Updated] (STORM-1538) TTransportException thrown when trying to visit topology page in UI
[ https://issues.apache.org/jira/browse/STORM-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1538: - Summary: TTransportException thrown when trying to visit topology page in UI (was: TTransportException thrown when trying to visit topology page) > TTransportException thrown when trying to visit topology page in UI > --- > > Key: STORM-1538 > URL: https://issues.apache.org/jira/browse/STORM-1538 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Priority: Critical > > When trying to access the topology page on the storm ui, exceptions are > thrown. > Visiting a topology page displays a TTransportException. > The cause can be seen in nimbus.log: > java.lang.IllegalArgumentException: Don't know how to create ISeq from: > clojure.core$_PLUS_ > at clojure.lang.RT.seqFrom(RT.java:528) > at clojure.lang.RT.seq(RT.java:509) > at clojure.core$seq__4128.invoke(core.clj:137) > at clojure.core.protocols$seq_reduce.invoke(protocols.clj:26) > at clojure.core.protocols$fn__6500.invoke(protocols.clj:83) > at > clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) > at clojure.core$reduce.invoke(core.clj:6515) > at > org.apache.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:564) > at > org.apache.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:744) > at org.apache.storm.stats$fn__1257.invoke(stats.clj:792) > at clojure.lang.MultiFn.invoke(MultiFn.java:243) > at clojure.core$partial$fn__4529.invoke(core.clj:2501) > at clojure.core.protocols$fn__6523.invoke(protocols.clj:167) > at > clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19) > at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) > at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) > at > clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) > at clojure.core$reduce.invoke(core.clj:6519) > at org.apache.storm.stats$aggregate_topo_stats.invoke(stats.clj:874) > at org.apache.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1037) > at > org.apache.storm.daemon.nimbus$fn__4817$exec_fn__2064__auto__$reify__4846.getTopologyPageInfo(nimbus.clj:2105) > at > org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3800) > at > org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3784) > at > org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) > at > org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) > at > org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) > at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1538) Exception being thrown after Utils conversion to java
[ https://issues.apache.org/jira/browse/STORM-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1538: - Summary: Exception being thrown after Utils conversion to java (was: TTransportException thrown when trying to visit topology page in UI) > Exception being thrown after Utils conversion to java > - > > Key: STORM-1538 > URL: https://issues.apache.org/jira/browse/STORM-1538 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Priority: Critical > > TTransportException thrown when trying to visit topology page in UI: > When trying to access the topology page on the storm ui, exceptions are > thrown. > Visiting a topology page displays a TTransportException. > The cause can be seen in nimbus.log: > java.lang.IllegalArgumentException: Don't know how to create ISeq from: > clojure.core$_PLUS_ > at clojure.lang.RT.seqFrom(RT.java:528) > at clojure.lang.RT.seq(RT.java:509) > at clojure.core$seq__4128.invoke(core.clj:137) > at clojure.core.protocols$seq_reduce.invoke(protocols.clj:26) > at clojure.core.protocols$fn__6500.invoke(protocols.clj:83) > at > clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) > at clojure.core$reduce.invoke(core.clj:6515) > at > org.apache.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:564) > at > org.apache.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:744) > at org.apache.storm.stats$fn__1257.invoke(stats.clj:792) > at clojure.lang.MultiFn.invoke(MultiFn.java:243) > at clojure.core$partial$fn__4529.invoke(core.clj:2501) > at clojure.core.protocols$fn__6523.invoke(protocols.clj:167) > at > clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19) > at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) > at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) > at > clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) > at clojure.core$reduce.invoke(core.clj:6519) > at org.apache.storm.stats$aggregate_topo_stats.invoke(stats.clj:874) > at org.apache.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1037) > at > org.apache.storm.daemon.nimbus$fn__4817$exec_fn__2064__auto__$reify__4846.getTopologyPageInfo(nimbus.clj:2105) > at > org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3800) > at > org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3784) > at > org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) > at > org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) > at > org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) > at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1538) TTransportException thrown when trying to visit topology page in UI
[ https://issues.apache.org/jira/browse/STORM-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1538: - Description: TTransportException thrown when trying to visit topology page in UI: When trying to access the topology page on the storm ui, exceptions are thrown. Visiting a topology page displays a TTransportException. The cause can be seen in nimbus.log: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.core$_PLUS_ at clojure.lang.RT.seqFrom(RT.java:528) at clojure.lang.RT.seq(RT.java:509) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:26) at clojure.core.protocols$fn__6500.invoke(protocols.clj:83) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6515) at org.apache.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:564) at org.apache.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:744) at org.apache.storm.stats$fn__1257.invoke(stats.clj:792) at clojure.lang.MultiFn.invoke(MultiFn.java:243) at clojure.core$partial$fn__4529.invoke(core.clj:2501) at clojure.core.protocols$fn__6523.invoke(protocols.clj:167) at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6519) at org.apache.storm.stats$aggregate_topo_stats.invoke(stats.clj:874) at org.apache.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1037) at org.apache.storm.daemon.nimbus$fn__4817$exec_fn__2064__auto__$reify__4846.getTopologyPageInfo(nimbus.clj:2105) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3800) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3784) at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) was: When trying to access the topology page on the storm ui, exceptions are thrown. Visiting a topology page displays a TTransportException. The cause can be seen in nimbus.log: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.core$_PLUS_ at clojure.lang.RT.seqFrom(RT.java:528) at clojure.lang.RT.seq(RT.java:509) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:26) at clojure.core.protocols$fn__6500.invoke(protocols.clj:83) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6515) at org.apache.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:564) at org.apache.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:744) at org.apache.storm.stats$fn__1257.invoke(stats.clj:792) at clojure.lang.MultiFn.invoke(MultiFn.java:243) at clojure.core$partial$fn__4529.invoke(core.clj:2501) at clojure.core.protocols$fn__6523.invoke(protocols.clj:167) at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6519) at org.apache.storm.stats$aggregate_topo_stats.invoke(stats.clj:874) at org.apache.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1037) at org.apache.storm.daemon.nimbus$fn__4817$exec_fn__2064__auto__$reify__4846.getTopologyPageInfo(nimbus.clj:2105) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3800) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3784) at org.apache.storm.thrift.ProcessFunction.proc
[jira] [Updated] (STORM-1538) Exception being thrown after Utils conversion to java
[ https://issues.apache.org/jira/browse/STORM-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1538: - Description: 1. TTransportException thrown when trying to visit topology page in UI: When trying to access the topology page on the storm ui, exceptions are thrown. Visiting a topology page displays a TTransportException. The cause can be seen in nimbus.log: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.core$_PLUS_ at clojure.lang.RT.seqFrom(RT.java:528) at clojure.lang.RT.seq(RT.java:509) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:26) at clojure.core.protocols$fn__6500.invoke(protocols.clj:83) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6515) at org.apache.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:564) at org.apache.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:744) at org.apache.storm.stats$fn__1257.invoke(stats.clj:792) at clojure.lang.MultiFn.invoke(MultiFn.java:243) at clojure.core$partial$fn__4529.invoke(core.clj:2501) at clojure.core.protocols$fn__6523.invoke(protocols.clj:167) at clojure.core.protocols$fn__6478$G__6473__6487.invoke(protocols.clj:19) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6519) at org.apache.storm.stats$aggregate_topo_stats.invoke(stats.clj:874) at org.apache.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1037) at org.apache.storm.daemon.nimbus$fn__4817$exec_fn__2064__auto__$reify__4846.getTopologyPageInfo(nimbus.clj:2105) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3800) at org.apache.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3784) at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2. Exception thrown in supervisor: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String at org.apache.storm.daemon.supervisor$create_artifacts_link.invoke(supervisor.clj:1083) at org.apache.storm.daemon.supervisor$fn__7494.invoke(supervisor.clj:1180) at clojure.lang.MultiFn.invoke(MultiFn.java:251) at org.apache.storm.daemon.supervisor$get_valid_new_worker_ids$iter__7081__7085$fn__7086.invoke(supervisor.clj:393) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core$dorun.invoke(core.clj:3009) at clojure.core$doall.invoke(core.clj:3025) at org.apache.storm.daemon.supervisor$get_valid_new_worker_ids.invoke(supervisor.clj:380) at org.apache.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:447) at clojure.core$partial$fn__4527.invoke(core.clj:2492) at org.apache.storm.event$event_manager$fn__6781.invoke(event.clj:40) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) was: TTransportException thrown when trying to visit topology page in UI: When trying to access the topology page on the storm ui, exceptions are thrown. Visiting a topology page displays a TTransportException. The cause can be seen in nimbus.log: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.core$_PLUS_ at clojure.lang.RT.seqFrom(RT.java:528) at clojure.lang.RT.seq(RT.java:509) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core.protocols$seq_reduce.invoke(protocols.clj:26) at clojure.core.protocols$fn__6500.invoke(protocols.clj:83) at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) at clojure.core$reduce.invoke(core.clj:6515) at org.apache.storm.st
[jira] [Created] (STORM-1554) Improvement on Storm's timer code by finding a better method to notify with simulate time, so we don't need this 1 second polling.
Boyang Jerry Peng created STORM-1554: Summary: Improvement on Storm's timer code by finding a better method to notify with simulate time, so we don't need this 1 second polling. Key: STORM-1554 URL: https://issues.apache.org/jira/browse/STORM-1554 Project: Apache Storm Issue Type: Improvement Reporter: Boyang Jerry Peng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1336) Evalute/Port JStorm cgroup support
[ https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng resolved STORM-1336. -- Resolution: Done > Evalute/Port JStorm cgroup support > -- > > Key: STORM-1336 > URL: https://issues.apache.org/jira/browse/STORM-1336 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Labels: jstorm-merger > > Supports controlling the upper limit of CPU core usage for a worker using > cgroups > Sounds like a good start, will be nice to integrate it with RAS requests too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (STORM-1336) Evalute/Port JStorm cgroup support
[ https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng closed STORM-1336. Resolution: Done > Evalute/Port JStorm cgroup support > -- > > Key: STORM-1336 > URL: https://issues.apache.org/jira/browse/STORM-1336 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Labels: jstorm-merger > > Supports controlling the upper limit of CPU core usage for a worker using > cgroups > Sounds like a good start, will be nice to integrate it with RAS requests too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (STORM-1336) Evalute/Port JStorm cgroup support
[ https://issues.apache.org/jira/browse/STORM-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng reopened STORM-1336: -- > Evalute/Port JStorm cgroup support > -- > > Key: STORM-1336 > URL: https://issues.apache.org/jira/browse/STORM-1336 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Boyang Jerry Peng > Labels: jstorm-merger > > Supports controlling the upper limit of CPU core usage for a worker using > cgroups > Sounds like a good start, will be nice to integrate it with RAS requests too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1140) Add Cgroup support for Storm supervisor node
[ https://issues.apache.org/jira/browse/STORM-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng resolved STORM-1140. -- Resolution: Duplicate dup https://issues.apache.org/jira/browse/STORM-1336 > Add Cgroup support for Storm supervisor node > > > Key: STORM-1140 > URL: https://issues.apache.org/jira/browse/STORM-1140 > Project: Apache Storm > Issue Type: New Feature >Reporter: Xin Wang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1140) Add Cgroup support for Storm supervisor node
[ https://issues.apache.org/jira/browse/STORM-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178454#comment-15178454 ] Boyang Jerry Peng commented on STORM-1140: -- dup https://issues.apache.org/jira/browse/STORM-1336 > Add Cgroup support for Storm supervisor node > > > Key: STORM-1140 > URL: https://issues.apache.org/jira/browse/STORM-1140 > Project: Apache Storm > Issue Type: New Feature >Reporter: Xin Wang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1146) Enforced resource isolation for Resource Aware Scheduler
[ https://issues.apache.org/jira/browse/STORM-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng resolved STORM-1146. -- Resolution: Duplicate dup https://issues.apache.org/jira/browse/STORM-1336 > Enforced resource isolation for Resource Aware Scheduler > > > Key: STORM-1146 > URL: https://issues.apache.org/jira/browse/STORM-1146 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng > > Currently Resource Aware Scheduler has not means of enforcing the resource > usage of workers on nodes. We implement a mechanism (maybe through c groups) > to enforce workers to not use more resources than was given to them -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1631) Storm CGroup bug when launching workers as the user that submitted the topology
[ https://issues.apache.org/jira/browse/STORM-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng reassigned STORM-1631: Assignee: Boyang Jerry Peng > Storm CGroup bug when launching workers as the user that submitted the > topology > --- > > Key: STORM-1631 > URL: https://issues.apache.org/jira/browse/STORM-1631 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > In secure multitenant storm, topology workers are launched with permission of > the user that submitted the topology. This causes a problem with cgroups > since workers are launched with permissions of the topology user which does > not have permissions to modify cgroups storm is using -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1631) Storm CGroup bug when launching workers as the user that submitted the topology
Boyang Jerry Peng created STORM-1631: Summary: Storm CGroup bug when launching workers as the user that submitted the topology Key: STORM-1631 URL: https://issues.apache.org/jira/browse/STORM-1631 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng In secure multitenant storm, topology workers are launched with permission of the user that submitted the topology. This causes a problem with cgroups since workers are launched with permissions of the topology user which does not have permissions to modify cgroups storm is using -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1631) Storm CGroup bug when launching workers as the user that submitted the topology
[ https://issues.apache.org/jira/browse/STORM-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195765#comment-15195765 ] Boyang Jerry Peng commented on STORM-1631: -- One bug is in function kill-existing-workers-with-change-in-components in supervisor.clj: The function tries to detect whether there is a change in assignment. The bug in this function is that the ordering of the assignment matters but it shouldn't. For example, if a worker assignment is [[1 1] [2 2]] and it changed to [[2 2] [1 1]] it will cause the supervisor to restart the worker > Storm CGroup bug when launching workers as the user that submitted the > topology > --- > > Key: STORM-1631 > URL: https://issues.apache.org/jira/browse/STORM-1631 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > In secure multitenant storm, topology workers are launched with permission of > the user that submitted the topology. This causes a problem with cgroups > since workers are launched with permissions of the topology user which does > not have permissions to modify cgroups storm is using -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (STORM-1631) Storm CGroup bug when launching workers as the user that submitted the topology
[ https://issues.apache.org/jira/browse/STORM-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1631: - Comment: was deleted (was: One bug is in function kill-existing-workers-with-change-in-components in supervisor.clj: The function tries to detect whether there is a change in assignment. The bug in this function is that the ordering of the assignment matters but it shouldn't. For example, if a worker assignment is [[1 1] [2 2]] and it changed to [[2 2] [1 1]] it will cause the supervisor to restart the worker) > Storm CGroup bug when launching workers as the user that submitted the > topology > --- > > Key: STORM-1631 > URL: https://issues.apache.org/jira/browse/STORM-1631 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > In secure multitenant storm, topology workers are launched with permission of > the user that submitted the topology. This causes a problem with cgroups > since workers are launched with permissions of the topology user which does > not have permissions to modify cgroups storm is using -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1631) torm CGroup bugs 1) when launching workers as the user that submitted the topology 2) when initial cleanup of cgroup fails
[ https://issues.apache.org/jira/browse/STORM-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1631: - Summary: torm CGroup bugs 1) when launching workers as the user that submitted the topology 2) when initial cleanup of cgroup fails (was: Storm CGroup bug when launching workers as the user that submitted the topology) > torm CGroup bugs 1) when launching workers as the user that submitted the > topology 2) when initial cleanup of cgroup fails > -- > > Key: STORM-1631 > URL: https://issues.apache.org/jira/browse/STORM-1631 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > In secure multitenant storm, topology workers are launched with permission of > the user that submitted the topology. This causes a problem with cgroups > since workers are launched with permissions of the topology user which does > not have permissions to modify cgroups storm is using -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1631) torm CGroup bugs 1) when launching workers as the user that submitted the topology 2) when initial cleanup of cgroup fails
[ https://issues.apache.org/jira/browse/STORM-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1631: - Description: In secure multitenant storm, topology workers are launched with permission of the user that submitted the topology. This causes a problem with cgroups since workers are launched with permissions of the topology user which does not have permissions to modify cgroups storm is using Also, the clean up code is not trying to clean up cgroups of killed workers if the initial attempt failed was:In secure multitenant storm, topology workers are launched with permission of the user that submitted the topology. This causes a problem with cgroups since workers are launched with permissions of the topology user which does not have permissions to modify cgroups storm is using > torm CGroup bugs 1) when launching workers as the user that submitted the > topology 2) when initial cleanup of cgroup fails > -- > > Key: STORM-1631 > URL: https://issues.apache.org/jira/browse/STORM-1631 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > In secure multitenant storm, topology workers are launched with permission of > the user that submitted the topology. This causes a problem with cgroups > since workers are launched with permissions of the topology user which does > not have permissions to modify cgroups storm is using > Also, the clean up code is not trying to clean up cgroups of killed workers > if the initial attempt failed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1636) Supervisor shutdown with worker id pass in being nil
Boyang Jerry Peng created STORM-1636: Summary: Supervisor shutdown with worker id pass in being nil Key: STORM-1636 URL: https://issues.apache.org/jira/browse/STORM-1636 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng Assignee: Boyang Jerry Peng In function kill-existing-workers-with-change-in-components in supervisor.clj: The function tries to detect whether there is a change in assignment. The bug in this function is that the ordering of the assignment matters but it shouldn't. For example, if a worker assignment is [[1 1] [2 2]] and it changed to [[2 2] [1 1]] it will cause the supervisor to restart the worker -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1634) Minor Refactoring of Resource Aware Scheduler
[ https://issues.apache.org/jira/browse/STORM-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng reassigned STORM-1634: Assignee: Boyang Jerry Peng > Minor Refactoring of Resource Aware Scheduler > - > > Key: STORM-1634 > URL: https://issues.apache.org/jira/browse/STORM-1634 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Refactored the following: > 1. The API interface to define custom scheduling strategies. The API exposed > from RAS to a per topology scheduling strategy is messy and needs to be > cleaned up. > 2. In RAS, the state of the scheduler is held by several member variables. > Its cleaner if the scheduler state is represented by a single object. This > will help we reduce the amount of code need when checkpointing the state of > the scheduler and potentially restoring the state in the future when a bad > scheduling happens -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1636) Supervisor shutdown with worker id pass in being nil
[ https://issues.apache.org/jira/browse/STORM-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1636: - Affects Version/s: 2.0.0 1.0.0 > Supervisor shutdown with worker id pass in being nil > -- > > Key: STORM-1636 > URL: https://issues.apache.org/jira/browse/STORM-1636 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.0, 2.0.0 >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > In function kill-existing-workers-with-change-in-components in supervisor.clj: > The function tries to detect whether there is a change in assignment. The bug > in this function is that the ordering of the assignment matters but it > shouldn't. For example, if a worker assignment is [[1 1] [2 2]] and it > changed to [[2 2] [1 1]] it will cause the supervisor to restart the worker -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1635) Master branch broken! Exception thrown immediately after submitting topology
[ https://issues.apache.org/jira/browse/STORM-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1635: - Description: Exception thrown: 2016-03-16 22:20:14.940 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer [ERROR] Unexpected throwable while invoking! java.lang.NullPointerException at clojure.lang.RT.intCast(RT.java:1163) at org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191$iter__2267__2271$fn__2272.invoke(nimbus.clj:1837) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core$dorun.invoke(core.clj:3009) at clojure.core$doall.invoke(core.clj:3025) at org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191.getClusterInfo(nimbus.clj:1822) at org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3724) at org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3708) at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) This is also causing UI to not work. was: Exception thrown: 2016-03-16 22:20:14.940 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer [ERROR] Unexpected throwable while invoking! java.lang.NullPointerException at clojure.lang.RT.intCast(RT.java:1163) at org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191$iter__2267__2271$fn__2272.invoke(nimbus.clj:1837) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core$dorun.invoke(core.clj:3009) at clojure.core$doall.invoke(core.clj:3025) at org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191.getClusterInfo(nimbus.clj:1822) at org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3724) at org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3708) at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) > Master branch broken! Exception thrown immediately after submitting topology > > > Key: STORM-1635 > URL: https://issues.apache.org/jira/browse/STORM-1635 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Priority: Critical > > Exception thrown: > 2016-03-16 22:20:14.940 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer > [ERROR] Unexpected throwable while invoking! > java.lang.NullPointerException > at clojure.lang.RT.intCast(RT.java:1163) > at > org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191$iter__2267__2271$fn__2272.invoke(nimbus.clj:1837) > at clojure.lang.LazySeq.sval(LazySeq.java:40) > at clojure.lang.LazySeq.seq(LazySeq.java:49) > at clojure.lang.RT.seq(RT.java:507) > at clojure.core$seq__4128.invoke(core.clj:137) > at clojure.core$dorun.invoke(core.clj:3009) > at clojure.core$doall.invoke(core.clj:3025) > at > org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191.getClusterInfo(nimbus.clj:1822) > at > org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3724) >
[jira] [Resolved] (STORM-1635) Master branch broken! Exception thrown immediately after submitting topology
[ https://issues.apache.org/jira/browse/STORM-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng resolved STORM-1635. -- Resolution: Duplicate Dup of STORM-1623 > Master branch broken! Exception thrown immediately after submitting topology > > > Key: STORM-1635 > URL: https://issues.apache.org/jira/browse/STORM-1635 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: John Fang >Priority: Critical > > Exception thrown: > 2016-03-16 22:20:14.940 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer > [ERROR] Unexpected throwable while invoking! > java.lang.NullPointerException > at clojure.lang.RT.intCast(RT.java:1163) > at > org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191$iter__2267__2271$fn__2272.invoke(nimbus.clj:1837) > at clojure.lang.LazySeq.sval(LazySeq.java:40) > at clojure.lang.LazySeq.seq(LazySeq.java:49) > at clojure.lang.RT.seq(RT.java:507) > at clojure.core$seq__4128.invoke(core.clj:137) > at clojure.core$dorun.invoke(core.clj:3009) > at clojure.core$doall.invoke(core.clj:3025) > at > org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191.getClusterInfo(nimbus.clj:1822) > at > org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3724) > at > org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3708) > at > org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) > at > org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) > at > org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) > at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > This is also causing UI to not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-890) Should we move the worker to java?
[ https://issues.apache.org/jira/browse/STORM-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-890: Assignee: (was: Boyang Jerry Peng) > Should we move the worker to java? > -- > > Key: STORM-890 > URL: https://issues.apache.org/jira/browse/STORM-890 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Robert Joseph Evans > > Both JStorm and Heron have the worker running in pure java with noticable > performance improvements. Java is also a lot easier for new developers to > pick up on. We should profile the worker and at a minimum move slow portions > to java, and possible most/all of it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1634) Minor Refactoring of Resource Aware Scheduler
[ https://issues.apache.org/jira/browse/STORM-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1634: - Summary: Minor Refactoring of Resource Aware Scheduler (was: Minor Refactoring Resource Aware Scheduler) > Minor Refactoring of Resource Aware Scheduler > - > > Key: STORM-1634 > URL: https://issues.apache.org/jira/browse/STORM-1634 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Priority: Minor > > Refactored the following: > 1. The API interface to define custom scheduling strategies. The API exposed > from RAS to a per topology scheduling strategy is messy and needs to be > cleaned up. > 2. In RAS, the state of the scheduler is held by several member variables. > Its cleaner if the scheduler state is represented by a single object. This > will help we reduce the amount of code need when checkpointing the state of > the scheduler and potentially restoring the state in the future when a bad > scheduling happens -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1634) Minor Refactoring Resource Aware Scheduler
Boyang Jerry Peng created STORM-1634: Summary: Minor Refactoring Resource Aware Scheduler Key: STORM-1634 URL: https://issues.apache.org/jira/browse/STORM-1634 Project: Apache Storm Issue Type: Improvement Reporter: Boyang Jerry Peng Priority: Minor Refactored the following: 1. The API interface to define custom scheduling strategies. The API exposed from RAS to a per topology scheduling strategy is messy and needs to be cleaned up. 2. In RAS, the state of the scheduler is held by several member variables. Its cleaner if the scheduler state is represented by a single object. This will help we reduce the amount of code need when checkpointing the state of the scheduler and potentially restoring the state in the future when a bad scheduling happens -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1623) nimbus.clj's minor bug
[ https://issues.apache.org/jira/browse/STORM-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng resolved STORM-1623. -- Resolution: Fixed Fix Version/s: 2.0.0 > nimbus.clj's minor bug > -- > > Key: STORM-1623 > URL: https://issues.apache.org/jira/browse/STORM-1623 > Project: Apache Storm > Issue Type: Bug >Reporter: John Fang >Assignee: John Fang > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1635) Master branch broken! Exception thrown immediately after submitting topology
Boyang Jerry Peng created STORM-1635: Summary: Master branch broken! Exception thrown immediately after submitting topology Key: STORM-1635 URL: https://issues.apache.org/jira/browse/STORM-1635 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng Priority: Critical Exception thrown: 2016-03-16 22:20:14.940 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer [ERROR] Unexpected throwable while invoking! java.lang.NullPointerException at clojure.lang.RT.intCast(RT.java:1163) at org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191$iter__2267__2271$fn__2272.invoke(nimbus.clj:1837) at clojure.lang.LazySeq.sval(LazySeq.java:40) at clojure.lang.LazySeq.seq(LazySeq.java:49) at clojure.lang.RT.seq(RT.java:507) at clojure.core$seq__4128.invoke(core.clj:137) at clojure.core$dorun.invoke(core.clj:3009) at clojure.core$doall.invoke(core.clj:3025) at org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191.getClusterInfo(nimbus.clj:1822) at org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3724) at org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3708) at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1681) Bug in scheduling cyclic topologies when scheduling with RAS
Boyang Jerry Peng created STORM-1681: Summary: Bug in scheduling cyclic topologies when scheduling with RAS Key: STORM-1681 URL: https://issues.apache.org/jira/browse/STORM-1681 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng Assignee: Boyang Jerry Peng Priority: Critical -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1681) Bug in scheduling cyclic topologies when scheduling with RAS
[ https://issues.apache.org/jira/browse/STORM-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1681: - Affects Version/s: 2.0.0 1.0.0 > Bug in scheduling cyclic topologies when scheduling with RAS > > > Key: STORM-1681 > URL: https://issues.apache.org/jira/browse/STORM-1681 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.0, 2.0.0 >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1681) Bug in scheduling cyclic topologies when scheduling with RAS
[ https://issues.apache.org/jira/browse/STORM-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1681: - Description: There is a bug in the bfs algorithm in RAS that does not correctly account for components already visited during the breadth first traveral > Bug in scheduling cyclic topologies when scheduling with RAS > > > Key: STORM-1681 > URL: https://issues.apache.org/jira/browse/STORM-1681 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.0, 2.0.0 >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Critical > > There is a bug in the bfs algorithm in RAS that does not correctly account > for components already visited during the breadth first traveral -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269888#comment-15269888 ] Boyang Jerry Peng commented on STORM-1757: -- I think this would be a worth while project. The google dataflow API is very well thought out. We could potentially increase the user base of Storm by doing this. There is also demand within Yahoo for this feature > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1766) Find a better getBestClustering algorithm for RAS
Boyang Jerry Peng created STORM-1766:
Summary: Find a better getBestClustering algorithm for RAS
Key: STORM-1766
URL: https://issues.apache.org/jira/browse/STORM-1766
Project: Apache Storm
Issue Type: Improvement
Reporter: Boyang Jerry Peng
Assignee: Boyang Jerry Peng
Currently the getBestClustering algorithm for RAS finds the "Best" cluster/rack
based on which rack has the most available resources this may be insufficient
and may cause topologies not to be able to be scheduled successfully even
though there are enough resources to schedule it in the cluster. We attempt to
find the rack with the most resources by find the rack with the biggest sum of
available memory + available cpu. This method is not effective since it does
not consider the number of slots available. This method also fails in
identifying racks that are not schedulable due to the exhaustion of one of the
resources either memory, cpu, or slots. The current implementation also tries
the initial scheduling on one rack and not try to schedule on all the racks
before giving up which may cause topologies to be failed to be scheduled due to
the above mentioned shortcomings in the current method. Also the current method
does not consider failures of workers. When executors of a topology gets
unassigned and needs to be scheduled again, the current logic in
getBestClustering may be inadequate if not complete wrong. When executors needs
to rescheduled due to a fault, getBestClustering will likely return a cluster
that is different from where the majority of executors from the topology is
originally scheduling in.
Thus, I propose a different strategy/algorithm to find the "best" cluster. I
have come up with a ordering strategy I dub subordinate resource availability
ordering (inspired by Dominant Resource Fairness) that sorts racks by the
subordinate (not dominant) resource availability.
For example given 4 racks with the following resource availabilities:
//generate some that has alot of memory but little of cpu
rack-3 Avail [ CPU 100.0 MEM 20.0 Slots 40 ] Total [ CPU 100.0 MEM 20.0
Slots 40 ]
//generate some supervisors that are depleted of one resource
rack-2 Avail [ CPU 0.0 MEM 8.0 Slots 40 ] Total [ CPU 0.0 MEM 8.0 Slots
40 ]
//generate some that has a lot of cpu but little of memory
rack-4 Avail [ CPU 6100.0 MEM 1.0 Slots 40 ] Total [ CPU 6100.0 MEM 1.0
Slots 40 ]
//generate another rack of supervisors with less resources than rack-0
rack-1 Avail [ CPU 2000.0 MEM 4.0 Slots 40 ] Total [ CPU 2000.0 MEM 4.0
Slots 40 ]
rack-0 Avail [ CPU 4000.0 MEM 8.0 Slots 40( ] Total [ CPU 4000.0 MEM
8.0 Slots 40 ]
Cluster Overall Avail [ CPU 12200.0 MEM 41.0 Slots 200 ] Total [ CPU
12200.0 MEM 41.0 Slots 200 ]
It is clear that rack-0 is the best cluster since its the most balanced and can
potentially schedule the most executors, while rack-2 is the worst rack since
rack-2 is depleted of cpu resource thus rendering it unschedulable even though
there are other resources available
We first calculate the resource availability percentage of all the racks for
each resource by computing: (resource available on rack) / (resource available
in cluster)
We do this calculation to normalize the values otherwise the resource values
would not be comparable. So For our example:
rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ]
effective resources: 0.00819672131147541
rack-2 Avail [ 0.0% MEM 19.51219512195122% Slots 20.0% ] effective resources:
0.0
rack-4 Avail [ CPU 50.0% MEM 2.4390243902439024% Slots 20.0% ] effective
resources: 0.024390243902439025
rack-1 Avail [ CPU 16.39344262295082% MEM 9.75609756097561% Slots 20.0% ]
effective resources: 0.0975609756097561
rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ]
effective resources: 0.1951219512195122
The effective resource of a rack, which is also the subordinate resource, is
computed by: MIN ( resource availability percentage of
{CPU, Memory, # of free Slots}
. Then we order the racks by the the effective resource. Thus for our example:
Sorted rack: [rack-0, rack-1, rack-4, rack-3, rack-2]
Also to deal with the presence of failures, if a topology is partially
scheduled, we find the rack with the most scheduled executors for the topology
and we try to schedule on that rack first.
Thus for the sorting for racks. We first sort by the number of executors
already scheduled on the rack and then by the subordinate resource availability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-1766) A better algorithm server rack selection for RAS
[
https://issues.apache.org/jira/browse/STORM-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boyang Jerry Peng updated STORM-1766:
-
Summary: A better algorithm server rack selection for RAS (was: Find a
better getBestClustering algorithm for RAS)
> A better algorithm server rack selection for RAS
>
>
> Key: STORM-1766
> URL: https://issues.apache.org/jira/browse/STORM-1766
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>
> Currently the getBestClustering algorithm for RAS finds the "Best"
> cluster/rack based on which rack has the most available resources this may be
> insufficient and may cause topologies not to be able to be scheduled
> successfully even though there are enough resources to schedule it in the
> cluster. We attempt to find the rack with the most resources by find the rack
> with the biggest sum of available memory + available cpu. This method is not
> effective since it does not consider the number of slots available. This
> method also fails in identifying racks that are not schedulable due to the
> exhaustion of one of the resources either memory, cpu, or slots. The current
> implementation also tries the initial scheduling on one rack and not try to
> schedule on all the racks before giving up which may cause topologies to be
> failed to be scheduled due to the above mentioned shortcomings in the current
> method. Also the current method does not consider failures of workers. When
> executors of a topology gets unassigned and needs to be scheduled again, the
> current logic in getBestClustering may be inadequate if not complete wrong.
> When executors needs to rescheduled due to a fault, getBestClustering will
> likely return a cluster that is different from where the majority of
> executors from the topology is originally scheduling in.
> Thus, I propose a different strategy/algorithm to find the "best" cluster. I
> have come up with a ordering strategy I dub subordinate resource availability
> ordering (inspired by Dominant Resource Fairness) that sorts racks by the
> subordinate (not dominant) resource availability.
> For example given 4 racks with the following resource availabilities:
> //generate some that has alot of memory but little of cpu
> rack-3 Avail [ CPU 100.0 MEM 20.0 Slots 40 ] Total [ CPU 100.0 MEM
> 20.0 Slots 40 ]
> //generate some supervisors that are depleted of one resource
> rack-2 Avail [ CPU 0.0 MEM 8.0 Slots 40 ] Total [ CPU 0.0 MEM 8.0
> Slots 40 ]
> //generate some that has a lot of cpu but little of memory
> rack-4 Avail [ CPU 6100.0 MEM 1.0 Slots 40 ] Total [ CPU 6100.0 MEM
> 1.0 Slots 40 ]
> //generate another rack of supervisors with less resources than rack-0
> rack-1 Avail [ CPU 2000.0 MEM 4.0 Slots 40 ] Total [ CPU 2000.0 MEM
> 4.0 Slots 40 ]
> rack-0 Avail [ CPU 4000.0 MEM 8.0 Slots 40( ] Total [ CPU 4000.0 MEM
> 8.0 Slots 40 ]
> Cluster Overall Avail [ CPU 12200.0 MEM 41.0 Slots 200 ] Total [ CPU
> 12200.0 MEM 41.0 Slots 200 ]
> It is clear that rack-0 is the best cluster since its the most balanced and
> can potentially schedule the most executors, while rack-2 is the worst rack
> since rack-2 is depleted of cpu resource thus rendering it unschedulable even
> though there are other resources available
> We first calculate the resource availability percentage of all the racks for
> each resource by computing: (resource available on rack) / (resource
> available in cluster)
> We do this calculation to normalize the values otherwise the resource values
> would not be comparable. So For our example:
> rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ]
> effective resources: 0.00819672131147541
> rack-2 Avail [ 0.0% MEM 19.51219512195122% Slots 20.0% ] effective resources:
> 0.0
> rack-4 Avail [ CPU 50.0% MEM 2.4390243902439024% Slots 20.0% ] effective
> resources: 0.024390243902439025
> rack-1 Avail [ CPU 16.39344262295082% MEM 9.75609756097561% Slots 20.0% ]
> effective resources: 0.0975609756097561
> rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ]
> effective resources: 0.1951219512195122
> The effective resource of a rack, which is also the subordinate resource, is
> computed by: MIN ( resource availability percentage of
> {CPU, Memory, # of free Slots}
> . Then we order the racks by the the effective resource. Thus for our example:
> Sorted rack: [rack-0, rack-1, rack-4, rack-3, rack-2]
> Also to deal with the presence of failures, if a topology is partially
> scheduled, we find the rack with the most scheduled executors for the
> topology and we try to schedule on that rack first.
> Thus for the sorting for racks. We first sort by the number of exec
[jira] [Commented] (STORM-1779) Incorrect bfs-ordering in DefaultResourceAwareStrategy
[ https://issues.apache.org/jira/browse/STORM-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280494#comment-15280494 ] Boyang Jerry Peng commented on STORM-1779: -- dup of https://issues.apache.org/jira/browse/STORM-1681. Already fixed in master-security as well as 1.x branch > Incorrect bfs-ordering in DefaultResourceAwareStrategy > -- > > Key: STORM-1779 > URL: https://issues.apache.org/jira/browse/STORM-1779 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0 >Reporter: Pavel Smirnov >Priority: Minor > Labels: bug, scheduler > > I have designed a simple diamond topology: spout, N parallel processingBolts, > joinBolt. > https://github.com/smirnp/storm-benchmark/blob/1.0.0/src/storm/benchmark/DiamondTopology.java > For 6-components topopoly the breadth-first search (bfs method in > DefaultResourceAwareStrategy.java:323) returns 13 components, which is > obviously incorrect for further scheduling (it fails on memory limits trying > to schedule already scheduled tasks in priorityToExecutorMap). > I have reimplemented bfs with the comparison with the original-one > (https://github.com/smirnp/storm-benchmark/blob/1.0.0/bfs.txt) > Please check it and fix the bug in next releases. > P.S. storm.yaml user for scheduling is here: > https://github.com/smirnp/storm-benchmark/blob/1.0.0/storm.yaml > Best regards, > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1866) Update Resource Aware Scheduler Documentation
Boyang Jerry Peng created STORM-1866: Summary: Update Resource Aware Scheduler Documentation Key: STORM-1866 URL: https://issues.apache.org/jira/browse/STORM-1866 Project: Apache Storm Issue Type: Story Reporter: Boyang Jerry Peng Assignee: Boyang Jerry Peng Priority: Trivial The new features for RAS needs to be documented and there are a couple links in the RAS documentation that does not work anymore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1866) Update Resource Aware Scheduler Documentation
[ https://issues.apache.org/jira/browse/STORM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-1866: - Description: The new features for RAS (i.e. STORM-1766) needs to be documented and there are a couple links in the RAS documentation that does not work anymore (was: The new features for RAS needs to be documented and there are a couple links in the RAS documentation that does not work anymore) > Update Resource Aware Scheduler Documentation > - > > Key: STORM-1866 > URL: https://issues.apache.org/jira/browse/STORM-1866 > Project: Apache Storm > Issue Type: Story >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Trivial > > The new features for RAS (i.e. STORM-1766) needs to be documented and there > are a couple links in the RAS documentation that does not work anymore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1766) A better algorithm server rack selection for RAS
[
https://issues.apache.org/jira/browse/STORM-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boyang Jerry Peng resolved STORM-1766.
--
Resolution: Done
> A better algorithm server rack selection for RAS
>
>
> Key: STORM-1766
> URL: https://issues.apache.org/jira/browse/STORM-1766
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>
> Currently the getBestClustering algorithm for RAS finds the "Best"
> cluster/rack based on which rack has the most available resources this may be
> insufficient and may cause topologies not to be able to be scheduled
> successfully even though there are enough resources to schedule it in the
> cluster. We attempt to find the rack with the most resources by find the rack
> with the biggest sum of available memory + available cpu. This method is not
> effective since it does not consider the number of slots available. This
> method also fails in identifying racks that are not schedulable due to the
> exhaustion of one of the resources either memory, cpu, or slots. The current
> implementation also tries the initial scheduling on one rack and not try to
> schedule on all the racks before giving up which may cause topologies to be
> failed to be scheduled due to the above mentioned shortcomings in the current
> method. Also the current method does not consider failures of workers. When
> executors of a topology gets unassigned and needs to be scheduled again, the
> current logic in getBestClustering may be inadequate if not complete wrong.
> When executors needs to rescheduled due to a fault, getBestClustering will
> likely return a cluster that is different from where the majority of
> executors from the topology is originally scheduling in.
> Thus, I propose a different strategy/algorithm to find the "best" cluster. I
> have come up with a ordering strategy I dub subordinate resource availability
> ordering (inspired by Dominant Resource Fairness) that sorts racks by the
> subordinate (not dominant) resource availability.
> For example given 4 racks with the following resource availabilities
> {code}
> //generate some that has alot of memory but little of cpu
> rack-3 Avail [ CPU 100.0 MEM 20.0 Slots 40 ] Total [ CPU 100.0 MEM
> 20.0 Slots 40 ]
> //generate some supervisors that are depleted of one resource
> rack-2 Avail [ CPU 0.0 MEM 8.0 Slots 40 ] Total [ CPU 0.0 MEM 8.0
> Slots 40 ]
> //generate some that has a lot of cpu but little of memory
> rack-4 Avail [ CPU 6100.0 MEM 1.0 Slots 40 ] Total [ CPU 6100.0 MEM
> 1.0 Slots 40 ]
> //generate another rack of supervisors with less resources than rack-0
> rack-1 Avail [ CPU 2000.0 MEM 4.0 Slots 40 ] Total [ CPU 2000.0 MEM
> 4.0 Slots 40 ]
> rack-0 Avail [ CPU 4000.0 MEM 8.0 Slots 40( ] Total [ CPU 4000.0 MEM
> 8.0 Slots 40 ]
> Cluster Overall Avail [ CPU 12200.0 MEM 41.0 Slots 200 ] Total [ CPU
> 12200.0 MEM 41.0 Slots 200 ]
> {code}
> It is clear that rack-0 is the best cluster since its the most balanced and
> can potentially schedule the most executors, while rack-2 is the worst rack
> since rack-2 is depleted of cpu resource thus rendering it unschedulable even
> though there are other resources available.
> We first calculate the resource availability percentage of all the racks for
> each resource by computing:
> {code}
> (resource available on rack) / (resource available in cluster)
> {code}
> We do this calculation to normalize the values otherwise the resource values
> would not be comparable.
> So for our example:
> {code}
> rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ]
> effective resources: 0.00819672131147541
> rack-2 Avail [ 0.0% MEM 19.51219512195122% Slots 20.0% ] effective resources:
> 0.0
> rack-4 Avail [ CPU 50.0% MEM 2.4390243902439024% Slots 20.0% ] effective
> resources: 0.024390243902439025
> rack-1 Avail [ CPU 16.39344262295082% MEM 9.75609756097561% Slots 20.0% ]
> effective resources: 0.0975609756097561
> rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ]
> effective resources: 0.1951219512195122
> {code}
> The effective resource of a rack, which is also the subordinate resource, is
> computed by:
> {code}
> MIN(resource availability percentage of {CPU, Memory, # of free Slots}).
> {code}
> Then we order the racks by the effective resource.
> Thus for our example:
> {code}
> Sorted rack: [rack-0, rack-1, rack-4, rack-3, rack-2]
> {code}
> Also to deal with the presence of failures, if a topology is partially
> scheduled, we find the rack with the most scheduled executors for the
> topology and we try to schedule on that rack first.
> Thus for the sorting for racks. We first sort by the number of executors
> a
[jira] [Commented] (STORM-1866) Update Resource Aware Scheduler Documentation
[ https://issues.apache.org/jira/browse/STORM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345044#comment-15345044 ] Boyang Jerry Peng commented on STORM-1866: -- [~ptgoetz] Sorry I have been busy for the last couple of week. I have submitted a PR for the updates to the documentation: https://github.com/apache/storm/pull/1510 > Update Resource Aware Scheduler Documentation > - > > Key: STORM-1866 > URL: https://issues.apache.org/jira/browse/STORM-1866 > Project: Apache Storm > Issue Type: Story >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Trivial > > The new features for RAS (i.e. STORM-1766) needs to be documented and there > are a couple links in the RAS documentation that does not work anymore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1998) Dynamic Topology Config and Resources Updates
Boyang Jerry Peng created STORM-1998: Summary: Dynamic Topology Config and Resources Updates Key: STORM-1998 URL: https://issues.apache.org/jira/browse/STORM-1998 Project: Apache Storm Issue Type: New Feature Reporter: Boyang Jerry Peng Assignee: Boyang Jerry Peng With this new feature, users will be able to update any topology config for any topology on the fly without having to restart their topology. For users running the Resource Aware Scheduler (RAS), it will allow the user to dynamically scale-in and scale-out resource usage on a per component basis. When RAS is used, this new feature will also allow users to dynamically change the scheduling strategy for a topology. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1866) Update Resource Aware Scheduler Documentation
[ https://issues.apache.org/jira/browse/STORM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng resolved STORM-1866. -- Resolution: Done > Update Resource Aware Scheduler Documentation > - > > Key: STORM-1866 > URL: https://issues.apache.org/jira/browse/STORM-1866 > Project: Apache Storm > Issue Type: Story >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Trivial > > The new features for RAS (i.e. STORM-1766) needs to be documented and there > are a couple links in the RAS documentation that does not work anymore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-2036) Fix minor bug in RAS Tests
Boyang Jerry Peng created STORM-2036: Summary: Fix minor bug in RAS Tests Key: STORM-2036 URL: https://issues.apache.org/jira/browse/STORM-2036 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng Assignee: Boyang Jerry Peng Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2036) Fix minor bug in RAS Tests
[ https://issues.apache.org/jira/browse/STORM-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2036: - Fix Version/s: 2.0.0 1.0.0 > Fix minor bug in RAS Tests > -- > > Key: STORM-2036 > URL: https://issues.apache.org/jira/browse/STORM-2036 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Trivial > Fix For: 1.0.0, 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (STORM-2036) Fix minor bug in RAS Tests
[ https://issues.apache.org/jira/browse/STORM-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng closed STORM-2036. Resolution: Fixed > Fix minor bug in RAS Tests > -- > > Key: STORM-2036 > URL: https://issues.apache.org/jira/browse/STORM-2036 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Trivial > Fix For: 1.0.0, 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-2056) Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer
Boyang Jerry Peng created STORM-2056: Summary: Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Key: STORM-2056 URL: https://issues.apache.org/jira/browse/STORM-2056 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng Assignee: Boyang Jerry Peng Priority: Minor Example: http://openstorm3blue-n2.blue.ygrid.yahoo.com:8000/log?file=scheduler.log&start=0&length=51200 should be: http://openstorm3blue-n2.blue.ygrid.yahoo.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2056) Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: Example: http://storm.cluster.com:8000/log?file=scheduler.log&start=0&length=51200 should be: http:/storm.cluster.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 was: Example: http://openstorm3blue-n2.blue.ygrid.yahoo.com:8000/log?file=scheduler.log&start=0&length=51200 should be: http://openstorm3blue-n2.blue.ygrid.yahoo.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 > Incorrect url for prev,first,last,next buttons when viewing daemon logs via > logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Example: > http://storm.cluster.com:8000/log?file=scheduler.log&start=0&length=51200 > should be: > http:/storm.cluster.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2056) Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: Example: http://storm.cluster.com:8000/log?file=scheduler.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 was: Example: http://storm.cluster.com:8000/log?file=scheduler.log&start=0&length=51200 should be: http:.//storm.cluster.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 > Incorrect url for prev,first,last,next buttons when viewing daemon logs via > logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Example: > http://storm.cluster.com:8000/log?file=scheduler.log&start=0&length=51200 > should be: > http://storm.cluster.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2056) Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: Example: http://storm.cluster.com:8000/log?file=scheduler.log&start=0&length=51200 should be: http:.//storm.cluster.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 was: Example: http://storm.cluster.com:8000/log?file=scheduler.log&start=0&length=51200 should be: http:/storm.cluster.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 > Incorrect url for prev,first,last,next buttons when viewing daemon logs via > logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Example: > http://storm.cluster.com:8000/log?file=scheduler.log&start=0&length=51200 > should be: > http:.//storm.cluster.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2056) Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 was: Example: http://storm.cluster.com:8000/log?file=scheduler.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=scheduler.log&start=0&length=51200 > Incorrect url for prev,first,last,next buttons when viewing daemon logs via > logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Example: > http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 > should be: > http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2056) Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 was: Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 > Incorrect url for prev,first,last,next buttons when viewing daemon logs via > logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Example: > http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 > should be: > http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 > Function with bug: > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2056) Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 was: Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 > Incorrect url for prev,first,last,next buttons when viewing daemon logs via > logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Incorrect url for prev,first,last,next buttons when viewing daemon logs via > logviewer > Example: > http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 > should be: > http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 > Function with bug: > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2056) Bugs in logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Summary: Bugs in logviewer (was: Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer) > Bugs in logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Incorrect url for prev,first,last,next buttons when viewing daemon logs via > logviewer > Example: > http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 > should be: > http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 > Function with bug: > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2056) Bugs in logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Priority: Major (was: Minor) > Bugs in logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs > via logviewer > Example: > http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 > should be: > http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 > Function with bug: > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 > 2. Downloading daemon files causes exception to be thrown because of function > download-log-file checks authorization via worker.yaml. Obviously daemon log > root will not have this file. > java.io.FileNotFoundException: > /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file > or directory) > at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) > at java.io.FileInputStream.(FileInputStream.java:138) > at java.io.FileReader.(FileReader.java:72) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) > at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) > at > backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) > at > backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) > at > backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) > at backtype.storm.daemon.logviewer$fn__11528.invoke(logviewer.clj:1024) > at > org.apache.storm.shade.compojure.core$make_route$fn__6445.invoke(core.clj:93) > at > org.apache.storm.shade.compojure.core$if_route$fn__6433.invoke(core.clj:39) > at > org.apache.storm.shade.compojure.core$if_method$fn__6426.invoke(core.clj:24) > at > org.apache.storm.shade.compojure.core$routing$fn__6451.invoke(core.clj:106) > at clojure.core$some.invoke(core.clj:2515) > at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106) > at clojure.lang.RestFn.applyTo(RestFn.java:139) > at clojure.core$apply.invoke(core.clj:626) > at > org.apache.storm.shade.compojure.core$routes$fn__6455.invoke(core.clj:111) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2056) Bugs in logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 2. Downloading daemon files causes exception to be thrown because of function download-log-file checks authorization via worker.yaml. Obviously daemon log root will not have this file. java.io.FileNotFoundException: /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileReader.(FileReader.java:72) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) at backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) at backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) at backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) at backtype.storm.daemon.logviewer$fn__11528.invoke(logviewer.clj:1024) at org.apache.storm.shade.compojure.core$make_route$fn__6445.invoke(core.clj:93) at org.apache.storm.shade.compojure.core$if_route$fn__6433.invoke(core.clj:39) at org.apache.storm.shade.compojure.core$if_method$fn__6426.invoke(core.clj:24) at org.apache.storm.shade.compojure.core$routing$fn__6451.invoke(core.clj:106) at clojure.core$some.invoke(core.clj:2515) at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106) at clojure.lang.RestFn.applyTo(RestFn.java:139) at clojure.core$apply.invoke(core.clj:626) at org.apache.storm.shade.compojure.core$routes$fn__6455.invoke(core.clj:111) was: Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 > Bugs in logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs > via logviewer > Example: > http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 > should be: > http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 > Function with bug: > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 > 2. Downloading daemon files causes exception to be thrown because of function > download-log-file checks authorization via worker.yaml. Obviously daemon log > root will not have this file. > java.io.FileNotFoundException: > /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file > or directory) > at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) > at java.io.FileInputStream.(FileInputStream.java:138) > at java.io.FileReader.(FileReader.java:72) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) > at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) > at > backtype.storm.daemon.logviewer$get_log_user_group_whitel
[jira] [Updated] (STORM-2056) Bugs in logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 2. Downloading daemon files causes exception to be thrown because of function download-log-file checks authorization via worker.yaml. Obviously daemon log root will not have this file. java.io.FileNotFoundException: /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileReader.(FileReader.java:72) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) at backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) at backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) at backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) at backtype.storm.daemon.logviewer$fn__11528.invoke(logviewer.clj:1024) at org.apache.storm.shade.compojure.core$make_route$fn__6445.invoke(core.clj:93) at org.apache.storm.shade.compojure.core$if_route$fn__6433.invoke(core.clj:39) at org.apache.storm.shade.compojure.core$if_method$fn__6426.invoke(core.clj:24) at org.apache.storm.shade.compojure.core$routing$fn__6451.invoke(core.clj:106) at clojure.core$some.invoke(core.clj:2515) at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106) at clojure.lang.RestFn.applyTo(RestFn.java:139) at clojure.core$apply.invoke(core.clj:626) at org.apache.storm.shade.compojure.core$routes$fn__6455.invoke(core.clj:111) 3. Function incorrect: (defn- daemon-download-link [fname] [[:p (link-to (UIHelpers/urlFormat "/daemondownload/%s" (to-array [fname])) "Download Full File")]]) https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L404 should be: (defn- daemon-download-link [fname] [[:p (link-to (UIHelpers/urlFormat "/daemondownload/%s" fname]) "Download Full File")]]) was: 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 2. Downloading daemon files causes exception to be thrown because of function download-log-file checks authorization via worker.yaml. Obviously daemon log root will not have this file. java.io.FileNotFoundException: /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileReader.(FileReader.java:72) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) at backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) at backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) at backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) at backtype.storm.daemon.logviewer$fn__11528.invoke(logviewer.clj:1024) at org.apache.storm.s
[jira] [Updated] (STORM-2056) Bugs in logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 2. Downloading daemon files causes exception to be thrown because of function download-log-file checks authorization via worker.yaml. Obviously daemon log root will not have this file. java.io.FileNotFoundException: /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileReader.(FileReader.java:72) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) at backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) at backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) at backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) at backtype.storm.daemon.logviewer$fn__11528.invoke(logviewer.clj:1024) at org.apache.storm.shade.compojure.core$make_route$fn__6445.invoke(core.clj:93) at org.apache.storm.shade.compojure.core$if_route$fn__6433.invoke(core.clj:39) at org.apache.storm.shade.compojure.core$if_method$fn__6426.invoke(core.clj:24) at org.apache.storm.shade.compojure.core$routing$fn__6451.invoke(core.clj:106) at clojure.core$some.invoke(core.clj:2515) at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106) at clojure.lang.RestFn.applyTo(RestFn.java:139) at clojure.core$apply.invoke(core.clj:626) at org.apache.storm.shade.compojure.core$routes$fn__6455.invoke(core.clj:111) 3. Function incorrect: (defn- daemon-download-link [fname] [[:p (link-to (UIHelpers/urlFormat "/daemondownload/%s" (to-array [fname])) "Download Full File")]]) https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L404 Not sure why we are putting fname into an array should be: (defn- daemon-download-link [fname] [[:p (link-to (UIHelpers/urlFormat "/daemondownload/%s" fname]) "Download Full File")]]) was: 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 2. Downloading daemon files causes exception to be thrown because of function download-log-file checks authorization via worker.yaml. Obviously daemon log root will not have this file. java.io.FileNotFoundException: /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileReader.(FileReader.java:72) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) at backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) at backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) at backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) at backtype.storm.daemon.logviewer$fn__11528.invoke(lo
[jira] [Updated] (STORM-2056) Bugs in logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 2. Downloading daemon files causes exception to be thrown because of function download-log-file checks authorization via worker.yaml. Obviously daemon log root will not have this file. java.io.FileNotFoundException: /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileReader.(FileReader.java:72) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) at backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) at backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) at backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) at backtype.storm.daemon.logviewer$fn__11528.invoke(logviewer.clj:1024) at org.apache.storm.shade.compojure.core$make_route$fn__6445.invoke(core.clj:93) at org.apache.storm.shade.compojure.core$if_route$fn__6433.invoke(core.clj:39) at org.apache.storm.shade.compojure.core$if_method$fn__6426.invoke(core.clj:24) at org.apache.storm.shade.compojure.core$routing$fn__6451.invoke(core.clj:106) at clojure.core$some.invoke(core.clj:2515) at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106) at clojure.lang.RestFn.applyTo(RestFn.java:139) at clojure.core$apply.invoke(core.clj:626) at org.apache.storm.shade.compojure.core$routes$fn__6455.invoke(core.clj:111) 3. Function incorrect: (defn- daemon-download-link [fname] [[:p (link-to (UIHelpers/urlFormat "/daemondownload/%s" (to-array [fname])) "Download Full File")]]) https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L404 Not sure why we are putting fname into an array should be: (defn- daemon-download-link [fname] [[:p (link-to (UIHelpers/urlFormat "/daemondownload/%s" fname]) "Download Full File")]]) 4. search-log-file should not check for authorized users in worker.yaml https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L833 was: 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 2. Downloading daemon files causes exception to be thrown because of function download-log-file checks authorization via worker.yaml. Obviously daemon log root will not have this file. java.io.FileNotFoundException: /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileReader.(FileReader.java:72) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) at backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) at backtype.storm.daemon.logviewer$authorized_log_user_QMA
[jira] [Created] (STORM-2060) Consolidate duplicated code in logviewer
Boyang Jerry Peng created STORM-2060: Summary: Consolidate duplicated code in logviewer Key: STORM-2060 URL: https://issues.apache.org/jira/browse/STORM-2060 Project: Apache Storm Issue Type: Sub-task Reporter: Boyang Jerry Peng Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2060) Consolidate duplicated code in logviewer
[ https://issues.apache.org/jira/browse/STORM-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2060: - Description: lots of duplicated code in daemonlog-page and log-page functions > Consolidate duplicated code in logviewer > > > Key: STORM-2060 > URL: https://issues.apache.org/jira/browse/STORM-2060 > Project: Apache Storm > Issue Type: Sub-task >Reporter: Boyang Jerry Peng >Priority: Trivial > > lots of duplicated code in daemonlog-page and log-page functions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (STORM-2060) Consolidate duplicated code in logviewer
[ https://issues.apache.org/jira/browse/STORM-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng closed STORM-2060. Resolution: Won't Fix > Consolidate duplicated code in logviewer > > > Key: STORM-2060 > URL: https://issues.apache.org/jira/browse/STORM-2060 > Project: Apache Storm > Issue Type: Sub-task >Reporter: Boyang Jerry Peng >Priority: Trivial > > lots of duplicated code in daemonlog-page and log-page functions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-2056) Bugs in logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Description: 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 2. Downloading daemon files causes exception to be thrown because of function download-log-file checks authorization via worker.yaml. Obviously daemon log root will not have this file. java.io.FileNotFoundException: /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileReader.(FileReader.java:72) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) at backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) at backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) at backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) at backtype.storm.daemon.logviewer$fn__11528.invoke(logviewer.clj:1024) at org.apache.storm.shade.compojure.core$make_route$fn__6445.invoke(core.clj:93) at org.apache.storm.shade.compojure.core$if_route$fn__6433.invoke(core.clj:39) at org.apache.storm.shade.compojure.core$if_method$fn__6426.invoke(core.clj:24) at org.apache.storm.shade.compojure.core$routing$fn__6451.invoke(core.clj:106) at clojure.core$some.invoke(core.clj:2515) at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106) at clojure.lang.RestFn.applyTo(RestFn.java:139) at clojure.core$apply.invoke(core.clj:626) at org.apache.storm.shade.compojure.core$routes$fn__6455.invoke(core.clj:111) 3. search-log-file should not check for authorized users in worker.yaml https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L833 was: 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs via logviewer Example: http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 should be: http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 Function with bug: https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 2. Downloading daemon files causes exception to be thrown because of function download-log-file checks authorization via worker.yaml. Obviously daemon log root will not have this file. java.io.FileNotFoundException: /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileReader.(FileReader.java:72) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) at backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) at backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) at backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) at backtype.storm.daemon.logviewer$fn__11528.invoke(logviewer.clj:1024) at org.apache.storm.shade.compojure.core$make_route$fn__6445.invoke(core.clj:93) at org.apache.storm.shade.compojure.core$if_route$fn__6433.invoke(core.clj:39) at org.apache.storm.shade.compojure.core$if_method$fn__6426.invoke(core.clj:
[jira] [Commented] (STORM-2056) Bugs in logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15455843#comment-15455843 ] Boyang Jerry Peng commented on STORM-2056: -- [~kabhwan], Thanks for merging in the fix! What about 2.x branch? Seems like you merged the fix into that branch as well > Bugs in logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > Fix For: 2.0.0, 1.1.0, 1.0.3 > > Time Spent: 40m > Remaining Estimate: 0h > > 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs > via logviewer > Example: > http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 > should be: > http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 > Function with bug: > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 > 2. Downloading daemon files causes exception to be thrown because of function > download-log-file checks authorization via worker.yaml. Obviously daemon log > root will not have this file. > java.io.FileNotFoundException: > /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file > or directory) > at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) > at java.io.FileInputStream.(FileInputStream.java:138) > at java.io.FileReader.(FileReader.java:72) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) > at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) > at > backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) > at > backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) > at > backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) > at backtype.storm.daemon.logviewer$fn__11528.invoke(logviewer.clj:1024) > at > org.apache.storm.shade.compojure.core$make_route$fn__6445.invoke(core.clj:93) > at > org.apache.storm.shade.compojure.core$if_route$fn__6433.invoke(core.clj:39) > at > org.apache.storm.shade.compojure.core$if_method$fn__6426.invoke(core.clj:24) > at > org.apache.storm.shade.compojure.core$routing$fn__6451.invoke(core.clj:106) > at clojure.core$some.invoke(core.clj:2515) > at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106) > at clojure.lang.RestFn.applyTo(RestFn.java:139) > at clojure.core$apply.invoke(core.clj:626) > at > org.apache.storm.shade.compojure.core$routes$fn__6455.invoke(core.clj:111) > 3. search-log-file should not check for authorized users in worker.yaml > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L833 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (STORM-2056) Bugs in logviewer
[ https://issues.apache.org/jira/browse/STORM-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-2056: - Comment: was deleted (was: [~kabhwan], Thanks for merging in the fix! What about 2.x branch? Seems like you merged the fix into that branch as well) > Bugs in logviewer > - > > Key: STORM-2056 > URL: https://issues.apache.org/jira/browse/STORM-2056 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > Fix For: 2.0.0, 1.1.0, 1.0.3 > > Time Spent: 40m > Remaining Estimate: 0h > > 1. Incorrect url for prev,first,last,next buttons when viewing daemon logs > via logviewer > Example: > http://storm.cluster.com:8000/log?file=nimbus.log&start=0&length=51200 > should be: > http://storm.cluster.com:8000/daemonlog?file=nimbus.log&start=0&length=51200 > Function with bug: > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L374 > 2. Downloading daemon files causes exception to be thrown because of function > download-log-file checks authorization via worker.yaml. Obviously daemon log > root will not have this file. > java.io.FileNotFoundException: > /home/y/var/storm/workers-artifacts/supervisor.log/worker.yaml (No such file > or directory) > at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) > at java.io.FileInputStream.(FileInputStream.java:138) > at java.io.FileReader.(FileReader.java:72) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) > at backtype.storm.util$clojure_from_yaml_file.invoke(util.clj:1066) > at > backtype.storm.daemon.logviewer$get_log_user_group_whitelist.invoke(logviewer.clj:310) > at > backtype.storm.daemon.logviewer$authorized_log_user_QMARK_.invoke(logviewer.clj:326) > at > backtype.storm.daemon.logviewer$download_log_file.invoke(logviewer.clj:497) > at backtype.storm.daemon.logviewer$fn__11528.invoke(logviewer.clj:1024) > at > org.apache.storm.shade.compojure.core$make_route$fn__6445.invoke(core.clj:93) > at > org.apache.storm.shade.compojure.core$if_route$fn__6433.invoke(core.clj:39) > at > org.apache.storm.shade.compojure.core$if_method$fn__6426.invoke(core.clj:24) > at > org.apache.storm.shade.compojure.core$routing$fn__6451.invoke(core.clj:106) > at clojure.core$some.invoke(core.clj:2515) > at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106) > at clojure.lang.RestFn.applyTo(RestFn.java:139) > at clojure.core$apply.invoke(core.clj:626) > at > org.apache.storm.shade.compojure.core$routes$fn__6455.invoke(core.clj:111) > 3. search-log-file should not check for authorized users in worker.yaml > https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/logviewer.clj#L833 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-2079) Unneccessary readStormConfig operation
Boyang Jerry Peng created STORM-2079: Summary: Unneccessary readStormConfig operation Key: STORM-2079 URL: https://issues.apache.org/jira/browse/STORM-2079 Project: Apache Storm Issue Type: Bug Reporter: Boyang Jerry Peng Assignee: Boyang Jerry Peng Priority: Trivial https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/topology/BaseConfigurationDeclarer.java#L26 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-966) ConfigValidation.DoubleValidator doesn't really validate whether the type of the object is a double
Boyang Jerry Peng created STORM-966: --- Summary: ConfigValidation.DoubleValidator doesn't really validate whether the type of the object is a double Key: STORM-966 URL: https://issues.apache.org/jira/browse/STORM-966 Project: Apache Storm Issue Type: Improvement Reporter: Boyang Jerry Peng Priority: Minor ConfigValidation.DoubleValidator code only checks if the object is null whether if the object is a instance of Number which is a parent class of Double. DoubleValidator is only used once in Config.java and in that instance: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = ConfigValidation.DoubleValidator; can just be set to: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = Double.class; Then we can just get rid of the misleading function ConfigValidation.DoubleValidator since it doesn't really check if a object is of double type -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-966) ConfigValidation.DoubleValidator doesn't really validate whether the type of the object is a double
[ https://issues.apache.org/jira/browse/STORM-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng reassigned STORM-966: --- Assignee: Boyang Jerry Peng > ConfigValidation.DoubleValidator doesn't really validate whether the type of > the object is a double > --- > > Key: STORM-966 > URL: https://issues.apache.org/jira/browse/STORM-966 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > ConfigValidation.DoubleValidator code only checks if the object is null > whether if the object is a instance of Number which is a parent class of > Double. > DoubleValidator is only used once in Config.java and in that instance: > public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = > ConfigValidation.DoubleValidator; > can just be set to: > public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = Double.class; > Then we can just get rid of the misleading function > ConfigValidation.DoubleValidator since it doesn't really check if a object is > of double type -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-966) ConfigValidation.DoubleValidator doesn't really validate whether the type of the object is a double
[ https://issues.apache.org/jira/browse/STORM-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-966: Description: ConfigValidation.DoubleValidator code only checks if the object is null whether if the object is a instance of Number which is a parent class of Double. DoubleValidator is only used once in Config.java and in that instance: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = ConfigValidation.DoubleValidator; can just be set to: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; Then we can just get rid of the misleading function ConfigValidation.DoubleValidator since it doesn't really check if a object is of double type. was: ConfigValidation.DoubleValidator code only checks if the object is null whether if the object is a instance of Number which is a parent class of Double. DoubleValidator is only used once in Config.java and in that instance: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = ConfigValidation.DoubleValidator; can just be set to: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = Double.class; Then we can just get rid of the misleading function ConfigValidation.DoubleValidator since it doesn't really check if a object is of double type > ConfigValidation.DoubleValidator doesn't really validate whether the type of > the object is a double > --- > > Key: STORM-966 > URL: https://issues.apache.org/jira/browse/STORM-966 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > ConfigValidation.DoubleValidator code only checks if the object is null > whether if the object is a instance of Number which is a parent class of > Double. > DoubleValidator is only used once in Config.java and in that instance: > public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = > ConfigValidation.DoubleValidator; > can just be set to: > public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; > Then we can just get rid of the misleading function > ConfigValidation.DoubleValidator since it doesn't really check if a object is > of double type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-966) ConfigValidation.DoubleValidator doesn't really validate whether the type of the object is a double
[ https://issues.apache.org/jira/browse/STORM-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-966: Description: ConfigValidation.DoubleValidator code only checks if the object is null whether if the object is a instance of Number which is a parent class of Double. DoubleValidator is only used once in Config.java and in that instance: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = ConfigValidation.DoubleValidator; can just be set to: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; Then we can just get rid of the misleading function ConfigValidation.DoubleValidator since it doesn't really check if a object is of double type. In previous commit 214ee7454548b884c591991b1faea770d1478cec Number.Class was used anyway was: ConfigValidation.DoubleValidator code only checks if the object is null whether if the object is a instance of Number which is a parent class of Double. DoubleValidator is only used once in Config.java and in that instance: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = ConfigValidation.DoubleValidator; can just be set to: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; Then we can just get rid of the misleading function ConfigValidation.DoubleValidator since it doesn't really check if a object is of double type. > ConfigValidation.DoubleValidator doesn't really validate whether the type of > the object is a double > --- > > Key: STORM-966 > URL: https://issues.apache.org/jira/browse/STORM-966 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > ConfigValidation.DoubleValidator code only checks if the object is null > whether if the object is a instance of Number which is a parent class of > Double. > DoubleValidator is only used once in Config.java and in that instance: > public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = > ConfigValidation.DoubleValidator; > can just be set to: > public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; > Then we can just get rid of the misleading function > ConfigValidation.DoubleValidator since it doesn't really check if a object is > of double type. In previous commit 214ee7454548b884c591991b1faea770d1478cec > Number.Class was used anyway -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-966) ConfigValidation.DoubleValidator doesn't really validate whether the type of the object is a double
[ https://issues.apache.org/jira/browse/STORM-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-966: Description: ConfigValidation.DoubleValidator code only checks if the object is null whether if the object is a instance of Number which is a parent class of Double. DoubleValidator is only used once in Config.java and in that instance: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = ConfigValidation.DoubleValidator; can just be set to: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; Then we can just get rid of the misleading function ConfigValidation.DoubleValidator since it doesn't really check if a object is of double type thus the validator function doesn't really do anything and the name is misleading. In previous commit 214ee7454548b884c591991b1faea770d1478cec Number.Class was used anyway was: ConfigValidation.DoubleValidator code only checks if the object is null whether if the object is a instance of Number which is a parent class of Double. DoubleValidator is only used once in Config.java and in that instance: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = ConfigValidation.DoubleValidator; can just be set to: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; Then we can just get rid of the misleading function ConfigValidation.DoubleValidator since it doesn't really check if a object is of double type. In previous commit 214ee7454548b884c591991b1faea770d1478cec Number.Class was used anyway > ConfigValidation.DoubleValidator doesn't really validate whether the type of > the object is a double > --- > > Key: STORM-966 > URL: https://issues.apache.org/jira/browse/STORM-966 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > ConfigValidation.DoubleValidator code only checks if the object is null > whether if the object is a instance of Number which is a parent class of > Double. > DoubleValidator is only used once in Config.java and in that instance: > public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = > ConfigValidation.DoubleValidator; > can just be set to: > public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; > Then we can just get rid of the misleading function > ConfigValidation.DoubleValidator since it doesn't really check if a object is > of double type thus the validator function doesn't really do anything and the > name is misleading. In previous commit > 214ee7454548b884c591991b1faea770d1478cec Number.Class was used anyway -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-966) ConfigValidation.DoubleValidator doesn't really validate whether the type of the object is a double
[ https://issues.apache.org/jira/browse/STORM-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated STORM-966: Description: ConfigValidation.DoubleValidator code only checks if the object is null whether if the object is a instance of Number which is a parent class of Double. DoubleValidator is only used once in Config.java and in that instance: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = ConfigValidation.DoubleValidator; can just be set to: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; Then we can just get rid of the misleading function ConfigValidation.DoubleValidator since it doesn't really check if a object is of double type thus the validator function doesn't really do anything and the name is misleading. In previous commit https://github.com/apache/storm/commit/214ee7454548b884c591991b1faea770d1478cec Number.Class was used anyway was: ConfigValidation.DoubleValidator code only checks if the object is null whether if the object is a instance of Number which is a parent class of Double. DoubleValidator is only used once in Config.java and in that instance: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = ConfigValidation.DoubleValidator; can just be set to: public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; Then we can just get rid of the misleading function ConfigValidation.DoubleValidator since it doesn't really check if a object is of double type thus the validator function doesn't really do anything and the name is misleading. In previous commit 214ee7454548b884c591991b1faea770d1478cec Number.Class was used anyway > ConfigValidation.DoubleValidator doesn't really validate whether the type of > the object is a double > --- > > Key: STORM-966 > URL: https://issues.apache.org/jira/browse/STORM-966 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > ConfigValidation.DoubleValidator code only checks if the object is null > whether if the object is a instance of Number which is a parent class of > Double. > DoubleValidator is only used once in Config.java and in that instance: > public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = > ConfigValidation.DoubleValidator; > can just be set to: > public static final Object TOPOLOGY_STATS_SAMPLE_RATE_SCHEMA = NUMBER.class; > Then we can just get rid of the misleading function > ConfigValidation.DoubleValidator since it doesn't really check if a object is > of double type thus the validator function doesn't really do anything and the > name is misleading. In previous commit > https://github.com/apache/storm/commit/214ee7454548b884c591991b1faea770d1478cec > Number.Class was used anyway -- This message was sent by Atlassian JIRA (v6.3.4#6332)
