secure storm UI
Is there a way to secure the storm UI page. Like enable logging to access the page so only authorized people can only access it. -- Kushan Maskey 817.403.7500 M. Miller Associates http://mmillerassociates.com/ kushan.mas...@mmillerassociates.com
Re: secure storm UI
This is available in the security branch. See https://github.com/apache/storm/blob/security/SECURITY.md You do not need to enable all of the security features to get UI auth. For authentication, look at ui.filter and ui.filter.params. For authorization, nimbus.admins, ui.users, logs.users, and topology.users -- Derek On 9/26/14 14:34, Kushan Maskey wrote: Is there a way to secure the storm UI page. Like enable logging to access the page so only authorized people can only access it. -- Kushan Maskey 817.403.7500 M. Miller Associates http://mmillerassociates.com/ kushan.mas...@mmillerassociates.com mailto:kushan.mas...@mmillerassociates.com
Storm UI Authentication
Hi Can anyone help me out in setting up the authentication for Storm UI Thanks Alex
Re: Storm UI Authentication
Hi Alex I am not sure about your exact requirements here but I guess any authentication scheme such as access management stack can be used. Thanks Deepak On 20/09/2014 6:57 pm, Alex Alexander alex62...@gmail.com wrote: Hi Can anyone help me out in setting up the authentication for Storm UI Thanks Alex
RE: Storm UI REST API documentation
Amazing thanks - if only it was easy to find! From: Ratay, Steve [mailto:steve.ra...@teradata.com] Sent: September-10-14 12:37 PM To: user@storm.incubator.apache.org Subject: Re: Storm UI REST API documentation There is actually some pretty good documentation at https://github.com/apache/incubator-storm/blob/master/STORM-UI-REST-API.md From: Dave Ellery dave.ell...@leonardo.commailto:dave.ell...@leonardo.com Reply-To: user@storm.incubator.apache.orgmailto:user@storm.incubator.apache.org user@storm.incubator.apache.orgmailto:user@storm.incubator.apache.org Date: Wednesday, September 10, 2014 at 12:33 PM To: user@storm.incubator.apache.orgmailto:user@storm.incubator.apache.org user@storm.incubator.apache.orgmailto:user@storm.incubator.apache.org Subject: Storm UI REST API documentation I have been trying to find some but for the life of me can't - I have a suspicion there is not any yet.
Storm UI REST API documentation
I have been trying to find some but for the life of me can't - I have a suspicion there is not any yet.
Re: Storm UI REST API documentation
There is actually some pretty good documentation at https://github.com/apache/incubator-storm/blob/master/STORM-UI-REST-API.md From: Dave Ellery dave.ell...@leonardo.commailto:dave.ell...@leonardo.com Reply-To: user@storm.incubator.apache.orgmailto:user@storm.incubator.apache.org user@storm.incubator.apache.orgmailto:user@storm.incubator.apache.org Date: Wednesday, September 10, 2014 at 12:33 PM To: user@storm.incubator.apache.orgmailto:user@storm.incubator.apache.org user@storm.incubator.apache.orgmailto:user@storm.incubator.apache.org Subject: Storm UI REST API documentation I have been trying to find some but for the life of me can’t – I have a suspicion there is not any yet.
STORM UI Issues
Hi Guys, We are running our topology on Local Mode. The Storm UI runs fine but unable to display the submitted topologies. We are new to STORM. Would appreciate help. Regards Sridhar The information contained in this transmission may contain privileged and confidential information of Microland Limited, including information protected by privacy laws. It is intended only for the use of Microland Limited. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: STORM UI Issues
Only topologies submitted to the Storm cluster using StormSubmitter are displayed in the UI. When topologies are submitted in Local Mode, the Localcluster class just simulates a Storm cluster using a VoltDB instance to store data in memory. Your topologies are not submitted to the actual storm cluster and hence are not displayed on your UI. Local Mode is usually used to develop the topology and test whether it is working correctly. Hope this helps, Palak Shah On Mon, Sep 1, 2014 at 3:31 PM, Sridhar G sridh...@microland.com wrote: Hi Guys, We are running our topology on Local Mode. The Storm UI runs fine but unable to display the submitted topologies. We are new to STORM. Would appreciate help. Regards Sridhar The information contained in this transmission may contain privileged and confidential information of Microland Limited, including information protected by privacy laws. It is intended only for the use of Microland Limited. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: STORM UI Issues
Hi, Thats true.. In local mode the UI is static in nature and will just indicate all the daemons are running fine. In order to monitor the storm cluster on UI the cluster should be in distributed mode. Thanks, Siddharth Ubale On Mon, Sep 1, 2014 at 3:56 PM, Palak Shah spala...@gmail.com wrote: Only topologies submitted to the Storm cluster using StormSubmitter are displayed in the UI. When topologies are submitted in Local Mode, the Localcluster class just simulates a Storm cluster using a VoltDB instance to store data in memory. Your topologies are not submitted to the actual storm cluster and hence are not displayed on your UI. Local Mode is usually used to develop the topology and test whether it is working correctly. Hope this helps, Palak Shah On Mon, Sep 1, 2014 at 3:31 PM, Sridhar G sridh...@microland.com wrote: Hi Guys, We are running our topology on Local Mode. The Storm UI runs fine but unable to display the submitted topologies. We are new to STORM. Would appreciate help. Regards Sridhar The information contained in this transmission may contain privileged and confidential information of Microland Limited, including information protected by privacy laws. It is intended only for the use of Microland Limited. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: Location of last error details seen in storm UI
Click on the link of the bolt/spout that is all the way on the left side. On Sun, Aug 24, 2014 at 11:19 PM, Jason Kania jason.ka...@ymail.com wrote: Hello, I am trying to get more detail on an error that is being displayed in the Storm UI under the Last Error column but unfortunately, I am not seeing it captured anywhere else. Does anyone know where this text could be seen? The problem is that the error text is insufficient to diagnose the problem. Thanks, Jason
Re: Location of last error details seen in storm UI
Better would be to view log files under /var/log/storm. Any issue with worker would be logged into /var/log/storm/worker-6700.log and /var/log/storm/worker-6701.log. On Mon, Aug 25, 2014 at 8:00 PM, Vincent Russell vincent.russ...@gmail.com wrote: Click on the link of the bolt/spout that is all the way on the left side. On Sun, Aug 24, 2014 at 11:19 PM, Jason Kania jason.ka...@ymail.com wrote: Hello, I am trying to get more detail on an error that is being displayed in the Storm UI under the Last Error column but unfortunately, I am not seeing it captured anywhere else. Does anyone know where this text could be seen? The problem is that the error text is insufficient to diagnose the problem. Thanks, Jason -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
Re: Location of last error details seen in storm UI
Thanks for the help. Unfortunately, the content shown when the spout/bolt is clicked does not show any information about any errors. From: Vincent Russell vincent.russ...@gmail.com To: user@storm.incubator.apache.org; Jason Kania jason.ka...@ymail.com Sent: Monday, August 25, 2014 10:30:32 AM Subject: Re: Location of last error details seen in storm UI Click on the link of the bolt/spout that is all the way on the left side. On Sun, Aug 24, 2014 at 11:19 PM, Jason Kania jason.ka...@ymail.com wrote: Hello, I am trying to get more detail on an error that is being displayed in the Storm UI under the Last Error column but unfortunately, I am not seeing it captured anywhere else. Does anyone know where this text could be seen? The problem is that the error text is insufficient to diagnose the problem. Thanks, Jason
Re: Location of last error details seen in storm UI
Thanks for the response. Unfortunately, I have no /var/log/storm on my system. Where is the path to these logs specified. I am guessing it is pointing somewhere else by default. Thanks, Jason From: Vikas Agarwal vi...@infoobjects.com To: user@storm.incubator.apache.org Cc: Jason Kania jason.ka...@ymail.com Sent: Monday, August 25, 2014 10:34:00 AM Subject: Re: Location of last error details seen in storm UI Better would be to view log files under /var/log/storm. Any issue with worker would be logged into /var/log/storm/worker-6700.log and /var/log/storm/worker-6701.log. On Mon, Aug 25, 2014 at 8:00 PM, Vincent Russell vincent.russ...@gmail.com wrote: Click on the link of the bolt/spout that is all the way on the left side. On Sun, Aug 24, 2014 at 11:19 PM, Jason Kania jason.ka...@ymail.com wrote: Hello, I am trying to get more detail on an error that is being displayed in the Storm UI under the Last Error column but unfortunately, I am not seeing it captured anywhere else. Does anyone know where this text could be seen? The problem is that the error text is insufficient to diagnose the problem. Thanks, Jason -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
Re: Location of last error details seen in storm UI
Jason, Default is under your storm installation check for logs dir. -Harsha On Mon, Aug 25, 2014, at 07:54 AM, Jason Kania wrote: Thanks for the response. Unfortunately, I have no /var/log/storm on my system. Where is the path to these logs specified. I am guessing it is pointing somewhere else by default. Thanks, Jason __ From: Vikas Agarwal vi...@infoobjects.com To: user@storm.incubator.apache.org Cc: Jason Kania jason.ka...@ymail.com Sent: Monday, August 25, 2014 10:34:00 AM Subject: Re: Location of last error details seen in storm UI Better would be to view log files under /var/log/storm. Any issue with worker would be logged into /var/log/storm/worker-6700.log and /var/log/storm/worker-6701.log. On Mon, Aug 25, 2014 at 8:00 PM, Vincent Russell [1]vincent.russ...@gmail.com wrote: Click on the link of the bolt/spout that is all the way on the left side. On Sun, Aug 24, 2014 at 11:19 PM, Jason Kania [2]jason.ka...@ymail.com wrote: Hello, I am trying to get more detail on an error that is being displayed in the Storm UI under the Last Error column but unfortunately, I am not seeing it captured anywhere else. Does anyone know where this text could be seen? The problem is that the error text is insufficient to diagnose the problem. Thanks, Jason -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters [3]http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax References 1. mailto:vincent.russ...@gmail.com 2. mailto:jason.ka...@ymail.com 3. http://www.infoobjects.com/
RE: Location of last error details seen in storm UI
The log would be in the corresponding worker node for that bolt or spout . Ssh onto the node , inside storm installation directory there are logs like worker-6700.log and so on . You can get all the logs from there. -Original Message- From: Jason Kania Sent: 25-08-2014 AM 08:52 To: user@storm.incubator.apache.org Subject: Location of last error details seen in storm UI Hello, I am trying to get more detail on an error that is being displayed in the Storm UI under the Last Error column but unfortunately, I am not seeing it captured anywhere else. Does anyone know where this text could be seen? The problem is that the error text is insufficient to diagnose the problem. Thanks, Jason
Re: Location of last error details seen in storm UI
current version of storm doesn't have a way to define storm log dir. One way to do is to edit logback/cluster.xml under storm installation. Upcoming release will have a config option storm.log.dir to redirect the logs from default dir. -Harsha On Mon, Aug 25, 2014, at 08:16 AM, Jason Kania wrote: Thanks for that. I looked to find which property or configuration parameter sets it but could not find it. Is there such a parameter? Thanks, Jason __ From: Harsha st...@harsha.io To: user@storm.incubator.apache.org Sent: Monday, August 25, 2014 11:10:17 AM Subject: Re: Location of last error details seen in storm UI Jason, Default is under your storm installation check for logs dir. -Harsha On Mon, Aug 25, 2014, at 07:54 AM, Jason Kania wrote: Thanks for the response. Unfortunately, I have no /var/log/storm on my system. Where is the path to these logs specified. I am guessing it is pointing somewhere else by default. Thanks, Jason __ From: Vikas Agarwal vi...@infoobjects.com To: user@storm.incubator.apache.org Cc: Jason Kania jason.ka...@ymail.com Sent: Monday, August 25, 2014 10:34:00 AM Subject: Re: Location of last error details seen in storm UI Better would be to view log files under /var/log/storm. Any issue with worker would be logged into /var/log/storm/worker-6700.log and /var/log/storm/worker-6701.log. On Mon, Aug 25, 2014 at 8:00 PM, Vincent Russell [1]vincent.russ...@gmail.com wrote: Click on the link of the bolt/spout that is all the way on the left side. On Sun, Aug 24, 2014 at 11:19 PM, Jason Kania [2]jason.ka...@ymail.com wrote: Hello, I am trying to get more detail on an error that is being displayed in the Storm UI under the Last Error column but unfortunately, I am not seeing it captured anywhere else. Does anyone know where this text could be seen? The problem is that the error text is insufficient to diagnose the problem. Thanks, Jason -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters [3]http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax References 1. mailto:vincent.russ...@gmail.com 2. mailto:jason.ka...@ymail.com 3. http://www.infoobjects.com/
Re: Location of last error details seen in storm UI
I have used HortonWorks distribution for Storm and other Hadoop ecosystem and it has property for setting storm.log.dir. It is using storm 0.9.1. On Mon, Aug 25, 2014 at 9:02 PM, Harsha st...@harsha.io wrote: current version of storm doesn't have a way to define storm log dir. One way to do is to edit logback/cluster.xml under storm installation. Upcoming release will have a config option storm.log.dir to redirect the logs from default dir. -Harsha On Mon, Aug 25, 2014, at 08:16 AM, Jason Kania wrote: Thanks for that. I looked to find which property or configuration parameter sets it but could not find it. Is there such a parameter? Thanks, Jason -- *From:* Harsha st...@harsha.io *To:* user@storm.incubator.apache.org *Sent:* Monday, August 25, 2014 11:10:17 AM *Subject:* Re: Location of last error details seen in storm UI Jason, Default is under your storm installation check for logs dir. -Harsha On Mon, Aug 25, 2014, at 07:54 AM, Jason Kania wrote: Thanks for the response. Unfortunately, I have no /var/log/storm on my system. Where is the path to these logs specified. I am guessing it is pointing somewhere else by default. Thanks, Jason -- *From:* Vikas Agarwal vi...@infoobjects.com *To:* user@storm.incubator.apache.org *Cc:* Jason Kania jason.ka...@ymail.com *Sent:* Monday, August 25, 2014 10:34:00 AM *Subject:* Re: Location of last error details seen in storm UI Better would be to view log files under /var/log/storm. Any issue with worker would be logged into /var/log/storm/worker-6700.log and /var/log/storm/worker-6701.log. On Mon, Aug 25, 2014 at 8:00 PM, Vincent Russell vincent.russ...@gmail.com wrote: Click on the link of the bolt/spout that is all the way on the left side. On Sun, Aug 24, 2014 at 11:19 PM, Jason Kania jason.ka...@ymail.com wrote: Hello, I am trying to get more detail on an error that is being displayed in the Storm UI under the Last Error column but unfortunately, I am not seeing it captured anywhere else. Does anyone know where this text could be seen? The problem is that the error text is insufficient to diagnose the problem. Thanks, Jason -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
Storm UI 0.9.2 bug (num workers displaying the num tasks and vice-versa)
Hello! In the Storm UI 0.9.2 I have observed that the num executors field is displaying the number of workers and the num workers is displaying the num executors. Is this a reported bug? Best regards, Florin
Re: storm ui show problem
hey do you find storm ui show wrong number of execute tuple sometimes the number of execute tuple is bigger than its real sometime is less anybody can help ? 2014-07-23 hushengbo 发件人： 唐思成 发送时间： 2014-07-17 19:24:08 收件人： storm_user 抄送： 主题： storm upgrade issue Hi all: I try to upgrade storm from 0.9.1 to 0.9.2-incubating, and when the worknode supervisor startup, the nimbus process goes down, here is what the nimbus.log say: Before upgrade, I already change storm.local.dir: to a new location and remove storm node in zookeeper using zkCli.sh, however that dont help. AnyIdea? 2014-07-17 19:15:29 b.s.d.nimbus [ERROR] Error when processing event java.lang.RuntimeException: java.io.InvalidClassException: backtype.storm.daemon.common.SupervisorInfo; local class incompatible: stream classdesc serialVersionUID = 7648414326720210054, local class serialVersionUID = 7463898661547835557 at backtype.storm.utils.Utils.deserialize(Utils.java:93) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.cluster$maybe_deserialize.invoke(cluster.clj:200) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.cluster$mk_storm_cluster_state$reify__2284.supervisor_info(cluster.clj:299) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:na] at java.lang.reflect.Method.invoke(Method.java:597) ~[na:na] at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.5.1.jar:na] at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.5.1.jar:na] at backtype.storm.daemon.nimbus$all_supervisor_info$fn__4715.invoke(nimbus.clj:277) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.core$map$fn__4207.invoke(core.clj:2487) ~[clojure-1.5.1.jar:na] at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na] at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na] at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.5.1.jar:na] at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.5.1.jar:na] at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na] at clojure.core$mapcat.doInvoke(core.clj:2514) ~[clojure-1.5.1.jar:na] at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.5.1.jar:na] at backtype.storm.daemon.nimbus$all_supervisor_info.invoke(nimbus.clj:275) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.nimbus$all_scheduling_slots.invoke(nimbus.clj:288) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.nimbus$compute_new_topology__GT_executor__GT_node_PLUS_port.invoke(nimbus.clj:580) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:660) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.5.1.jar:na] at backtype.storm.daemon.nimbus$fn__5210$exec_fn__1396__auto5211$fn__5216$fn__5217.invoke(nimbus.clj:905) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.nimbus$fn__5210$exec_fn__1396__auto5211$fn__5216.invoke(nimbus.clj:904) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.timer$schedule_recurring$this__1134.invoke(timer.clj:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.timer$mk_timer$fn__1117$fn__1118.invoke(timer.clj:50) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.timer$mk_timer$fn__1117.invoke(timer.clj:42) [storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:619) [na:na] Caused by: java.io.InvalidClassException: backtype.storm.daemon.common.SupervisorInfo; local class incompatible: stream classdesc serialVersionUID = 7648414326720210054, local class serialVersionUID = 7463898661547835557 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:562) ~[na:na] at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583) ~[na:na] at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496) ~[na:na] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732) ~[na:na] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) ~[na:na] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) ~[na:na] at backtype.storm.utils.Utils.deserialize(Utils.java:89) ~[storm-core-0.9.2
Re: NotAliveException via Storm UI
I solved my own issue I think. There is a space in my topology name. So I believe the storm UI is not escaping the link. On Wed, Jul 23, 2014 at 6:49 PM, Vincent Russell vincent.russ...@gmail.com wrote: I'm using Storm 0.9.0.1 and I'm getting a NotAliveException via the Storm UI: NotAliveException(msg:Ingest-33-1406154838) at backtype.storm.generated.Nimbus$getTopologyInfo_result.read(Nimbus.java:11330) at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78) at backtype.storm.generated.Nimbus$Client.recv_getTopologyInfo(Nimbus.java:474) at backtype.storm.generated.Nimbus$Client.getTopologyInfo(Nimbus.java:461) at backtype.storm.ui.core$topology_page.invoke(core.clj:481) at backtype.storm.ui.core$fn__7877.invoke(core.clj:745) at compojure.core$make_route$fn__3855.invoke(core.clj:93) at compojure.core$if_route$fn__3843.invoke(core.clj:39) at compojure.core$if_method$fn__3836.invoke(core.clj:24) at compojure.core$routing$fn__3861.invoke(core.clj:106) at clojure.core$some.invoke(core.clj:2390) at compojure.core$routing.doInvoke(core.clj:106) at clojure.lang.RestFn.applyTo(RestFn.java:139) at clojure.core$apply.invoke(core.clj:603) at compojure.core$routes$fn__3865.invoke(core.clj:111) at ring.middleware.reload$wrap_reload$fn__7170.invoke(reload.clj:14) at backtype.storm.ui.core$catch_errors$fn__7912.invoke(core.clj:798) at ring.middleware.keyword_params$wrap_keyword_params$fn__4391.invoke(keyword_params.clj:27) at ring.middleware.nested_params$wrap_nested_params$fn__4428.invoke(nested_params.clj:65) at ring.middleware.params$wrap_params$fn__4365.invoke(params.clj:55) at ring.middleware.multipart_params$wrap_multipart_params$fn__4454.invoke(multipart_params.clj:103) at ring.middleware.flash$wrap_flash$fn__4729.invoke(flash.clj:14) at ring.middleware.session$wrap_session$fn__4720.invoke(session.clj:43) at ring.middleware.cookies$wrap_cookies$fn__4657.invoke(cookies.clj:160) at ring.adapter.jetty$proxy_handler$fn__4204.invoke(jetty.clj:16) at ring.adapter.jetty.proxy$org.mortbay.jetty.handler.AbstractHandler$0.handle(Unknown Source) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) but if I get the status from the command line it looks fine: # storm list Running: java -client -Dstorm.options= -Dstorm.home=/opt/storm/storm-0.9.0.1 -Djava.library.path=/usr/lib64:/usr/lib -Dstorm.conf.file= -cp /opt/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/opt/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/opt/storm/storm-0.9.0.1/lib/core.incubator-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/opt/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/opt/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/opt/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/opt/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/opt/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/opt/storm/storm-0.9.0.1/lib/asm-4.0.jar:/opt/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/opt/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/opt/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/opt/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/opt/storm/storm-0.9.0.1/lib/guava-13.0.jar:/opt/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/opt/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/opt/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/json-simple-1.1.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/opt/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/opt/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/opt/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/opt/storm/storm-0.9.0.1/lib/objenesis-1.2.jar:/opt/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5-20081211.jar:/opt/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/opt/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/opt/storm/storm-0.9.0.1
Re: Storm UI : handle custom stream as a system one
Thanks Harsha. I created https://issues.apache.org/jira/browse/STORM-416 On 21 July 2014 17:05, Harsha st...@harsha.io wrote: the code is in core.clj in mk-include-sys-fn which calls system-id?(common.clj). I think UI code is fine but nimbus won't accept a topology which has stream-id with __ at the beginning. We can probably add a exception in nimbus if a stream starts with __ and its in storm.user.system.streams. Can you please file a jira to track this. Thanks. -Harsha On Mon, Jul 21, 2014, at 08:59 AM, Julien Nioche wrote: Yes, that's also the conclusion I came to. I could not find where in the UI code is the call to Utils.isSystemId(String). One option would be to be able to define in the configuration a list of streams to treat as system. Does the UI code access the configuration files at all? Thanks Harsha On 21 July 2014 16:43, Harsha st...@harsha.io wrote: thats caused by validate-ids! function which checks if the users stream id is system id and throws that exception. So looks like __ reserved for system streams only not allowed for users. On Mon, Jul 21, 2014, at 08:30 AM, Julien Nioche wrote: Hi Harsha Am getting : 5935 [main] WARN backtype.storm.daemon.nimbus - Topology submission exception. (topology name='QueuePopulator') #InvalidTopologyException InvalidTopologyException(msg:__log is not a valid stream id) 5941 [main] ERROR org.apache.zookeeper.server.NIOServerCnxnFactory - Thread Thread[main,5,main] died backtype.storm.generated.InvalidTopologyException: null at backtype.storm.daemon.common$validate_ids_BANG_.invoke(common.clj:126) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.common$validate_basic_BANG_.invoke(common.clj:142) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.common$system_topology_BANG_.invoke(common.clj:297) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] Thanks Julien On 21 July 2014 16:22, Harsha st...@harsha.io wrote: Hi Julien, UI code calls Utils.isSystemId(String) which checks if the stream id starts with __. What error are you seeing when you renamed into __log. -Harsha On Mon, Jul 21, 2014, at 03:45 AM, Julien Nioche wrote: Hi, I have a custom stream for handling logs (called '_log') and send them to ElasticSearch for indexing. The log tuples are generated by my spouts and bolts. My pipeline also uses the default stream for the normal processing of tuples from RabbitMQ. Everything works fine but I would like to be able to treat this _log stream as one of the system ones (e.g. __metrics) and be able to hide them from the stats. The summary of Emitted / Transferred currently takes these log events into account which is not very useful. I tried renaming the stream into '__log' but this resulted in an error when trying to start the topoloy. Any idea of how I could do that? Thanks Julien -- Open Source Solutions for Text Engineering http://digitalpebble.blogspot.com/ http://www.digitalpebble.com http://twitter.com/digitalpebble -- Open Source Solutions for Text Engineering http://digitalpebble.blogspot.com/ http://www.digitalpebble.com http://twitter.com/digitalpebble -- Open Source Solutions for Text Engineering http://digitalpebble.blogspot.com/ http://www.digitalpebble.com http://twitter.com/digitalpebble -- Open Source Solutions for Text Engineering http://digitalpebble.blogspot.com/ http://www.digitalpebble.com http://twitter.com/digitalpebble
Re: Storm UI
Thanks for the answer, But my problem is that I am on storm incubator ( https://github.com/apache/incubator-storm) and I don't find anything like you said, except at this path : storm-core/src/ui/public/index.html where I suppose you can see your storm ui. I understand I have to start Storm UI via a command line thanks to STORM_HOME, but this environment variable is not defined in my case ... Maybe I missed something ... Any idea what I have to do?? Thanks 2014-07-11 17:37 GMT+02:00 Harsha st...@harsha.io: Storm UI provides metrics about topologies on the cluster and no.of tuples emitted, transferred and any last known errors etc.. you can start storm ui by running STORM_HOME/bin/storm ui which runs daemon at port 8080. If you hover over the table headers in Storm UI it will show you a text which talks about that particular value. If you are trying to add custom metrics to your topology please refer to this page http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to On Fri, Jul 11, 2014, at 02:38 AM, Benjamin SOULAS wrote: Hi everyone, Actually intern for my master's degree, I have to implement topologies and see what's happening. I am trying to see those data via Storm UI; My problem is that I don't find enough documentation on that... I installed the splunk interface, but I don't know how to implement it on my topologies ... Does the Metrics interfaces are used for this??? Please I really need help ... Regards
Re: Storm UI
Hi benjamin, If i understand your question right, u are unable to see the UI after installing the Storm tar.gz. you need to capture ui.port:8080 on the storm.yaml file present in storm_home/conf/ location then u should be able to see the UI at port 8080. Also in local mode (if u r using it) u can just see the UI but i dont think u can capture any changes in there while topology is running. it will be a clean webpage with parameters displayed. if u want to monitor the metrics i think u will need a distributed storm environment. Thanks, Siddharth Ubale On Tue, Jul 15, 2014 at 1:29 PM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Thanks for the answer, But my problem is that I am on storm incubator ( https://github.com/apache/incubator-storm) and I don't find anything like you said, except at this path : storm-core/src/ui/public/index.html where I suppose you can see your storm ui. I understand I have to start Storm UI via a command line thanks to STORM_HOME, but this environment variable is not defined in my case ... Maybe I missed something ... Any idea what I have to do?? Thanks 2014-07-11 17:37 GMT+02:00 Harsha st...@harsha.io: Storm UI provides metrics about topologies on the cluster and no.of tuples emitted, transferred and any last known errors etc.. you can start storm ui by running STORM_HOME/bin/storm ui which runs daemon at port 8080. If you hover over the table headers in Storm UI it will show you a text which talks about that particular value. If you are trying to add custom metrics to your topology please refer to this page http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to On Fri, Jul 11, 2014, at 02:38 AM, Benjamin SOULAS wrote: Hi everyone, Actually intern for my master's degree, I have to implement topologies and see what's happening. I am trying to see those data via Storm UI; My problem is that I don't find enough documentation on that... I installed the splunk interface, but I don't know how to implement it on my topologies ... Does the Metrics interfaces are used for this??? Please I really need help ... Regards
Re: Storm UI
Hi benjamin, Just a word of caution.. Please do not use the incubating versions... instead take the storm-0.9.0.1.tar.gz https://dl.dropboxusercontent.com/s/tqdpoif32gufapo/storm-0.9.0.1.tar.gz for installation. Issues for these versions have ben documented. Also u can refer to http://10jumps.com/blog/storm-installation-single-machine for installation. Thanks, Siddharth On Tue, Jul 15, 2014 at 3:18 PM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Hi Suddharth, In fact, i took the last version of storm, i installed it thanks to maven. Then i developed topologies to test how it works. i also recompile the src because i could need to modify could in the future. My development environment is that one : - linux 32bits in a VM - Eclipse juno - the last maven (3.2.1) When i trie to launch storm ui, i got that error : ** The storm client can only be run from within a release. You appear to be trying to run the client from a checkout of Storm's source code. You can download a Storm release at http://storm-project.net/downloads.html ** So i thought i had a wrong version, so now i am trying to do the same thing with the last storm version (with sources) from the apache website (0.9.2) but i have the same error. I don't understand, maybe i misunderstood everything about storm implementation. But i will follow your instructions now, maybe it will work. Thanks a lot for your help. Kind regards. 2014-07-15 11:31 GMT+02:00 siddharth ubale siddharth.ub...@gmail.com: Hi benjamin, If i understand your question right, u are unable to see the UI after installing the Storm tar.gz. you need to capture ui.port:8080 on the storm.yaml file present in storm_home/conf/ location then u should be able to see the UI at port 8080. Also in local mode (if u r using it) u can just see the UI but i dont think u can capture any changes in there while topology is running. it will be a clean webpage with parameters displayed. if u want to monitor the metrics i think u will need a distributed storm environment. Thanks, Siddharth Ubale On Tue, Jul 15, 2014 at 1:29 PM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Thanks for the answer, But my problem is that I am on storm incubator ( https://github.com/apache/incubator-storm) and I don't find anything like you said, except at this path : storm-core/src/ui/public/index.html where I suppose you can see your storm ui. I understand I have to start Storm UI via a command line thanks to STORM_HOME, but this environment variable is not defined in my case ... Maybe I missed something ... Any idea what I have to do?? Thanks 2014-07-11 17:37 GMT+02:00 Harsha st...@harsha.io: Storm UI provides metrics about topologies on the cluster and no.of tuples emitted, transferred and any last known errors etc.. you can start storm ui by running STORM_HOME/bin/storm ui which runs daemon at port 8080. If you hover over the table headers in Storm UI it will show you a text which talks about that particular value. If you are trying to add custom metrics to your topology please refer to this page http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to On Fri, Jul 11, 2014, at 02:38 AM, Benjamin SOULAS wrote: Hi everyone, Actually intern for my master's degree, I have to implement topologies and see what's happening. I am trying to see those data via Storm UI; My problem is that I don't find enough documentation on that... I installed the splunk interface, but I don't know how to implement it on my topologies ... Does the Metrics interfaces are used for this??? Please I really need help ... Regards
Re: Storm UI
Ok, But i don't understand why i can't start storm ui on a local mode, actually i make some copy/paste for watching my topologies result ... Any idea? Are we obliged to launch storm on a cluster ti have storm UI??? Regards 2014-07-15 12:55 GMT+02:00 siddharth ubale siddharth.ub...@gmail.com: Hi benjamin, Just a word of caution.. Please do not use the incubating versions... instead take the storm-0.9.0.1.tar.gz https://dl.dropboxusercontent.com/s/tqdpoif32gufapo/storm-0.9.0.1.tar.gz for installation. Issues for these versions have ben documented. Also u can refer to http://10jumps.com/blog/storm-installation-single-machine for installation. Thanks, Siddharth On Tue, Jul 15, 2014 at 3:18 PM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Hi Suddharth, In fact, i took the last version of storm, i installed it thanks to maven. Then i developed topologies to test how it works. i also recompile the src because i could need to modify could in the future. My development environment is that one : - linux 32bits in a VM - Eclipse juno - the last maven (3.2.1) When i trie to launch storm ui, i got that error : ** The storm client can only be run from within a release. You appear to be trying to run the client from a checkout of Storm's source code. You can download a Storm release at http://storm-project.net/downloads.html ** So i thought i had a wrong version, so now i am trying to do the same thing with the last storm version (with sources) from the apache website (0.9.2) but i have the same error. I don't understand, maybe i misunderstood everything about storm implementation. But i will follow your instructions now, maybe it will work. Thanks a lot for your help. Kind regards. 2014-07-15 11:31 GMT+02:00 siddharth ubale siddharth.ub...@gmail.com: Hi benjamin, If i understand your question right, u are unable to see the UI after installing the Storm tar.gz. you need to capture ui.port:8080 on the storm.yaml file present in storm_home/conf/ location then u should be able to see the UI at port 8080. Also in local mode (if u r using it) u can just see the UI but i dont think u can capture any changes in there while topology is running. it will be a clean webpage with parameters displayed. if u want to monitor the metrics i think u will need a distributed storm environment. Thanks, Siddharth Ubale On Tue, Jul 15, 2014 at 1:29 PM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Thanks for the answer, But my problem is that I am on storm incubator ( https://github.com/apache/incubator-storm) and I don't find anything like you said, except at this path : storm-core/src/ui/public/index.html where I suppose you can see your storm ui. I understand I have to start Storm UI via a command line thanks to STORM_HOME, but this environment variable is not defined in my case ... Maybe I missed something ... Any idea what I have to do?? Thanks 2014-07-11 17:37 GMT+02:00 Harsha st...@harsha.io: Storm UI provides metrics about topologies on the cluster and no.of tuples emitted, transferred and any last known errors etc.. you can start storm ui by running STORM_HOME/bin/storm ui which runs daemon at port 8080. If you hover over the table headers in Storm UI it will show you a text which talks about that particular value. If you are trying to add custom metrics to your topology please refer to this page http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to On Fri, Jul 11, 2014, at 02:38 AM, Benjamin SOULAS wrote: Hi everyone, Actually intern for my master's degree, I have to implement topologies and see what's happening. I am trying to see those data via Storm UI; My problem is that I don't find enough documentation on that... I installed the splunk interface, but I don't know how to implement it on my topologies ... Does the Metrics interfaces are used for this??? Please I really need help ... Regards
Re: Storm UI
Nope, u can have a UI .. but to monitor u need us storm topolgy to run on the cluster. On Tue, Jul 15, 2014 at 5:24 PM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Ok, But i don't understand why i can't start storm ui on a local mode, actually i make some copy/paste for watching my topologies result ... Any idea? Are we obliged to launch storm on a cluster ti have storm UI??? Regards 2014-07-15 12:55 GMT+02:00 siddharth ubale siddharth.ub...@gmail.com: Hi benjamin, Just a word of caution.. Please do not use the incubating versions... instead take the storm-0.9.0.1.tar.gz https://dl.dropboxusercontent.com/s/tqdpoif32gufapo/storm-0.9.0.1.tar.gz for installation. Issues for these versions have ben documented. Also u can refer to http://10jumps.com/blog/storm-installation-single-machine for installation. Thanks, Siddharth On Tue, Jul 15, 2014 at 3:18 PM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Hi Suddharth, In fact, i took the last version of storm, i installed it thanks to maven. Then i developed topologies to test how it works. i also recompile the src because i could need to modify could in the future. My development environment is that one : - linux 32bits in a VM - Eclipse juno - the last maven (3.2.1) When i trie to launch storm ui, i got that error : ** The storm client can only be run from within a release. You appear to be trying to run the client from a checkout of Storm's source code. You can download a Storm release at http://storm-project.net/downloads.html ** So i thought i had a wrong version, so now i am trying to do the same thing with the last storm version (with sources) from the apache website (0.9.2) but i have the same error. I don't understand, maybe i misunderstood everything about storm implementation. But i will follow your instructions now, maybe it will work. Thanks a lot for your help. Kind regards. 2014-07-15 11:31 GMT+02:00 siddharth ubale siddharth.ub...@gmail.com: Hi benjamin, If i understand your question right, u are unable to see the UI after installing the Storm tar.gz. you need to capture ui.port:8080 on the storm.yaml file present in storm_home/conf/ location then u should be able to see the UI at port 8080. Also in local mode (if u r using it) u can just see the UI but i dont think u can capture any changes in there while topology is running. it will be a clean webpage with parameters displayed. if u want to monitor the metrics i think u will need a distributed storm environment. Thanks, Siddharth Ubale On Tue, Jul 15, 2014 at 1:29 PM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Thanks for the answer, But my problem is that I am on storm incubator ( https://github.com/apache/incubator-storm) and I don't find anything like you said, except at this path : storm-core/src/ui/public/index.html where I suppose you can see your storm ui. I understand I have to start Storm UI via a command line thanks to STORM_HOME, but this environment variable is not defined in my case ... Maybe I missed something ... Any idea what I have to do?? Thanks 2014-07-11 17:37 GMT+02:00 Harsha st...@harsha.io: Storm UI provides metrics about topologies on the cluster and no.of tuples emitted, transferred and any last known errors etc.. you can start storm ui by running STORM_HOME/bin/storm ui which runs daemon at port 8080. If you hover over the table headers in Storm UI it will show you a text which talks about that particular value. If you are trying to add custom metrics to your topology please refer to this page http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to On Fri, Jul 11, 2014, at 02:38 AM, Benjamin SOULAS wrote: Hi everyone, Actually intern for my master's degree, I have to implement topologies and see what's happening. I am trying to see those data via Storm UI; My problem is that I don't find enough documentation on that... I installed the splunk interface, but I don't know how to implement it on my topologies ... Does the Metrics interfaces are used for this??? Please I really need help ... Regards
Re: Storm UI
Hi, We are planning to use 0.9.2-incubating for the production environment? Why we should not use 'incubating'. From Apache we don't have any other versions which are latest and stable Regards, On Tue, Jul 15, 2014 at 7:09 PM, Derek Dagit der...@yahoo-inc.com wrote: Siddharth, Just a word of caution.. Please do not use the incubating versions... Is there a blocking issue that we should be aware of? As far as I know, the Apache releases should be usable. -- Derek On 7/15/14, 5:55, siddharth ubale wrote: Hi benjamin, Just a word of caution.. Please do not use the incubating versions... instead take the storm-0.9.0.1.tar.gz https://dl.dropboxusercontent.com/s/tqdpoif32gufapo/storm-0.9.0.1.tar.gz for installation. Issues for these versions have ben documented. Also u can refer to http://10jumps.com/blog/storm-installation-single-machine for installation. Thanks, Siddharth On Tue, Jul 15, 2014 at 3:18 PM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Hi Suddharth, In fact, i took the last version of storm, i installed it thanks to maven. Then i developed topologies to test how it works. i also recompile the src because i could need to modify could in the future. My development environment is that one : - linux 32bits in a VM - Eclipse juno - the last maven (3.2.1) When i trie to launch storm ui, i got that error : ** The storm client can only be run from within a release. You appear to be trying to run the client from a checkout of Storm's source code. You can download a Storm release at http://storm-project.net/downloads.html ** So i thought i had a wrong version, so now i am trying to do the same thing with the last storm version (with sources) from the apache website (0.9.2) but i have the same error. I don't understand, maybe i misunderstood everything about storm implementation. But i will follow your instructions now, maybe it will work. Thanks a lot for your help. Kind regards. 2014-07-15 11:31 GMT+02:00 siddharth ubale siddharth.ub...@gmail.com: Hi benjamin, If i understand your question right, u are unable to see the UI after installing the Storm tar.gz. you need to capture ui.port:8080 on the storm.yaml file present in storm_home/conf/ location then u should be able to see the UI at port 8080. Also in local mode (if u r using it) u can just see the UI but i dont think u can capture any changes in there while topology is running. it will be a clean webpage with parameters displayed. if u want to monitor the metrics i think u will need a distributed storm environment. Thanks, Siddharth Ubale On Tue, Jul 15, 2014 at 1:29 PM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Thanks for the answer, But my problem is that I am on storm incubator ( https://github.com/apache/incubator-storm) and I don't find anything like you said, except at this path : storm-core/src/ui/public/ index.html where I suppose you can see your storm ui. I understand I have to start Storm UI via a command line thanks to STORM_HOME, but this environment variable is not defined in my case ... Maybe I missed something ... Any idea what I have to do?? Thanks 2014-07-11 17:37 GMT+02:00 Harsha st...@harsha.io: Storm UI provides metrics about topologies on the cluster and no.of tuples emitted, transferred and any last known errors etc.. you can start storm ui by running STORM_HOME/bin/storm ui which runs daemon at port 8080. If you hover over the table headers in Storm UI it will show you a text which talks about that particular value. If you are trying to add custom metrics to your topology please refer to this page http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to On Fri, Jul 11, 2014, at 02:38 AM, Benjamin SOULAS wrote: Hi everyone, Actually intern for my master's degree, I have to implement topologies and see what's happening. I am trying to see those data via Storm UI; My problem is that I don't find enough documentation on that... I installed the splunk interface, but I don't know how to implement it on my topologies ... Does the Metrics interfaces are used for this??? Please I really need help ... Regards
Re: Storm UI
incubating just means apache incubating project, not mean to beta, not stable, etc. 2014년 7월 15일 화요일, siddharth ubalesiddharth.ub...@gmail.com님이 작성한 메시지: Hi benjamin, Just a word of caution.. Please do not use the incubating versions... instead take the storm-0.9.0.1.tar.gz https://dl.dropboxusercontent.com/s/tqdpoif32gufapo/storm-0.9.0.1.tar.gz for installation. Issues for these versions have ben documented. Also u can refer to http://10jumps.com/blog/storm-installation-single-machine for installation. Thanks, Siddharth On Tue, Jul 15, 2014 at 3:18 PM, Benjamin SOULAS benjamin.soula...@gmail.com javascript:_e(%7B%7D,'cvml','benjamin.soula...@gmail.com'); wrote: Hi Suddharth, In fact, i took the last version of storm, i installed it thanks to maven. Then i developed topologies to test how it works. i also recompile the src because i could need to modify could in the future. My development environment is that one : - linux 32bits in a VM - Eclipse juno - the last maven (3.2.1) When i trie to launch storm ui, i got that error : ** The storm client can only be run from within a release. You appear to be trying to run the client from a checkout of Storm's source code. You can download a Storm release at http://storm-project.net/downloads.html ** So i thought i had a wrong version, so now i am trying to do the same thing with the last storm version (with sources) from the apache website (0.9.2) but i have the same error. I don't understand, maybe i misunderstood everything about storm implementation. But i will follow your instructions now, maybe it will work. Thanks a lot for your help. Kind regards. 2014-07-15 11:31 GMT+02:00 siddharth ubale siddharth.ub...@gmail.com javascript:_e(%7B%7D,'cvml','siddharth.ub...@gmail.com');: Hi benjamin, If i understand your question right, u are unable to see the UI after installing the Storm tar.gz. you need to capture ui.port:8080 on the storm.yaml file present in storm_home/conf/ location then u should be able to see the UI at port 8080. Also in local mode (if u r using it) u can just see the UI but i dont think u can capture any changes in there while topology is running. it will be a clean webpage with parameters displayed. if u want to monitor the metrics i think u will need a distributed storm environment. Thanks, Siddharth Ubale On Tue, Jul 15, 2014 at 1:29 PM, Benjamin SOULAS benjamin.soula...@gmail.com javascript:_e(%7B%7D,'cvml','benjamin.soula...@gmail.com'); wrote: Thanks for the answer, But my problem is that I am on storm incubator ( https://github.com/apache/incubator-storm) and I don't find anything like you said, except at this path : storm-core/src/ui/public/index.html where I suppose you can see your storm ui. I understand I have to start Storm UI via a command line thanks to STORM_HOME, but this environment variable is not defined in my case ... Maybe I missed something ... Any idea what I have to do?? Thanks 2014-07-11 17:37 GMT+02:00 Harsha st...@harsha.io javascript:_e(%7B%7D,'cvml','st...@harsha.io');: Storm UI provides metrics about topologies on the cluster and no.of tuples emitted, transferred and any last known errors etc.. you can start storm ui by running STORM_HOME/bin/storm ui which runs daemon at port 8080. If you hover over the table headers in Storm UI it will show you a text which talks about that particular value. If you are trying to add custom metrics to your topology please refer to this page http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to On Fri, Jul 11, 2014, at 02:38 AM, Benjamin SOULAS wrote: Hi everyone, Actually intern for my master's degree, I have to implement topologies and see what's happening. I am trying to see those data via Storm UI; My problem is that I don't find enough documentation on that... I installed the splunk interface, but I don't know how to implement it on my topologies ... Does the Metrics interfaces are used for this??? Please I really need help ... Regards -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
Re: Storm UI: not displayed Executors
Thanks Derek. I was able to solve about this problem. Concretely, I referred to the following page. https://git-wip-us.apache.org/repos/asf?p=incubator-storm.git;a=blob;f=storm-core/src/ui/public/component.html;hb=31c786c6 Thank you! 2014-07-11 22:42 GMT+09:00 Derek Dagit der...@yahoo-inc.com: This should be fixed with either STORM-370 (merged) or STORM-379 (pull request open). The script that sorts the table did not check that the list was of non-zero size before it attempted to sort, and that resulted in an exception that halted subsequent rendering on the page. You can: - checkout the latest storm and use that - cherry-pick commit 31c786c into the version you are using. -- Derek On 7/11/14, 5:54, 川原駿 wrote: Hello. Recently I upgraded storm to 0.9.2. In Component summary page of Storm UI, Executors is not displayed only when emitted of its spout/bolt is 0. Please tell me the solution. Thanks.
Storm UI: not displayed Executors
Hello. Recently I upgraded storm to 0.9.2. In Component summary page of Storm UI, Executors is not displayed only when emitted of its spout/bolt is 0. Please tell me the solution. Thanks.
Re: Storm UI: not displayed Executors
This should be fixed with either STORM-370 (merged) or STORM-379 (pull request open). The script that sorts the table did not check that the list was of non-zero size before it attempted to sort, and that resulted in an exception that halted subsequent rendering on the page. You can: - checkout the latest storm and use that - cherry-pick commit 31c786c into the version you are using. -- Derek On 7/11/14, 5:54, 川原駿 wrote: Hello. Recently I upgraded storm to 0.9.2. In Component summary page of Storm UI, Executors is not displayed only when emitted of its spout/bolt is 0. Please tell me the solution. Thanks.
Re: Storm UI
Storm UI provides metrics about topologies on the cluster and no.of tuples emitted, transferred and any last known errors etc.. you can start storm ui by running STORM_HOME/bin/storm ui which runs daemon at port 8080. If you hover over the table headers in Storm UI it will show you a text which talks about that particular value. If you are trying to add custom metrics to your topology please refer to this page [1]http://www.bigdata-cookbook.com/post/72320512609/storm- metrics-how-to On Fri, Jul 11, 2014, at 02:38 AM, Benjamin SOULAS wrote: Hi everyone, Actually intern for my master's degree, I have to implement topologies and see what's happening. I am trying to see those data via Storm UI; My problem is that I don't find enough documentation on that... I installed the splunk interface, but I don't know how to implement it on my topologies ... Does the Metrics interfaces are used for this??? Please I really need help ... Regards References 1. http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
Re: what does each field of storm UI mean?
Adrian, If you hover over the title of the field, there should appear a pop-up to explain what it means. -- Derek On 6/17/14, 21:05, 이승진 wrote: Dear storm users I want to see performance of each bolt and decide the number of parallelism. In storm UI there are several fields which is confusing, so would be glad if you can tell me. Capacity(last 10m) - average capacity per one second in last 10 minute of a single executor? For example, if Capcity is 1.2, does that mean single executor processed 1.2 messages per second in average? Execute latency and Process latency - Is it average value or value of last processed message? and what is the difference between them? and what is the difference between them and Capacity? Sincerly, Adrian SJ Lee
Storm UI - Custom time window
Hi, I was looking at the Storm UI and wanted to check if we can configure the time window property so that it can give me a data for a smaller time window. I found that in the code within stats.clj, I could configure the below lines and get the UI to show the stats for time window which I need. (def NUM-STAT-BUCKETS 1) ;; 10 minutes, 3 hours, 1 day (def STAT-BUCKETS [1 30 540 4320]) Is there any way we can externalize this configuration? Is there any impact related to storm's performance with this change? Regards, Pradeep V.B. This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.
Re: Storm UI
it displays different stats depending on whether or not I show/hide system stats. This is expected. There should be a tool tip on that button that says something like toggle inclusion of system components. How much the stats change seems to differ between Chrome and IE. That button is setting a cookie's value to true/false. The math is done on the server side and not in the browser, so the difference in the browser used should not matter beyond setting the cookie. If the toggle is not taking effect at all for some browsers, then we should create a new Issue to take a look. -- Derek On 4/2/14, 8:04, David Crossland wrote: I have a curious issue with the UI, it displays different stats depending on whether or not I show/hide system stats. How much the stats change seems to differ between Chrome and IE. Is this a known issue? Thanks David
Re: Storm UI
Unluckily I've taken the cluster down now otherwise I would take some screenshots. As soon as I have run one up I will provide some. But from what I remember it seemed to double in Chrome, in IE it was much larger. It just made it hard to know which stats to interpret as correct, although I reasoned it must be when the system stats are hidden. I did not see a tooltip.. I will check that too. I will reply probably tomorrow with the info Thanks David From: Derek Dagitmailto:der...@yahoo-inc.com Sent: ?Wednesday?, ?2? ?April? ?2014 ?15?:?42 To: user@storm.incubator.apache.orgmailto:user@storm.incubator.apache.org it displays different stats depending on whether or not I show/hide system stats. This is expected. There should be a tool tip on that button that says something like toggle inclusion of system components. How much the stats change seems to differ between Chrome and IE. That button is setting a cookie's value to true/false. The math is done on the server side and not in the browser, so the difference in the browser used should not matter beyond setting the cookie. If the toggle is not taking effect at all for some browsers, then we should create a new Issue to take a look. -- Derek On 4/2/14, 8:04, David Crossland wrote: I have a curious issue with the UI, it displays different stats depending on whether or not I show/hide system stats. How much the stats change seems to differ between Chrome and IE. Is this a known issue? Thanks David
RE: Storm ui when using trident is confusing
Hi. I am not familiar with this tool so it will take me some time to see if it is suited for my needs. I will check this out. thanks *From:* Otis Gospodnetic [mailto:otis.gospodne...@gmail.com] *Sent:* Friday, January 24, 2014 9:27 AM *To:* user@storm.incubator.apache.org *Subject:* Re: Storm ui when using trident is confusing Hi Michal, I hope somebody else can answer your various questions, but you seem to be interested in monitoring Storm, so let me point you to to this mini announcement: http://blog.sematext.com/2013/12/18/announce-spm/ - in short, SPM can monitor your Storm cluster and show you various storm metrics as graphs. I hope this helps. Otis On Wed, Jan 22, 2014 at 5:09 AM, Michal Singer mic...@leadspace.com wrote: Hi, I see that when using trident the storm ui is much harder to understand. The spout is named differently and if I am not mistaken it is based defined under the bolts? And there are a lot of messages transferred I assume of the CoordinateBolt or other management messages and it is hard to distinguish them from the applicative messages. Can someone explain how the ui can be used for trident? Is there an intention on improving this so it can be used as monitoring system for trident? Thanks, Michal
Re: Storm ui when using trident is confusing
Hi Michal, I hope somebody else can answer your various questions, but you seem to be interested in monitoring Storm, so let me point you to to this mini announcement: http://blog.sematext.com/2013/12/18/announce-spm/ - in short, SPM can monitor your Storm cluster and show you various storm metrics as graphs. I hope this helps. Otis On Wed, Jan 22, 2014 at 5:09 AM, Michal Singer mic...@leadspace.com wrote: Hi, I see that when using trident the storm ui is much harder to understand. The spout is named differently and if I am not mistaken it is based defined under the bolts? And there are a lot of messages transferred I assume of the CoordinateBolt or other management messages and it is hard to distinguish them from the applicative messages. Can someone explain how the ui can be used for trident? Is there an intention on improving this so it can be used as monitoring system for trident? Thanks, Michal
Re: TOPOLOGY INFORMATION IS NOT APPEARING IN STORM UI
storm ui will only display the information of topologies submited to the same storm cluster local model will not work. 2014/1/22 Richards Peter hbkricha...@gmail.com Hi, I am not sure what exactly is happening in your case. I assume that you are submitting the topology in local cluster mode. If so you need not use the storm kill command. Please go through the pages I have shared in my previous mail to get a better idea. Richards Peter.
Re: TOPOLOGY INFORMATION IS NOT APPEARING IN STORM UI
There are two things to notice here: 1. Whether you are submitting the topology to the storm cluster or to the local cluster. If you submit the topology to the storm cluster, you will be able to find the topology information on storm UI. 2. Whether storm nimbus and storm ui parameters are correctly configured in program and storm.yaml file respectively. Thanks, Richards Peter.
RE: TOPOLOGY INFORMATION IS NOT APPEARING IN STORM UI
OutputCollector collector;/*** At the end of the spout (when the
cluster is shutdown* We will show the word counters*/@Overridepublic void
cleanup() {System.out.println(-- Word Counter [+name+-+id+]
--);for(Map.EntryString, Integer entry :
counters.entrySet()){System.out.println(entry.getKey()+:
+entry.getValue());}}/*** On each word We will count*/@Overridepublic void
execute(Tuple input) {String str = input.getString(0);/*** If the word dosn't
exist in the map we will create* this, if not We will add
1*/if(!counters.containsKey(str)){counters.put(str, 1);}else{Integer c =
counters.get(str) + 1;counters.put(str, c);}//Set the tuple as
Acknowledgecollector.ack(input);}/*** On create*/@Overridepublic void
prepare(Map stormConf, TopologyContext context,OutputCollector collector)
{this.counters = new HashMapString, Integer();this.collector =
collector;this.name = context.getThisComponentId();this.id =
context.getThisTaskId();}@Overridepublic void
declareOutputFields(OutputFieldsDeclarer declarer) {}
public void deactivate(){}
public MapString, Object getComponentConfiguration(){return count;}
}Date: Tue, 21 Jan 2014 19:37:10 +0530
Subject: Re: TOPOLOGY INFORMATION IS NOT APPEARING IN STORM UI
From: hbkricha...@gmail.com
To: user@storm.incubator.apache.org
There are two things to notice here:
1. Whether you are submitting the topology to the storm cluster or to the local
cluster. If you submit the topology to the storm cluster, you will be able to
find the topology information on storm UI.
2. Whether storm nimbus and storm ui parameters are correctly configured in
program and storm.yaml file respectively.
Thanks,
Richards Peter.
Re: TOPOLOGY INFORMATION IS NOT APPEARING IN STORM UI
, Object getComponentConfiguration(){return count;}
}
*Bolt program - word counter:*
import java.util.HashMap;
import java.util.Map;
import backtype.storm.task.OutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.IRichBolt;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.tuple.Tuple;
public class WordCounter implements IRichBolt {
Integer id;
String name;
MapString, Integer counters;
MapString, Object count;
private OutputCollector collector;
/**
* At the end of the spout (when the cluster is shutdown
* We will show the word counters
*/
@Override
public void cleanup() {
System.out.println(-- Word Counter [+name+-+id+] --);
for(Map.EntryString, Integer entry : counters.entrySet()){
System.out.println(entry.getKey()+: +entry.getValue());
}
}
/**
* On each word We will count
*/
@Override
public void execute(Tuple input) {
String str = input.getString(0);
/**
* If the word dosn't exist in the map we will create
* this, if not We will add 1
*/
if(!counters.containsKey(str)){
counters.put(str, 1);
}else{
Integer c = counters.get(str) + 1;
counters.put(str, c);
}
//Set the tuple as Acknowledge
collector.ack(input);
}
/**
* On create
*/
@Override
public void prepare(Map stormConf, TopologyContext context,
OutputCollector collector) {
this.counters = new HashMapString, Integer();
this.collector = collector;
this.name = context.getThisComponentId();
this.id = context.getThisTaskId();
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {}
public void deactivate(){}
public MapString, Object getComponentConfiguration(){return count;}
}
--
Date: Tue, 21 Jan 2014 19:37:10 +0530
Subject: Re: TOPOLOGY INFORMATION IS NOT APPEARING IN STORM UI
From: hbkricha...@gmail.com
To: user@storm.incubator.apache.org
There are two things to notice here:
1. Whether you are submitting the topology to the storm cluster or to the
local cluster. If you submit the topology to the storm cluster, you will be
able to find the topology information on storm UI.
2. Whether storm nimbus and storm ui parameters are correctly configured
in program and storm.yaml file respectively.
Thanks,
Richards Peter.
RE: TOPOLOGY INFORMATION IS NOT APPEARING IN STORM UI
Hi Peter,Thanks for your kind reply. We killed the topology by giving cluster.killtopology(word count). We have the doubt that should we also give shutdown command in spite of kill topology command. We are running the topology once, it is working fine. 2nd time if we run the topology it is not running. Third time it is running. We are not able to resolve this. 2nd time the entire process is getting shutdown without showing the output. But 3rd time we are getting the output. Another problem is the process is not terminating automatically in the command prompt. Please do guide us. Thanks,Priya Date: Wed, 22 Jan 2014 11:21:00 +0530 Subject: Re: TOPOLOGY INFORMATION IS NOT APPEARING IN STORM UI From: hbkricha...@gmail.com To: user@storm.incubator.apache.org Hi, When you run the topology in local mode: https://github.com/nathanmarz/storm/wiki/Local-mode - You can call cluster.shutdown(); to kill the topology. - If you are running it from eclipse, it is fine even if you kill the running application. When you run the topology in cluster mode: https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster -You have to issue the storm kill topologyname command to kill the topology Once you kill the topology running in cluster mode, you can restart the same topology in that cluster without changing its name. Richards Peter.
Re: TOPOLOGY INFORMATION IS NOT APPEARING IN STORM UI
Hi, I am not sure what exactly is happening in your case. I assume that you are submitting the topology in local cluster mode. If so you need not use the storm kill command. Please go through the pages I have shared in my previous mail to get a better idea. Richards Peter.
How to capture custom Storm UI related metrics using Ganglia
Hi, I am trying to use Ganglia to capture all the metrics that are displayed in Storm UI. I am not quite sure on how to do that.I was able to integrate Cassandra with Ganglia,where I just need to specify the metric name I want to capture in a cassandra config file and Ganglia displays that metric.Is there anything similar for Storm as well? I know its possible to capture the JVM,Memory stats of the Storm cluster,but I am more interested in Storm specific parameters such as Nimbus config,Number of spouts/bolts,Tuples emitted,Tuples acked,Latency,etc. Any help in this regard will be highly appreciated.
