Re: Storm UI: not displayed Executors
Thanks Derek. I was able to solve about this problem. Concretely, I referred to the following page. https://git-wip-us.apache.org/repos/asf?p=incubator-storm.git;a=blob;f=storm-core/src/ui/public/component.html;hb=31c786c6 Thank you! 2014-07-11 22:42 GMT+09:00 Derek Dagit der...@yahoo-inc.com: This should be fixed with either STORM-370 (merged) or STORM-379 (pull request open). The script that sorts the table did not check that the list was of non-zero size before it attempted to sort, and that resulted in an exception that halted subsequent rendering on the page. You can: - checkout the latest storm and use that - cherry-pick commit 31c786c into the version you are using. -- Derek On 7/11/14, 5:54, 川原駿 wrote: Hello. Recently I upgraded storm to 0.9.2. In Component summary page of Storm UI, Executors is not displayed only when emitted of its spout/bolt is 0. Please tell me the solution. Thanks.
Need help to use storm with mysql.
I made a storm topoogy where spout was fetching data from mysql using select query. The select query was fired after every 30 msec but because the size of the table is more than 20 GB the select query takes more than 10 sec to execute therefore this is not working. I need to know what are the possible alternatives for this situation. Kindly reply as soon as possible. Thanks,
Re: Need help to use storm with mysql.
Maybe batch execution? 2014-07-14 18:39 GMT+08:00 amjad khan amjadkhan987...@gmail.com: I made a storm topoogy where spout was fetching data from mysql using select query. The select query was fired after every 30 msec but because the size of the table is more than 20 GB the select query takes more than 10 sec to execute therefore this is not working. I need to know what are the possible alternatives for this situation. Kindly reply as soon as possible. Thanks,
Re: Need help to use storm with mysql.
I suggest to keep spout as light as possible just to accept request trigger and fetch data from bolt. On 14 Jul 2014 16:48, Telles Nobrega tellesnobr...@gmail.com wrote: I believe that batch matchs your usecase, but if you really want to use real time here, you could have a deamon that fetchs this informations from db and push them to a queue and have the spout read from this queue. If there is nothing to send, just sleep the spout until information is available. On Mon, Jul 14, 2014 at 8:06 AM, yuanjun Li csu...@gmail.com wrote: Maybe batch execution? 2014-07-14 18:39 GMT+08:00 amjad khan amjadkhan987...@gmail.com: I made a storm topoogy where spout was fetching data from mysql using select query. The select query was fired after every 30 msec but because the size of the table is more than 20 GB the select query takes more than 10 sec to execute therefore this is not working. I need to know what are the possible alternatives for this situation. Kindly reply as soon as possible. Thanks, -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG
Re: Does Storm support JDK7 ??
Hi Haralds, Have you build it with JDK8 and run with JDK8, or you just downloaded Storm (which is build with JDK6) and run it with JDK8? On Mon, Jul 14, 2014 at 3:24 PM, Haralds Ulmanis hara...@evilezh.net wrote: Do not know about jdk7, I'm running on jdk8 and seems fine. On 14 July 2014 13:11, Veder Lope add...@gmail.com wrote: Storm is a great project!!! We are trying to build a project, where Storm holds a crucial role in our architecture. As I see in pom.xml (in maven-compiler-plugin), source and target are set to Java 1.6. 1) Is Storm compatible with JDK7? 2) I know we can download Storm (build for JDK6) and run it using JDK7, but there are a few incompatibilities* between JDK7 and JDK6. Will these incompatibilities affect Storm or not? 3) Do you plan to move to JDK7? 4) What is the restriction that holds are back to JDK6? (we are now stuck to JDK6 compile and runtime because of Storm) 5) Can we just build Storm with JDK7 (alter both source and target in pom.xml) and then use JDK7 for runtime or not? Have you seen any errors with this road? *incompatibilities: Check this: http://www.oracle.com/technetwork/java/javase/compatibility-417013.html#incompatibilities Regards, Adrianos Dadis.
Re: Does Storm support JDK7 ??
You need only run the existing releases on JDK 7 or 8. On Jul 14, 2014 7:15 AM, Haralds Ulmanis hara...@evilezh.net wrote: Actually now I've customized a bit storm and recompiled as I needed some changes in it. But initially I just downloaded and run. On 14 July 2014 14:02, Adrianos Dadis add...@gmail.com wrote: Hi Haralds, Have you build it with JDK8 and run with JDK8, or you just downloaded Storm (which is build with JDK6) and run it with JDK8? On Mon, Jul 14, 2014 at 3:24 PM, Haralds Ulmanis hara...@evilezh.net wrote: Do not know about jdk7, I'm running on jdk8 and seems fine. On 14 July 2014 13:11, Veder Lope add...@gmail.com wrote: Storm is a great project!!! We are trying to build a project, where Storm holds a crucial role in our architecture. As I see in pom.xml (in maven-compiler-plugin), source and target are set to Java 1.6. 1) Is Storm compatible with JDK7? 2) I know we can download Storm (build for JDK6) and run it using JDK7, but there are a few incompatibilities* between JDK7 and JDK6. Will these incompatibilities affect Storm or not? 3) Do you plan to move to JDK7? 4) What is the restriction that holds are back to JDK6? (we are now stuck to JDK6 compile and runtime because of Storm) 5) Can we just build Storm with JDK7 (alter both source and target in pom.xml) and then use JDK7 for runtime or not? Have you seen any errors with this road? *incompatibilities: Check this: http://www.oracle.com/technetwork/java/javase/compatibility-417013.html#incompatibilities Regards, Adrianos Dadis.
Re: Does Storm support JDK7 ??
We are running Storm on JDK 7. On Mon, Jul 14, 2014 at 9:51 AM, Michael Rose mich...@fullcontact.com wrote: You need only run the existing releases on JDK 7 or 8. On Jul 14, 2014 7:15 AM, Haralds Ulmanis hara...@evilezh.net wrote: Actually now I've customized a bit storm and recompiled as I needed some changes in it. But initially I just downloaded and run. On 14 July 2014 14:02, Adrianos Dadis add...@gmail.com wrote: Hi Haralds, Have you build it with JDK8 and run with JDK8, or you just downloaded Storm (which is build with JDK6) and run it with JDK8? On Mon, Jul 14, 2014 at 3:24 PM, Haralds Ulmanis hara...@evilezh.net wrote: Do not know about jdk7, I'm running on jdk8 and seems fine. On 14 July 2014 13:11, Veder Lope add...@gmail.com wrote: Storm is a great project!!! We are trying to build a project, where Storm holds a crucial role in our architecture. As I see in pom.xml (in maven-compiler-plugin), source and target are set to Java 1.6. 1) Is Storm compatible with JDK7? 2) I know we can download Storm (build for JDK6) and run it using JDK7, but there are a few incompatibilities* between JDK7 and JDK6. Will these incompatibilities affect Storm or not? 3) Do you plan to move to JDK7? 4) What is the restriction that holds are back to JDK6? (we are now stuck to JDK6 compile and runtime because of Storm) 5) Can we just build Storm with JDK7 (alter both source and target in pom.xml) and then use JDK7 for runtime or not? Have you seen any errors with this road? *incompatibilities: Check this: http://www.oracle.com/technetwork/java/javase/compatibility-417013.html#incompatibilities Regards, Adrianos Dadis. -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com
Re: Does Storm support JDK7 ??
We will use latest JDK7. Thank you all :) On Mon, Jul 14, 2014 at 5:23 PM, siddharth ubale siddharth.ub...@gmail.com wrote: We are currently running storm on JDk 1.7 . On Mon, Jul 14, 2014 at 7:46 PM, Supun Kamburugamuva supu...@gmail.com wrote: We are running Storm on JDK 7. On Mon, Jul 14, 2014 at 9:51 AM, Michael Rose mich...@fullcontact.com wrote: You need only run the existing releases on JDK 7 or 8. On Jul 14, 2014 7:15 AM, Haralds Ulmanis hara...@evilezh.net wrote: Actually now I've customized a bit storm and recompiled as I needed some changes in it. But initially I just downloaded and run. On 14 July 2014 14:02, Adrianos Dadis add...@gmail.com wrote: Hi Haralds, Have you build it with JDK8 and run with JDK8, or you just downloaded Storm (which is build with JDK6) and run it with JDK8? On Mon, Jul 14, 2014 at 3:24 PM, Haralds Ulmanis hara...@evilezh.net wrote: Do not know about jdk7, I'm running on jdk8 and seems fine. On 14 July 2014 13:11, Veder Lope add...@gmail.com wrote: Storm is a great project!!! We are trying to build a project, where Storm holds a crucial role in our architecture. As I see in pom.xml (in maven-compiler-plugin), source and target are set to Java 1.6. 1) Is Storm compatible with JDK7? 2) I know we can download Storm (build for JDK6) and run it using JDK7, but there are a few incompatibilities* between JDK7 and JDK6. Will these incompatibilities affect Storm or not? 3) Do you plan to move to JDK7? 4) What is the restriction that holds are back to JDK6? (we are now stuck to JDK6 compile and runtime because of Storm) 5) Can we just build Storm with JDK7 (alter both source and target in pom.xml) and then use JDK7 for runtime or not? Have you seen any errors with this road? *incompatibilities: Check this: http://www.oracle.com/technetwork/java/javase/compatibility-417013.html#incompatibilities Regards, Adrianos Dadis. -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com
Re: Does Storm support JDK7 ??
Source and target are set to 1.6 to ensure backward compatibility for those who, for whatever reason, need to use Java 1.6. 1. Yes, Storm is compatible with JDK 7. 2. No. 3. I imagine this will happen at some point. But there’s nothing to stop you from running Storm with JDK 7. 4. No restriction. For now we do not rely on any JDK 7-specific features, so providing backward compatibility with 1.6 is a no-brainer. 5. Storm binary releases are compatible with JDK 6 and 7, both Oracle and OpenJDK. There’s no need to recompile. - Taylor On Jul 14, 2014, at 8:11 AM, Veder Lope add...@gmail.com wrote: Storm is a great project!!! We are trying to build a project, where Storm holds a crucial role in our architecture. As I see in pom.xml (in maven-compiler-plugin), source and target are set to Java 1.6. 1) Is Storm compatible with JDK7? 2) I know we can download Storm (build for JDK6) and run it using JDK7, but there are a few incompatibilities* between JDK7 and JDK6. Will these incompatibilities affect Storm or not? 3) Do you plan to move to JDK7? 4) What is the restriction that holds are back to JDK6? (we are now stuck to JDK6 compile and runtime because of Storm) 5) Can we just build Storm with JDK7 (alter both source and target in pom.xml) and then use JDK7 for runtime or not? Have you seen any errors with this road? *incompatibilities: Check this: http://www.oracle.com/technetwork/java/javase/compatibility-417013.html#incompatibilities Regards, Adrianos Dadis. signature.asc Description: Message signed with OpenPGP using GPGMail
回复: How to specify StreamID in storm-kafka spout?
Thanks for your reply, but i mean that KafkaSpout don't provide a way to set stream id, and from it's source code, i found this statement in PartitionManager class: collector.emit(tup, new KafkaMessageId(partition, offset); And this is the only emit call. Vladi Feigin vladi...@gmail.com编写： You need to pass the stream ID (you define it ) in SpoutOutputCollector emit method (first parameter) And you need also to pass stream id when you build a topology (TopologyBuilder) . For example when you call shuffleGrouping method Vladi On Mon, Jul 14, 2014 at 11:50 AM, Link Wang link.wang...@gmail.com wrote: Dear all, I'm using KafkaSpout of 0.9.2-incubating, I want my KafkaSpout to emit more than one stream with given streamID, but It seems that there's not way to do this. any help? bellow is my pom dependency of storm-kafka: dependency groupIdorg.apache.storm/groupId artifactIdstorm-kafka/artifactId version0.9.2-incubating/version /dependency Link Wang
Max Spout Pending
What is the optimal max spout pending to use in a topology ? I found this thread here: http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3cca+avhzatfg_s88lkombvommkh-rafwr6szy0i8b8tm3rfab...@mail.gmail.com%3E that didn't seem to have a follow up. Part of it says to Start with a max spout pending that is for sure too small -- one for trident, or the number of executors for storm -- and increase it until you stop seeing changes in the flow. You'll probably end up with something near 2*(throughput in recs/sec)*(end-to-end latency) (2x the Little's law capacity). Does this make sense for a Max Spout Pending value ? I expect my topology to have a throughput of around 80,000/s and I've been seeing a complete latency of around 300ms, so given this formula, I'd want 2*8*.3 = 48,000 Max Spout Pending. This seems absurdly high to me.. -- Raphael Hsieh
Re: Max Spout Pending
Max spout pending config specifies how many *batches* can be processed simultaneously by your topology. Thats why 48,000 seems absurdly high to you. Divide it between the batch size and you'll get the max spout pending config that you were expecting. 2014-07-14 19:00 GMT+02:00 Raphael Hsieh raffihs...@gmail.com: What is the optimal max spout pending to use in a topology ? I found this thread here: http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3cca+avhzatfg_s88lkombvommkh-rafwr6szy0i8b8tm3rfab...@mail.gmail.com%3E that didn't seem to have a follow up. Part of it says to Start with a max spout pending that is for sure too small -- one for trident, or the number of executors for storm -- and increase it until you stop seeing changes in the flow. You'll probably end up with something near 2*(throughput in recs/sec)*(end-to-end latency) (2x the Little's law capacity). Does this make sense for a Max Spout Pending value ? I expect my topology to have a throughput of around 80,000/s and I've been seeing a complete latency of around 300ms, so given this formula, I'd want 2*8*.3 = 48,000 Max Spout Pending. This seems absurdly high to me.. -- Raphael Hsieh -- Carlos Rodríguez Developer at ENEO Tecnología http://redborder.net/ http://lnkd.in/bgfCVF9
Re: Need help to use storm with mysql.
Can you expand on your use case? What is the query selecting on? Is the column you are querying on indexed? Do you really need to look at the entire 20 gb every 20ms? On Jul 14, 2014 6:39 AM, amjad khan amjadkhan987...@gmail.com wrote: I made a storm topoogy where spout was fetching data from mysql using select query. The select query was fired after every 30 msec but because the size of the table is more than 20 GB the select query takes more than 10 sec to execute therefore this is not working. I need to know what are the possible alternatives for this situation. Kindly reply as soon as possible. Thanks,
Re: Max Spout Pending
Is there a way to tell how many batches my topology processes per second ? Or for that matter how many tuples are processed per second ? Aside from creating a new bolt purely for that aggregation ? On Mon, Jul 14, 2014 at 2:08 PM, Carlos Rodriguez crodrig...@redborder.net wrote: Max spout pending config specifies how many *batches* can be processed simultaneously by your topology. Thats why 48,000 seems absurdly high to you. Divide it between the batch size and you'll get the max spout pending config that you were expecting. 2014-07-14 19:00 GMT+02:00 Raphael Hsieh raffihs...@gmail.com: What is the optimal max spout pending to use in a topology ? I found this thread here: http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3cca+avhzatfg_s88lkombvommkh-rafwr6szy0i8b8tm3rfab...@mail.gmail.com%3E that didn't seem to have a follow up. Part of it says to Start with a max spout pending that is for sure too small -- one for trident, or the number of executors for storm -- and increase it until you stop seeing changes in the flow. You'll probably end up with something near 2*(throughput in recs/sec)*(end-to-end latency) (2x the Little's law capacity). Does this make sense for a Max Spout Pending value ? I expect my topology to have a throughput of around 80,000/s and I've been seeing a complete latency of around 300ms, so given this formula, I'd want 2*8*.3 = 48,000 Max Spout Pending. This seems absurdly high to me.. -- Raphael Hsieh -- Carlos Rodríguez Developer at ENEO Tecnología http://redborder.net/ http://lnkd.in/bgfCVF9 -- Raphael Hsieh
Re: Measuring a topology performance
What's the recommended way to measure the avg. time of the tuple spending in the topology until its full processing? You can do this with acking enabled. In the UI, go to a spout and look for Complete Latency. -- Derek On 7/13/14, 7:03, 唐 思成 wrote: UI has metric called latency means how long a bolt take to process a tuple 在 2014年7月13日，下午5:49，Vladi Feigin vladi...@gmail.com 写道： Hi All, What's the recommended way to measure the avg. time of the tuple spending in the topology until its full processing? We use Storm version 0.8.2 and have the topologies with acks and without. Thank you, Vladi
Re: Re: Max Spout Pending
https://gist.github.com/mrflip/5958028#provisionings Max-pending (TOPOLOGY_MAX_SPOUT_PENDING) sets the number of tuple trees live in the system at any one time. maybe this is useful for you 2014-07-15 唐思成 发件人： Raphael Hsieh 发送时间： 2014-07-15 05:54:28 收件人： user 抄送： 主题： Re: Max Spout Pending Is there a way to tell how many batches my topology processes per second ? Or for that matter how many tuples are processed per second ? Aside from creating a new bolt purely for that aggregation ? On Mon, Jul 14, 2014 at 2:08 PM, Carlos Rodriguez crodrig...@redborder.net wrote: Max spout pending config specifies how many *batches* can be processed simultaneously by your topology. Thats why 48,000 seems absurdly high to you. Divide it between the batch size and you'll get the max spout pending config that you were expecting. 2014-07-14 19:00 GMT+02:00 Raphael Hsieh raffihs...@gmail.com: What is the optimal max spout pending to use in a topology ? I found this thread here: http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3cca+avhzatfg_s88lkombvommkh-rafwr6szy0i8b8tm3rfab...@mail.gmail.com%3E that didn't seem to have a follow up. Part of it says to Start with a max spout pending that is for sure too small -- one for trident, or the number of executors for storm -- and increase it until you stop seeing changes in the flow. You'll probably end up with something near 2*(throughput in recs/sec)*(end-to-end latency) (2x the Little's law capacity). Does this make sense for a Max Spout Pending value ? I expect my topology to have a throughput of around 80,000/s and I've been seeing a complete latency of around 300ms, so given this formula, I'd want 2*8*.3 = 48,000 Max Spout Pending. This seems absurdly high to me.. -- Raphael Hsieh -- Carlos Rodríguez Developer at ENEO Tecnología http://redborder.net/ http://lnkd.in/bgfCVF9 -- Raphael Hsieh
