Storm use case
We are trying to build a webservices application that can support *10k TPS*. I'm trying to do some POC's on strom, but I'm a bit concerned if using storm is the right fit here. Here is the scenario: Client will send a webservice request, which we will receive it (using apache CXF) and push it into JMS (probably kafka or RabbitMQ). From JMS storm spout will receive it and sent it to the bolt. In bolt we will be performing the validation that involves db calls, and once done we will persist the data in no-sql db. I understand the advantages of using storm, but my concern is that we are not performing some complex bolt chaining and might be using one or two bolts. I'm confused whether storm fits well in this case? P.S. we are planning to deploy webservices on the application server in cluster setup to support 10k TPS. Not sure if cluster setup is good approach, but I'll look into it later.
Decouple topology configuration from code
Hi folks, Apache Camel has a number of DSL which allow its topologies (routes wrt. Camel terminology) to be set up and configured easily. I am interested in such approach for Storm. I found java beans usage in: https://github.com/granthenke/storm-spring/ but sounds fairly limited to me. Is there any other DSL like initiative for Storm ? My second concern is storm cluster management: we'd like to have a registry of topologies and be able to register/destroy/launch/suspend/kill/update registered topologies using a REST API. Is there any tool/initiative to support that ? Thx, /DV Dominique Villard Architecte logiciel / Lead Developer Orange/OF/DTSI/DSI/DFY/SDFY tél. 04 97 46 30 03 dominique.vill...@orange.commailto:dominique.vill...@orange-ftgroup.com _ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
Re: Can I make Nimbus failsafe
Hi all, can you please guide me the Algo or any inputs how i can make nimbus highly available? I have seen the code of JStorm 9.0 where they make nimbus High available using zookeeper. Any one who is done something in it please guide me. regards Tarkeshwar On Wed, Sep 10, 2014 at 2:16 PM, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Ok we can do it together. How can we plan it? On Tue, Sep 9, 2014 at 6:09 PM, 潘臻轩 zhenxuan...@gmail.com wrote: hi, I can not direct git you source code, it mix our company code, I have do many change for community branch. I will implement it on community branch later, if you interest, do together 2014-09-09 20:05 GMT+08:00 M.Tarkeshwar Rao tarkeshwa...@gmail.com: Hi all, How we can achieve high availability of nimbus. Can you please help us to achieve this? Regards Tarkeshwar
Re: Trident transactional topology stuck re-emitting batches with Netty, but running fine with ZMQ (was Re: Topology is stuck)
Yes, it's been fixed in 'master' for some time now. Danijel On Tuesday, September 16, 2014, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Hi Danijel, Is the issue resolved in any version of the storm? Regards Tarkeshwar On Thu, Jul 17, 2014 at 6:57 PM, Danijel Schiavuzzi dani...@schiavuzzi.com javascript:_e(%7B%7D,'cvml','dani...@schiavuzzi.com'); wrote: I've filled a bug report for this under https://issues.apache.org/jira/browse/STORM-406 The issue is 100% reproducible with, it seems, any Trident topology and across multiple Storm versions with Netty transport enabled. 0MQ is working fine. You can try with TridentWordCount from storm-starter, for example. Your insight seems correct: when the killed worker re-spawns on the same slot (port), the topology stops processing. See the above JIRA for additional info. Danijel On Thu, Jul 17, 2014 at 7:20 AM, M.Tarkeshwar Rao tarkeshwa...@gmail.com javascript:_e(%7B%7D,'cvml','tarkeshwa...@gmail.com'); wrote: Thanks Danijel for helping me. On Thu, Jul 17, 2014 at 1:37 AM, Danijel Schiavuzzi dani...@schiavuzzi.com javascript:_e(%7B%7D,'cvml','dani...@schiavuzzi.com'); wrote: I see no issues with your cluster configuration. You should definitely share the (simplified if possible) topology code and the steps to reproduce the blockage, better yet you should file a JIRA task on Apache's JIRA web -- be sure to include your Trident internals modifications. Unfortunately, seems I'm having the same issues now with Storm 0.9.2 too, so I might get back here with some updates soon. It's not so fast and easily reproducible as it was under 0.9.1, but the bug seems nonetheless still present. I'll reduce the number of Storm slots and topology workers as per your insights, hopefully this might make it easier to reproduce the bug with a simplified Trident topology. On Tuesday, July 15, 2014, M.Tarkeshwar Rao tarkeshwa...@gmail.com javascript:_e(%7B%7D,'cvml','tarkeshwa...@gmail.com'); wrote: Hi Denijel, We have done few changes in the the trident core framework code as per our need which is working fine with zeromq. I am sharing configuration which we are using. Can you please suggest our config is fine or not? Code part is so large so we are writing some sample topology and trying to reproduce the issue, which we will share with you. What are the steps to reproduce the issue: - 1. we deployed our topology with one linux machine, two workers and one acker with batch size 2. 2. both the worker are up and start the processing. 3. after few seconds i killed one of the worker kill -9. 4. when the killed worker spawned on the same port it is getting hanged. 5. only retries going on. 6. when the killed worker spawned on the another port everything working fine. machine conf: -- [root@sb6270x1637-2 conf]# uname -a Linux bl460cx2378 2.6.32-431.5.1.el6.x86_64 #1 SMP Fri Jan 10 14:46:43 EST 2014 x86_64 x86_64 x86_64 GNU/Linux *storm.yaml* which we are using to launch nimbus, supervisor and ui ## These MUST be filled in for a storm configuration storm.zookeeper.servers: - 10.61.244.86 storm.zookeeper.port: 2000 supervisor.slots.ports: - 6788 - 6789 - 6800 - 6801 - 6802 - 6803 nimbus.host: 10.61.244.86 storm.messaging.transport: backtype.storm.messaging.netty.Context storm.messaging.netty.server_worker_threads: 10 storm.messaging.netty.client_worker_threads: 10 storm.messaging.netty.buffer_size: 5242880 storm.messaging.netty.max_retries: 100 storm.messaging.netty.max_wait_ms: 1000 storm.messaging.netty.min_wait_ms: 100 storm.local.dir: /root/home_98/home/enavgoy/storm-local storm.scheduler: com.ericsson.storm.scheduler.TopologyScheduler topology.acker.executors: 1 topology.message.timeout.secs: 30 supervisor.scheduler.meta: name: supervisor1 worker.childopts: -Xmx2048m mm.hdfs.ipaddress: 10.61.244.7 mm.hdfs.port: 9000 topology.batch.size: 2 topology.batch.timeout: 1 topology.workers: 2 topology.debug: true Regards Tarkeshwar On Mon, Jul 7, 2014 at 1:22 PM, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: Hi Tarkeshwar, Could you provide a code sample of your topology? Do you have any special configs enabled? Thanks, Danijel On Mon, Jul 7, 2014 at 9:01 AM, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Hi Danijel, We are able to reproduce this issue with 0.9.2 as well. We have two worker setup to run the trident topology. When we kill one of the worker and again when that killed worker spawn on same port(same slot) then that worker not able to communicate with 2nd worker. only transaction attempts are increasing continuously. But if the killed worker spawn on new slot(new communication port) then it working fine. Same behavior as in storm 9.0.1. Please update me if you
Re: Can I make Nimbus failsafe
I am working on this right now , can’t promise the dates because I have some other tasks as well. The general approach is already described in the JIRA. You basically need 3 things: Leader election between all nimbus hosts. (zookeeper is what I am going with), checks on nimbus background threads and state changing APIS to ensure only leader is changing states. Some way to distribute the local nimbus data directory to all the other nimbus and keep it consistent. Simplest way is to not have a local directory but instead use something like HDFS, if you can take that kind of dependency. Nimbus discovery, could be done either by adding/modifying current rest/thrift APIS or by allowing supervisors or clients to read from zookeeper. I have some code on leader election part with preliminary testing and unit-test here. This is by no means ready for anyone’s use but if you want to take a look to get an idea you can take a look. Thanks Parth On Sep 16, 2014, at 6:04 AM, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: something -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Storm 0.9.2-incubating - num workers and num executors switched?
We recently upgraded to Storm 0.9.2-incubating, and found that on the UI, Num workers and Num executors switched. Example: In older version (0.9.0.1): [cid:image001.png@01CFD1C4.B986FB20] In new version (0.9.2-incubating): [cid:image002.png@01CFD1C4.B986FB20] Is this a UI bug? Or did something change in Storm core functionality? Thanks, Jing
Re: Storm 0.9.2-incubating - num workers and num executors switched?
Hi Jing, Its the UI bug fixed in the trunk. [1]https://issues.apache.org/jira/browse/STORM-369 -Harsha On Tue, Sep 16, 2014, at 12:45 PM, Tao, Jing wrote: We recently upgraded to Storm 0.9.2-incubating, and found that on the UI, Num workers and Num executors switched. Example: In older version (0.9.0.1): cid:image001.png@01CFD1C4.B986FB20 In new version (0.9.2-incubating): cid:image002.png@01CFD1C4.B986FB20 Is this a UI bug? Or did something change in Storm core functionality? Thanks, Jing Email had 2 attachments: * image001.png 13k (image/png) * image002.png 14k (image/png) References 1. https://issues.apache.org/jira/browse/STORM-369
Trying to run test Storm App on Windows but getting problems with POM file
[INFO] Scanning for projects... [INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1.193 s [INFO] Finished at: 2014-09-16T22:39:58+02:00 [INFO] Final Memory: 8M/113M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException = I have tried the solution provided by http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqR7HeZ=k2cdrtbq_ntw52yppokaska_hzrjgf+qrh2...@mail.gmail.com%3E by addind the given rows and by changing the version to 0.9.3 (because in the solution email is 0.9.1). Thanks in advance!
muliple-nodes kafka cluster
Hi, All I have been using kafka cluster in single server with three brokers, but I am thinking to build a larger kafka cluster, say 4 nodes (server), and 3 brokers in each node, so totally 12 brokers, would that be better than single node cluster? Or single node will be fair enough, since web api may push million rows into kafka cluster every day, I am kinda worry if the cluster is capable to take such much data without losing data. If I am using multiple-nodes, the tradeoff is the connection time among different nodes? thanks Alec
Re: muliple-nodes kafka cluster
Hi Alec, Single node kafka cluster not recommended apart from using it for development. I highly recommend using multinode cluster and create a partitioned topic with replication. This not only makes it optimal to take in more data at faster rates also allows your cluster running if there is a node failure as the topic is replicated there wouldn't be huge data loss. If I am using multiple-nodes, the tradeoff is the connection time among different nodes? kafka producer api sends a message to broker either round-robin or based on partition function. please go through the kafka docs here [1]http://kafka.apache.org/documentation.html for simple consumer and also how the replication works among multiple nodes. -Harsha On Tue, Sep 16, 2014, at 02:06 PM, Sa Li wrote: Hi, All I have been using kafka cluster in single server with three brokers, but I am thinking to build a larger kafka cluster, say 4 nodes (server), and 3 brokers in each node, so totally 12 brokers, would that be better than single node cluster? Or single node will be fair enough, since web api may push million rows into kafka cluster every day, I am kinda worry if the cluster is capable to take such much data without losing data. If I am using multiple-nodes, the tradeoff is the connection time among different nodes? thanks Alec References 1. http://kafka.apache.org/documentation.html
Re: muliple-nodes kafka cluster
[INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] Reactor Summary: [INFO] [INFO] Storm .. SUCCESS [ 2.816 s] [INFO] maven-shade-clojure-transformer SUCCESS [ 30.954 s] [INFO] Storm Core . SUCCESS [01:34 min] [INFO] storm-starter .. FAILURE [ 0.128 s] [INFO] storm-kafka SKIPPED [INFO] storm-hdfs . SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 02:08 min [INFO] Finished at: 2014-09-16T23:51:57+02:00 [INFO] Final Memory: 39M/411M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :storm-starter On Tue, Sep 16, 2014 at 11:46 PM, Harsha st...@harsha.io wrote: Hi Alec, Single node kafka cluster not recommended apart from using it for development. I highly recommend using multinode cluster and create a partitioned topic with replication. This not only makes it optimal to take in more data at faster rates also allows your cluster running if there is a node failure as the topic is replicated there wouldn't be huge data loss. If I am using multiple-nodes, the tradeoff is the connection time among different nodes? kafka producer api sends a message to broker either round-robin or based on partition function. please go through the kafka docs here http://kafka.apache.org/documentation.html for simple consumer and also how the replication works among multiple nodes. -Harsha On Tue, Sep 16, 2014, at 02:06 PM, Sa Li wrote: Hi, All I have been using kafka cluster in single server with three brokers, but I am thinking to build a larger kafka cluster, say 4 nodes (server), and 3 brokers in each node, so totally 12 brokers, would that be better than single node cluster? Or single node will be fair enough, since web api may push million rows into kafka cluster every day, I am kinda worry if the cluster is capable to take such much data without losing data. If I am using multiple-nodes, the tradeoff is the connection time among different nodes? thanks Alec
Re: Trying to run test Storm App on Windows but getting problems with POM file
I think you'll need to build and install in your local maven repo, i dont think 0.9.3 is in maven central. On Tue, Sep 16, 2014 at 4:47 PM, Gezim Musliaj gmusl...@gmail.com wrote: [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1.193 s [INFO] Finished at: 2014-09-16T22:39:58+02:00 [INFO] Final Memory: 8M/113M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException = I have tried the solution provided by http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqR7HeZ=k2cdrtbq_ntw52yppokaska_hzrjgf+qrh2...@mail.gmail.com%3E by addind the given rows and by changing the version to 0.9.3 (because in the solution email is 0.9.1). Thanks in advance!
Re: Trying to run test Storm App on Windows but getting problems with POM file
I have been following these instructions: If you are using the latest development version of Storm, e.g. by having cloned the Storm git repository, then you must first perform a local build of Storm itself. Otherwise you will run into Maven errors such as Could not resolve dependencies for project org.apache.storm:storm-starter:storm-version-SNAPSHOT. # Must be run from the top-level directory of the Storm code repository $ mvn clean install -DskipTests=true This command will build Storm locally and install its jar files to your user's $HOME/.m2/repository/. When you run the Maven command to build and run storm-starter (see below), Maven will then be able to find the corresponding version of Storm in this local Maven repository at $HOME/.m2/repository. From https://github.com/apache/incubator-storm/tree/master/examples/storm-starter On Wed, Sep 17, 2014 at 12:02 AM, Nick Beenham nick.been...@gmail.com wrote: I think you'll need to build and install in your local maven repo, i dont think 0.9.3 is in maven central. On Tue, Sep 16, 2014 at 4:47 PM, Gezim Musliaj gmusl...@gmail.com wrote: [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1.193 s [INFO] Finished at: 2014-09-16T22:39:58+02:00 [INFO] Final Memory: 8M/113M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException = I have tried the solution provided by http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqR7HeZ=k2cdrtbq_ntw52yppokaska_hzrjgf+qrh2...@mail.gmail.com%3E by addind the given rows and by changing the version to 0.9.3 (because in the solution email is 0.9.1). Thanks in advance!
Re: Decouple topology configuration from code
Awhile ago I had written a camel adapter for storm so that spout inputs could come from camel. Not sure how useful it would be for you but its located here: https://github.com/calrissian/storm-recipes/blob/master/camel/src/main/java/org/calrissian/recipes/camel/spout/CamelConsumerSpout.java Hi folks, Apache Camel has a number of DSL which allow its topologies (routes wrt. Camel terminology) to be set up and configured easily. I am interested in such approach for Storm. I found java beans usage in: https://github.com/granthenke/storm-spring/ but sounds fairly limited to me. Is there any other DSL like initiative for Storm ? My second concern is storm cluster management: we’d like to have a registry of topologies and be able to register/destroy/launch/suspend/kill/update registered topologies using a REST API. Is there any tool/initiative to support that ? Thx, /DV *Dominique Villard* *Architecte logiciel / Lead Developer* Orange/OF/DTSI/DSI/DFY/SDFY *tél. 04 97 46 30 03* dominique.vill...@orange.com dominique.vill...@orange-ftgroup.com _ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
Re: Decouple topology configuration from code
Also, Trident is a DSL for rapidly producing useful analytics in Storm and I've been working on a DSL that makes streams processing for complex event processing possible. That one is located here: https://github.com/calrissian/flowmix On Sep 16, 2014 4:29 AM, dominique.vill...@orange.com wrote: Hi folks, Apache Camel has a number of DSL which allow its topologies (routes wrt. Camel terminology) to be set up and configured easily. I am interested in such approach for Storm. I found java beans usage in: https://github.com/granthenke/storm-spring/ but sounds fairly limited to me. Is there any other DSL like initiative for Storm ? My second concern is storm cluster management: we’d like to have a registry of topologies and be able to register/destroy/launch/suspend/kill/update registered topologies using a REST API. Is there any tool/initiative to support that ? Thx, /DV *Dominique Villard* *Architecte logiciel / Lead Developer* Orange/OF/DTSI/DSI/DFY/SDFY *tél. 04 97 46 30 03* dominique.vill...@orange.com dominique.vill...@orange-ftgroup.com _ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
Re: Trying to run test Storm App on Windows but getting problems with POM file
Hi , Did you changed storm/pom.xml groupIdorg.apache.storm/groupId artifactIdstorm/artifactId version0.9.3-incubating-SNAPSHOT/version and also are your running mvn install from top-level dir not from storm-starter. -Harsha On Tue, Sep 16, 2014, at 03:12 PM, Gezim Musliaj wrote: I have been following these instructions: If you are using the latest development version of Storm, e.g. by having cloned the Storm git repository, then you must first perform a local build of Storm itself. Otherwise you will run into Maven errors such as Could not resolve dependencies for project org.apache.storm:storm-starter:storm-version-SNAPSHOT . # Must be run from the top-level directory of the Storm code repository $ mvn clean install -DskipTests=true This command will build Storm locally and install its jar files to your user's $HOME/.m2/repository/. When you run the Maven command to build and run storm-starter (see below), Maven will then be able to find the corresponding version of Storm in this local Maven repository at $HOME/.m2/repository. From [1]https://github.com/apache/incubator-storm/tree/master/e xamples/storm-starter On Wed, Sep 17, 2014 at 12:02 AM, Nick Beenham [2]nick.been...@gmail.com wrote: I think you'll need to build and install in your local maven repo, i dont think 0.9.3 is in maven central. On Tue, Sep 16, 2014 at 4:47 PM, Gezim Musliaj [3]gmusl...@gmail.com wrote: [INFO] Scanning for projects... [INFO] [INFO] --- - [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] --- - [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] --- - [INFO] BUILD FAILURE [INFO] --- - [INFO] Total time: 1.193 s [INFO] Finished at: 2014-09-16T22:39:58+02:00 [INFO] Final Memory: 8M/113M [INFO] --- - [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in [4]http://rep [5]o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] [6]http://cwiki.apache.org/confluence/display/MAVEN/Dependen cyReso lutionException = I have tried the solution provided by [7]http://mail-archives.apache.org/mod_mbox/storm-user/201404 .mbox/%3CCALFqTqR7HeZ=k2CdrTbq_NTW52YpPOkAsKa_HZrJGF+QRH2pDg@ma il.gmail.com%3E by addind the given rows and by changing the version to 0.9.3 (because in the solution email is 0.9.1). Thanks in advance! References 1. https://github.com/apache/incubator-storm/tree/master/examples/storm-starter 2. mailto:nick.been...@gmail.com 3. mailto:gmusl...@gmail.com 4. http://rep/ 5. http://o1.maven.org/maven2/ 6. http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso 7. http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqR7HeZ=k2cdrtbq_ntw52yppokaska_hzrjgf+qrh2...@mail.gmail.com%3E
Re: Trying to run test Storm App on Windows but getting problems with POM file
Yes, Im running it from top level dir (incubator-storm) with the following command: mvn clean install -DskipTests=true and still cant get it to work properly, I have tried to build it from the storm-starter and still the same, keeps telling me there is an error with dependencies. On Wed, Sep 17, 2014 at 1:45 AM, Harsha st...@harsha.io wrote: Hi , Did you changed storm/pom.xml groupIdorg.apache.storm/groupId artifactIdstorm/artifactId version0.9.3-incubating-SNAPSHOT/version and also are your running mvn install from top-level dir not from storm-starter. -Harsha On Tue, Sep 16, 2014, at 03:12 PM, Gezim Musliaj wrote: I have been following these instructions: If you are using the latest development version of Storm, e.g. by having cloned the Storm git repository, then you must first perform a local build of Storm itself. Otherwise you will run into Maven errors such as Could not resolve dependencies for project org.apache.storm:storm-starter:storm-version-SNAPSHOT. # Must be run from the top-level directory of the Storm code repository $ mvn clean install -DskipTests=true This command will build Storm locally and install its jar files to your user's $HOME/.m2/repository/. When you run the Maven command to build and run storm-starter (see below), Maven will then be able to find the corresponding version of Storm in this local Maven repository at $HOME/.m2/repository. From https://github.com/apache/incubator-storm/tree/master/examples/storm-starter On Wed, Sep 17, 2014 at 12:02 AM, Nick Beenham nick.been...@gmail.com wrote: I think you'll need to build and install in your local maven repo, i dont think 0.9.3 is in maven central. On Tue, Sep 16, 2014 at 4:47 PM, Gezim Musliaj gmusl...@gmail.com wrote: [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1.193 s [INFO] Finished at: 2014-09-16T22:39:58+02:00 [INFO] Final Memory: 8M/113M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException = I have tried the solution provided by http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqR7HeZ=k2cdrtbq_ntw52yppokaska_hzrjgf+qrh2...@mail.gmail.com%3E by addind the given rows and by changing the version to 0.9.3 (because in the solution email is 0.9.1). Thanks in advance!
Is anyone using Scala/ ScalaStorm to write Storm Bolts?
have you written Storm Bolts in Scala and used in production? Did you use https://github.com/velvia/ScalaStorm How was the experience?
Re: Trying to run test Storm App on Windows but getting problems with POM file
Finally I solved the problem, I had tried building storm-starter before the main storm project. I just removed the previous folder with all of its files, extracted a new one from the incubator-storm.rar and did the build succesfully. But when I make the eclipse project files via the command mvn eclipse:eclipse inside the storm-starter folder and then I open it on Eclipse IDE Indigo I get errors like Class missing, Fix Project Setup etc. I figured it out that in the build path were a bunch of M2_HOME .jar files but I dont know if the problem has to do with the ordering or anything else. On Wed, Sep 17, 2014 at 1:57 AM, Gezim Musliaj gmusl...@gmail.com wrote: Yes, Im running it from top level dir (incubator-storm) with the following command: mvn clean install -DskipTests=true and still cant get it to work properly, I have tried to build it from the storm-starter and still the same, keeps telling me there is an error with dependencies. On Wed, Sep 17, 2014 at 1:45 AM, Harsha st...@harsha.io wrote: Hi , Did you changed storm/pom.xml groupIdorg.apache.storm/groupId artifactIdstorm/artifactId version0.9.3-incubating-SNAPSHOT/version and also are your running mvn install from top-level dir not from storm-starter. -Harsha On Tue, Sep 16, 2014, at 03:12 PM, Gezim Musliaj wrote: I have been following these instructions: If you are using the latest development version of Storm, e.g. by having cloned the Storm git repository, then you must first perform a local build of Storm itself. Otherwise you will run into Maven errors such as Could not resolve dependencies for project org.apache.storm:storm-starter:storm-version-SNAPSHOT. # Must be run from the top-level directory of the Storm code repository $ mvn clean install -DskipTests=true This command will build Storm locally and install its jar files to your user's $HOME/.m2/repository/. When you run the Maven command to build and run storm-starter (see below), Maven will then be able to find the corresponding version of Storm in this local Maven repository at $HOME/.m2/repository. From https://github.com/apache/incubator-storm/tree/master/examples/storm-starter On Wed, Sep 17, 2014 at 12:02 AM, Nick Beenham nick.been...@gmail.com wrote: I think you'll need to build and install in your local maven repo, i dont think 0.9.3 is in maven central. On Tue, Sep 16, 2014 at 4:47 PM, Gezim Musliaj gmusl...@gmail.com wrote: [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building storm-starter 0.9.3-incubating-SNAPSHOT [INFO] [WARNING] The POM for org.apache.storm:storm-core:jar:0.9.3-incubating is missin g, no dependency information available [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1.193 s [INFO] Finished at: 2014-09-16T22:39:58+02:00 [INFO] Final Memory: 8M/113M [INFO] [ERROR] Failed to execute goal on project storm-starter: Could not resolve depen dencies for project org.apache.storm:storm-starter:jar:0.9.3-incubating-SNAPSHOT : Failure to find org.apache.storm:storm-core:jar:0.9.3-incubating in http://rep o1.maven.org/maven2/ was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forc ed - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException = I have tried the solution provided by http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3CCALFqTqR7HeZ=k2cdrtbq_ntw52yppokaska_hzrjgf+qrh2...@mail.gmail.com%3E by addind the given rows and by changing the version to 0.9.3 (because in the solution email is 0.9.1). Thanks in advance!
Re: Trident transactional topology stuck re-emitting batches with Netty, but running fine with ZMQ (was Re: Topology is stuck)
Thanks for reply.can I pulll this fix or can I download it? On 17 Sep 2014 09:58, tarkeshwa...@gmail.com wrote: In which version it is available. On 16 Sep 2014 19:01, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: Yes, it's been fixed in 'master' for some time now. Danijel On Tuesday, September 16, 2014, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Hi Danijel, Is the issue resolved in any version of the storm? Regards Tarkeshwar On Thu, Jul 17, 2014 at 6:57 PM, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: I've filled a bug report for this under https://issues.apache.org/jira/browse/STORM-406 The issue is 100% reproducible with, it seems, any Trident topology and across multiple Storm versions with Netty transport enabled. 0MQ is working fine. You can try with TridentWordCount from storm-starter, for example. Your insight seems correct: when the killed worker re-spawns on the same slot (port), the topology stops processing. See the above JIRA for additional info. Danijel On Thu, Jul 17, 2014 at 7:20 AM, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Thanks Danijel for helping me. On Thu, Jul 17, 2014 at 1:37 AM, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: I see no issues with your cluster configuration. You should definitely share the (simplified if possible) topology code and the steps to reproduce the blockage, better yet you should file a JIRA task on Apache's JIRA web -- be sure to include your Trident internals modifications. Unfortunately, seems I'm having the same issues now with Storm 0.9.2 too, so I might get back here with some updates soon. It's not so fast and easily reproducible as it was under 0.9.1, but the bug seems nonetheless still present. I'll reduce the number of Storm slots and topology workers as per your insights, hopefully this might make it easier to reproduce the bug with a simplified Trident topology. On Tuesday, July 15, 2014, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Hi Denijel, We have done few changes in the the trident core framework code as per our need which is working fine with zeromq. I am sharing configuration which we are using. Can you please suggest our config is fine or not? Code part is so large so we are writing some sample topology and trying to reproduce the issue, which we will share with you. What are the steps to reproduce the issue: - 1. we deployed our topology with one linux machine, two workers and one acker with batch size 2. 2. both the worker are up and start the processing. 3. after few seconds i killed one of the worker kill -9. 4. when the killed worker spawned on the same port it is getting hanged. 5. only retries going on. 6. when the killed worker spawned on the another port everything working fine. machine conf: -- [root@sb6270x1637-2 conf]# uname -a Linux bl460cx2378 2.6.32-431.5.1.el6.x86_64 #1 SMP Fri Jan 10 14:46:43 EST 2014 x86_64 x86_64 x86_64 GNU/Linux *storm.yaml* which we are using to launch nimbus, supervisor and ui ## These MUST be filled in for a storm configuration storm.zookeeper.servers: - 10.61.244.86 storm.zookeeper.port: 2000 supervisor.slots.ports: - 6788 - 6789 - 6800 - 6801 - 6802 - 6803 nimbus.host: 10.61.244.86 storm.messaging.transport: backtype.storm.messaging.netty.Context storm.messaging.netty.server_worker_threads: 10 storm.messaging.netty.client_worker_threads: 10 storm.messaging.netty.buffer_size: 5242880 storm.messaging.netty.max_retries: 100 storm.messaging.netty.max_wait_ms: 1000 storm.messaging.netty.min_wait_ms: 100 storm.local.dir: /root/home_98/home/enavgoy/storm-local storm.scheduler: com.ericsson.storm.scheduler.TopologyScheduler topology.acker.executors: 1 topology.message.timeout.secs: 30 supervisor.scheduler.meta: name: supervisor1 worker.childopts: -Xmx2048m mm.hdfs.ipaddress: 10.61.244.7 mm.hdfs.port: 9000 topology.batch.size: 2 topology.batch.timeout: 1 topology.workers: 2 topology.debug: true Regards Tarkeshwar On Mon, Jul 7, 2014 at 1:22 PM, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: Hi Tarkeshwar, Could you provide a code sample of your topology? Do you have any special configs enabled? Thanks, Danijel On Mon, Jul 7, 2014 at 9:01 AM, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Hi Danijel, We are able to reproduce this issue with 0.9.2 as well. We have two worker setup to run the trident topology. When we kill one of the worker and again when that killed worker spawn on same port(same slot) then that worker not able to communicate with 2nd worker. only transaction attempts are increasing continuously. But if the killed worker spawn on new slot(new communication port) then it working fine. Same behavior as in storm 9.0.1. Please update me if you
Re: Trident transactional topology stuck re-emitting batches with Netty, but running fine with ZMQ (was Re: Topology is stuck)
In which version it is available. On 16 Sep 2014 19:01, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: Yes, it's been fixed in 'master' for some time now. Danijel On Tuesday, September 16, 2014, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Hi Danijel, Is the issue resolved in any version of the storm? Regards Tarkeshwar On Thu, Jul 17, 2014 at 6:57 PM, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: I've filled a bug report for this under https://issues.apache.org/jira/browse/STORM-406 The issue is 100% reproducible with, it seems, any Trident topology and across multiple Storm versions with Netty transport enabled. 0MQ is working fine. You can try with TridentWordCount from storm-starter, for example. Your insight seems correct: when the killed worker re-spawns on the same slot (port), the topology stops processing. See the above JIRA for additional info. Danijel On Thu, Jul 17, 2014 at 7:20 AM, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Thanks Danijel for helping me. On Thu, Jul 17, 2014 at 1:37 AM, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: I see no issues with your cluster configuration. You should definitely share the (simplified if possible) topology code and the steps to reproduce the blockage, better yet you should file a JIRA task on Apache's JIRA web -- be sure to include your Trident internals modifications. Unfortunately, seems I'm having the same issues now with Storm 0.9.2 too, so I might get back here with some updates soon. It's not so fast and easily reproducible as it was under 0.9.1, but the bug seems nonetheless still present. I'll reduce the number of Storm slots and topology workers as per your insights, hopefully this might make it easier to reproduce the bug with a simplified Trident topology. On Tuesday, July 15, 2014, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Hi Denijel, We have done few changes in the the trident core framework code as per our need which is working fine with zeromq. I am sharing configuration which we are using. Can you please suggest our config is fine or not? Code part is so large so we are writing some sample topology and trying to reproduce the issue, which we will share with you. What are the steps to reproduce the issue: - 1. we deployed our topology with one linux machine, two workers and one acker with batch size 2. 2. both the worker are up and start the processing. 3. after few seconds i killed one of the worker kill -9. 4. when the killed worker spawned on the same port it is getting hanged. 5. only retries going on. 6. when the killed worker spawned on the another port everything working fine. machine conf: -- [root@sb6270x1637-2 conf]# uname -a Linux bl460cx2378 2.6.32-431.5.1.el6.x86_64 #1 SMP Fri Jan 10 14:46:43 EST 2014 x86_64 x86_64 x86_64 GNU/Linux *storm.yaml* which we are using to launch nimbus, supervisor and ui ## These MUST be filled in for a storm configuration storm.zookeeper.servers: - 10.61.244.86 storm.zookeeper.port: 2000 supervisor.slots.ports: - 6788 - 6789 - 6800 - 6801 - 6802 - 6803 nimbus.host: 10.61.244.86 storm.messaging.transport: backtype.storm.messaging.netty.Context storm.messaging.netty.server_worker_threads: 10 storm.messaging.netty.client_worker_threads: 10 storm.messaging.netty.buffer_size: 5242880 storm.messaging.netty.max_retries: 100 storm.messaging.netty.max_wait_ms: 1000 storm.messaging.netty.min_wait_ms: 100 storm.local.dir: /root/home_98/home/enavgoy/storm-local storm.scheduler: com.ericsson.storm.scheduler.TopologyScheduler topology.acker.executors: 1 topology.message.timeout.secs: 30 supervisor.scheduler.meta: name: supervisor1 worker.childopts: -Xmx2048m mm.hdfs.ipaddress: 10.61.244.7 mm.hdfs.port: 9000 topology.batch.size: 2 topology.batch.timeout: 1 topology.workers: 2 topology.debug: true Regards Tarkeshwar On Mon, Jul 7, 2014 at 1:22 PM, Danijel Schiavuzzi dani...@schiavuzzi.com wrote: Hi Tarkeshwar, Could you provide a code sample of your topology? Do you have any special configs enabled? Thanks, Danijel On Mon, Jul 7, 2014 at 9:01 AM, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: Hi Danijel, We are able to reproduce this issue with 0.9.2 as well. We have two worker setup to run the trident topology. When we kill one of the worker and again when that killed worker spawn on same port(same slot) then that worker not able to communicate with 2nd worker. only transaction attempts are increasing continuously. But if the killed worker spawn on new slot(new communication port) then it working fine. Same behavior as in storm 9.0.1. Please update me if you get any new development. Regards Tarkeshwar On Thu, Jul 3, 2014 at 7:06 PM, Danijel Schiavuzzi
