Although, implementing the Storm cluster manually would be really nice learning, I would suggest using HortonWorks distribution which comes with Storm as OOTB solution and you can configure everything from Ambari UI. We are using Storm on Amazon EC2 machine, though it is right now in beta stage. We are going to move to production in coming 2-3 months.
On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell <hartzell.step...@gmail.com > wrote: > All, > > I implemented the suggestions given by Parh and Harsha. I am now using > the default.yaml but I changed the storm.zookeeper.servers to the nimbus > machine's ip address: 54.68.149.181. I also changed the nimbus.host to > 54.68.149.181. I also opened up port 6627. Now, the UI web page gives the > following error: org.apache.thrift7.transport.TTransportException: > java.net.ConnectException: Connection refused > > You should be able to see the error it gives by going to the web page > yourself at: http://54.68.149.181:8080. I am only using this account to > test and see if I can even get storm to work, so these machines are only > for testing. Perhaps someone could tell me what the storm.yaml file should > look like for this setup? > > -Thanks, Stephne > > On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell < > hartzell.step...@gmail.com> wrote: > >> I'm getting kind of confused by the storm.yaml file. Should I be using >> the default.yaml and just modify the zookeeper and nimbus ip, or should I >> use a bran new storm.yaml? >> >> My nimbus machine has the ip address: 54.68.149.181. My zookeeper is on >> the nimbus machine. what should the storm.yaml look like on my worker and >> nimbus machine? Will the storm.yaml be the same on my worker and nimbus >> machine? I am not trying to do anything fancy, I am just trying to get a >> very basic cluster up and running. >> >> -Thanks, Stephen >> >> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell < >> hartzell.step...@gmail.com> wrote: >> >>> All Thanks so much for your help. I cannot tell you how much I >>> appreciate it. I'm going to try out your suggestions and keep banging my >>> head again the wall : D. I've spent an enormous amount of time trying to >>> get this to work. I'll let you know what happens after I try to implement >>> your suggestions. It would be really cool if someone had a tutorial that >>> detailed this part. (I'll make it myself if I ever get this to work!) It >>> seems like trying to get a two-machine cluster setup on AWS would be a VERY >>> common use-case. I've read and watched everything I can on the topic and >>> nothing got it working for me! >>> >>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt < >>> pbrahmbh...@hortonworks.com> wrote: >>> >>>> The worker connects to the thrift port and not the ui port. You need to >>>> open port 6627 or whatever is the value being set in storm.yaml using >>>> property “nimbus.thrift.port”. >>>> >>>> Based on the configuration that you have pointed so far it seems your >>>> nimbus host has nimbus,ui,supervisor working because you actually have >>>> zookeeper running locally on that host. As Harsha pointed out you need to >>>> change it to a value that is the public ip instead of loopback interface. >>>> >>>> Thanks >>>> Parth >>>> >>>> >>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote: >>>> >>>> storm.zookeeper.servers: >>>> - "127.0.0.1" >>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback >>>> interface , instead either use your public ip or 0.0.0.0*) >>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a different >>>> folder probably /home/storm, /tmp/storm will get deleted if your machine is >>>> restarted)* >>>> >>>> make sure you zookeeper is also listening in 0.0.0.0 or public ip not >>>> 127.0.0.1. >>>> >>>> "No, I cannot ping my host which has a public ip address of >>>> 54.68.149.181" >>>> you are not able to reach this ip form worker node but able to access >>>> the UI using it? >>>> -Harsha >>>> >>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote: >>>> >>>> Harsha, >>>> >>>> The storm.yaml on the host machine looks like this: >>>> >>>> storm.zookeeper.servers: >>>> - "127.0.0.1" >>>> >>>> >>>> nimbus.host: "127.0.0.1" >>>> >>>> storm.local.dir: /tmp/storm >>>> >>>> >>>> The storm.yaml on the worker machine looks like this: >>>> >>>> storm.zookeeper.servers: >>>> - "54.68.149.181" >>>> >>>> >>>> nimbus.host: "54.68.149.181" >>>> >>>> storm.local.dir: /tmp/storm >>>> >>>> No, I cannot ping my host which has a public ip address of >>>> 54.68.149.181 although I can connect to the UI web page when it is hosted. >>>> I don't know how I would go about connecting to zookeeper on the nimbus >>>> host. >>>> -Thanks, Stephen >>>> >>>> >>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote: >>>> >>>> >>>> There aren't any errors in worker machine supervisor logs. Are you >>>> using the same storm.yaml for both the machines and also are you able to >>>> ping your nimbus host or connect to zookeeper on nimbus host. >>>> -Harsha >>>> >>>> >>>> >>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote: >>>> >>>> Harsha, >>>> >>>> Thanks so much for getting back with me. I will check the logs, but I >>>> don't seem to get any error messages. I have a nimbus AWS machine with >>>> zookeeper on it and a worker AWS machine. >>>> >>>> On the nimbus machine I start the zookeeper and then I run: >>>> >>>> bin/storm nimbus & >>>> bin/storm supervisor & >>>> bin/storm ui >>>> >>>> On the worker machine I run: >>>> bin/storm supervisor >>>> >>>> When I go to the UI page, I only see 1 supervisor (the one on the >>>> nimbus machine). So apparently, the worker machine isn't "registering" with >>>> the nimbus machine. >>>> >>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote: >>>> >>>> >>>> Hi Stephen, >>>> What are the issues you are seeing. >>>> "How do worker machines "know" how to connect to nimbus? Is it in the >>>> storm configuration file" >>>> >>>> Yes. make sure you the supervisor(worker) , nimbus nodes are able to >>>> connect to your zookeeper cluster. >>>> Check your logs under storm_inst/logs/ for any errors when you try to >>>> start nimbus or supervisors. >>>> If you are installing it manually try following these steps if you are >>>> not already done. >>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/ >>>> -Harsha >>>> >>>> >>>> >>>> >>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote: >>>> >>>> All, >>>> >>>> I would greatly appreciate any help that anyone would afford. I've been >>>> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2 >>>> machines. So far, I haven't been able to get a cluster built. I can get a >>>> supervisor and nimbus to run on a single machine, but I can't figure out >>>> how to get another worker to connect to nimbus. How do worker machines >>>> "know" how to connect to nimbus? Is it in the storm configuration file? >>>> I've gone through many tutorials and the official documentation, but this >>>> point doesn't seem to be covered anywhere in sufficient detail for a new >>>> guy like me. >>>> >>>> Some of you may be tempted to point me toward storm-deploy, but I >>>> spent four days trying to get that to work until I gave up. I'm having >>>> Issue #58 on github. Following the instructions exactly and other tutorials >>>> on a bran new AWS machine fails. So I gave up on storm-deploy and decided >>>> to try and setup a cluster manually. Thanks in advance to anyone willing to >>>> offer me any inputs you can! >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or >>>> entity to which it is addressed and may contain information that is >>>> confidential, privileged and exempt from disclosure under applicable law. >>>> If the reader of this message is not the intended recipient, you are hereby >>>> notified that any printing, copying, dissemination, distribution, >>>> disclosure or forwarding of this communication is strictly prohibited. If >>>> you have received this communication in error, please contact the sender >>>> immediately and delete it from your system. Thank You. >>>> >>> >>> >> > -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
