https://github.com/apache/storm/blob/master/conf/defaults.yaml
On Thu, Sep 25, 2014 at 12:48 PM, Gunderson, Richard-CW < richard.gunder...@bestbuy.com> wrote: > Thank you for the responses guys. > > > > I’ve done a little experimenting and discovered restarting the daemons > (nimbus, supervisor) is not such a big deal for our topologies. > > > > We already have a dedicated zookeeper server in our environment, so we had > no need to use the included zk library. It was easy to define the zk server > in storm.yaml. > > > > We dedicate one node (i.e. virtual server) to Nimbus and the Storm UI. > Nothing else runs on that server. > > > > We have three nodes dedicated to the supervisors. Nothing else runs on > those servers. > > > > They all communicate to Zookeeper without issues; The supervisors register > themselves when a new daemon starts, and nimbus finds them without issue. > > > > One issue we had were the supervisors were trying to communicate with > either other using the Netty client. When the supervisors registered in > Zookeeper, they would provide their hostname. Well, our Storm cluster is > built in AWS (amazon web services) and we spin up/down servers all the > time, so we can’t use hostnames and instead rely on IP addresses. After > some searching, a coworker found a property that you can define in > storm.yaml called storm.local.hostname. You can set that value equal to > the ip address. And the supervisor will register it’s ip address instead of > the hostname. > > > > After we made that change, we no longer got netty-client errors in the > supervisor.log file. That was a huge headache and I’m glad it is finally > resolved. > > > > It would be nice if there was a list somewhere that listed ALL possible > values allowed in the storm.yaml file. I have a feeling mine is missing > some. > > > > Richard Gunderson > > > > *From:* Nathan Leung [mailto:ncle...@gmail.com] > *Sent:* Wednesday, September 24, 2014 8:09 AM > *To:* user > *Subject:* Re: Configuration changes and storm cluster > > > > In my experience storm.yaml changes require daemon restarts, while > cluster.xml changes get picked up dynamically by running topologies. > > > > On Wed, Sep 24, 2014 at 2:44 AM, Richards Peter <hbkricha...@gmail.com> > wrote: > > Answers inline. > > Regards, > Richards Peter. > > > > On Tue, Sep 23, 2014 at 8:55 PM, Gunderson, Richard-CW < > richard.gunder...@bestbuy.com> wrote: > > Hi Storm users. > > > > I’ve recently started managing a Storm cluster. I’m very new to Storm and > have a lot to learn. I’ve been tasked to create Chef scripts to automate > two things: > > 1) Update our storm cluster if we have changes to our Storm > configuration (this will primarily be changes to properties in the > storm.yaml file) > > 2) Updates to our topologies. > > > > My question: Does storm notice if the properties file (storm.yaml, or > cluster.xml) has been updated and automatically “incorporate” the changes > while it’s running? Or must I shut down the entire cluster (*nimbus and > all supervisors*) and restart everything? I didn’t notice anywhere in the > official apache storm documentation about what to do when after updating > the configuration file. > > > > > From my experience, storm.yaml file is read only once - when storm > daemons(nimbus, supervisor, ui) are started. If you are adding only new > supervisors machines into the cluster you need not restart any of the > running services. You just have to start the supervisor on the newly added > machine. In storm.yaml file you do not specify the ip address of any of the > supervisor machines. The supervisors automatically talk to the nimbus > server configured in the storm.yaml file. Storm is stateless and all state > information is stored in zookeeper. So you need not worry about adding any > new supervisor machines. > > I normally restart all the storm daemons if there is a change in my > storm.yaml file. You should take expert opinion from others in this forum. > > But let me bring one more thing to your attention. If you add more > supervisor ports in your storm yaml file, you should also increase a > configuration parameter in zoo.cfg file of zookeeper -> maxClientCnxns. > > http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_advancedConfiguration > > > > > > Also: What happens if Nimbus has a different storm.yaml file than the > supervisors? Does a supervisor process ever read that file? Or does Nimbus > control everything? > > > > I am not sure about your question. If the nimbus machine runs only nimbus, > then that file is read only by nimbus. However if you have nimbus and > supervisor running on same machine the file will be read by both the > processes. > > > > We have recently created two new topologies and will need to start tuning > the performance of these topologies, so we’ll have a lot of need to update > our configuration a lot in the coming days. > > > > Thank you, > > > > Richard Gunderson > > > > >
