RE: Create multiple supervisors on same node
Thanks Derek! I tried your suggestion, copied the entire storm home directory (which, in my case, is storm-0.8.2) to a new directory storm2, then in storm2 directory, I changed the conf/storm.yaml with different ports, and tried to create a new supervisor. Still, got the same supervisor ID as the 1st one (which I created from storm-0.8.2 directory). Did I do anything incorrectly? -Tao -Original Message- From: Derek Dagit [mailto:der...@yahoo-inc.com] Sent: Friday, August 22, 2014 11:01 AM To: user@storm.incubator.apache.org Subject: Re: Create multiple supervisors on same node The two supervisors are sharing the same state, and that is how they get the same randomly-generated ID. If I recall correctly, the default state directory is created in the current working directory of the process, so that is whatever directory you happen to be in when you start the supervisor. I think probably a good thing to do is copy the entire storm home directory, change the storm.yaml in the copy to be configured with different ports as you tried, and make sure to cd into the appropriate directory when you launch the supervisor. -- Derek On 8/22/14, 9:49, Yu, Tao wrote: Hi all, Anyone knows what's the requirement to generate multiple supervisors on the same node (for same topology)? I can create the 1st supervisor, then I update the supervisor.slots.ports to different ports, and tried to create the 2nd supervisor on same node, it ends up creating a new supervisor but with same supervisor ID as the 1st one, so it still only has one supervisor on that node and storm UI shows 1 supervisor as well. Any suggestion on how to create the 2nd, 3rd supervisor on the same node? Any help is appreciated! thanks, -Tao
Re: Create multiple supervisors on same node
Instead of increasing supervisors, isn't it possible to increase the works per supervisor. By default storm creates 4 workers per supervisors. I think you can override this in storm.ymal file. thanks, Amila. On Fri, Aug 22, 2014 at 8:49 AM, Yu, Tao ta...@aware.com wrote: Hi all, Anyone knows what's the requirement to generate multiple supervisors on the same node (for same topology)? I can create the 1st supervisor, then I update the supervisor.slots.ports to different ports, and tried to create the 2nd supervisor on same node, it ends up creating a new supervisor but with same supervisor ID as the 1st one, so it still only has one supervisor on that node and storm UI shows 1 supervisor as well. Any suggestion on how to create the 2nd, 3rd supervisor on the same node? Any help is appreciated! thanks, -Tao -- Amila Suriarachchi blog: http://amilachinthaka.blogspot.com/
Re: Create multiple supervisors on same node
Tao, I tried the above steps I am able to run two supervisors on the same node. Did you check the logs for supervisor under storm2. If it didn't created a local_dir/storm dir than your supervisor daemon might not be running. check for logs if there are any errors. -Harsha On Fri, Aug 22, 2014, at 09:20 AM, Yu, Tao wrote: Thanks Harsha! I tried your way, and here is what I have (major parts) in my storm.yaml: storm.local.dir: /opt/grid/tao/storm/storm-0.8.2/local_data/storm supervisor.slots.ports: - 6700 - 6701 1) I created the 1st supervisor, and I can see specified sub-folder local_data/storm/supervisor was created under opt/grid/tao/storm/storm-0.8.2. That's OK! 2) then I copied the entire storm-0.8.2 folder to a new storm2 (/opt/grid/tao/storm/storm2) 3) delete the sub-folder local_data under storm2 4) updated the storm.yaml under storm2 with below change: storm.local.dir: /opt/grid/tao/storm/storm2/local_data/storm supervisor.slots.ports: - 8700 - 8701 5) under storm2, create a new supervisor. Then the new supervisor still has the 1st supervisor's ID. And under storm2, the sub-folder local_data/storm was not created. Does storm still use the 1st storm home directory (storm/storm-0.8.2) local_data folder? Thanks, -Tao -Original Message- From: Harsha [mailto:st...@harsha.io] Sent: Friday, August 22, 2014 11:28 AM To: user@storm.incubator.apache.org Subject: Re: Create multiple supervisors on same node Tao, you need to delete the storm-local dir under your copied over storm dir ( storm2). Otherwise it will still pick up the same supervisor-id. -Harsha On Fri, Aug 22, 2014, at 08:16 AM, Yu, Tao wrote: Thanks Derek! I tried your suggestion, copied the entire storm home directory (which, in my case, is storm-0.8.2) to a new directory storm2, then in storm2 directory, I changed the conf/storm.yaml with different ports, and tried to create a new supervisor. Still, got the same supervisor ID as the 1st one (which I created from storm-0.8.2 directory). Did I do anything incorrectly? -Tao -Original Message- From: Derek Dagit [mailto:der...@yahoo-inc.com] Sent: Friday, August 22, 2014 11:01 AM To: user@storm.incubator.apache.org Subject: Re: Create multiple supervisors on same node The two supervisors are sharing the same state, and that is how they get the same randomly-generated ID. If I recall correctly, the default state directory is created in the current working directory of the process, so that is whatever directory you happen to be in when you start the supervisor. I think probably a good thing to do is copy the entire storm home directory, change the storm.yaml in the copy to be configured with different ports as you tried, and make sure to cd into the appropriate directory when you launch the supervisor. -- Derek On 8/22/14, 9:49, Yu, Tao wrote: Hi all, Anyone knows what's the requirement to generate multiple supervisors on the same node (for same topology)? I can create the 1st supervisor, then I update the supervisor.slots.ports to different ports, and tried to create the 2nd supervisor on same node, it ends up creating a new supervisor but with same supervisor ID as the 1st one, so it still only has one supervisor on that node and storm UI shows 1 supervisor as well. Any suggestion on how to create the 2nd, 3rd supervisor on the same node? Any help is appreciated! thanks, -Tao
Re: Create multiple supervisors on same node
I also tried another scenario: instead of copying the entire storm home directory, I only use one storm home, but different storm-local dir and ports, which both are specified in storm.yaml, I can still create multiple supervisors. (Of course, every time before I start a new supervisor, I have to update the storm.yaml for different storm-local dir and ports). You will have two supervisors writing to the same log. I recommend creating two distinct storm home directories unless you have a good reason to have them shared. I think the code assumes it is the only supervisor writing in storm home. -- Derek On 8/22/14, 14:08, Yu, Tao wrote: Thanks Harsha! Just cleaned zookeeper data (stop and re-start zookeeper) and tried again, now I can create multiple supervisors successfully! I also tried another scenario: instead of copying the entire storm home directory, I only use one storm home, but different storm-local dir and ports, which both are specified in storm.yaml, I can still create multiple supervisors. (Of course, every time before I start a new supervisor, I have to update the storm.yaml for different storm-local dir and ports). So my new questions are: 1) what's the best approach to create multiple supervisors on same node: a) each supervisor has it's own storm home directory;Or b) all supervisors have common storm home directory. in both approaches, supervisors have its own storm-local dir and ports. 2) when start supervisor, can we specify storm to use custom configuration (.yaml)? For example, like: $bin/storm supervisor --config conf/myConfig.yaml Seems like storm will always use conf/storm.yaml, and I do not see any document mentions about specifying custom config file. Thanks, -Tao -Original Message- From: Harsha [mailto:st...@harsha.io] Sent: Friday, August 22, 2014 12:57 PM To: user@storm.incubator.apache.org Subject: Re: Create multiple supervisors on same node Tao, I tried the above steps I am able to run two supervisors on the same node. Did you check the logs for supervisor under storm2. If it didn't created a local_dir/storm dir than your supervisor daemon might not be running. check for logs if there are any errors. -Harsha On Fri, Aug 22, 2014, at 09:20 AM, Yu, Tao wrote: Thanks Harsha! I tried your way, and here is what I have (major parts) in my storm.yaml: storm.local.dir: /opt/grid/tao/storm/storm-0.8.2/local_data/storm supervisor.slots.ports: - 6700 - 6701 1) I created the 1st supervisor, and I can see specified sub-folder local_data/storm/supervisor was created under opt/grid/tao/storm/storm-0.8.2. That's OK! 2) then I copied the entire storm-0.8.2 folder to a new storm2 (/opt/grid/tao/storm/storm2) 3) delete the sub-folder local_data under storm2 4) updated the storm.yaml under storm2 with below change: storm.local.dir: /opt/grid/tao/storm/storm2/local_data/storm supervisor.slots.ports: - 8700 - 8701 5) under storm2, create a new supervisor. Then the new supervisor still has the 1st supervisor's ID. And under storm2, the sub-folder local_data/storm was not created. Does storm still use the 1st storm home directory (storm/storm-0.8.2) local_data folder? Thanks, -Tao -Original Message- From: Harsha [mailto:st...@harsha.io] Sent: Friday, August 22, 2014 11:28 AM To: user@storm.incubator.apache.org Subject: Re: Create multiple supervisors on same node Tao, you need to delete the storm-local dir under your copied over storm dir ( storm2). Otherwise it will still pick up the same supervisor-id. -Harsha On Fri, Aug 22, 2014, at 08:16 AM, Yu, Tao wrote: Thanks Derek! I tried your suggestion, copied the entire storm home directory (which, in my case, is storm-0.8.2) to a new directory storm2, then in storm2 directory, I changed the conf/storm.yaml with different ports, and tried to create a new supervisor. Still, got the same supervisor ID as the 1st one (which I created from storm-0.8.2 directory). Did I do anything incorrectly? -Tao -Original Message- From: Derek Dagit [mailto:der...@yahoo-inc.com] Sent: Friday, August 22, 2014 11:01 AM To: user@storm.incubator.apache.org Subject: Re: Create multiple supervisors on same node The two supervisors are sharing the same state, and that is how they get the same randomly-generated ID. If I recall correctly, the default state directory is created in the current working directory of the process, so that is whatever directory you happen to be in when you start the supervisor. I think probably a good thing to do is copy the entire storm home directory, change the storm.yaml in the copy to be configured with different ports as you tried, and make sure to cd into the appropriate directory when you launch the supervisor. -- Derek On 8/22/14, 9:49, Yu, Tao wrote: Hi all, Anyone knows what's the requirement
RE: Create multiple supervisors on same node
Thanks Derek, that makes sense for me! Br, -Tao -Original Message- From: Derek Dagit [mailto:der...@yahoo-inc.com] Sent: Friday, August 22, 2014 3:34 PM To: user@storm.incubator.apache.org Subject: Re: Create multiple supervisors on same node I also tried another scenario: instead of copying the entire storm home directory, I only use one storm home, but different storm-local dir and ports, which both are specified in storm.yaml, I can still create multiple supervisors. (Of course, every time before I start a new supervisor, I have to update the storm.yaml for different storm-local dir and ports). You will have two supervisors writing to the same log. I recommend creating two distinct storm home directories unless you have a good reason to have them shared. I think the code assumes it is the only supervisor writing in storm home. -- Derek On 8/22/14, 14:08, Yu, Tao wrote: Thanks Harsha! Just cleaned zookeeper data (stop and re-start zookeeper) and tried again, now I can create multiple supervisors successfully! I also tried another scenario: instead of copying the entire storm home directory, I only use one storm home, but different storm-local dir and ports, which both are specified in storm.yaml, I can still create multiple supervisors. (Of course, every time before I start a new supervisor, I have to update the storm.yaml for different storm-local dir and ports). So my new questions are: 1) what's the best approach to create multiple supervisors on same node: a) each supervisor has it's own storm home directory;Or b) all supervisors have common storm home directory. in both approaches, supervisors have its own storm-local dir and ports. 2) when start supervisor, can we specify storm to use custom configuration (.yaml)? For example, like: $bin/storm supervisor --config conf/myConfig.yaml Seems like storm will always use conf/storm.yaml, and I do not see any document mentions about specifying custom config file. Thanks, -Tao -Original Message- From: Harsha [mailto:st...@harsha.io] Sent: Friday, August 22, 2014 12:57 PM To: user@storm.incubator.apache.org Subject: Re: Create multiple supervisors on same node Tao, I tried the above steps I am able to run two supervisors on the same node. Did you check the logs for supervisor under storm2. If it didn't created a local_dir/storm dir than your supervisor daemon might not be running. check for logs if there are any errors. -Harsha On Fri, Aug 22, 2014, at 09:20 AM, Yu, Tao wrote: Thanks Harsha! I tried your way, and here is what I have (major parts) in my storm.yaml: storm.local.dir: /opt/grid/tao/storm/storm-0.8.2/local_data/storm supervisor.slots.ports: - 6700 - 6701 1) I created the 1st supervisor, and I can see specified sub-folder local_data/storm/supervisor was created under opt/grid/tao/storm/storm-0.8.2. That's OK! 2) then I copied the entire storm-0.8.2 folder to a new storm2 (/opt/grid/tao/storm/storm2) 3) delete the sub-folder local_data under storm2 4) updated the storm.yaml under storm2 with below change: storm.local.dir: /opt/grid/tao/storm/storm2/local_data/storm supervisor.slots.ports: - 8700 - 8701 5) under storm2, create a new supervisor. Then the new supervisor still has the 1st supervisor's ID. And under storm2, the sub-folder local_data/storm was not created. Does storm still use the 1st storm home directory (storm/storm-0.8.2) local_data folder? Thanks, -Tao -Original Message- From: Harsha [mailto:st...@harsha.io] Sent: Friday, August 22, 2014 11:28 AM To: user@storm.incubator.apache.org Subject: Re: Create multiple supervisors on same node Tao, you need to delete the storm-local dir under your copied over storm dir ( storm2). Otherwise it will still pick up the same supervisor-id. -Harsha On Fri, Aug 22, 2014, at 08:16 AM, Yu, Tao wrote: Thanks Derek! I tried your suggestion, copied the entire storm home directory (which, in my case, is storm-0.8.2) to a new directory storm2, then in storm2 directory, I changed the conf/storm.yaml with different ports, and tried to create a new supervisor. Still, got the same supervisor ID as the 1st one (which I created from storm-0.8.2 directory). Did I do anything incorrectly? -Tao -Original Message- From: Derek Dagit [mailto:der...@yahoo-inc.com] Sent: Friday, August 22, 2014 11:01 AM To: user@storm.incubator.apache.org Subject: Re: Create multiple supervisors on same node The two supervisors are sharing the same state, and that is how they get the same randomly-generated ID. If I recall correctly, the default state directory is created in the current working directory of the process, so that is whatever directory you happen to be in when you start
