Hi kishore, Thank you for preservance and helpful support.
Do you have any direction how can we implement zookeeper clustering programmatically in my aforementioned suggestion? As far as I know we need to use quorum. Do you know any example that can help us. Thanks, Erwin On Jun 15, 2013 3:46 AM, "kishore g" <[email protected]> wrote: > Hi Erwin, > > Yes thats possible. Good idea. > > thanks, > Kishore G > > > On Fri, Jun 14, 2013 at 3:42 PM, Erwin Karbasi <[email protected]> wrote: > >> Kishore, >> >> Thank you a lot for detailed explanation. >> >> I thought to define in configuration of each participant whether it will >> act as controller or Zookeeper as well. All participants will know those >> controllers and zookeepers that defined. >> For example I will define in 5 participants that they also should act as >> controller and zookeepers as well on the same process of the participants. >> Is it possible? >> >> Cheers, >> Erwin >> On Jun 14, 2013 6:21 PM, "kishore g" <[email protected]> wrote: >> >>> Hi Erwin, >>> >>> Here are the options: >>> >>> 1. participant controller and zookeeper on one process on all 50 boxes: >>> This is not possible, you need only 3 or 5 zookeepers running and all >>> participant/controllers need to know the machinename: port of the zookeeper >>> boxes upfront. >>> >>> 2. separate zookeeper but participant and controller in one process: >>> This is possible, controller can be started as part of all participants but >>> only one controller will be active. This is fine until you have like 100 >>> nodes, after which its better to have a separate controller. >>> >>> 3. zookeeper and controller on same box but separate process, >>> participants in different: This is possible, note that once you this >>> configuration all additional clusters you deploy can share the same >>> zookeeper/controller. So if you plan to have many clusters this is a better >>> option and much easier to manage the clusters. You can use standard >>> commands to start zookeeper and helix controller. >>> >>> 4. zookeeper and controller on same box and in one process: This also >>> works and the advantages of 3 holds good here as well. You will have to >>> start >>> both of them programmatically. Any time you restart a process, both >>> zookeeper and controller will be upgraded. >>> >>> >>> My preference is 3) 4) 2), you cannot do 1). >>> >>> I agree with you that it requires additional monitoring but actual >>> experience suggests that co-locating zookeeper with participants causes >>> un-wanted problems. For example, the participants are heavily loaded and >>> your application might have large GC pauses/consume al cpu/memory on that >>> box. This will adversely impact the functionality of zookeeper. >>> >>> Thanks, >>> Kishore G >>> >>> >>> >>> >>> >>> On Fri, Jun 14, 2013 at 5:44 AM, Erwin Karbasi <[email protected]>wrote: >>> >>>> My question is, whether is it possible to collocate participant, >>>> controller and zookeeper on one process, is there any limitations in term >>>> of Helix or Zookeeper? >>>> Any direction regarding above approach would be appreciated. >>>> On Jun 14, 2013 3:23 PM, "Ming Fang" <[email protected]> wrote: >>>> >>>>> That's not possible at this time. >>>>> ZAC currently must run as a standalone process and only allow one >>>>> instance. >>>>> >>>>> Sent from my iPad >>>>> >>>>> On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <[email protected]> wrote: >>>>> >>>>> If I have 40 app server instances (JVMs) could I run participants on >>>>> all of them, controller and zookeeper just on 5 of then, but on the same >>>>> process of the 5 participants? >>>>> On Jun 14, 2013 2:29 PM, "Ming Fang" <[email protected]> wrote: >>>>> >>>>>> ZAC currently runs Zookeeper in standalone mode. >>>>>> Therefore you can only run one instance. >>>>>> If you also run the participant in it then that limits you to one >>>>>> instance of the participant also. >>>>>> >>>>>> >>>>>> Sent from my iPad >>>>>> >>>>>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <[email protected]> >>>>>> wrote: >>>>>> >>>>>> Hi Ming, >>>>>> >>>>>> Thank you for fast turnaround response. >>>>>> >>>>>> Could I run my app that uses Helix (participant), controller and >>>>>> zookeeper on the same process? >>>>>> >>>>>> Cheers, >>>>>> Erwin >>>>>> On Jun 14, 2013 2:16 PM, "Ming Fang" <[email protected]> wrote: >>>>>> >>>>>>> Erwin >>>>>>> >>>>>>> Here is an example of running Zookeeper, and Controller in the same >>>>>>> process, >>>>>>> >>>>>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java >>>>>>> We have this in production. >>>>>>> >>>>>>> Sent from my iPad >>>>>>> >>>>>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> I know that it's better that participants, controller and zookeeper >>>>>>> will have separate java processes but we have more then 40 boxes with >>>>>>> app >>>>>>> server that we'd like to manage them with Helix Leader election and >>>>>>> cannot >>>>>>> add more processes because each additional java process requires >>>>>>> management >>>>>>> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional >>>>>>> java processes. We'll not manage to approve additional processes >>>>>>> >>>>>>> Your insight would appreciated. >>>>>>> >>>>>>> Thanks, >>>>>>> Erwin >>>>>>> On Jun 14, 2013 9:58 AM, "kishore g" <[email protected]> wrote: >>>>>>> >>>>>>>> Hi Erwin, >>>>>>>> >>>>>>>> This is how I would setup your cluster. Allocate 3 boxes that only >>>>>>>> run zookeeper and Helix controller. Its better that they are two >>>>>>>> separate >>>>>>>> process. These boxes need not be powerful but isolating zookeeper and >>>>>>>> helix >>>>>>>> controller from actual server is a good deployment practice. See >>>>>>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon >>>>>>>> how to use zookeeper clustering ensemble. >>>>>>>> >>>>>>>> >>>>>>>> Run the real servers on the rest of the boxes. Do not start >>>>>>>> zookeeper as part of these processes, instead simply provide the >>>>>>>> zookeeper >>>>>>>> address (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix Api. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Kishore G >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi Kishore, >>>>>>>>> >>>>>>>>> We have more then 50 nodes. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Erwin >>>>>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Erwin, >>>>>>>>>> >>>>>>>>>> How many nodes do you have in your cluster ? >>>>>>>>>> >>>>>>>>>> thanks, >>>>>>>>>> Kishore G >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hello Maters, >>>>>>>>>>> >>>>>>>>>>> We don't want to have separate/specific JAVA process for >>>>>>>>>>> Zookeeper. >>>>>>>>>>> As I figured out from the Master-Slave sample we can start the >>>>>>>>>>> Zookeeper programmatically using the Zookeeper API. >>>>>>>>>>> >>>>>>>>>>> I have several questions: >>>>>>>>>>> >>>>>>>>>>> 1. If we are using the Zookeeper API for initializing the >>>>>>>>>>> Zookeeper from the Helix app (out app that uses Helix) would we have >>>>>>>>>>> specific process for Zookeeper? Would we have one process for our >>>>>>>>>>> app and >>>>>>>>>>> another for the Zookeeper? >>>>>>>>>>> What happens if we'll not initialize the Zookeeper from the >>>>>>>>>>> Helix app programmatically but using the commands, whether then >>>>>>>>>>> would we >>>>>>>>>>> have specific process for Zookeeper? >>>>>>>>>>> >>>>>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix app, >>>>>>>>>>> programmatically, How can we use Zookeeper clustering (ensemble) , >>>>>>>>>>> is there >>>>>>>>>>> any API for Zookeeper clustering, any direction would highly >>>>>>>>>>> appreciated. >>>>>>>>>>> >>>>>>>>>>> 3. Is it possible to start the Zookeeper by using commands and >>>>>>>>>>> not initializing specific process for it but using the Helix app >>>>>>>>>>> (our app >>>>>>>>>>> that uses Helix API)? >>>>>>>>>>> >>>>>>>>>>> I hope my questions are enough clear. >>>>>>>>>>> >>>>>>>>>>> Thanks in advance, >>>>>>>>>>> Erwin Karbasi >>>>>>>>>>> AT&T, Senior Software Architect >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>> >
