Hey Sonali, You will need to setup separately in order to configure your yarn-site.xml files for the NMs to point to the RM's host/port. They default to localhost, which is what hello-samza is using.
On the Kafka side, the same things applies- you'll need to configure each broker with a unique broker id, etc. Cheers, Chris On 2/3/14 11:25 AM, "[email protected]" <[email protected]> wrote: >Ah, makes sense > >So to have a cluster setup with RM and NMs running on different nodes, >Can I reuse the "grid" script from "hello-samza"? or will I have to do >the setup separately and then change the config files on samza? > >Thanks, >Sonali > >-----Original Message----- >From: Chris Riccomini [mailto:[email protected]] >Sent: Monday, February 03, 2014 11:02 AM >To: [email protected] >Subject: Re: Cluster Installation > >Hey Sonali, > >I believe the point at which YARN became version compatible for 2.* as at >2.1.0-beta. I believe 2.0.5 is not API compatible with later versions of >YARN (e.g. 2.2). For this reason, you'll need to upgrade your YARN grid, >or use a different one with a higher version. > >For its part, Samza should work with YARN grids 2.1.0-beta and beyond, >though I haven't tested this. The YARN community has given a commitment >to maintaining API compatibility going forward for YARN 2.*, which means >that future upgrades should not be required, until YARN 3 comes out. > >The rest of your understanding is correct. You can run a 1 RM, 2 NM kind >of cluster, throw some Kafka brokers on there, and you should be good to >go. You can also re-use your existing ZK, if you wish. > >Cheers, >Chris > >On 2/3/14 10:42 AM, "[email protected]" ><[email protected]> wrote: > >>Thanks Chris/Gary. >> >>I have an existing Zookeeper and YARN Cluster. However, the YARN >>version that I have (that came preinstalled with Pivotal HD) is 2.0.5. >>So from what you're saying I cannot reuse it for my Samza deployment. >> >>So then my option is: >>1. Reuse zookeeper. So I'll have to configure Samza to point to the >>right cluster 2. Run Samza with its YARN grid and Kafka Installation (I >>can do this on multiple servers right? 1 RM, 2 NM kind of situation) >> >>Thanks, >>Sonali >> >> >>-----Original Message----- >>From: Chris Riccomini [mailto:[email protected]] >>Sent: Friday, January 31, 2014 11:24 AM >>To: [email protected] >>Subject: Re: Cluster Installation >> >>Hey Sonali, >> >>Everything Gary said is correct. >> >>One other item of note is that if you're interested in running stuff >>locally in a dev-mode fashion, you don't need YARN. You can use the >>LocalJobFactory instead of the YarnJobFactory factory when configuring >>your job's "job.factory.class" setting. >> >>For "real" deployments, yes you'll need YARN, ZooKeeper, and Kafka. >>They can be deployed using any standard way of shipping software around >>to a cluster of machines. >> >>Cheers, >>Chris >> >>On 1/31/14 12:58 AM, "Garry Turkington" >><[email protected]> >>wrote: >> >>>Hi Sonali, >>> >>>This was something that I had some questions about originally as well. >>>In terms of required components then yes, for any size of Samza >>>deployment you will need all those pieces. >>> >>>In terms of actual deployment, from what I understand from the >>>LinkedIn guys they do run Samza on a dedicated YARN grid that also has >>>a Kafka broker collocated on each node. These decisions though appear >>>to be more down to convenience than a hard requirement. >>> >>>In my own setup I have existing ZooKeeper and Kafka clusters that I'm >>>pointing Samza at but do need to run a dedicated YARN grid because my >>>Hadoop cluster has a pre-2.2 version of YARN running on it. >>> >>>So if you have existing components you can reuse them, if not then >>>repurposing the Hello Samza package is a good starting point to get >>>all the things you want on the required hosts. Only caveat would be to >>>not drop a ZK node on each host, the ZK quorum should follow the usual >>>advice of an odd number of servers and likely no more than 3, 5 or 7 >>>depending on your deployment size. >>> >>>Garry >>> >>>-----Original Message----- >>>From: [email protected] >>>[mailto:[email protected]] >>>Sent: 30 January 2014 23:38 >>>To: [email protected] >>>Subject: Cluster Installation >>> >>>Hi All, >>> >>>I'm new to working with Samza and have been trying to figure out the >>>best cluster configuration. I understand that Samza comes with >>>yarn,kafka and zookeeper out of the box. Is that the model just for a >>>standalone/local configuration. What if I want a bigger cluster? Do I >>>have to install yarn, kafka and zookeeper separately? Any suggestions >>>would be great! >>> >>>Thanks, >>>Sonali >>> >>>Sonali Parthasarathy >>>R&D Developer, Data Insights >>>Accenture Technology Labs >>>703-341-7432 >>> >>> >>>________________________________ >>> >>>This message is for the designated recipient only and may contain >>>privileged, proprietary, or otherwise confidential information. If you >>>have received it in error, please notify the sender immediately and >>>delete the original. Any other use of the e-mail by you is prohibited. >>>Where allowed by local law, electronic communications with Accenture >>>and its affiliates, including e-mail and instant messaging (including >>>content), may be scanned by our systems for the purposes of >>>information security and assessment of internal compliance with >>>Accenture policy. . >>>______________________________________________________________________ >>>_ >>>___ >>>____________ >>> >>>www.accenture.com >>> >>>----- >>>No virus found in this message. >>>Checked by AVG - www.avg.com >>>Version: 2014.0.4259 / Virus Database: 3684/7046 - Release Date: >>>01/30/14 >> >> >> >>________________________________ >> >>This message is for the designated recipient only and may contain >>privileged, proprietary, or otherwise confidential information. If you >>have received it in error, please notify the sender immediately and >>delete the original. Any other use of the e-mail by you is prohibited. >>Where allowed by local law, electronic communications with Accenture >>and its affiliates, including e-mail and instant messaging (including >>content), may be scanned by our systems for the purposes of information >>security and assessment of internal compliance with Accenture policy. . >>_______________________________________________________________________ >>___ >>____________ >> >>www.accenture.com >> > > > >________________________________ > >This message is for the designated recipient only and may contain >privileged, proprietary, or otherwise confidential information. If you >have received it in error, please notify the sender immediately and >delete the original. Any other use of the e-mail by you is prohibited. >Where allowed by local law, electronic communications with Accenture and >its affiliates, including e-mail and instant messaging (including >content), may be scanned by our systems for the purposes of information >security and assessment of internal compliance with Accenture policy. . >__________________________________________________________________________ >____________ > >www.accenture.com >
