Ah, makes sense So to have a cluster setup with RM and NMs running on different nodes, Can I reuse the "grid" script from "hello-samza"? or will I have to do the setup separately and then change the config files on samza?
Thanks, Sonali -----Original Message----- From: Chris Riccomini [mailto:[email protected]] Sent: Monday, February 03, 2014 11:02 AM To: [email protected] Subject: Re: Cluster Installation Hey Sonali, I believe the point at which YARN became version compatible for 2.* as at 2.1.0-beta. I believe 2.0.5 is not API compatible with later versions of YARN (e.g. 2.2). For this reason, you'll need to upgrade your YARN grid, or use a different one with a higher version. For its part, Samza should work with YARN grids 2.1.0-beta and beyond, though I haven't tested this. The YARN community has given a commitment to maintaining API compatibility going forward for YARN 2.*, which means that future upgrades should not be required, until YARN 3 comes out. The rest of your understanding is correct. You can run a 1 RM, 2 NM kind of cluster, throw some Kafka brokers on there, and you should be good to go. You can also re-use your existing ZK, if you wish. Cheers, Chris On 2/3/14 10:42 AM, "[email protected]" <[email protected]> wrote: >Thanks Chris/Gary. > >I have an existing Zookeeper and YARN Cluster. However, the YARN >version that I have (that came preinstalled with Pivotal HD) is 2.0.5. >So from what you're saying I cannot reuse it for my Samza deployment. > >So then my option is: >1. Reuse zookeeper. So I'll have to configure Samza to point to the >right cluster 2. Run Samza with its YARN grid and Kafka Installation (I >can do this on multiple servers right? 1 RM, 2 NM kind of situation) > >Thanks, >Sonali > > >-----Original Message----- >From: Chris Riccomini [mailto:[email protected]] >Sent: Friday, January 31, 2014 11:24 AM >To: [email protected] >Subject: Re: Cluster Installation > >Hey Sonali, > >Everything Gary said is correct. > >One other item of note is that if you're interested in running stuff >locally in a dev-mode fashion, you don't need YARN. You can use the >LocalJobFactory instead of the YarnJobFactory factory when configuring >your job's "job.factory.class" setting. > >For "real" deployments, yes you'll need YARN, ZooKeeper, and Kafka. >They can be deployed using any standard way of shipping software around >to a cluster of machines. > >Cheers, >Chris > >On 1/31/14 12:58 AM, "Garry Turkington" ><[email protected]> >wrote: > >>Hi Sonali, >> >>This was something that I had some questions about originally as well. >>In terms of required components then yes, for any size of Samza >>deployment you will need all those pieces. >> >>In terms of actual deployment, from what I understand from the >>LinkedIn guys they do run Samza on a dedicated YARN grid that also has >>a Kafka broker collocated on each node. These decisions though appear >>to be more down to convenience than a hard requirement. >> >>In my own setup I have existing ZooKeeper and Kafka clusters that I'm >>pointing Samza at but do need to run a dedicated YARN grid because my >>Hadoop cluster has a pre-2.2 version of YARN running on it. >> >>So if you have existing components you can reuse them, if not then >>repurposing the Hello Samza package is a good starting point to get >>all the things you want on the required hosts. Only caveat would be to >>not drop a ZK node on each host, the ZK quorum should follow the usual >>advice of an odd number of servers and likely no more than 3, 5 or 7 >>depending on your deployment size. >> >>Garry >> >>-----Original Message----- >>From: [email protected] >>[mailto:[email protected]] >>Sent: 30 January 2014 23:38 >>To: [email protected] >>Subject: Cluster Installation >> >>Hi All, >> >>I'm new to working with Samza and have been trying to figure out the >>best cluster configuration. I understand that Samza comes with >>yarn,kafka and zookeeper out of the box. Is that the model just for a >>standalone/local configuration. What if I want a bigger cluster? Do I >>have to install yarn, kafka and zookeeper separately? Any suggestions >>would be great! >> >>Thanks, >>Sonali >> >>Sonali Parthasarathy >>R&D Developer, Data Insights >>Accenture Technology Labs >>703-341-7432 >> >> >>________________________________ >> >>This message is for the designated recipient only and may contain >>privileged, proprietary, or otherwise confidential information. If you >>have received it in error, please notify the sender immediately and >>delete the original. Any other use of the e-mail by you is prohibited. >>Where allowed by local law, electronic communications with Accenture >>and its affiliates, including e-mail and instant messaging (including >>content), may be scanned by our systems for the purposes of >>information security and assessment of internal compliance with Accenture >>policy. . >>______________________________________________________________________ >>_ >>___ >>____________ >> >>www.accenture.com >> >>----- >>No virus found in this message. >>Checked by AVG - www.avg.com >>Version: 2014.0.4259 / Virus Database: 3684/7046 - Release Date: >>01/30/14 > > > >________________________________ > >This message is for the designated recipient only and may contain >privileged, proprietary, or otherwise confidential information. If you >have received it in error, please notify the sender immediately and >delete the original. Any other use of the e-mail by you is prohibited. >Where allowed by local law, electronic communications with Accenture >and its affiliates, including e-mail and instant messaging (including >content), may be scanned by our systems for the purposes of information >security and assessment of internal compliance with Accenture policy. . >_______________________________________________________________________ >___ >____________ > >www.accenture.com > ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. . ______________________________________________________________________________________ www.accenture.com
