Re: Pre-requisites for hadoop 0.23/CDH4
You are better of trying hadoop-0.23.1 or even hadoop-0.23.2-rc0 since CDH4's version of YARN is very incomplete and you might get nasty surprises there. Settings: # Run 1 NodeManager with yarn.nodemanager.resource.memory-mb - 1024 # Use CapacityScheduler (significantly better tested) by setting yarn.resourcemanager.scheduler.class to org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler # Set min container size to 128M or 256M (upto you) by setting yarn.scheduler.capacity.minimum-allocation-mb # Set max container size to 1024M (max given to NM) by setting yarn.scheduler.capacity.maximum-allocation-mb Arun On Apr 18, 2012, at 8:00 PM, praveenesh kumar wrote: Hi, Sweet.. Can you please elaborate how can I tweak my configs to make CDH4/hadoop-0.23 run in 1.5GB RAM VM. Regards, Praveenesh On Wed, Apr 18, 2012 at 8:42 AM, Harsh J ha...@cloudera.com wrote: Praveenesh, Speaking minimally (and thereby requiring less tweaks on your end), 1.5 GB would be a good value to use for RAM if available (1.0 will do too, if you make sure to tweak your configs to not use too much heap memory). Single processor should do fine for testing purposes. On Tue, Apr 17, 2012 at 8:51 PM, praveenesh kumar praveen...@gmail.com wrote: I am looking to test hadoop 0.23 or CDH4 beta on my local VM. I am looking to execute the sample example codes in new architecture, play around with the containers/resource managers. Is there any pre-requisite on default memory/CPU/core settings I need to keep in mind before setting up the VM. Regards, Praveenesh -- Harsh J -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: Pre-requisites for hadoop 0.23/CDH4
Thanks Arun, Will try out those settings. Is there a good documentation on configuring/playing with hadoop 0.23 apart from the apache hadoop-0.23 page. I have already looked into that page.. Just wondering is there something more that I don't know. Regards, Praveenesh On Fri, Apr 20, 2012 at 12:45 AM, Arun C Murthy a...@hortonworks.com wrote: You are better of trying hadoop-0.23.1 or even hadoop-0.23.2-rc0 since CDH4's version of YARN is very incomplete and you might get nasty surprises there. Settings: # Run 1 NodeManager with yarn.nodemanager.resource.memory-mb - 1024 # Use CapacityScheduler (significantly better tested) by setting yarn.resourcemanager.scheduler.class to org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler # Set min container size to 128M or 256M (upto you) by setting yarn.scheduler.capacity.minimum-allocation-mb # Set max container size to 1024M (max given to NM) by setting yarn.scheduler.capacity.maximum-allocation-mb Arun On Apr 18, 2012, at 8:00 PM, praveenesh kumar wrote: Hi, Sweet.. Can you please elaborate how can I tweak my configs to make CDH4/hadoop-0.23 run in 1.5GB RAM VM. Regards, Praveenesh On Wed, Apr 18, 2012 at 8:42 AM, Harsh J ha...@cloudera.com wrote: Praveenesh, Speaking minimally (and thereby requiring less tweaks on your end), 1.5 GB would be a good value to use for RAM if available (1.0 will do too, if you make sure to tweak your configs to not use too much heap memory). Single processor should do fine for testing purposes. On Tue, Apr 17, 2012 at 8:51 PM, praveenesh kumar praveen...@gmail.com wrote: I am looking to test hadoop 0.23 or CDH4 beta on my local VM. I am looking to execute the sample example codes in new architecture, play around with the containers/resource managers. Is there any pre-requisite on default memory/CPU/core settings I need to keep in mind before setting up the VM. Regards, Praveenesh -- Harsh J -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
Re: Pre-requisites for hadoop 0.23/CDH4
Hi, Sweet.. Can you please elaborate how can I tweak my configs to make CDH4/hadoop-0.23 run in 1.5GB RAM VM. Regards, Praveenesh On Wed, Apr 18, 2012 at 8:42 AM, Harsh J ha...@cloudera.com wrote: Praveenesh, Speaking minimally (and thereby requiring less tweaks on your end), 1.5 GB would be a good value to use for RAM if available (1.0 will do too, if you make sure to tweak your configs to not use too much heap memory). Single processor should do fine for testing purposes. On Tue, Apr 17, 2012 at 8:51 PM, praveenesh kumar praveen...@gmail.com wrote: I am looking to test hadoop 0.23 or CDH4 beta on my local VM. I am looking to execute the sample example codes in new architecture, play around with the containers/resource managers. Is there any pre-requisite on default memory/CPU/core settings I need to keep in mind before setting up the VM. Regards, Praveenesh -- Harsh J
Pre-requisites for hadoop 0.23/CDH4
I am looking to test hadoop 0.23 or CDH4 beta on my local VM. I am looking to execute the sample example codes in new architecture, play around with the containers/resource managers. Is there any pre-requisite on default memory/CPU/core settings I need to keep in mind before setting up the VM. Regards, Praveenesh
