Re: Unable to set the heap size on Amazon elastic mapreduce
Hey Shrish If you are looking to do it on a job level, you can achive the same as hadoop jar -D mapred.map.java.opts=-Xmx1024m Also it is not recommended to change this on a per job level, normally in production environments this value is set in mapred-site.xml and is marked final, so that it can't be overridden at per job level. Overriding at job level can lead to out of memory issues, if not handled wisely at job level. * mapred.child.java.opts -Xmx2048m* true Regards Bejoy KS On Thu, Apr 5, 2012 at 10:12 PM, kasi subrahmanyam wrote: > Hi Shirsh, > > You need to increase the heap size of the JVM for the changes you made for > mapred.child to take effect > These changes need to be made in the files of the hadoop configuration > folder. > > In hadoop-env.sh increase the heap size to may be 2000.(I think this > property by default will be commented.) > Then in the mapred-default.xml change( in the src folder) overwrite the > other properties like mapred.map.child.java.opts. > > I hope this helps. > > > > On Thu, Apr 5, 2012 at 1:26 PM, shrish bajpai > wrote: > >> Hi, >> I have tried the following combinations of bootstrap actions to increase >> the heap size of my job but none of them seem to work: >> >> --mapred-key-value mapred.child.java.opts=-Xmx1024m >> --mapred-key-value mapred.child.ulimit=unlimited >> >> --mapred-key-value mapred.map.child.java.opts=-Xmx1024m >> --mapred-key-value mapred.map.child.ulimit=unlimited >> >> -m mapred.map.child.java.opts=-Xmx1024m >> -m mapred.map.child.ulimit=unlimited >> >> -m mapred.child.java.opts=-Xmx1024m >> -m mapred.child.ulimit=unlimited >> >> What is the right syntax? >> >> Thanks >> >> Shrish >> >> >
Re: Unable to set the heap size on Amazon elastic mapreduce
Hi Shirsh, You need to increase the heap size of the JVM for the changes you made for mapred.child to take effect These changes need to be made in the files of the hadoop configuration folder. In hadoop-env.sh increase the heap size to may be 2000.(I think this property by default will be commented.) Then in the mapred-default.xml change( in the src folder) overwrite the other properties like mapred.map.child.java.opts. I hope this helps. On Thu, Apr 5, 2012 at 1:26 PM, shrish bajpai wrote: > Hi, > I have tried the following combinations of bootstrap actions to increase > the heap size of my job but none of them seem to work: > > --mapred-key-value mapred.child.java.opts=-Xmx1024m > --mapred-key-value mapred.child.ulimit=unlimited > > --mapred-key-value mapred.map.child.java.opts=-Xmx1024m > --mapred-key-value mapred.map.child.ulimit=unlimited > > -m mapred.map.child.java.opts=-Xmx1024m > -m mapred.map.child.ulimit=unlimited > > -m mapred.child.java.opts=-Xmx1024m > -m mapred.child.ulimit=unlimited > > What is the right syntax? > > Thanks > > Shrish > >
Unable to set the heap size on Amazon elastic mapreduce
Hi, I have tried the following combinations of bootstrap actions to increase the heap size of my job but none of them seem to work: --mapred-key-value mapred.child.java.opts=-Xmx1024m --mapred-key-value mapred.child.ulimit=unlimited --mapred-key-value mapred.map.child.java.opts=-Xmx1024m --mapred-key-value mapred.map.child.ulimit=unlimited -m mapred.map.child.java.opts=-Xmx1024m -m mapred.map.child.ulimit=unlimited -m mapred.child.java.opts=-Xmx1024m -m mapred.child.ulimit=unlimited What is the right syntax? Thanks Shrish
