Hi Andy, This is a client side property that you can set in hadoop-env.sh (/etc/hadoop/conf). The hadoop script (/bin/hadoop) itself, which is a bash shell script, contains code like the following.
if [ "$HADOOP_HEAPSIZE" != "" ]; then #echo "run with heapsize $HADOOP_HEAPSIZE" JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m" #echo $JAVA_HEAP_MAX fi Use hadoop-env.sh. Best, Bjorn On Fri, Apr 18, 2014 at 7:20 PM, Andy Srine <andy.sr...@gmail.com> wrote: > I guess then the question is, how do I set the (system?) property and what > property exactly is this? > > "export HADOOP_HEAPSIZE=4096" - Works from the environment. > > And none of the following seem to work from the command line. > -Xmx4G > > -Dmapred.child.java.opts=-Xmx4G > > -DHADOOP_HEAPSIZE=4096 > > -DHADOOP_OPTS=-Xmx4G > > > I tried this from Java code as well and no luck. > > System.setProperty("HADOOP_HEAPSIZE", "4096"); > > > What am I missing? > > > Thanks, > > Andy > > > > > On Fri, Apr 18, 2014 at 11:28 AM, Shahab Yunus <shahab.yu...@gmail.com>wrote: > >> You can pass hadoop conf properties through the -D option. Have you seen >> this? >> >> http://stackoverflow.com/questions/15490090/how-to-specify-system-property-in-hadoop-except-modify-hadoop-env-sh >> >> This is not for system properties. The assumption is that you want to >> specify hadoop conf property through the command line. >> >> Regards, >> Shahab >> >> >> On Fri, Apr 18, 2014 at 2:12 PM, Andy Srine <andy.sr...@gmail.com> wrote: >> >>> Thanks for the tip. May be it's a different setting I need. Let me >>> explain the problem. >>> >>> I get an heap error on the command line - "Exception in thread "main" >>> java.lang.OutOfMemoryError: Java heap space". It woks when I set "export >>> HADOOP_HEAPSIZE=4096" in my env. But how can I set it via the hadoop >>> command line (hadoop jar ...)? Next choice would be to do it >>> programmatically in Java using hadoop conf. I can't really change the XML >>> files across the cluster. >>> >>> Regards, >>> Andy >>> >>> >>> On Thu, Apr 17, 2014 at 6:39 PM, Azuryy Yu <azury...@gmail.com> wrote: >>> >>>> Do you want add "-Xmx4g" to your MR tasks? if so, just add it as >>>> "mapred.child.java.opts" in the mapred-site.xml >>>> >>>> >>>> On Fri, Apr 18, 2014 at 9:35 AM, Andy Srine <andy.sr...@gmail.com>wrote: >>>> >>>>> Quick question. How would I pass the following JVM option to the >>>>> Hadoop command line? >>>>> >>>>> "-Xmx4G" >>>>> >>>>> hadoop jar <jar file> <class_name> <args> >>>>> >>>>> Thanks, >>>>> Andy >>>>> >>>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Andy >>> >>> >> > > > -- > Thanks, > Andy > >