Hi Andy,
This is a client side property that you can set in hadoop-env.sh
(/etc/hadoop/conf). The hadoop script (/bin/hadoop)
itself, which is a bash shell script, contains code like the following.
if [ "$HADOOP_HEAPSIZE" != "" ]; then
#echo "run with heapsize $HADOOP_HEAPSIZE"
JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"
#echo $JAVA_HEAP_MAX
fi
Use hadoop-env.sh.
Best,
Bjorn
On Fri, Apr 18, 2014 at 7:20 PM, Andy Srine <andy.sr...@gmail.com> wrote:
> I guess then the question is, how do I set the (system?) property and what
> property exactly is this?
>
> "export HADOOP_HEAPSIZE=4096" - Works from the environment.
>
> And none of the following seem to work from the command line.
> -Xmx4G
>
> -Dmapred.child.java.opts=-Xmx4G
>
> -DHADOOP_HEAPSIZE=4096
>
> -DHADOOP_OPTS=-Xmx4G
>
>
> I tried this from Java code as well and no luck.
>
> System.setProperty("HADOOP_HEAPSIZE", "4096");
>
>
> What am I missing?
>
>
> Thanks,
>
> Andy
>
>
>
>
> On Fri, Apr 18, 2014 at 11:28 AM, Shahab Yunus <shahab.yu...@gmail.com>wrote:
>
>> You can pass hadoop conf properties through the -D option. Have you seen
>> this?
>>
>> http://stackoverflow.com/questions/15490090/how-to-specify-system-property-in-hadoop-except-modify-hadoop-env-sh
>>
>> This is not for system properties. The assumption is that you want to
>> specify hadoop conf property through the command line.
>>
>> Regards,
>> Shahab
>>
>>
>> On Fri, Apr 18, 2014 at 2:12 PM, Andy Srine <andy.sr...@gmail.com> wrote:
>>
>>> Thanks for the tip. May be it's a different setting I need. Let me
>>> explain the problem.
>>>
>>> I get an heap error on the command line - "Exception in thread "main"
>>> java.lang.OutOfMemoryError: Java heap space". It woks when I set "export
>>> HADOOP_HEAPSIZE=4096" in my env. But how can I set it via the hadoop
>>> command line (hadoop jar ...)? Next choice would be to do it
>>> programmatically in Java using hadoop conf. I can't really change the XML
>>> files across the cluster.
>>>
>>> Regards,
>>> Andy
>>>
>>>
>>> On Thu, Apr 17, 2014 at 6:39 PM, Azuryy Yu <azury...@gmail.com> wrote:
>>>
>>>> Do you want add "-Xmx4g" to your MR tasks? if so, just add it as
>>>> "mapred.child.java.opts" in the mapred-site.xml
>>>>
>>>>
>>>> On Fri, Apr 18, 2014 at 9:35 AM, Andy Srine <andy.sr...@gmail.com>wrote:
>>>>
>>>>> Quick question. How would I pass the following JVM option to the
>>>>> Hadoop command line?
>>>>>
>>>>> "-Xmx4G"
>>>>>
>>>>> hadoop jar <jar file> <class_name> <args>
>>>>>
>>>>> Thanks,
>>>>> Andy
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Andy
>>>
>>>
>>
>
>
> --
> Thanks,
> Andy
>
>
