On Fri, May 8, 2009 at 1:11 PM, Ken Krugler <kkrugler_li...@transpac.com>wrote:
> You an set the mapred.child.java.opts on a per job basis
>> either via -D mapred.child.java.ops="java options" or via
>> conf.set("mapred.child.java.opts", "java options").
>>
>> Note: the conf.set must be done before the job is submitted.
>>
>> On Fri, May 8, 2009 at 11:57 AM, Philip Zeyliger <phi...@cloudera.com
>> >wrote:
>>
>> You could add "-Xss<n>" to the "mapred.child.java.opts" configuration
>>> setting. That's controlling the Java stack size, which I think is the
>>> relevant bit for you.
>>>
>>
> That's part of it, but there's also native memory used when you start a
> thread with most JREs.
It doesn't look like Hadoop lets you run an arbitrary ulimit command, but if
you take a look at Shell.getUlimitMemoryCommand(conf) (called from
TaskRunner.java), you'll see that it lets you specify "ulimit -v N"
commands. You could probably augment that for "ulimit -s" pretty easily.
