On 9/14/09 10:42 PM, "Chris Dyer" <redp...@umd.edu> wrote: > And worse, in > my task logs I see the message: > "attempt to override final parameter: mapred.child.ulimit; Ignoring." > which doesn't exactly inspire confidence that I'm on the right path.
Chances are the param has been marked final in the task tracker's running config which will prevent you overriding the value with a job specific configuration. > ps. I'm running an 18.3 cluster on Amazon EC2 (I've been using the > Cloudera convenience scripts, but I can abandon this if I need more > control). The instances have plenty of memory (7.5GB). Depending upon how many tasks per node, that may not be enough. Streaming jobs eat a crapton (I'm pretty sure that is an SI unit) of memory. If you are hitting 2gb+, that means you can probably run 3 tasks max without swapping. [Don't forget to count the size of the task tracker JVM, the streaming.jar JVM, etc, and be cognizant of the fact that JVM mem size != Java heap size.]
