On 6/26/08 10:14 AM, "Joydeep Sen Sarma" <[EMAIL PROTECTED]> wrote:
> However - in our environment - we spawn all streaming tasks through a
> wrapper program (with the wrapper being defaulted across all users). We
> can control resource uses from this wrapper (and also do some level of
> job control).

    This is essentially what we're doing via torque (and therefore hod).

> This does require some minor code changes and would be happy to contrib
> - but the approach doesn't quite fit in well with the approach in 2765
> that passes each one of these resource limits as a hadoop config param.

    As I said in HADOOP-3280, I'm still convinced that having Hadoop set
these types of restrictions directly is the wrong approach.  It is almost
always going to be better to use the OS controls that are specific to the
installation.  Enabling OS specific features means that, at a maximum,
hadoop should likely being calling a script rather than doing ulimits or
having the equivalent of #ifdef code everywhere. [After all, what if I want
to use Solaris-specific features like projects or privileges?]

    But I suspect most of the tunables that people will care about can
likely be managed at the OS level before hadoop is even involved.

Reply via email to