On 6/26/08 10:14 AM, "Joydeep Sen Sarma" <[EMAIL PROTECTED]> wrote: > However - in our environment - we spawn all streaming tasks through a > wrapper program (with the wrapper being defaulted across all users). We > can control resource uses from this wrapper (and also do some level of > job control).
This is essentially what we're doing via torque (and therefore hod). > This does require some minor code changes and would be happy to contrib > - but the approach doesn't quite fit in well with the approach in 2765 > that passes each one of these resource limits as a hadoop config param. As I said in HADOOP-3280, I'm still convinced that having Hadoop set these types of restrictions directly is the wrong approach. It is almost always going to be better to use the OS controls that are specific to the installation. Enabling OS specific features means that, at a maximum, hadoop should likely being calling a script rather than doing ulimits or having the equivalent of #ifdef code everywhere. [After all, what if I want to use Solaris-specific features like projects or privileges?] But I suspect most of the tunables that people will care about can likely be managed at the OS level before hadoop is even involved.