Guys,
Just curious, how does the h_vmem work on processes of MPI jobs(or
OPENMP, multi-threading)? I have some parallel jobs, the top command
shows "VET" of 40GB, while the "RES" is only 100MB.
On Mon, Jun 30, 2014 at 3:01 PM, Michael Stauffer <mgsta...@gmail.com> wrote:
>> Message: 4
>> Date: Mon, 30 Jun 2014 11:53:12 +0200
>> From: Txema Heredia <txema.llis...@gmail.com>
>> To: Derrick Lin <klin...@gmail.com>, SGE Mailing List
>> <users@gridengine.org>
>> Subject: Re: [gridengine users] Enforce users to use specific amount
>> of memory/slot
>> Message-ID: <53b13388.5060...@gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
>>
>>
>> Hi Derrick,
>>
>> You could either set h_vmem as a consumable (consumable=yes) attribute
>> and set a default value of 8GB for it. This way, whenever a job doesn't
>> request any amount of h_vmem, it will automatically request 8GB per
>> slot. This will affect all types of jobs.
>>
>> You could also define a JSV script that checks the username, and forces
>> a -l h_vmem=8G for his/her jobs (
>> jsv_sub_add_param('l_hard','h_vmem','8G') ). This will affect all jobs
>> for that user, but could turn into a pain to manage.
>>
>> Or, you could set a different policy and allow all users to request the
>> amount of memory they really need, trying to fit best the node. What is
>> the point of forcing the user to reserve 63 additional cores when they
>> only need 1 core and 500GB of memory? You could fit in that node one job
>> like this, and, say, two 30-core-6GB-memory jobs.
>>
>> Txema
>>
>>
>>
>> El 30/06/14 08:55, Derrick Lin escribi?:
>>
>> > Hi guys,
>> >
>> > A typical node on our cluster has 64 cores and 512GB memory. So it's
>> > about 8GB/core. Occasionally, we have some jobs that utilizes only 1
>> > core but 400-500GB of memory, that annoys lots of users. So I am
>> > seeking a way that can force jobs to run strictly below 8GB/core
>> > ration or it should be killed.
>> >
>> > For example, the above job should ask for 64 cores in order to use
>> > 500GB of memory (we have user quota for slots).
>> >
>> > I have been trying to play around h_vmem, set it to consumable and
>> > configure RQS
>> >
>> > {
>> > name max_user_vmem
>> > enabled true
>> > description "Each user can utilize more than 8GB/slot"
>> > limit users {bad_user} to h_vmem=8g
>> > }
>> >
>> > but it seems to be setting a total vmem bad_user can use per job.
>> >
>> > I would love to set it on users instead of queue or hosts because we
>> > have applications that utilize the same set of nodes and app should be
>> > unlimited.
>> >
>> > Thanks
>> > Derrick
>
>
> I've been dealing with this too. I'm using h_vmem to kill processes that go
> above the limit, and s_vmem set slightly lower by default to give
> well-behaved processes a chance first to exit gracefully.
>
> The issue is that these use virtual memory, which is (always, more or less)
> great than resident memory, i.e. the actual ram usage. And with java apps
> like Matlab, the amount of virtual memory reserved/used is HUGE compared to
> resident, by 10x give or take. So it makes it really impracticle actually.
> However so far I've just set the default h_vmem and s_vmem values high
> enough to accomadate jvm apps, and increased the per-host consumable
> appropriately. We don't get fine-grained memory control, but it definitely
> controls out-of-control users/procs that otherwise might gobble up enough
> ram to slow dow the entire node.
>
> We may switch to UVE just for this reason, to get memory limits based on
> resident memory, if it seems worth it enough in the end.
>
> -M
>
> _______________________________________________
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users
>
_______________________________________________
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users
