Re: process hangs on do_exit when oom happens

2012-10-26 Thread Mike Galbraith
On Fri, 2012-10-26 at 10:03 -0700, Mike Galbraith wrote: > The bug is in the patch that used sched_setscheduler_nocheck(). Plain > sched_setscheduler() would have replied -EGOAWAY. sched_setscheduler_nocheck() should say go away too methinks. This isn't about permissions, it's about not being

Re: process hangs on do_exit when oom happens

2012-10-26 Thread Mike Galbraith
On Fri, 2012-10-26 at 10:42 +0800, Qiang Gao wrote: > On Thu, Oct 25, 2012 at 5:57 PM, Michal Hocko wrote: > > On Wed 24-10-12 11:44:17, Qiang Gao wrote: > >> On Wed, Oct 24, 2012 at 1:43 AM, Balbir Singh > >> wrote: > >> > On Tue, Oct 23, 2012 at 3:45 PM, Michal Hocko wrote: > >> >> On Tue

Re: process hangs on do_exit when oom happens

2012-10-26 Thread Mike Galbraith
On Fri, 2012-10-26 at 10:42 +0800, Qiang Gao wrote: On Thu, Oct 25, 2012 at 5:57 PM, Michal Hocko mho...@suse.cz wrote: On Wed 24-10-12 11:44:17, Qiang Gao wrote: On Wed, Oct 24, 2012 at 1:43 AM, Balbir Singh bsinghar...@gmail.com wrote: On Tue, Oct 23, 2012 at 3:45 PM, Michal Hocko

Re: process hangs on do_exit when oom happens

2012-10-26 Thread Mike Galbraith
On Fri, 2012-10-26 at 10:03 -0700, Mike Galbraith wrote: The bug is in the patch that used sched_setscheduler_nocheck(). Plain sched_setscheduler() would have replied -EGOAWAY. sched_setscheduler_nocheck() should say go away too methinks. This isn't about permissions, it's about not being

Re: process hangs on do_exit when oom happens

2012-10-25 Thread Qiang Gao
On Thu, Oct 25, 2012 at 5:57 PM, Michal Hocko wrote: > On Wed 24-10-12 11:44:17, Qiang Gao wrote: >> On Wed, Oct 24, 2012 at 1:43 AM, Balbir Singh wrote: >> > On Tue, Oct 23, 2012 at 3:45 PM, Michal Hocko wrote: >> >> On Tue 23-10-12 18:10:33, Qiang Gao wrote: >> >>> On Tue, Oct 23, 2012 at

Re: process hangs on do_exit when oom happens

2012-10-25 Thread Michal Hocko
On Wed 24-10-12 11:44:17, Qiang Gao wrote: > On Wed, Oct 24, 2012 at 1:43 AM, Balbir Singh wrote: > > On Tue, Oct 23, 2012 at 3:45 PM, Michal Hocko wrote: > >> On Tue 23-10-12 18:10:33, Qiang Gao wrote: > >>> On Tue, Oct 23, 2012 at 5:50 PM, Michal Hocko wrote: > >>> > On Tue 23-10-12 15:18:48,

Re: process hangs on do_exit when oom happens

2012-10-25 Thread Michal Hocko
On Wed 24-10-12 11:44:17, Qiang Gao wrote: On Wed, Oct 24, 2012 at 1:43 AM, Balbir Singh bsinghar...@gmail.com wrote: On Tue, Oct 23, 2012 at 3:45 PM, Michal Hocko mho...@suse.cz wrote: On Tue 23-10-12 18:10:33, Qiang Gao wrote: On Tue, Oct 23, 2012 at 5:50 PM, Michal Hocko mho...@suse.cz

Re: process hangs on do_exit when oom happens

2012-10-25 Thread Qiang Gao
On Thu, Oct 25, 2012 at 5:57 PM, Michal Hocko mho...@suse.cz wrote: On Wed 24-10-12 11:44:17, Qiang Gao wrote: On Wed, Oct 24, 2012 at 1:43 AM, Balbir Singh bsinghar...@gmail.com wrote: On Tue, Oct 23, 2012 at 3:45 PM, Michal Hocko mho...@suse.cz wrote: On Tue 23-10-12 18:10:33, Qiang Gao

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Qiang Gao
On Wed, Oct 24, 2012 at 1:43 AM, Balbir Singh wrote: > On Tue, Oct 23, 2012 at 3:45 PM, Michal Hocko wrote: >> On Tue 23-10-12 18:10:33, Qiang Gao wrote: >>> On Tue, Oct 23, 2012 at 5:50 PM, Michal Hocko wrote: >>> > On Tue 23-10-12 15:18:48, Qiang Gao wrote: >>> >> This process was moved to

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Balbir Singh
On Tue, Oct 23, 2012 at 3:45 PM, Michal Hocko wrote: > On Tue 23-10-12 18:10:33, Qiang Gao wrote: >> On Tue, Oct 23, 2012 at 5:50 PM, Michal Hocko wrote: >> > On Tue 23-10-12 15:18:48, Qiang Gao wrote: >> >> This process was moved to RT-priority queue when global oom-killer >> >> happened to

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Michal Hocko
On Tue 23-10-12 18:10:33, Qiang Gao wrote: > On Tue, Oct 23, 2012 at 5:50 PM, Michal Hocko wrote: > > On Tue 23-10-12 15:18:48, Qiang Gao wrote: > >> This process was moved to RT-priority queue when global oom-killer > >> happened to boost the recovery of the system.. > > > > Who did that? oom

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Qiang Gao
On Tue, Oct 23, 2012 at 5:50 PM, Michal Hocko wrote: > On Tue 23-10-12 15:18:48, Qiang Gao wrote: >> This process was moved to RT-priority queue when global oom-killer >> happened to boost the recovery of the system.. > > Who did that? oom killer doesn't boost the priority (scheduling class) >

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Michal Hocko
On Tue 23-10-12 15:18:48, Qiang Gao wrote: > This process was moved to RT-priority queue when global oom-killer > happened to boost the recovery of the system.. Who did that? oom killer doesn't boost the priority (scheduling class) AFAIK. > but it wasn't get properily dealt with. I still have no

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Michal Hocko
On Tue 23-10-12 17:08:40, Qiang Gao wrote: > this is just an example to show how to reproduce. actually,the first time I > saw > this situation was on a machine with 288G RAM with many tasks running and > we limit 30G for each. but finanlly, no one exceeds this limit the the system > oom. Yes

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Qiang Gao
global-oom is the right thing to do. but oom-killed-process hanging on do_exit is not the normal behavior On Tue, Oct 23, 2012 at 5:01 PM, Sha Zhengju wrote: > On 10/23/2012 11:35 AM, Qiang Gao wrote: >> >> information about the system is in the attach file "information.txt" >> >> I can not

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Qiang Gao
this is just an example to show how to reproduce. actually,the first time I saw this situation was on a machine with 288G RAM with many tasks running and we limit 30G for each. but finanlly, no one exceeds this limit the the system oom. On Tue, Oct 23, 2012 at 4:35 PM, Michal Hocko wrote: > On

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Sha Zhengju
On 10/23/2012 11:35 AM, Qiang Gao wrote: information about the system is in the attach file "information.txt" I can not reproduce it in the upstream 3.6.0 kernel.. On Sat, Oct 20, 2012 at 12:04 AM, Michal Hocko wrote: On Wed 17-10-12 18:23:34, gaoqiang wrote: I looked up nothing useful with

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Michal Hocko
On Tue 23-10-12 11:35:52, Qiang Gao wrote: > I'm sure this is a global-oom,not cgroup-oom. [the dmesg output in the end] Yes this is the global oom killer because: > cglimit -M 700M ./tt > then after global-oom,the process hangs.. > 179184 pages RAM So you have ~700M of RAM so the memcg limit

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Qiang Gao
This process was moved to RT-priority queue when global oom-killer happened to boost the recovery of the system.. but it wasn't get properily dealt with. I still have no idea why where the problem is .. On Tue, Oct 23, 2012 at 12:40 PM, Balbir Singh wrote: > On Tue, Oct 23, 2012 at 9:05 AM,

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Qiang Gao
This process was moved to RT-priority queue when global oom-killer happened to boost the recovery of the system.. but it wasn't get properily dealt with. I still have no idea why where the problem is .. On Tue, Oct 23, 2012 at 12:40 PM, Balbir Singh bsinghar...@gmail.com wrote: On Tue, Oct 23,

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Michal Hocko
On Tue 23-10-12 11:35:52, Qiang Gao wrote: I'm sure this is a global-oom,not cgroup-oom. [the dmesg output in the end] Yes this is the global oom killer because: cglimit -M 700M ./tt then after global-oom,the process hangs.. 179184 pages RAM So you have ~700M of RAM so the memcg limit is

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Sha Zhengju
On 10/23/2012 11:35 AM, Qiang Gao wrote: information about the system is in the attach file information.txt I can not reproduce it in the upstream 3.6.0 kernel.. On Sat, Oct 20, 2012 at 12:04 AM, Michal Hockomho...@suse.cz wrote: On Wed 17-10-12 18:23:34, gaoqiang wrote: I looked up nothing

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Qiang Gao
this is just an example to show how to reproduce. actually,the first time I saw this situation was on a machine with 288G RAM with many tasks running and we limit 30G for each. but finanlly, no one exceeds this limit the the system oom. On Tue, Oct 23, 2012 at 4:35 PM, Michal Hocko

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Qiang Gao
global-oom is the right thing to do. but oom-killed-process hanging on do_exit is not the normal behavior On Tue, Oct 23, 2012 at 5:01 PM, Sha Zhengju handai@gmail.com wrote: On 10/23/2012 11:35 AM, Qiang Gao wrote: information about the system is in the attach file information.txt I can

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Michal Hocko
On Tue 23-10-12 17:08:40, Qiang Gao wrote: this is just an example to show how to reproduce. actually,the first time I saw this situation was on a machine with 288G RAM with many tasks running and we limit 30G for each. but finanlly, no one exceeds this limit the the system oom. Yes but

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Michal Hocko
On Tue 23-10-12 15:18:48, Qiang Gao wrote: This process was moved to RT-priority queue when global oom-killer happened to boost the recovery of the system.. Who did that? oom killer doesn't boost the priority (scheduling class) AFAIK. but it wasn't get properily dealt with. I still have no

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Qiang Gao
On Tue, Oct 23, 2012 at 5:50 PM, Michal Hocko mho...@suse.cz wrote: On Tue 23-10-12 15:18:48, Qiang Gao wrote: This process was moved to RT-priority queue when global oom-killer happened to boost the recovery of the system.. Who did that? oom killer doesn't boost the priority (scheduling

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Michal Hocko
On Tue 23-10-12 18:10:33, Qiang Gao wrote: On Tue, Oct 23, 2012 at 5:50 PM, Michal Hocko mho...@suse.cz wrote: On Tue 23-10-12 15:18:48, Qiang Gao wrote: This process was moved to RT-priority queue when global oom-killer happened to boost the recovery of the system.. Who did that? oom

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Balbir Singh
On Tue, Oct 23, 2012 at 3:45 PM, Michal Hocko mho...@suse.cz wrote: On Tue 23-10-12 18:10:33, Qiang Gao wrote: On Tue, Oct 23, 2012 at 5:50 PM, Michal Hocko mho...@suse.cz wrote: On Tue 23-10-12 15:18:48, Qiang Gao wrote: This process was moved to RT-priority queue when global oom-killer

Re: process hangs on do_exit when oom happens

2012-10-23 Thread Qiang Gao
On Wed, Oct 24, 2012 at 1:43 AM, Balbir Singh bsinghar...@gmail.com wrote: On Tue, Oct 23, 2012 at 3:45 PM, Michal Hocko mho...@suse.cz wrote: On Tue 23-10-12 18:10:33, Qiang Gao wrote: On Tue, Oct 23, 2012 at 5:50 PM, Michal Hocko mho...@suse.cz wrote: On Tue 23-10-12 15:18:48, Qiang Gao

Re: process hangs on do_exit when oom happens

2012-10-22 Thread Balbir Singh
On Tue, Oct 23, 2012 at 9:05 AM, Qiang Gao wrote: > information about the system is in the attach file "information.txt" > > I can not reproduce it in the upstream 3.6.0 kernel.. > > On Sat, Oct 20, 2012 at 12:04 AM, Michal Hocko wrote: >> On Wed 17-10-12 18:23:34, gaoqiang wrote: >>> I looked

Re: process hangs on do_exit when oom happens

2012-10-22 Thread Michal Hocko
On Mon 22-10-12 10:16:43, Qiang Gao wrote: > I don't know whether the process will exit finally, bug this stack lasts > for hours, which is obviously unnormal. > The situation: we use a command calld "cglimit" to fork-and-exec the > worker process,and the "cglimit" will > set some limitation on

Re: process hangs on do_exit when oom happens

2012-10-22 Thread Michal Hocko
On Mon 22-10-12 10:16:43, Qiang Gao wrote: I don't know whether the process will exit finally, bug this stack lasts for hours, which is obviously unnormal. The situation: we use a command calld cglimit to fork-and-exec the worker process,and the cglimit will set some limitation on the

Re: process hangs on do_exit when oom happens

2012-10-22 Thread Balbir Singh
On Tue, Oct 23, 2012 at 9:05 AM, Qiang Gao gaoqiangs...@gmail.com wrote: information about the system is in the attach file information.txt I can not reproduce it in the upstream 3.6.0 kernel.. On Sat, Oct 20, 2012 at 12:04 AM, Michal Hocko mho...@suse.cz wrote: On Wed 17-10-12 18:23:34,

Re: process hangs on do_exit when oom happens

2012-10-21 Thread Balbir Singh
On Mon, Oct 22, 2012 at 7:46 AM, Qiang Gao wrote: > I don't know whether the process will exit finally, bug this stack lasts > for hours, which is obviously unnormal. > The situation: we use a command calld "cglimit" to fork-and-exec the worker > process,and the "cglimit" will > set some

Re: process hangs on do_exit when oom happens

2012-10-21 Thread Qiang Gao
I don't know whether the process will exit finally, bug this stack lasts for hours, which is obviously unnormal. The situation: we use a command calld "cglimit" to fork-and-exec the worker process,and the "cglimit" will set some limitation on the worker with cgroup. for now,we limit the

Re: process hangs on do_exit when oom happens

2012-10-21 Thread Qiang Gao
I don't know whether the process will exit finally, bug this stack lasts for hours, which is obviously unnormal. The situation: we use a command calld cglimit to fork-and-exec the worker process,and the cglimit will set some limitation on the worker with cgroup. for now,we limit the memory,and

Re: process hangs on do_exit when oom happens

2012-10-21 Thread Balbir Singh
On Mon, Oct 22, 2012 at 7:46 AM, Qiang Gao gaoqiangs...@gmail.com wrote: I don't know whether the process will exit finally, bug this stack lasts for hours, which is obviously unnormal. The situation: we use a command calld cglimit to fork-and-exec the worker process,and the cglimit will

Re: process hangs on do_exit when oom happens

2012-10-19 Thread Michal Hocko
On Wed 17-10-12 18:23:34, gaoqiang wrote: > I looked up nothing useful with google,so I'm here for help.. > > when this happens: I use memcg to limit the memory use of a > process,and when the memcg cgroup was out of memory, > the process was oom-killed however,it cannot really complete the >

Re: process hangs on do_exit when oom happens

2012-10-19 Thread Michal Hocko
On Wed 17-10-12 18:23:34, gaoqiang wrote: I looked up nothing useful with google,so I'm here for help.. when this happens: I use memcg to limit the memory use of a process,and when the memcg cgroup was out of memory, the process was oom-killed however,it cannot really complete the

process hangs on do_exit when oom happens

2012-10-17 Thread gaoqiang
I looked up nothing useful with google,so I'm here for help.. when this happens: I use memcg to limit the memory use of a process,and when the memcg cgroup was out of memory, the process was oom-killed however,it cannot really complete the exiting. here is the some information OS

process hangs on do_exit when oom happens

2012-10-17 Thread gaoqiang
I looked up nothing useful with google,so I'm here for help.. when this happens: I use memcg to limit the memory use of a process,and when the memcg cgroup was out of memory, the process was oom-killed however,it cannot really complete the exiting. here is the some information OS