On Tue, Jul 30, 2013 at 9:28 AM, Eric W. Biederman <[email protected]> wrote: > > [email protected] (Eric W. Biederman) writes: > > Ok. I have been trying for an hour and I have not been able to > reproduce the weird hang with the memcg, and it used to be something I > could reproduce trivially. So it appears the patch below is the fix. > > After I sleep I will see if I can turn it into a proper patch.
Contributing with another data point: I am seeing similar issues with un-killable tasks inside LXC containers on a vanilla 3.8.11 kernel. The stack from zombie tasks look like this: # cat /proc/12499/stack [<ffffffff81186226>] __mem_cgroup_try_charge+0xa96/0xbf0 [<ffffffff8118670b>] __mem_cgroup_try_charge_swapin+0xab/0xd0 [<ffffffff8118678d>] mem_cgroup_try_charge_swapin+0x5d/0x70 [<ffffffff811524f5>] handle_pte_fault+0x315/0xac0 [<ffffffff81152f11>] handle_mm_fault+0x271/0x3d0 [<ffffffff815bbf3b>] __do_page_fault+0x20b/0x4c0 [<ffffffff815bc1fe>] do_page_fault+0xe/0x10 [<ffffffff815b8718>] page_fault+0x28/0x30 [<ffffffff81056327>] mm_release+0x127/0x140 [<ffffffff8105ece1>] do_exit+0x171/0xa70 [<ffffffff8105f635>] do_group_exit+0x55/0xd0 [<ffffffff8106fa8f>] get_signal_to_deliver+0x23f/0x5d0 [<ffffffff81014402>] do_signal+0x42/0x600 [<ffffffff81014a48>] do_notify_resume+0x88/0xc0 [<ffffffff815c0b92>] int_signal+0x12/0x17 [<ffffffffffffffff>] 0xffffffffffffffff Same symptoms that Eric described: a race condition in memcg when there is a page fault and the process is exiting. I went ahead and reproduced the bug described earlier here on the same 3.8.11 kernel, also using the Mesos framework (http://mesos.apache.org/) memory Ballooning tests. The call trace from zombie tasks in this case look very similar: # cat /proc/22827/stack [<ffffffff81186280>] __mem_cgroup_try_charge+0xaf0/0xbf0 [<ffffffff8118670b>] __mem_cgroup_try_charge_swapin+0xab/0xd0 [<ffffffff8118678d>] mem_cgroup_try_charge_swapin+0x5d/0x70 [<ffffffff811524f5>] handle_pte_fault+0x315/0xac0 [<ffffffff81152f11>] handle_mm_fault+0x271/0x3d0 [<ffffffff815bbf3b>] __do_page_fault+0x20b/0x4c0 [<ffffffff815bc1fe>] do_page_fault+0xe/0x10 [<ffffffff815b8718>] page_fault+0x28/0x30 [<ffffffff81056327>] mm_release+0x127/0x140 [<ffffffff8105ece1>] do_exit+0x171/0xa70 [<ffffffff8105f635>] do_group_exit+0x55/0xd0 [<ffffffff8106fa8f>] get_signal_to_deliver+0x23f/0x5d0 [<ffffffff81014402>] do_signal+0x42/0x600 [<ffffffff81014a48>] do_notify_resume+0x88/0xc0 [<ffffffff815c0b92>] int_signal+0x12/0x17 [<ffffffffffffffff>] 0xffffffffffffffff Then, I applied Eric's patch below, and I can't reproduce the problem anymore. Before the patch, it was very easy to reproduce it with some extra memory pressure from other processes in the instance (increasing the probability of page faults when processes are exiting). We also tried a vanilla 3.11.1 kernel, and we could reproduce the bug on it pretty easily. > > Eric > > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > > index 00a7a66..5998a57 100644 > > --- a/mm/memcontrol.c > > +++ b/mm/memcontrol.c > > @@ -1792,16 +1792,6 @@ static void mem_cgroup_out_of_memory(struct > > mem_cgroup *memcg, gfp_t gfp_mask, > > unsigned int points = 0; > > struct task_struct *chosen = NULL; > > > > - /* > > - * If current has a pending SIGKILL or is exiting, then > > automatically > > - * select it. The goal is to allow it to allocate so that it may > > - * quickly exit and free its memory. > > - */ > > - if (fatal_signal_pending(current) || current->flags & PF_EXITING) { > > - set_thread_flag(TIF_MEMDIE); > > - return; > > - } > > - > > check_panic_on_oom(CONSTRAINT_MEMCG, gfp_mask, order, NULL); > > totalpages = mem_cgroup_get_limit(memcg) >> PAGE_SHIFT ? : 1; > > for_each_mem_cgroup_tree(iter, memcg) { > > @@ -2220,7 +2210,15 @@ static bool mem_cgroup_handle_oom(struct mem_cgroup > > *memcg, gfp_t mask, > > mem_cgroup_oom_notify(memcg); > > spin_unlock(&memcg_oom_lock); > > > > - if (need_to_kill) { > > + /* > > + * If current has a pending SIGKILL or is exiting, then > > automatically > > + * select it. The goal is to allow it to allocate so that it may > > + * quickly exit and free its memory. > > + */ > > + if (fatal_signal_pending(current) || current->flags & PF_EXITING) { > > + set_thread_flag(TIF_MEMDIE); > > + finish_wait(&memcg_oom_waitq, &owait.wait); > > + } else if (need_to_kill) { > > finish_wait(&memcg_oom_waitq, &owait.wait); > > mem_cgroup_out_of_memory(memcg, mask, order); > > } else { > -- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
