Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-20 Thread Tejun Heo
On Wed, Mar 16, 2016 at 06:04:56PM +0100, Peter Zijlstra wrote: > > Hmmm... I don't think it'd be safe to merge the two ops. Nothing > > guarantees that the RCU callback of cpu controller is called after the > > cgroup core one and cgroup core one would do use-after-free. Just > > changing offlin

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-19 Thread Tejun Heo
Hello, again. On Wed, Mar 16, 2016 at 07:24:14AM -0700, Tejun Heo wrote: > I could be mistaken but AFAICS there doesn't seem to be anything > requiring bothering with the more specialized exit methods. Given > that no css iteration is used and everything is lock protected, the Ooh, missed the rc

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-19 Thread Tejun Heo
Hello, Peter. On Wed, Mar 16, 2016 at 04:22:45PM +0100, Peter Zijlstra wrote: > > css_online() > > > > The css is now guaranteed to be visible for css_for_each*() > > iterations. This distinction exists because some controllers > > need to propagate state changes downwards requiring

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-19 Thread Peter Zijlstra
On Wed, Mar 16, 2016 at 09:50:06AM -0700, Tejun Heo wrote: > > static void cpu_cgroup_css_free(struct cgroup_subsys_state *css) > > { > > struct task_group *tg = css_tg(css); > > > > + /* > > +* Relies on the RCU grace period between css_released() and this. > > +*/ > > + sched_

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-19 Thread Tejun Heo
Hello, Peter. Sorry about the delay. On Mon, Mar 14, 2016 at 01:09:03PM +0100, Peter Zijlstra wrote: > On Mon, Mar 14, 2016 at 12:20:57PM +0100, Peter Zijlstra wrote: > > So I would suggest TJ to revert that patch and queue it for stable. > > > > It it clearly borken, because cgroup_exit() is ca

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-19 Thread Niklas Cassel
On 03/16/2016 04:22 PM, Peter Zijlstra wrote: > Subject: sched: Fix/cleanup cgroup teardown/init > > The cpu controller hasn't kept up with the various changes in the whole > cgroup initialization / destruction sequence, and commit 2e91fa7f6d45 > ("cgroup: keep zombies associated with their origin

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-19 Thread Peter Zijlstra
On Wed, Mar 16, 2016 at 07:24:14AM -0700, Tejun Heo wrote: > So, the problem here is that cpu is using css_offline to tear down the > css. perf shouldn't have the same problem as it destroys its css from > css_free. The distinctions among different callbacks evolved over > time and we need to upd

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-14 Thread Peter Zijlstra
On Mon, Mar 14, 2016 at 12:20:57PM +0100, Peter Zijlstra wrote: > So I would suggest TJ to revert that patch and queue it for stable. > > It it clearly borken, because cgroup_exit() is called from preemptible > context, so _obviously_ we can (and clearly will) schedule after that, > which is somew

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-14 Thread Peter Zijlstra
On Sat, Mar 12, 2016 at 06:42:57PM +0900, Kazuki Yamaguchi wrote: > 2e91fa7 cgroup: keep zombies associated with their original cgroups So the below hackery yields: [ 192.814857] [ cut here ] [ 192.820025] WARNING: CPU: 38 PID: 3539 at ../kernel/sched/fair.c:288 enqueu

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-12 Thread Peter Zijlstra
On Sat, Mar 12, 2016 at 06:42:57PM +0900, Kazuki Yamaguchi wrote: > I can reproduce it on QEMU (qemu-system-x86_64 -smp 2). Most excellent! I'll go have a play.

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-12 Thread Kazuki Yamaguchi
Hello, I got similar kernel crashes after the patch, which went to 4.4: 2e91fa7 cgroup: keep zombies associated with their original cgroups I was just about to report, but maybe this is related? ^^^[0.761718] BUG: unable to handle kernel NULL pointer dereference at 08b0 [

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-11 Thread Tejun Heo
Hello, Peter. On Thu, Mar 10, 2016 at 01:54:17PM +0100, Peter Zijlstra wrote: > > I've reproduced this on v4.4, but I've also managed to reproduce the bug > > after cherry-picking the following patches > > (all but one were marked for v4.4 stable): > > > > 6fe1f34 sched/cgroup: Fix cgroup entity

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-11 Thread Peter Zijlstra
On Fri, Mar 11, 2016 at 06:02:07PM +0100, Niklas Cassel wrote: > > Can you describe a reliable reproducer? > > Right now I just do a reboot loop, systemd creates cgroups during boot > this dump took 84 reboots. Harder to reproduce with more trace_printks. Bugger, I don't have a single system with

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-11 Thread Niklas Cassel
On 03/10/2016 01:54 PM, Peter Zijlstra wrote: > On Fri, Mar 04, 2016 at 11:41:17AM +0100, Niklas Cassel wrote: > >> A snippet of the trace_printks I've added when analyzing the problem. >> The prints show that a certain cfs_rq gets readded after it has been removed, >> and that update_blocked_aver

Re: [BUG] sched: leaf_cfs_rq_list use after free

2016-03-10 Thread Peter Zijlstra
On Fri, Mar 04, 2016 at 11:41:17AM +0100, Niklas Cassel wrote: > A snippet of the trace_printks I've added when analyzing the problem. > The prints show that a certain cfs_rq gets readded after it has been removed, > and that update_blocked_averages uses the cfs_rq which has already been freed: >