Re: [PATCH] cpuset: adjust the lock sequence when rebuilding the sched domains.
On Thu, 5 Sep 2019 at 16:43, Juri Lelli wrote: > > Hi, > > On 05/09/19 10:18, Chunyan Zhang wrote: > > From: Vincent Wang > > > > A deadlock issue is found when executing a cpu hotplug stress test on > > android phones with cpuset and scheduil enabled. > > > > When CPUx is plugged out, the hotplug thread that calls cpu_down() > > will hold cpu_hotplug_lock and wait the thread cpuhp/x to finish > > hotplug. If the core is the last one in a cluster, cpuhp/x have to > > call cpuhp_cpufreq_offline() and the kernel thread sugov need to exit > > for schedutil governor. The exit of sugov need to hold > > cgroup_threadgroup_rwsem in exit_signals(). For example: > > > > PID: 150TASK: ffc0b9cad080 CPU: 0 COMMAND: "sprdhotplug" > > #0 [ff8009fcb9d0] __switch_to at ff80080858f0 > > #1 [ff8009fcb9f0] __schedule at ff80089f185c > > #2 [ff8009fcba80] schedule at ff80089f1b84 > > #3 [ff8009fcbaa0] schedule_timeout at ff80089f5124 > > #4 [ff8009fcbb40] wait_for_common at ff80089f2944 > > #5 [ff8009fcbbe0] wait_for_completion at ff80089f29a4 > > #6 [ff8009fcbc00] __cpuhp_kick_ap at ff80080ab030 > > #7 [ff8009fcbc20] cpuhp_kick_ap_work at ff80080ab154 > > #8 [ff8009fcbc70] _cpu_down at ff80089ee19c > > #9 [ff8009fcbcd0] cpu_down at ff80080ac144 > > > > PID: 26 TASK: ffc0bbe22080 CPU: 3 COMMAND: "cpuhp/3" > > #0 [ff8009693a30] __switch_to at ff80080858f0 > > #1 [ff8009693a50] __schedule at ff80089f185c > > #2 [ff8009693ae0] schedule at ff80089f1b84 > > #3 [ff8009693b00] schedule_timeout at ff80089f5124 > > #4 [ff8009693ba0] wait_for_common at ff80089f2944 > > #5 [ff8009693c40] wait_for_completion at ff80089f29a4 > > #6 [ff8009693c60] kthread_stop at ff80080ccd2c > > #7 [ff8009693c90] sugov_exit at ff8008102134 > > #8 [ff8009693cc0] cpufreq_exit_governor at ff80086c03bc > > #9 [ff8009693ce0] cpufreq_offline at ff80086c0634 > > > > PID: 13819 TASK: ffc0affb6080 CPU: 0 COMMAND: "sugov:3" > > #0 [ff800ee73c30] __switch_to at ff80080858f0 > > #1 [ff800ee73c50] __schedule at ff80089f185c > > #2 [ff800ee73ce0] schedule at ff80089f1b84 > > #3 [ff800ee73d00] rwsem_down_read_failed at ff80089f49d0 > > #4 [ff800ee73d80] __percpu_down_read at ff8008102ebc > > #5 [ff800ee73da0] exit_signals at ff80080bbd24 > > #6 [ff800ee73de0] do_exit at ff80080ae65c > > #7 [ff800ee73e60] kthread at ff80080cc550 > > > > Sometimes cgroup_threadgroup_rwsem is hold by another thread, for > > example Binder:681_2 on android, it wants to hold cpuset_mutex: > > > > PID: 732TASK: ffc09668b080 CPU: 2 COMMAND: "Binder:681_2" > > #0 [ff800cb7b8c0] __switch_to at ff80080858f0 > > #1 [ff800cb7b8e0] __schedule at ff80089f185c > > #2 [ff800cb7b970] schedule at ff80089f1b84 > > #3 [ff800cb7b990] schedule_preempt_disabled at ff80089f205c > > #4 [ff800cb7b9a0] __mutex_lock at ff80089f3118 > > #5 [ff800cb7ba40] __mutex_lock_slowpath at ff80089f3230 > > #6 [ff800cb7ba60] mutex_lock at ff80089f3278 > > #7 [ff800cb7ba80] cpuset_can_attach at ff8008152f84 > > #8 [ff800cb7bae0] cgroup_migrate_execute at ff800814ada8 > > > > On android, a thread kworker/3:0 will hold cpuset_mutex in > > rebuild_sched_domains() and want to hold cpu_hotplug_lock which > > is already hold by the hotplug thread. > > > > PID: 4847 TASK: ffc031a6a080 CPU: 3 COMMAND: "kworker/3:0" > > #0 [ff8016fd3ad0] __switch_to at ff80080858f0 > > #1 [ff8016fd3af0] __schedule at ff80089f185c > > #2 [ff8016fd3b80] schedule at ff80089f1b84 > > #3 [ff8016fd3ba0] rwsem_down_read_failed at ff80089f49d0 > > #4 [ff8016fd3c20] __percpu_down_read at ff8008102ebc > > #5 [ff8016fd3c40] cpus_read_lock at ff80080aa59c > > #6 [ff8016fd3c50] rebuild_sched_domains_locked at ff80081522a8 > > > > In order to fix the deadlock, this patch will adjust the lock sequence > > when rebuilding sched domains. After stress tests, it works well. > > > > Signed-off-by: Vincent Wang > > Signed-off-by: Chunyan Zhang > > --- > > kernel/cgroup/cpuset.c | 9 - > > 1 file changed, 4 insertions(+), 5 deletions(-) > > > > diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c > > index 5aa37531ce76..ef10d276da22 100644 > > --- a/kernel/cgroup/cpuset.c > > +++ b/kernel/cgroup/cpuset.c > > @@ -912,7 +912,6 @@ static void rebuild_sched_domains_locked(void) > > int ndoms; > > > > lockdep_assert_held(_mutex); > > - get_online_cpus(); > > > > /* > >* We have raced with CPU hotplug. Don't do anything to avoid > > @@ -921,19 +920,17 @@ static void rebuild_sched_domains_locked(void) > >*/ > > if (!top_cpuset.nr_subparts_cpus && > >
Re: [PATCH] cpuset: adjust the lock sequence when rebuilding the sched domains.
Hi, On 05/09/19 10:18, Chunyan Zhang wrote: > From: Vincent Wang > > A deadlock issue is found when executing a cpu hotplug stress test on > android phones with cpuset and scheduil enabled. > > When CPUx is plugged out, the hotplug thread that calls cpu_down() > will hold cpu_hotplug_lock and wait the thread cpuhp/x to finish > hotplug. If the core is the last one in a cluster, cpuhp/x have to > call cpuhp_cpufreq_offline() and the kernel thread sugov need to exit > for schedutil governor. The exit of sugov need to hold > cgroup_threadgroup_rwsem in exit_signals(). For example: > > PID: 150TASK: ffc0b9cad080 CPU: 0 COMMAND: "sprdhotplug" > #0 [ff8009fcb9d0] __switch_to at ff80080858f0 > #1 [ff8009fcb9f0] __schedule at ff80089f185c > #2 [ff8009fcba80] schedule at ff80089f1b84 > #3 [ff8009fcbaa0] schedule_timeout at ff80089f5124 > #4 [ff8009fcbb40] wait_for_common at ff80089f2944 > #5 [ff8009fcbbe0] wait_for_completion at ff80089f29a4 > #6 [ff8009fcbc00] __cpuhp_kick_ap at ff80080ab030 > #7 [ff8009fcbc20] cpuhp_kick_ap_work at ff80080ab154 > #8 [ff8009fcbc70] _cpu_down at ff80089ee19c > #9 [ff8009fcbcd0] cpu_down at ff80080ac144 > > PID: 26 TASK: ffc0bbe22080 CPU: 3 COMMAND: "cpuhp/3" > #0 [ff8009693a30] __switch_to at ff80080858f0 > #1 [ff8009693a50] __schedule at ff80089f185c > #2 [ff8009693ae0] schedule at ff80089f1b84 > #3 [ff8009693b00] schedule_timeout at ff80089f5124 > #4 [ff8009693ba0] wait_for_common at ff80089f2944 > #5 [ff8009693c40] wait_for_completion at ff80089f29a4 > #6 [ff8009693c60] kthread_stop at ff80080ccd2c > #7 [ff8009693c90] sugov_exit at ff8008102134 > #8 [ff8009693cc0] cpufreq_exit_governor at ff80086c03bc > #9 [ff8009693ce0] cpufreq_offline at ff80086c0634 > > PID: 13819 TASK: ffc0affb6080 CPU: 0 COMMAND: "sugov:3" > #0 [ff800ee73c30] __switch_to at ff80080858f0 > #1 [ff800ee73c50] __schedule at ff80089f185c > #2 [ff800ee73ce0] schedule at ff80089f1b84 > #3 [ff800ee73d00] rwsem_down_read_failed at ff80089f49d0 > #4 [ff800ee73d80] __percpu_down_read at ff8008102ebc > #5 [ff800ee73da0] exit_signals at ff80080bbd24 > #6 [ff800ee73de0] do_exit at ff80080ae65c > #7 [ff800ee73e60] kthread at ff80080cc550 > > Sometimes cgroup_threadgroup_rwsem is hold by another thread, for > example Binder:681_2 on android, it wants to hold cpuset_mutex: > > PID: 732TASK: ffc09668b080 CPU: 2 COMMAND: "Binder:681_2" > #0 [ff800cb7b8c0] __switch_to at ff80080858f0 > #1 [ff800cb7b8e0] __schedule at ff80089f185c > #2 [ff800cb7b970] schedule at ff80089f1b84 > #3 [ff800cb7b990] schedule_preempt_disabled at ff80089f205c > #4 [ff800cb7b9a0] __mutex_lock at ff80089f3118 > #5 [ff800cb7ba40] __mutex_lock_slowpath at ff80089f3230 > #6 [ff800cb7ba60] mutex_lock at ff80089f3278 > #7 [ff800cb7ba80] cpuset_can_attach at ff8008152f84 > #8 [ff800cb7bae0] cgroup_migrate_execute at ff800814ada8 > > On android, a thread kworker/3:0 will hold cpuset_mutex in > rebuild_sched_domains() and want to hold cpu_hotplug_lock which > is already hold by the hotplug thread. > > PID: 4847 TASK: ffc031a6a080 CPU: 3 COMMAND: "kworker/3:0" > #0 [ff8016fd3ad0] __switch_to at ff80080858f0 > #1 [ff8016fd3af0] __schedule at ff80089f185c > #2 [ff8016fd3b80] schedule at ff80089f1b84 > #3 [ff8016fd3ba0] rwsem_down_read_failed at ff80089f49d0 > #4 [ff8016fd3c20] __percpu_down_read at ff8008102ebc > #5 [ff8016fd3c40] cpus_read_lock at ff80080aa59c > #6 [ff8016fd3c50] rebuild_sched_domains_locked at ff80081522a8 > > In order to fix the deadlock, this patch will adjust the lock sequence > when rebuilding sched domains. After stress tests, it works well. > > Signed-off-by: Vincent Wang > Signed-off-by: Chunyan Zhang > --- > kernel/cgroup/cpuset.c | 9 - > 1 file changed, 4 insertions(+), 5 deletions(-) > > diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c > index 5aa37531ce76..ef10d276da22 100644 > --- a/kernel/cgroup/cpuset.c > +++ b/kernel/cgroup/cpuset.c > @@ -912,7 +912,6 @@ static void rebuild_sched_domains_locked(void) > int ndoms; > > lockdep_assert_held(_mutex); > - get_online_cpus(); > > /* >* We have raced with CPU hotplug. Don't do anything to avoid > @@ -921,19 +920,17 @@ static void rebuild_sched_domains_locked(void) >*/ > if (!top_cpuset.nr_subparts_cpus && > !cpumask_equal(top_cpuset.effective_cpus, cpu_active_mask)) > - goto out; > + return; > > if (top_cpuset.nr_subparts_cpus && > !cpumask_subset(top_cpuset.effective_cpus, cpu_active_mask)) > - goto out;
[PATCH] cpuset: adjust the lock sequence when rebuilding the sched domains.
From: Vincent Wang A deadlock issue is found when executing a cpu hotplug stress test on android phones with cpuset and scheduil enabled. When CPUx is plugged out, the hotplug thread that calls cpu_down() will hold cpu_hotplug_lock and wait the thread cpuhp/x to finish hotplug. If the core is the last one in a cluster, cpuhp/x have to call cpuhp_cpufreq_offline() and the kernel thread sugov need to exit for schedutil governor. The exit of sugov need to hold cgroup_threadgroup_rwsem in exit_signals(). For example: PID: 150TASK: ffc0b9cad080 CPU: 0 COMMAND: "sprdhotplug" #0 [ff8009fcb9d0] __switch_to at ff80080858f0 #1 [ff8009fcb9f0] __schedule at ff80089f185c #2 [ff8009fcba80] schedule at ff80089f1b84 #3 [ff8009fcbaa0] schedule_timeout at ff80089f5124 #4 [ff8009fcbb40] wait_for_common at ff80089f2944 #5 [ff8009fcbbe0] wait_for_completion at ff80089f29a4 #6 [ff8009fcbc00] __cpuhp_kick_ap at ff80080ab030 #7 [ff8009fcbc20] cpuhp_kick_ap_work at ff80080ab154 #8 [ff8009fcbc70] _cpu_down at ff80089ee19c #9 [ff8009fcbcd0] cpu_down at ff80080ac144 PID: 26 TASK: ffc0bbe22080 CPU: 3 COMMAND: "cpuhp/3" #0 [ff8009693a30] __switch_to at ff80080858f0 #1 [ff8009693a50] __schedule at ff80089f185c #2 [ff8009693ae0] schedule at ff80089f1b84 #3 [ff8009693b00] schedule_timeout at ff80089f5124 #4 [ff8009693ba0] wait_for_common at ff80089f2944 #5 [ff8009693c40] wait_for_completion at ff80089f29a4 #6 [ff8009693c60] kthread_stop at ff80080ccd2c #7 [ff8009693c90] sugov_exit at ff8008102134 #8 [ff8009693cc0] cpufreq_exit_governor at ff80086c03bc #9 [ff8009693ce0] cpufreq_offline at ff80086c0634 PID: 13819 TASK: ffc0affb6080 CPU: 0 COMMAND: "sugov:3" #0 [ff800ee73c30] __switch_to at ff80080858f0 #1 [ff800ee73c50] __schedule at ff80089f185c #2 [ff800ee73ce0] schedule at ff80089f1b84 #3 [ff800ee73d00] rwsem_down_read_failed at ff80089f49d0 #4 [ff800ee73d80] __percpu_down_read at ff8008102ebc #5 [ff800ee73da0] exit_signals at ff80080bbd24 #6 [ff800ee73de0] do_exit at ff80080ae65c #7 [ff800ee73e60] kthread at ff80080cc550 Sometimes cgroup_threadgroup_rwsem is hold by another thread, for example Binder:681_2 on android, it wants to hold cpuset_mutex: PID: 732TASK: ffc09668b080 CPU: 2 COMMAND: "Binder:681_2" #0 [ff800cb7b8c0] __switch_to at ff80080858f0 #1 [ff800cb7b8e0] __schedule at ff80089f185c #2 [ff800cb7b970] schedule at ff80089f1b84 #3 [ff800cb7b990] schedule_preempt_disabled at ff80089f205c #4 [ff800cb7b9a0] __mutex_lock at ff80089f3118 #5 [ff800cb7ba40] __mutex_lock_slowpath at ff80089f3230 #6 [ff800cb7ba60] mutex_lock at ff80089f3278 #7 [ff800cb7ba80] cpuset_can_attach at ff8008152f84 #8 [ff800cb7bae0] cgroup_migrate_execute at ff800814ada8 On android, a thread kworker/3:0 will hold cpuset_mutex in rebuild_sched_domains() and want to hold cpu_hotplug_lock which is already hold by the hotplug thread. PID: 4847 TASK: ffc031a6a080 CPU: 3 COMMAND: "kworker/3:0" #0 [ff8016fd3ad0] __switch_to at ff80080858f0 #1 [ff8016fd3af0] __schedule at ff80089f185c #2 [ff8016fd3b80] schedule at ff80089f1b84 #3 [ff8016fd3ba0] rwsem_down_read_failed at ff80089f49d0 #4 [ff8016fd3c20] __percpu_down_read at ff8008102ebc #5 [ff8016fd3c40] cpus_read_lock at ff80080aa59c #6 [ff8016fd3c50] rebuild_sched_domains_locked at ff80081522a8 In order to fix the deadlock, this patch will adjust the lock sequence when rebuilding sched domains. After stress tests, it works well. Signed-off-by: Vincent Wang Signed-off-by: Chunyan Zhang --- kernel/cgroup/cpuset.c | 9 - 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c index 5aa37531ce76..ef10d276da22 100644 --- a/kernel/cgroup/cpuset.c +++ b/kernel/cgroup/cpuset.c @@ -912,7 +912,6 @@ static void rebuild_sched_domains_locked(void) int ndoms; lockdep_assert_held(_mutex); - get_online_cpus(); /* * We have raced with CPU hotplug. Don't do anything to avoid @@ -921,19 +920,17 @@ static void rebuild_sched_domains_locked(void) */ if (!top_cpuset.nr_subparts_cpus && !cpumask_equal(top_cpuset.effective_cpus, cpu_active_mask)) - goto out; + return; if (top_cpuset.nr_subparts_cpus && !cpumask_subset(top_cpuset.effective_cpus, cpu_active_mask)) - goto out; + return; /* Generate domain masks and attrs */ ndoms = generate_sched_domains(, ); /* Have scheduler rebuild the domains */ partition_sched_domains(ndoms, doms, attr); -out: -
