On 7/9/19 3:42 PM, Peter Zijlstra wrote: > On Tue, Jul 09, 2019 at 03:24:36PM +0200, luca abeni wrote: >> Hi Peter, >> >> On Mon, 8 Jul 2019 15:55:36 +0200 >> Peter Zijlstra <pet...@infradead.org> wrote: >> >>> On Mon, May 06, 2019 at 06:48:33AM +0200, Luca Abeni wrote: >>>> @@ -1223,8 +1250,17 @@ static void update_curr_dl(struct rq *rq) >>>> dl_se->dl_overrun = 1; >>>> >>>> __dequeue_task_dl(rq, curr, 0); >>>> - if (unlikely(dl_se->dl_boosted >>>> || !start_dl_timer(curr))) >>>> + if (unlikely(dl_se->dl_boosted >>>> || !start_dl_timer(curr))) { enqueue_task_dl(rq, curr, >>>> ENQUEUE_REPLENISH); +#ifdef CONFIG_SMP >>>> + } else if (dl_se->dl_adjust) { >>>> + if (rq->migrating_task == NULL) { >>>> + queue_balance_callback(rq, >>>> &per_cpu(dl_migrate_head, rq->cpu), migrate_dl_task); >>> >>> I'm not entirely sure about this one. >>> >>> That is, we only do those callbacks from: >>> >>> schedule_tail() >>> __schedule() >>> rt_mutex_setprio() >>> __sched_setscheduler() >>> >>> and the above looks like it can happen outside of those. >> >> Sorry, I did not know the constraints or requirements for using >> queue_balance_callback()... >> >> I used it because I wanted to trigger a migration from >> update_curr_dl(), but invoking double_lock_balance() from this function >> obviously resulted in a warning. So, I probably misunderstood the >> purpose of the balance callback API, and I misused it. >> >> What would have been the "right way" to trigger a migration for a task >> when it is throttled? > > I'm thinking we'll end up in schedule() pretty soon after a throttle to > make 'current' go away, right? We could put the queue_balance_callback() > in dequeue_task_dl() or something.
Is this what you are concerned about? (2 Cpus (CPU1, CPU2), 4 deadline task (thread0-X)) with @@ -1137,6 +1137,13 @@ static inline void rq_pin_lock(struct rq *rq, struct rq_flags *rf) rf->cookie = lockdep_pin_lock(&rq->lock); #ifdef CONFIG_SCHED_DEBUG +#ifdef CONFIG_SMP + /* + * There should not be pending callbacks at the start of rq_lock(); + * all sites that handle them flush them at the end. + */ + WARN_ON_ONCE(rq->balance_callback); +#endif [ 87.251233] *** <--- queue_balance_callback(migrate_dl_task) p=[thread0-2 3626] on CPU1 [ 87.251237] *** <--- queue_balance_callback(migrate_dl_task) p=[thread0-3 3627] on CPU2 [ 87.251261] WARNING: CPU: 2 PID: 3627 at kernel/sched/sched.h:1145 __schedule+0x56c/0x690 [ 87.259173] WARNING: CPU: 1 PID: 3626 at kernel/sched/sched.h:1145 try_to_wake_up+0x68c/0x778 [ 87.615871] WARNING: CPU: 1 PID: 3626 at kernel/sched/sched.h:1145 scheduler_tick+0x110/0x118 [ 87.615882] WARNING: CPU: 2 PID: 3616 at kernel/sched/sched.h:1145 task_rq_lock+0xe8/0xf0 [ 88.012571] WARNING: CPU: 1 PID: 3626 at kernel/sched/sched.h:1145 update_blocked_averages+0x924/0x998 [ 88.176844] WARNING: CPU: 2 PID: 3616 at kernel/sched/sched.h:1145 load_balance+0x4d0/0xbc0 [ 88.381905] WARNING: CPU: 2 PID: 3616 at kernel/sched/sched.h:1145 load_balance+0x7d8/0xbc0 [ 88.586991] *** ---> migrate_dl_task() p=[thread0-3 3627] to CPU1 [ 88.587008] *** ---> migrate_dl_task() p=[thread0-2 3626] to CPU1 [ 89.335307] *** <--- queue_balance_callback(migrate_dl_task) p=[thread0-0 3624] on CPU2 [ 89.343252] *** ---> migrate_dl_task() p=[thread0-0 3624] to CPU1 [ 89.379309] *** <--- queue_balance_callback(migrate_dl_task) p=[thread0-1 3625] on CPU2 [ 89.387249] *** ---> migrate_dl_task() p=[thread0-1 3625] to CPU1 [ 89.591323] *** <--- queue_balance_callback(migrate_dl_task) p=[thread0-3 3627] on CPU2 [ 89.599263] *** ---> migrate_dl_task() p=[thread0-3 3627] to CPU2