sched/core: Call wakeup_preempt() for queued tasks during class change

In the Linux kernel, the following vulnerability has been resolved: sched/core: Call wakeup_preempt() for queued tasks during class change Paul reported recent RCU changes exposed a lost resched where CPU idles with a rcutorture thread waiting while being completely runnable. Dumping the rq state at the time of idling with runnable tasks showed the following: rq->next_class = &fair_sched_class rq->nr_running = 1 rq->cfs.h_nr_queued = 0 rq->rt.rt_nr_running = 1 Although the waiting task was an RT task, "rq->next_class" did not point to rt_sched_class and was stuck at the lower priority fair_sched_class. Inspecting sched_change_end() revealed the following race with __sched_setscheduler() for queued, but non-running tasks: CPU0 CPU1 ==== ==== __schedule() /* * rq->donor is a fair task that blocks. * rq->next_class = &fair_sched_class */ rq_lock(rq0) pick_next_task() /* nr_running = 0 */ pick_task_fair() sched_balance_newidle() raw_spin_rq_unlock(rq0) rq_lock(rq0) activate_task(rq0, p) /* p is a fair task */ rq0->nr_running++; rq0->cfs.h_nr_queued++; wakeup_preempt(rq, p) rq->next_class = &fair_sched_class resched_curr() rq_unlock(rq0) __sched_setscheduler(p) /* p is moved to SCHED_RR */ task_rq_lock(p) sched_change_begin() rq0->cfs.h_nr_queued--; rq0->rt.rt_nr_running++; sched_change_end() /* * p->on_rq = 1 (ctx->queued) * but p is not rq->donor * (!ctx->running) hence * resched_curr() is and * rq->next_class upgrade is * skipped. */ task_rq_unlock() raw_spin_rq_lock(rq0) /* * rq->nr_running = 1 * rq->cfs.h_nr_queued = 0 * rq->rt.rt_nr_running = 1 * rq->next_class = &fair_sched_class */ if (rq->cfs.h_nr_queued && ...) /* Above is false */ if (rq_modified_above(this_rq, &fair_sched_class)) /* Also false since rq->next_class is still fair */ /* * sched_balance_newidle() is considered to * have failed to pull any fair tasks and no * other wakeups were detected. */ return NULL; /* pick_task_fair() */ pick_task_idle() return rq->idle; next = rq->idle; /* Preemption indicators are lost */ clear_tsk_need_resched(prev); clear_preempt_need_resched(); rq->curr = rq->idle; !!! rq->idle switches in with rq->running == 1 but !need_resched() !!! In simple words, enqueue, followed by a class change of the newly queued task goes unnoticed in rq->next_class indicator and sched_balance_newidle() sees no fair task, nor any higher priority task enqueued. The above sequence will lead to CPU idling with a runnable task on the CPU. Call wakeup_preempt() for ctx->queued cases in sched_change_end() to re-evaluate rq->next_class after class change (ENQUEUE_CLASS) and set the NEED_RESCHED indicator accordingly. Newidle balance will correctly see an enqueue of a higher priority t ---truncated---