On Tue, Jan 12, 2021 at 07:57:26AM -0700, Jens Axboe wrote: > On 1/11/21 12:21 PM, Valentin Schneider wrote: > > On 11/01/21 18:16, Peter Zijlstra wrote: > >> Sadly it appears like io_uring() uses kthread_create_on_cpu() without > >> then having any hotplug crud on, so that needs additinoal frobbing. > >> > > > > I noticed that as well sometime ago, and I believed then (still do) this > > usage is broken. I don't think usage of kthread_create_on_cpu() outside > > of smpboot makes sense, because without any hotplug step to park the > > thread, its affinity can end up being reset after its dedicated CPU gets > > offlined. > > > > I'm clueless about io_uring, but if it *actually* has a good reason to > > use some pcpu kthreads (it seems it doesn't have to be on all CPUs?), > > then it needs to register some hotplug step to park them / do something > > sensible on hotplug. > > For io_uring, it's purely used by the SQPOLL mode, which sets aside a > kernel thread for submissions so the application doesn't have to do > anything but write new SQE entries to submit. The thread then notices > these, and submits them. There's an option to affinitize that thread > to a single CPU, which often makes sense for setups like that. Think > of it like a strong hint.
OK, that matches what I could make of it earlier today. > Things aren't going to break if this CPU goes away and we end up being > affinitized to some other CPU, though it is suboptimal. So I guess we > might need some notifiers to ensure that we reset the CPU back again > if it's gone offline+online again? I can take a look at that. Indeed, that would make sense. Thanks!
