On Tue, May 27, 2025 at 4:01 PM Uday Shankar <ushan...@purestorage.com> wrote: > > Currently, ublk_drv associates to each hardware queue (hctx) a unique > task (called the queue's ubq_daemon) which is allowed to issue > COMMIT_AND_FETCH commands against the hctx. If any other task attempts > to do so, the command fails immediately with EINVAL. When considered > together with the block layer architecture, the result is that for each > CPU C on the system, there is a unique ublk server thread which is > allowed to handle I/O submitted on CPU C. This can lead to suboptimal > performance under imbalanced load generation. For an extreme example, > suppose all the load is generated on CPUs mapping to a single ublk > server thread. Then that thread may be fully utilized and become the > bottleneck in the system, while other ublk server threads are totally > idle. > > This issue can also be addressed directly in the ublk server without > kernel support by having threads dequeue I/Os and pass them around to > ensure even load. But this solution requires inter-thread communication > at least twice for each I/O (submission and completion), which is > generally a bad pattern for performance. The problem gets even worse > with zero copy, as more inter-thread communication would be required to > have the buffer register/unregister calls to come from the correct > thread. > > Therefore, address this issue in ublk_drv by allowing each I/O to have > its own daemon task. Two I/Os in the same queue are now allowed to be > serviced by different daemon tasks - this was not possible before. > Imbalanced load can then be balanced across all ublk server threads by > having the ublk server threads issue FETCH_REQs in a round-robin manner. > As a small toy example, consider a system with a single ublk device > having 2 queues, each of depth 4. A ublk server having 4 threads could > issue its FETCH_REQs against this device as follows (where each entry is > the qid,tag pair that the FETCH_REQ targets): > > ublk server thread: T0 T1 T2 T3 > 0,0 0,1 0,2 0,3 > 1,3 1,0 1,1 1,2 > > This setup allows for load that is concentrated on one hctx/ublk_queue > to be spread out across all ublk server threads, alleviating the issue > described above. > > Add the new UBLK_F_PER_IO_DAEMON feature to ublk_drv, which ublk servers > can use to essentially test for the presence of this change and tailor > their behavior accordingly. > > Signed-off-by: Uday Shankar <ushan...@purestorage.com> > Reviewed-by: Caleb Sander Mateos <csan...@purestorage.com>
Still looks good to me. Best, Caleb
