On Mon, Oct 08, 2018 at 02:58:21PM +0800, Dongli Zhang wrote:
> I got the same result when emulating nvme with qemu: the VM has 12 cpu, while
> the num_queues of nvme is 8.
>
> # uname -r
> 4.14.1
> # ll /sys/block/nvme*n1/mq/*/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/0/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/1/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/2/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/3/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/4/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/5/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/6/cpu_list
>
>
> # uname -r
> 4.18.10
> # ll /sys/block/nvme*n1/mq/*/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/0/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/1/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/2/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/3/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/4/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/5/cpu_list
> -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/6/cpu_list
>
> From below qemu source code, when n->num_queues is 8, the handler of
> NVME_FEAT_NUM_QUEUES returns 0x60006.
>
> 719 static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest
> *req)
> 720 {
> 721 uint32_t dw10 = le32_to_cpu(cmd->cdw10);
> 722 uint32_t dw11 = le32_to_cpu(cmd->cdw11);
> 723
> 724 switch (dw10) {
> 725 case NVME_VOLATILE_WRITE_CACHE:
> 726 blk_set_enable_write_cache(n->conf.blk, dw11 & 1);
> 727 break;
> 728 case NVME_NUMBER_OF_QUEUES:
> 729 trace_nvme_setfeat_numq((dw11 & 0xFFFF) + 1,
> 730 ((dw11 >> 16) & 0xFFFF) + 1,
> 731 n->num_queues - 1, n->num_queues - 1);
> 732 req->cqe.result =
> 733 cpu_to_le32((n->num_queues - 2) | ((n->num_queues - 2) <<
> 16));
> ----> returns 0x60006 when num_queues is 8.
>
>
> Finally, nr_io_queues is set to 6+1=7 in nvme_set_queue_count() in VM kernel.
>
> I do not know how to paraphrase this in the world of nvme.
>
> Dongli Zhang
>
> On 10/08/2018 01:59 PM, Dongli Zhang wrote:
> > I can reproduce with qemu:
> >
> > # ls /sys/block/nvme*n1/mq/*/cpu_list
> > /sys/block/nvme0n1/mq/0/cpu_list
> > /sys/block/nvme0n1/mq/1/cpu_list
> > /sys/block/nvme0n1/mq/2/cpu_list
> > /sys/block/nvme0n1/mq/3/cpu_list
> > /sys/block/nvme0n1/mq/4/cpu_list
> > /sys/block/nvme0n1/mq/5/cpu_list
> > /sys/block/nvme0n1/mq/6/cpu_list
> >
> > Here is the qemu cmdline emulating 8-queue nvme while the VM has 12 cpu:
> >
> > # qemu-system-x86_64 -m 4096 -smp 12 \
> > -kernel /path-to-kernel/linux-4.18.10/arch/x86_64/boot/bzImage \
> > -hda /path-to-img/ubuntu1804.qcow2 \
> > -append "root=/dev/sda1 init=/sbin/init text" -enable-kvm \
> > -net nic -net user,hostfwd=tcp::5022-:22 \
> > -device nvme,drive=nvme1,serial=deadbeaf1,num_queues=8 \
> > -drive file=/path-to-img/nvme.disk,if=none,id=nvme1
> >
> > Dongli Zhang
Qemu counts one of those queues as the admin queue.
> > On 10/08/2018 01:05 PM, Prasun Ratn wrote:
> >> Hi
> >>
> >> I have an NVMe SSD that has 8 hw queues and on older kernels I see all
> >> 8 show up. However on a recent kernel (I tried 4.18), I only see 7. Is
> >> this a known issue?
That probably means you only have 8 MSI-x vectors, one of which is
reserved for the admin queue. We used to share an IO vector with the
admin queue, however some people figured out you can break your controller
that way with the linux irq spread.
