On Tue, 09/04 08:48, Nikola Ciprich wrote:
> Hi,
>
> I've seen this discussed few times without any real conclusion,
> so I'd like to try asking again..
>
> we're seeing very bad performance on ceph volumes mapped to qemu using
> librbd,
> while block access to krbd is multiple times better (when comparing fast
> storage iops).
>
> rough numbers I'm getting:
>
> qemu, block access to KRBD mapped volume:
> read: > ~100K IOPS
> write: > ~100K IOPS
>
> qemu: librbd access to volume:
> read: ~25K IOPS
> write: ~20K IOPS
>
>
> when I run FIO on host, using librbd, I'm getting ~140K IOPS on reads, ~50K
> IOPS
> on write.
>
> trying different cache modes does not have any important impact. I'm using
> virtio-scsi,
>
> no special settings, host is 4.14.60 x86_64, qemu-kvm-2.12.0, guest has 16
> cores assigned.
>
> fio cmdline:
>
> on host:
> fio --readwrite=randread/randwrite --randrepeat=1 --ioengine=rbd --direct=1
> --numjobs=16 --gtod_reduce=1 --name=test --pool=${pool} --rbdname=${vol}
> --invalidate=0 --bs=4k --iodepth=64 --time_based --runtime=60
> --group_reporting
>
>
> in guest:
> fio --readwrite=randread/randwrite --filename=/dev/sdx --randrepeat=1
> --ioengine=libaio --direct=1 --numjobs=16 --gtod_reduce=1 --name=test
> --invalidate=0 --bs=4k --iodepth=64 --time_based --runtime=60
> --group_reporting
>
> trying numjobs=1 leads to similar results (qemu krbd much faster then librbd)
>
> no special rbd features enabled on volume, apart from layering.
>
> any ideas on why the difference might be so huge?
Any different pattern in host mpstat comparing the krbd and librbd runs? IIUC
even with numjobs=1, iodepth=64 may still allow krbd to benefit from blk-mq's
scalability.
Fam
>
> thanks a lot in advance
>
> BR
>
> nik
>
>
> --
> -------------------------------------
> Ing. Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28.rijna 168, 709 00 Ostrava
>
> tel.: +420 591 166 214
> fax: +420 596 621 273
> mobil: +420 777 093 799
> www.linuxbox.cz
>
> mobil servis: +420 737 238 656
> email servis: [email protected]
> -------------------------------------
>
