Well, just FYI, I changed cache_mode from NULL (none), to writethrough directly in DB and the performance boosted greatly. It may be an important feature for NVME drives.
Currently, on 4.11, the user can set cache-mode for disk offerings, but cannot for service offerings, which are translated to cache=none corresponding disk offerings. The only way is to use SQL to change that for root disk disk offerings. CreateServiceOffering API doesn't support cache mode. It can be a serious limitation for NVME users, because by default they could meet poor read/write performance. пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <kudryavtsev...@bw-sw.com>: > Darius, thanks for your participation, > > first, I used 4.14 kernel which is the default one for my cluster. Next, > switched to 4.15 with dist-upgrade. > > Do you have an idea how to turn on amount of queues for virtio-scsi with > Cloudstack? > > пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <daz...@gmail.com>: > >> Hi, >> >> I can see a few issues with your xml file. You can try using "queues" >> inside your disk definitions. This should help a little, not sure by >> how much for your case, but for my specific it went up by almost the >> number of queues. Also try cache directsync or writethrough. You >> should switch kernel if bugs are still there with 4.15 kernel. >> >> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev >> <kudryavtsev...@bw-sw.com> wrote: >> > >> > Hello, colleagues. >> > >> > Hope, someone could help me. I just deployed a new VM host with Intel >> P4500 >> > local storage NVMe drive. >> > >> > From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs >> with >> > FIO, write performance is also high as expected. >> > >> > I've created a new KVM VM Service offering with virtio-scsi controller >> > (tried virtio as well) and VM is deployed. Now I try to benchmark it >> with >> > FIO. Results are very strange: >> > >> > 1. Read/Write with large blocks (1M) shows expected performance (my >> limits >> > are R=1000/W=500 MBs). >> > >> > 2. Write with direct=0 leads to expected 50K IOPS, while write with >> > direct=1 leads to very moderate 2-3K IOPS. >> > >> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS. >> > >> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0% which >> is >> > strange. >> > >> > So, basically, from inside VM my NVMe works very slow when small IOPS >> are >> > executed. From the host, it works great. >> > >> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read >> > performance is nice. Maybe someone managed to use NVME with KVM with >> small >> > IOPS? >> > >> > The filesystem is XFS, previously tried with EXT4 - results are the >> same. >> > >> > This is the part of VM XML definition generated by CloudStack: >> > >> > <devices> >> > <emulator>/usr/bin/kvm-spice</emulator> >> > <disk type='file' device='disk'> >> > <driver name='qemu' type='qcow2' cache='none' discard='unmap'/> >> > <source >> > file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/> >> > <backingStore type='file' index='1'> >> > <format type='raw'/> >> > <source >> > file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/> >> > <backingStore/> >> > </backingStore> >> > <target dev='sda' bus='scsi'/> >> > <iotune> >> > <read_bytes_sec>1048576000</read_bytes_sec> >> > <write_bytes_sec>524288000</write_bytes_sec> >> > <read_iops_sec>100000</read_iops_sec> >> > <write_iops_sec>50000</write_iops_sec> >> > </iotune> >> > <serial>6809dbd04a1540149322</serial> >> > <alias name='scsi0-0-0-0'/> >> > <address type='drive' controller='0' bus='0' target='0' unit='0'/> >> > </disk> >> > <disk type='file' device='cdrom'> >> > <driver name='qemu' type='raw'/> >> > <backingStore/> >> > <target dev='hdc' bus='ide'/> >> > <readonly/> >> > <alias name='ide0-1-0'/> >> > <address type='drive' controller='0' bus='1' target='0' unit='0'/> >> > </disk> >> > <controller type='scsi' index='0' model='virtio-scsi'> >> > <alias name='scsi0'/> >> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09' >> > function='0x0'/> >> > </controller> >> > >> > So, what I see now, is that it works slower than couple of two Samsung >> 960 >> > PRO which is extremely strange. >> > >> > Thanks in advance. >> > >> > >> > -- >> > With best regards, Ivan Kudryavtsev >> > Bitworks LLC >> > Cell RU: +7-923-414-1515 >> > Cell USA: +1-201-257-1512 >> > WWW: http://bitworks.software/ <http://bw-sw.com/> >> >
