On 8/22/19 3:59 PM, Jason Dillaman wrote:
> On Thu, Aug 22, 2019 at 9:23 AM Wido den Hollander <w...@42on.com> wrote:
>>
>> Hi,
>>
>> In a couple of situations I have encountered that Virtual Machines
>> running on RBD had a high I/O-wait, nearly 100%, on their vdX (VirtIO)
>> or sdX (Virtio-SCSI) devices while they were performing CPU intensive tasks.
>>
>> These servers would be running a very CPU intensive application while
>> *not* doing that many disk I/O.
>>
>> I however noticed that the I/O-wait of the disk(s) in the VM went up to
>> 100%.
>>
>> This VM is CPU limited by Libvirt by putting that KVM process in it's
>> own cgroup with a CPU limitation.
>>
>> Now, my theory is:
>>
>> KVM (qemu-kvm) is completely userspace and librbd runs inside qemu-kvm
>> as a library. All threads for disk I/O are part of the same PID and thus
>> part of that cgroup.
>>
>> If a process inside the Virtual Machine now starts to consume all CPU
>> time there is nothing left for librbd which slows it down.
>>
>> This then causes a increased I/O-wait inside the Virtual Machine. Even
>> though the VM is not performing a lot of disk I/O. The wait of the I/O
>> goes up due to this.
>>
>>
>> Is my theory sane?
>
> Yes, I would say that your theory is sane. Have you looked into
> libvirt's cgroup controls for limiting the emulator portion vs the
> vCPUs [1]? I'd hope the librbd code and threads should be running in
> the emulator cgroup (in a perfect world).
>
I checked with 'virsh schedinfo X' and this is the output I got:
Scheduler : posix
cpu_shares : 1000
vcpu_period : 100000
vcpu_quota : -1
emulator_period: 100000
emulator_quota : -1
global_period : 100000
global_quota : -1
iothread_period: 100000
iothread_quota : -1
How can we confirm if the librbd code runs inside the Emulator part?
Wido
>> Can somebody confirm this?
>>
>> Thanks,
>>
>> Wido
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> [1] https://libvirt.org/cgroups.html
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
