[Kernel-packages] [Bug 1346687] Re: Ubuntu 14 KVM Guest I/O Elevator Non-configurable
any news ? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1346687 Title: Ubuntu 14 KVM Guest I/O Elevator Non-configurable Status in linux package in Ubuntu: Invalid Bug description: Ubuntu 14.04 guest runs pretty much standard standard configuration: KVM hypervisor, ACPI and APIC enabled, virtio disk bus, raw storage format... While running this project, I am also involved in testing SSD drives across different Linux distributions and their support for discard option (TRIM). To cut the story short, Ubuntu-based KVM guest did not set any scheduler on the boot disk vda): # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom deadline 1 0B 0B vda disk 1 0B 0B vda1 part 1 0B 0B vda2 part 1 0B 0B vda5 part 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B That is in stark contrast with other distributions running almost identical KVM guest configuration (with the exception of the root file system using BTRFS instead of EXT4 on SUSE). Oracle Linux 6.5: # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom deadline 1 0B 0B vda disk deadline 1 0B 0B vda1 part deadline 1 0B 0B vda2 part deadline 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B OpenSUSE 13.1: # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom cfq 1 0B 0B vda disk cfq 1 0B 0B vda1 part cfq 1 0B 0B vda2 part cfq 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B Indeed, checking elevator capabilities for boot disk on Ubuntu KVM guest showed: # cat /sys/block/vda/queue/scheduler none Other Linux distributions show more options: # cat /sys/block/vda/queue/scheduler noop deadline [cfq] As well, attempts to change the elevator on Ubuntu guest fail. For example: # echo noop > /sys/block/vda/queue/scheduler # echo $? 0 # cat /sys/block/vda/queue/scheduler none Setting it globally in /etc/default/grub and updating it via update- grub2 fails too. In the meantime, to automate check of elevators (schedulers) and discard support for SSD drives and thin-provisioned volumes on Linux, I wrote simple Perl script: http://www.circlingcycle.com.au/Unix-sources/Linux-check-IO-scheduler- and-discard-support.pl.txt Part of the results on RHEL 6.5 server would look like: INFO: I/O elevator (scheduler) and discard support summary INFO: Hard Disk sdb configured with I/O scheduler "cfq" INFO: SSD sda configured with I/O scheduler "deadline" supports discard operation INFO: Hard Disk sdc configured with I/O scheduler "cfq" INFO: Hard Disk sdi configured with I/O scheduler "cfq" INFO: Hard Disk sdh configured with I/O scheduler "cfq" The Ubuntu KVM guest runs latest paches: Linux ubuntu14-vm2.circlingcycle.com.au 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Regards, Dusan Baljevic VK2COT To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346687/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1346687] Re: Ubuntu 14 KVM Guest I/O Elevator Non-configurable
I hate to reopen an old bug, but I haven't seen this discussed anywhere else, and I think there is some validity in this. Without the use of the cfq scheduler on block devices (virtual or not), the blkio cgroup controller doesn't perform any IO accounting (see below). Without that, tools like systemd-cgtop don't show any I/O stats. Determining which (group of) processes is using the most IO can be very valuable even in a virtual environment. It also prevents the use of ionice (which depends on cfq) on certain tasks within the VM, which again still have merit inside a VM, and the hypervisor can't affect once it reaches that level in the I/O path. Now, for accounting at least, whether that's a fix that should be done in the choice of block device queue scheduler or the blkio cgroup controller, is perhaps up for debate. But as it stands, users/operators are prevented from making the choice of elevator for themselves which seems wrong to me. If I want to pay the CPU cost for performing IO accounting if not scheduling inside a VM, even if the underlying storage may turn around and reorder things with its own more global optimization view, I should still be allowed to. Thanks, Brian This was done on a VMware VM since obviously I can't demonstrate it in a KVM VM, but the idea applies to physical machines as well. Tested running both 3.16 and 4.5 kernels. (get the current process' blkio cgroup) # grep blkio /proc/self/cgroup 7:blkio:/system.slice/ssh.service (yeah, I know, I was missing pam_systemd in the stack, but whatever) (check the current scheduler being used) # cat /sys/block/sda/queue/scheduler [noop] deadline cfq (check the current stats) # cat /sys/fs/cgroup/blkio/system.slice/ssh.service/blkio.io_serviced Total 0 (induce some IO) # time find /var/log -type f -print0 2>/dev/null | xargs -0 cat > /dev/null 2>/dev/null real0m4.539s user0m0.000s sys 0m0.048s (check the new blk io stats) # cat /sys/fs/cgroup/blkio/system.slice/ssh.service/blkio.io_serviced Total 0 (no change) (switch to the cfq scheduler) # echo cfq | sudo tee /sys/block/sda/queue/scheduler cfq (verify that the change happened) # cat /sys/block/sda/queue/scheduler noop deadline [cfq] (check the current blkio stats) # cat /sys/fs/cgroup/blkio/system.slice/ssh.service/blkio.io_serviced 8:0 Read 0 8:0 Write 0 8:0 Sync 0 8:0 Async 0 8:0 Total 0 Total 0 (look new fields) (clear the page cache to make sure our next test actually causes io) # echo 1 | sudo tee /proc/sys/vm/drop_caches 1 (induce io again) # time find /var/log -type f -print0 2>/dev/null | xargs -0 cat > /dev/null 2>/dev/null real0m2.386s user0m0.000s sys 0m0.040s (check the blkio stats again) # cat /sys/fs/cgroup/blkio/system.slice/ssh.service/blkio.io_serviced 8:0 Read 1827 8:0 Write 0 8:0 Sync 0 8:0 Async 1827 8:0 Total 1827 Total 1827 (look, data!) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1346687 Title: Ubuntu 14 KVM Guest I/O Elevator Non-configurable Status in linux package in Ubuntu: Invalid Bug description: Ubuntu 14.04 guest runs pretty much standard standard configuration: KVM hypervisor, ACPI and APIC enabled, virtio disk bus, raw storage format... While running this project, I am also involved in testing SSD drives across different Linux distributions and their support for discard option (TRIM). To cut the story short, Ubuntu-based KVM guest did not set any scheduler on the boot disk vda): # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom deadline 1 0B 0B vda disk 1 0B 0B vda1 part 1 0B 0B vda2 part 1 0B 0B vda5 part 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B That is in stark contrast with other distributions running almost identical KVM guest configuration (with the exception of the root file system using BTRFS instead of EXT4 on SUSE). Oracle Linux 6.5: # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom deadline 1 0B 0B vda disk deadline 1 0B 0B vda1 part deadline 1 0B 0B vda2 part deadline 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B OpenSUSE 13.1: # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom cfq 1 0B 0B vda disk cfq 1 0B 0B vda1 part cfq 1 0B 0B vda2 part cfq 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B Indeed, checking elevator capabilities for boot disk on Ubuntu KVM guest showed: # cat /sys/block/vda/queue/scheduler none Other Linux distributions show more options: # cat /sys/block/vda/queue/scheduler noop deadline [cfq] As well, attempts to change the elevator
[Kernel-packages] [Bug 1346687] Re: Ubuntu 14 KVM Guest I/O Elevator Non-configurable
This is a change in the kernel between 3.13 and 3.13-rc1. It also persists in the most recent kernel version. I believe this is still an invalid bug as what benefit would setting an I/O scheduler to anything be if the device is virtual? On a physical machine you are still able to set the I/O scheduler properly. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1346687 Title: Ubuntu 14 KVM Guest I/O Elevator Non-configurable Status in “linux” package in Ubuntu: Invalid Bug description: Ubuntu 14.04 guest runs pretty much standard standard configuration: KVM hypervisor, ACPI and APIC enabled, virtio disk bus, raw storage format... While running this project, I am also involved in testing SSD drives across different Linux distributions and their support for discard option (TRIM). To cut the story short, Ubuntu-based KVM guest did not set any scheduler on the boot disk vda): # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom deadline 1 0B 0B vda disk 1 0B 0B vda1 part 1 0B 0B vda2 part 1 0B 0B vda5 part 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B That is in stark contrast with other distributions running almost identical KVM guest configuration (with the exception of the root file system using BTRFS instead of EXT4 on SUSE). Oracle Linux 6.5: # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom deadline 1 0B 0B vda disk deadline 1 0B 0B vda1 part deadline 1 0B 0B vda2 part deadline 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B OpenSUSE 13.1: # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom cfq 1 0B 0B vda disk cfq 1 0B 0B vda1 part cfq 1 0B 0B vda2 part cfq 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B Indeed, checking elevator capabilities for boot disk on Ubuntu KVM guest showed: # cat /sys/block/vda/queue/scheduler none Other Linux distributions show more options: # cat /sys/block/vda/queue/scheduler noop deadline [cfq] As well, attempts to change the elevator on Ubuntu guest fail. For example: # echo noop > /sys/block/vda/queue/scheduler # echo $? 0 # cat /sys/block/vda/queue/scheduler none Setting it globally in /etc/default/grub and updating it via update- grub2 fails too. In the meantime, to automate check of elevators (schedulers) and discard support for SSD drives and thin-provisioned volumes on Linux, I wrote simple Perl script: http://www.circlingcycle.com.au/Unix-sources/Linux-check-IO-scheduler- and-discard-support.pl.txt Part of the results on RHEL 6.5 server would look like: INFO: I/O elevator (scheduler) and discard support summary INFO: Hard Disk sdb configured with I/O scheduler "cfq" INFO: SSD sda configured with I/O scheduler "deadline" supports discard operation INFO: Hard Disk sdc configured with I/O scheduler "cfq" INFO: Hard Disk sdi configured with I/O scheduler "cfq" INFO: Hard Disk sdh configured with I/O scheduler "cfq" The Ubuntu KVM guest runs latest paches: Linux ubuntu14-vm2.circlingcycle.com.au 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Regards, Dusan Baljevic VK2COT To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346687/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1346687] Re: Ubuntu 14 KVM Guest I/O Elevator Non-configurable
** Package changed: util-linux (Ubuntu) => linux (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1346687 Title: Ubuntu 14 KVM Guest I/O Elevator Non-configurable Status in “linux” package in Ubuntu: Invalid Bug description: Ubuntu 14.04 guest runs pretty much standard standard configuration: KVM hypervisor, ACPI and APIC enabled, virtio disk bus, raw storage format... While running this project, I am also involved in testing SSD drives across different Linux distributions and their support for discard option (TRIM). To cut the story short, Ubuntu-based KVM guest did not set any scheduler on the boot disk vda): # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom deadline 1 0B 0B vda disk 1 0B 0B vda1 part 1 0B 0B vda2 part 1 0B 0B vda5 part 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B That is in stark contrast with other distributions running almost identical KVM guest configuration (with the exception of the root file system using BTRFS instead of EXT4 on SUSE). Oracle Linux 6.5: # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom deadline 1 0B 0B vda disk deadline 1 0B 0B vda1 part deadline 1 0B 0B vda2 part deadline 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B OpenSUSE 13.1: # lsblk -io KNAME,TYPE,SCHED,ROTA,DISC-GRAN,DISC-MAX KNAME TYPE SCHED ROTA DISC-GRAN DISC-MAX sr0 rom cfq 1 0B 0B vda disk cfq 1 0B 0B vda1 part cfq 1 0B 0B vda2 part cfq 1 0B 0B dm-0 lvm 1 0B 0B dm-1 lvm 1 0B 0B Indeed, checking elevator capabilities for boot disk on Ubuntu KVM guest showed: # cat /sys/block/vda/queue/scheduler none Other Linux distributions show more options: # cat /sys/block/vda/queue/scheduler noop deadline [cfq] As well, attempts to change the elevator on Ubuntu guest fail. For example: # echo noop > /sys/block/vda/queue/scheduler # echo $? 0 # cat /sys/block/vda/queue/scheduler none Setting it globally in /etc/default/grub and updating it via update- grub2 fails too. In the meantime, to automate check of elevators (schedulers) and discard support for SSD drives and thin-provisioned volumes on Linux, I wrote simple Perl script: http://www.circlingcycle.com.au/Unix-sources/Linux-check-IO-scheduler- and-discard-support.pl.txt Part of the results on RHEL 6.5 server would look like: INFO: I/O elevator (scheduler) and discard support summary INFO: Hard Disk sdb configured with I/O scheduler "cfq" INFO: SSD sda configured with I/O scheduler "deadline" supports discard operation INFO: Hard Disk sdc configured with I/O scheduler "cfq" INFO: Hard Disk sdi configured with I/O scheduler "cfq" INFO: Hard Disk sdh configured with I/O scheduler "cfq" The Ubuntu KVM guest runs latest paches: Linux ubuntu14-vm2.circlingcycle.com.au 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Regards, Dusan Baljevic VK2COT To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346687/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp