Re: Terrible disk/network performance with kvm
On 08/15/2011 10:51 PM, Nico Kadel-Garcia wrote: On Mon, Aug 15, 2011 at 6:05 PM, Orion Poplawskior...@cora.nwra.com wrote: I have a server that I recently migrated from CentOS 5.6 to SL6.1. The host install serves as a basic disk server and runs our nightly backups with amanda. We also have several virtual machines running on it, so of which are mostly idle and others which can be quite busy. Since making the move I'm seeing the disk and network response/performance go to hell if I have any non-idle VMs running. One barometer - I have a raid10 rebuild running at the moment. With no VMs I can reach 200,000K/sec resync speed. I'm getting 131,000K with two idle VMs and some other basic activity. If I start a busy vm, it drops to a few hundred K/sec and all disk access is very slow. Response to keystrokes in a network shell can be very slow as well. As anyone else experienced problems like this? I'm not seeing any error messages on the host or VMs. Have you rebuilt the virtual images? And does the underlying RAID, and the disks, have 4096 byte blocks? If so, you will need to be quite cautious in partitioning your disk images, because no installer in the world can tell from scratch, inside a virtualized host, what the block size is on the underlying hardware, and this can *kill* your performance if it's mis-set. Nope, everything is as before - I installed SL6 into the existing volume group and preserved the VMs logical volumes. Thanks for the heads up though. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA DivisionFAX: 303-415-9702 3380 Mitchell Lane or...@cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com
Re: Terrible disk/network performance with kvm
On Mon, 15 Aug 2011 16:05:17 -0600, Orion Poplawski wrote: I have a server that I recently migrated from CentOS 5.6 to SL6.1. The host install serves as a basic disk server and runs our nightly backups with amanda. We also have several virtual machines running on it, so of which are mostly idle and others which can be quite busy. Since making the move I'm seeing the disk and network response/performance go to hell if I have any non-idle VMs running. One barometer - I have a raid10 rebuild running at the moment. With no VMs I can reach 200,000K/sec resync speed. I'm getting 131,000K with two idle VMs and some other basic activity. If I start a busy vm, it drops to a few hundred K/sec and all disk access is very slow. Response to keystrokes in a network shell can be very slow as well. As anyone else experienced problems like this? I'm not seeing any error messages on the host or VMs. Hi, The default settings are bad. Please tell us more about your settings. Otherwise, I recommend the following points: - controller: virtio - kvm option: cache=none - qcow2 disk format with metadata preallocation - create your disk image with: qemu-img create -f qcow2 -o size=400,preallocation=metadata vdisk.img Here's a good link to the topic: http://www.ilsistemista.net/index.php/virtualization/11-kvm-io-slowness-on-rhel-6.html Best regards, Morten
Re: Terrible disk/network performance with kvm
On 16/08/2011 8:29 AM, Orion Poplawski wrote: I'm using LVM volumes for the disk images, stored on my 8 disk raid10 array. While this is not exactly related to KVM, the underlying principles will more than likely be the same for KVM... http://www.crc.id.au/xen-performance-guide/ The key is not to layer too many things on top of each other or you'll run into issues with block boundaries etc. It also shows a different way to install CentOS 5 in this case - however it should be easily adjustable to just about any EL OS. -- Steven Haigh Email: net...@crc.id.au Web: http://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299
Re: Terrible disk/network performance with kvm
On Mon, Aug 15, 2011 at 6:05 PM, Orion Poplawski or...@cora.nwra.com wrote: I have a server that I recently migrated from CentOS 5.6 to SL6.1. The host install serves as a basic disk server and runs our nightly backups with amanda. We also have several virtual machines running on it, so of which are mostly idle and others which can be quite busy. Since making the move I'm seeing the disk and network response/performance go to hell if I have any non-idle VMs running. One barometer - I have a raid10 rebuild running at the moment. With no VMs I can reach 200,000K/sec resync speed. I'm getting 131,000K with two idle VMs and some other basic activity. If I start a busy vm, it drops to a few hundred K/sec and all disk access is very slow. Response to keystrokes in a network shell can be very slow as well. As anyone else experienced problems like this? I'm not seeing any error messages on the host or VMs. Have you rebuilt the virtual images? And does the underlying RAID, and the disks, have 4096 byte blocks? If so, you will need to be quite cautious in partitioning your disk images, because no installer in the world can tell from scratch, inside a virtualized host, what the block size is on the underlying hardware, and this can *kill* your performance if it's mis-set.
