Re: Workload spikes on KVM host when doing IO on a guest...
Avi Kivity wrote: On 05/20/2012 08:29 PM, Erik Brakkee wrote: Avi Kivity wrote: On 05/20/2012 08:02 PM, Erik Brakkee wrote: [...] Thanks for this information. Unfortunately, io="native" in domain.xml is not supported by opensuse 11.3. It is supported in 12.1 so it appears that the version of KVM I have on the server is too old. I tried it on a system running the newer version and indeed, as you say the load disappears completely when using io="native". I am going to update the host now (probably to centos 6.2) to get rid of this problem. To be clear: it's not a problem. It's completely normal, and doesn't affect anything. The only problem with it is that it leads to high workload spikes, which is normally a reason to have a good look at what is going on. In this case, the newer version of KVM should help eliminate these spikes, so that the next time I see a spike in the workload I know that I have to look into something. In the mean time I have migrated the host machine to run centos 6.2 and later 6.3. There I could use the aio=native option together with cache=none and I saw a significant drop in thw workload while doing a lot of IO on the guest. However, after upgrading to centos 6.4, I now see huge workload spikes again, but the virtual machine configuration is the same. Also, I am seeing a high usage of swap on the host machine. It almost looks as if the aio=native option is being ignored or could it be a different problem? Here is one of the processes: qemu 3026 4.6 17.9 4751536 4419208 ? Sl Mar27 136:21 /usr/libexec/qemu-kvm -name sparrow -S -M rhel6.2.0 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -uuid 1389eb3f-8f26-686e-93c4-21267a66ec52 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/sparrow.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/bootdisks/sparrow,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/sparrow/disk,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:01:01:03,bus=pci.0,addr=0x5 -netdev tap,fd=27,id=hostnet1,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:01:01:04,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:2 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 I can also see that the swap space is exclusively used by qemu-kvm: [root@falcon bin]# ./examineswap ... Overall swap used: 1264924 kB kB pid name 575092 3026qemu-kvm 391468 2882qemu-kvm 298364 2953qemu-kvm (see at the end of the examineswap script). This snapshot of the swap space is taken shortly after enabling swap and it continues to fill up the entire swap space, with still a lot of memory free [root@falcon bin]# free total used free sharedbuffers cached Mem: 24662972 24460920 202052 0 10904048 40712 -/+ buffers/cache: 13516160 11146812 Swap: 20971441508516 588628 Also, I have set the swappiness to 0 but it does not help: [root@falcon bin]# sysctl vm.swappiness vm.swappiness = 0 Details on the OS and KVM: [root@falcon ~]# uname -a Linux falcon.fritz.box 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Mar 13 00:26:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [root@falcon ~]# cat /etc/issue CentOS release 6.4 (Final) Kernel \r on an \m [root@falcon ~]# rpm -qa | grep kvm qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.2.x86_64 I am also getting a workload on the host of approximately 20 now whereas the workload on the VM doing the IO is just 6. What I am doing at this time is creating a backup over iSCSI with a target on the physical host and an initiator on the VM. The VM has created snapshot logical volumes and is copying data to the iSCSI target. More spcifically, I am using this script that I developed myself: http://wamblee.org/snapshot.html Do you have any idea what this could be? Cheers Erik PS. The examineswap script I downloaded #!/bin/bash # find-out-what-is-using-your-swap.sh # -- Get current swap usage for all running processes # -- # -- rev.0.3, 2012-09-03, Jan Smid - alignment and intendation, sorting # -- rev.0.2, 2012-08-09, Mikko Rantalainen - pipe the output to "sort -nk3" to get sorted output
Re: KVM performance Java server/MySQL...
> >> Other optimizations people are testing out there. >> >> - use "nohz=off" in the kernel loading line y menu.lst >> - Disable Cgroups completely. Using cgclear, and turning off cgred >> cg-config daemons. I also tried this option but it did not have a significant effect and degraded performance a bit, although it is difficult to tell because of the variations in the measured times. On a positive note, I have managed to get a significant performance improvement by checking the BIOS. When we got the server, it was configured to use the Performance Per Watt profile. All current tests were run with CPU scaling enabled (and also recognized by the linux kernel). What I did to get increased performance was to enable the 'Performance' profile instead and let the hardware do the CPU scaling (this is what I understand that the option means). Now the measured times have dropped from a consistent 19 seconds on the physical host to a consistent 10 seconds on the physical host. On the virtual host, the variable times from 21-27 seconds have now turned into consistent times of approximately 13.5 seconds. This is with the nohz=off kernel option. With the nohz=off option removed, I get consistent times that are mostly a bit under 13 seconds (12.8 upwards). These results are for a VM with 4 virtual CPUs that are also pinned to cores of the first CPU. >From the looks of it, the kernel we are using does not fully understand the CPU it is dealing with. Also, I am doubtful that we ever saw the actual CPU frequences when doing 'cat /proc/cpuinfo'. Also, positively, increasing the VCPU count from 4 to 8 and also not pinning them does not deteriorate performance. In fact, it looks like it increases performance even a bit, leading to an overhead of 20% approximately. All in all, the performance results are acceptable now looking at the absolute figures. In fact, I am quite happy with this because it means we can continue to take this setup in production. However, there still appears to be a 20% penalty for virtualization in our specific case. Do you have any ideas on how to get to the bottom of this? Being able to squeeze the last 20% out of it would really make us even more happy (my day is already made with this BIOS setting). Any ideas? > However, I got a performance degradation of running the same test with > MySQL and JBoss colocated and increasing the CPU count from 4 to 8. Then I > did various tricks to improve performance. Pinning the processes did not > have a significant effect. What appeared to have an effect was to remove > memory ballooning. > > But now I get inconsistent results, varying from 21 seconds to 27 seconds. > The 21 figure is acceptable but the 27 figure is not. The tests are being > done on new hardware (see previous posts), with basically the only thing > running the virtual machine. > > Summary of configuration: > - host and guest: centos 6.3, transparent hugepages > - vm: cpu mode is host-passthrough, pinning to the cores of one processor, > removed tablet, sound, and USB devices > - host: hyperthreading switched off in BIOS > > Do you have any idea what this could be? I expect it is somehow NUMA > related, but how would I troubleshoot this? Ideally I would like to make > sure the entire VM runs on one CPU and allocates memory from that CPU and > never moves (or both CPU and memory move together). I saw some > presentations on the internet about NUMA work being done for linux. Do you > have any suggestions? > > My domain.xml is given below: > > > master-data05-v50 > 79ddd84d-937e-357b-8e57-c7f487dc3464 > 8388608 > 8388608 > 8 > > > > > > > > > > > > hvm > > > > > > > > > > > destroy > restart > restart > > /usr/libexec/qemu-kvm > > > > > > > > > > > > >function='0x0'/> > > > > > > >function='0x0'/> > > > > > > >function='0x0'/> > > > > > > >function='0x0'/> > > > >function='0x2'/> > > > >function='0x1'/> > > > > > > > >function='0x0'/> > > > > > > > > > > > > > > > > >function='0x0'/> > > > > > > The output of virsh capabilities: > > > > > 44454c4c-4600-1031-8043-b9c04f47354a > > x86_64 > SandyBridge > Intel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tcp > > > > > >
Re: KVM performance Java server/MySQL...
> Other optimizations people are testing out there. > > - use "nohz=off" in the kernel loading line y menu.lst > - Disable Cgroups completely. Using cgclear, and turning off cgred > cg-config daemons. > > And from a Personal point of view, we've always tried to use MySQL in > a different server from JBoss. > 99% of the times is far better for performance and tuning. I am still having problems. Running mysql and JBoss on different VMs is significantly slower, but that appears to be a pure application issue relating to the network overhead in a huge amount of queries. This is of course not an issue for this mailing list. However, I got a performance degradation of running the same test with MySQL and JBoss colocated and increasing the CPU count from 4 to 8. Then I did various tricks to improve performance. Pinning the processes did not have a significant effect. What appeared to have an effect was to remove memory ballooning. But now I get inconsistent results, varying from 21 seconds to 27 seconds. The 21 figure is acceptable but the 27 figure is not. The tests are being done on new hardware (see previous posts), with basically the only thing running the virtual machine. Summary of configuration: - host and guest: centos 6.3, transparent hugepages - vm: cpu mode is host-passthrough, pinning to the cores of one processor, removed tablet, sound, and USB devices - host: hyperthreading switched off in BIOS Do you have any idea what this could be? I expect it is somehow NUMA related, but how would I troubleshoot this? Ideally I would like to make sure the entire VM runs on one CPU and allocates memory from that CPU and never moves (or both CPU and memory move together). I saw some presentations on the internet about NUMA work being done for linux. Do you have any suggestions? My domain.xml is given below: master-data05-v50 79ddd84d-937e-357b-8e57-c7f487dc3464 8388608 8388608 8 hvm destroy restart restart /usr/libexec/qemu-kvm The output of virsh capabilities: 44454c4c-4600-1031-8043-b9c04f47354a x86_64 SandyBridge Intel tcp hvm 32 /usr/libexec/qemu-kvm rhel6.3.0 pc rhel6.2.0 rhel6.1.0 rhel6.0.0 rhel5.5.0 rhel5.4.4 rhel5.4.0 /usr/libexec/qemu-kvm hvm 64 /usr/libexec/qemu-kvm rhel6.3.0 pc rhel6.2.0 rhel6.1.0 rhel6.0.0 rhel5.5.0 rhel5.4.4 rhel5.4.0 /usr/libexec/qemu-kvm The output of virsh vcpuinfo master-data05-v50: VCPU: 0 CPU:0 State: running CPU time: 119.4s CPU Affinity: y--- VCPU: 1 CPU:2 State: running CPU time: 78.9s CPU Affinity: --y- VCPU: 2 CPU:4 State: running CPU time: 74.4s CPU Affinity: y--- VCPU: 3 CPU:6 State: running CPU time: 61.6s CPU Affinity: --y- VCPU: 4 CPU:8 State: running CPU time: 91.4s CPU Affinity: y--- VCPU: 5 CPU:10 State: running CPU time: 67.4s CPU Affinity: --y- VCPU: 6 CPU:12 State: running CPU time: 52.8s CPU Affinity: y--- VCPU: 7 CPU:14 State: running CPU time: 66.8s CPU Affinity: --y- I also checked x2apic using 'grep -i x2apic /var/log/messages*'. this shows x2apic is not working on the host but it is working on the guest. Should I enable it on the host as well as suggested by the output? On the host I get: /var/log/messages-20130210:Feb 8 08:51:00
Re: KVM performance Java server/MySQL...
> >> On Thu, Feb 07, 2013 at 04:41:31PM +0100, Erik Brakkee wrote: >>> Hi, >>> >>> >>> We have been benchmarking a java server application (java 6 update 29) >>> that requires a mysql database. The scenario is quite simple. We open a >>> web page which displays a lot of search results. To get the content of >>> the >>> page one big query is done with many smaller queries to retrieve the >>> data. >>> The test from the java side is single threaded. >>> >>> We have used the following deployment scenarios: >>> 1. JBoss in VM, MySql in separate VM >>> 2. JBoss in VM, MySQL native >>> 3. JBoss native, MySQL in vm. >>> 4. JBoss native and MySQL native on the same physical machine >>> 5. JBoss and MySQL virtualized on the same VM. >>> >>> What we see is that the performance (time to execute) is practically >>> the >>> same for all scenarios (approx. 30 seconds), except for scenario 4 that >>> takes approx. 21 seconds. This difference is quite large and contrasts >>> many other test on the internet and other benchmarks we did previously. >>> >>> We have tried pinning the VMs, turning hyperthreading off, varying the >>> CPU >>> model (including host-passthrough), but this did not have any >>> significant >>> impact. >>> >>> The hardware on which we are running is a dual socket E5-2650 machine >>> with >>> 64 GB memory. The server is a Dell poweredge R720 server with SAS >>> disks, >>> RAID controller with battery backup (writeback cache). Transparent huge >>> pages is turned on. >>> >>> We are at a loss to explain the differences in the test. In particular, >>> we >>> would have expected the least performance when both were running >>> virtualized and we would have expected a better performance when JBoss >>> and >>> MySQL were running virtualized in the same VM as compared to JBoss and >>> MySQL both running in different virtual machines. It looks like we are >>> dealing with multiple issues here and not just one. >>> >>> Right now we have a 30% penalty for running virtualized which is too >>> much >>> for us; 10% would be allright. What would you suggest to do to >>> troubleshoot this further? >>> >> >> What is you kernel/qemu versions and command line you are using to start >> a VM? > > Centos 6.3, 2.6.32-279.22.1.el6.x86_64 >> rpm -qf /usr/libexec/qemu-kvm > qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 > > The guest is also running centos 6.3 with the same settings. Settings that > can influence Java performance (such as transparent huge pages) are turned > on on both the host and guest (see the remark on hugepages below). > > The command-line is as follows: > > /usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 8192 -mem-prealloc > -mem-path /hugepages/libvirt/qemu -smp 4,sockets=4,cores=1,threads=1 -name > master-data05-v50 -uuid 79ddd84d-937e-357b-8e57-c7f487dc3464 -nodefconfig > -nodefaults -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/master-data05-v50.monitor,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc > -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive > if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device > ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 > -drive > file=/dev/raid5/v50disk1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 > -drive > file=/dev/vg_raid1/v50disk2,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1 > -drive > file=/dev/raid5/v50disk3,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk2,id=virtio-disk2 > -drive > file=/var/data/images/configdisks/v50/configdisk.img,if=none,id=drive-virtio-disk25,format=raw,cache=none > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk25,id=virtio-disk25 > -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:00:01:50,bus=pci.0,addr=0x3 > -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 > > > > This vi
Re: KVM performance Java server/MySQL...
> The IO scheduler on the host and on the guest is CFS. We also tried with > deadline scheduler on the host but this did not make any measurable > difference. We did not try no-op on the host. I mean of course that we did not try no-op on the guest (not on the host). -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM performance Java server/MySQL...
> On Thu, Feb 07, 2013 at 04:41:31PM +0100, Erik Brakkee wrote: >> Hi, >> >> >> We have been benchmarking a java server application (java 6 update 29) >> that requires a mysql database. The scenario is quite simple. We open a >> web page which displays a lot of search results. To get the content of >> the >> page one big query is done with many smaller queries to retrieve the >> data. >> The test from the java side is single threaded. >> >> We have used the following deployment scenarios: >> 1. JBoss in VM, MySql in separate VM >> 2. JBoss in VM, MySQL native >> 3. JBoss native, MySQL in vm. >> 4. JBoss native and MySQL native on the same physical machine >> 5. JBoss and MySQL virtualized on the same VM. >> >> What we see is that the performance (time to execute) is practically the >> same for all scenarios (approx. 30 seconds), except for scenario 4 that >> takes approx. 21 seconds. This difference is quite large and contrasts >> many other test on the internet and other benchmarks we did previously. >> >> We have tried pinning the VMs, turning hyperthreading off, varying the >> CPU >> model (including host-passthrough), but this did not have any >> significant >> impact. >> >> The hardware on which we are running is a dual socket E5-2650 machine >> with >> 64 GB memory. The server is a Dell poweredge R720 server with SAS disks, >> RAID controller with battery backup (writeback cache). Transparent huge >> pages is turned on. >> >> We are at a loss to explain the differences in the test. In particular, >> we >> would have expected the least performance when both were running >> virtualized and we would have expected a better performance when JBoss >> and >> MySQL were running virtualized in the same VM as compared to JBoss and >> MySQL both running in different virtual machines. It looks like we are >> dealing with multiple issues here and not just one. >> >> Right now we have a 30% penalty for running virtualized which is too >> much >> for us; 10% would be allright. What would you suggest to do to >> troubleshoot this further? >> > > What is you kernel/qemu versions and command line you are using to start > a VM? Centos 6.3, 2.6.32-279.22.1.el6.x86_64 > rpm -qf /usr/libexec/qemu-kvm qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64 The guest is also running centos 6.3 with the same settings. Settings that can influence Java performance (such as transparent huge pages) are turned on on both the host and guest (see the remark on hugepages below). The command-line is as follows: /usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 8192 -mem-prealloc -mem-path /hugepages/libvirt/qemu -smp 4,sockets=4,cores=1,threads=1 -name master-data05-v50 -uuid 79ddd84d-937e-357b-8e57-c7f487dc3464 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/master-data05-v50.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/dev/raid5/v50disk1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/dev/vg_raid1/v50disk2,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/raid5/v50disk3,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/var/data/images/configdisks/v50/configdisk.img,if=none,id=drive-virtio-disk25,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk25,id=virtio-disk25 -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:00:01:50,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 This virtual machine has three virtio disks, and one file based disk. The last disk is about 100MB in size and is used only during startup (contains configurationd data for initializing the vm) and is only read and never written. It has one CDrom which is not used. It also uses old-style hugepages. Apparently this did not have any significant effect on performance over transparent hugepages (as would be expected). We configured these old style hugepages just to
KVM performance Java server/MySQL...
Hi, We have been benchmarking a java server application (java 6 update 29) that requires a mysql database. The scenario is quite simple. We open a web page which displays a lot of search results. To get the content of the page one big query is done with many smaller queries to retrieve the data. The test from the java side is single threaded. We have used the following deployment scenarios: 1. JBoss in VM, MySql in separate VM 2. JBoss in VM, MySQL native 3. JBoss native, MySQL in vm. 4. JBoss native and MySQL native on the same physical machine 5. JBoss and MySQL virtualized on the same VM. What we see is that the performance (time to execute) is practically the same for all scenarios (approx. 30 seconds), except for scenario 4 that takes approx. 21 seconds. This difference is quite large and contrasts many other test on the internet and other benchmarks we did previously. We have tried pinning the VMs, turning hyperthreading off, varying the CPU model (including host-passthrough), but this did not have any significant impact. The hardware on which we are running is a dual socket E5-2650 machine with 64 GB memory. The server is a Dell poweredge R720 server with SAS disks, RAID controller with battery backup (writeback cache). Transparent huge pages is turned on. We are at a loss to explain the differences in the test. In particular, we would have expected the least performance when both were running virtualized and we would have expected a better performance when JBoss and MySQL were running virtualized in the same VM as compared to JBoss and MySQL both running in different virtual machines. It looks like we are dealing with multiple issues here and not just one. Right now we have a 30% penalty for running virtualized which is too much for us; 10% would be allright. What would you suggest to do to troubleshoot this further? Cheers Erik -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
kvm-clock clocksource efficiency versus tsc...
OS: Centos 6.2 KVM version: qemu-kvm-tools-0.12.1.2-2.209.el6_2.4.x86_64 qemu-kvm-0.12.1.2-2.209.el6_2.4.x86_64 uname -a: Linux myhost 2.6.32-220.7.1.el6.x86_64 #1 SMP Wed Mar 7 00:52:02 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux Hi, I have been performance testing a time tracing utiliity for a Java enterprise application at work. The idea is that we measure time for different parts of our application and build time trees for this information. The time tree can be viewed and analysed in case of problems. It is also automatically output in case a certain operation is taking longer than a threshold. As part of these tests we found that it matters a lot what type of clocksource is used. In particular, with hpet and acpi_pm the execution is very slow (700ns per call (similar results using clock_gettime() in a C program). In addition, hpet and acpi_pm sycnrhonize the application. This is of course a disaster for server applications that tend to query the current time quite a lot. What works quite well is the tsc clocksource. In that case, the time drops to about 38ns on one of our systems and we can prove that there is no synchronization anymore which is good. When running inside a KVM VM the default clock source is kvm-clock. This clock takes about 160ns per call and also does not synchronise the application. However, using the tsc clock source delivers similar performance on the virtual machine as on the host. I now have the following questions: * can the performance of kvm-clock be optimized further to be (almost) identical to that of the host's clock source? * what are the consequences of using the tsc clock in combination with NTPD? Will this result in system instability or larger than usual clock skew? Cheers Erik -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: UDP problem with virtio...
Stefan Hajnoczi wrote: On Sat, May 26, 2012 at 11:23 AM, Erik Brakkee wrote: I have done some more experiments and it does work when using a Centos 6.2 guest. Therefore, it is most likely a compatibility issues between the guest and host virtio implementation. I read somewhere there is some sort of compatibility mode for virtio but could not find how to activate it. Is there a way to make these different virtio host/guest implementations talk to each other? They should be compatible. I noticed in your domain XML the VM has two virtio-net interfaces. Could that be affecting your mount success? Did you collect a packet capture inside the guest? (tcpdump -nli eth0 -s0 -w /tmp/dump.pcap) It would be interesting to understand what the guest sees during the failed and successful UDP NFS mounts. Can you be more specific about the mount failure. Is there an error message? Anything interesting in the guest's dmesg? Did you collect a packet capture on the host? Stefan Hi all, I sent a reply earlier but apparently not to this mailing list. The problem went away after installing the latest software updates on the openSuSE 11.3 guest. I tried reproducing it from a backup of the VM running the older kernel but was unable to do this on a openSuSE 12.1 host. To reproduce it I would have had to use the centos 6.2 host but that would be quite a lot more work to do. So I am leaving the issue now as resolved. It was probably some earlier bug that was fixed. Cheers Erik -- Nonsense and other useful things: http://brakkee.org MountainHoppers: http://mountainhoppers.nl Track Detective: http://trackdetective.com Twitter: http://twitter.com/ErikBrakkee -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: UDP problem with virtio...
Hi all, I have done some more experiments and it does work when using a Centos 6.2 guest. Therefore, it is most likely a compatibility issues between the guest and host virtio implementation. I read somewhere there is some sort of compatibility mode for virtio but could not find how to activate it. Is there a way to make these different virtio host/guest implementations talk to each other? Cheers Erik Erik Brakkee wrote: Hi, I am running a guest that exposes a NFS share to be used by a TViX box. The TViX box is not that advanced and uses NFS over UDP instead of TCP. What I am seeing is that when I use the virtio network driver, the TViX box cannot mount the filesystem. However, when I use rtl8139 instead, it can. The problem is easily reproduced: 1. Create a guest that uses a virtio network interface (bridging setup) 2. Expose an NFS share on the guest 3. mount the NFS share using 'mount -o udp host:/share /localdir' 4. verify that the NFS share cannot be mounted. 5. Modify the guest to use rtl8139 device emulation and stop the guest and start it 6. mount the NFS share using 'mount -o udp host:/share /localdir' 7. Verify tha the NFS share can be mounted Now the bug reporting guidelines say that I should always use the latest KVM version but compiling KVM from source and using that is not really an option. The server is important enough not to mess around with. Therefore, my question is whether this problem has been seen before or not. My configuration is as follows: *Host* centos 6.2 uname -a gives "Linux falcon.fritz.box 2.6.32-220.17.1.el6.x86_64 #1 SMP Wed May 16 00:01:37 BST 2012 x86_64 x86_64 x86_64 GNU/Linux" KVM software: qemu-kvm-0.12.1.2-2.209.el6_2.5.x86_64 *Guest* OpenSuSE 11.3 uname -a gives "Linux shikra 2.6.34.7-0.7-default #1 SMP 2010-12-13 11:13:53 +0100 x86_64 x86_64 x86_64 GNU/Linux" I also see in the instructions I should try running qemu with some alternative switches, but I am not running qemu directly. I am using virt-manager and a domain.xml to configure how qemu runs. How should I modify the domain.xml to help in troubleshooting this issue? Cheers Erik PS. the domain.xml is attached. -- Nonsense and other useful things: http://brakkee.org MountainHoppers: http://mountainhoppers.nl Track Detective: http://trackdetective.com Twitter: http://twitter.com/ErikBrakkee -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
UDP problem with virtio...
Hi, I am running a guest that exposes a NFS share to be used by a TViX box. The TViX box is not that advanced and uses NFS over UDP instead of TCP. What I am seeing is that when I use the virtio network driver, the TViX box cannot mount the filesystem. However, when I use rtl8139 instead, it can. The problem is easily reproduced: 1. Create a guest that uses a virtio network interface (bridging setup) 2. Expose an NFS share on the guest 3. mount the NFS share using 'mount -o udp host:/share /localdir' 4. verify that the NFS share cannot be mounted. 5. Modify the guest to use rtl8139 device emulation and stop the guest and start it 6. mount the NFS share using 'mount -o udp host:/share /localdir' 7. Verify tha the NFS share can be mounted Now the bug reporting guidelines say that I should always use the latest KVM version but compiling KVM from source and using that is not really an option. The server is important enough not to mess around with. Therefore, my question is whether this problem has been seen before or not. My configuration is as follows: *Host* centos 6.2 uname -a gives "Linux falcon.fritz.box 2.6.32-220.17.1.el6.x86_64 #1 SMP Wed May 16 00:01:37 BST 2012 x86_64 x86_64 x86_64 GNU/Linux" KVM software: qemu-kvm-0.12.1.2-2.209.el6_2.5.x86_64 *Guest* OpenSuSE 11.3 uname -a gives "Linux shikra 2.6.34.7-0.7-default #1 SMP 2010-12-13 11:13:53 +0100 x86_64 x86_64 x86_64 GNU/Linux" I also see in the instructions I should try running qemu with some alternative switches, but I am not running qemu directly. I am using virt-manager and a domain.xml to configure how qemu runs. How should I modify the domain.xml to help in troubleshooting this issue? Cheers Erik PS. the domain.xml is attached. shikra 3a524d9f-a9df-4280-d0d7-e7562a860147 6291456 6291456 8 hvm destroy restart restart /usr/libexec/qemu-kvm
Workload spikes on KVM host when doing IO on a guest...
Hi, I am seeing high workload spikes of approx. 15 when I do IO inside a KVM guest, for instance dd if=/dev/zero bs=1G count=1 of=hog When I execute a similar command on the host to write a file on the same physical disk, the workload only goes to about 3. I am using virtio on the guest with cache mode none. Also, I am using the noop IO scheduler on the guest and the deadline IO scheduler on the host. The guest is allocated a logical volume from the host. When I execute the dd command on the guest, it finishes almost instantaneously but when I execute it on the host I have to wait for approx 10 seconds. Specifically, on the guest I see a transfer speed of approx. 600 MB/s and on the host I get 75.9MB/s. The figure for the host is most reliable as this is close to what the hard disks can handle (WD enterprise class SATA hard disks). What appears to be happening is that somehow it forwards all IO from the guest immediately to the host, just as if write back caching was used. When I look at the output of 'virsh dumpxm ' I get this as part of the output which indicates that cache="none" is actually used The host is an opensuse 11.3 system (Linux falcon 2.6.34.10-0.6-default #1 SMP 2011-12-13 18:27:38 +0100 x86_64 x86_64 x86_64 GNU/Linux). The kvm version is falcon:~ # rpm -qa | grep kvm kvm-0.12.5-1.8.1.x86_64 Is this some known issue in this version of KVM and should I simply upgrade (or replace the host with a centos 6.2 system). Or is there a simple configuration that can fix this? Cheers Erik -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
target dev name ignored?
Hi, When I define a domain XML with a virtio disk then the target device setting is ignored. For instance, given this declaration the target device name is still 'vda' and not 'sda': I ran into this while experimenting. In practice I will only run into this problem when trying to mount the boot sector (which is the only file system which is not an LVM) so this is not a major issue. For this reason I usually do not mount /boot and also disable kernel updates. But... it would be nice if it worked. Use case is switching between virtio and scsi setups. Right now I have to either have separate boot entries in grub for virtio and scsi or I have to edit the root kernel parameter in grub during boot. Cheers Erik -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: USB Passthrough 1.1 performance problem...
Erik Brakkee wrote: Erik Brakkee wrote: 2010/12/14 Erik Brakkee: Daniel P. Berrange wrote: On Tue, Dec 14, 2010 at 12:55:04PM +0100, Kenni Lund wrote: 2010/12/14 Erik Brakkee: From: Kenni Lund 2010/12/14 Erik Brakkee: From: Kenni Lund Does this mean I have a chance now that PCI passthrough of my WinTV PVR-500 might work now? Passthrough of a PVR-500 has been working for a long time. I've been running with passthrough of a PVR-500 in my HTPC, since November/December 2009...so it should work with any recent kernel and any recent version of qemu-kvm you can find today - No patching needed. The only issue I had with the PVR-500 card, was when *I* didn't free up the shared interrupts...once I fixed that, it "just worked". How did you free up those shared interrupts then? I tried different slots but always get conflicts with the USB irqs. I did an unbind of the conflicting device (eg. disabled it). I moved the PVR-500 card around in the different slots and once I got a conflict with the integrated sound card, I left the PVR-500 card in that slot (it's a headless machine, so no need for sound) and configured unbind of the sound card at boot time. On my old system I think it was conflicting with one of the USB controllers as well, but it didn't really matter, as I only lost a few of the ports on the back of the computer for that particular USB controller - I still had plenty of USB ports left and if I really needed more ports, I could just plug in an extra USB PCI card. My /etc/rc.local boot script looks like the following today: -- #Remove HDA conflicting with ivtv1 echo ":00:1b.0"> /sys/bus/pci/drivers/HDA\ Intel/unbind # ivtv0 echo " 0016"> /sys/bus/pci/drivers/pci-stub/new_id echo ":04:08.0"> /sys/bus/pci/drivers/ivtv/unbind echo ":04:08.0"> /sys/bus/pci/drivers/pci-stub/bind echo " 0016"> /sys/bus/pci/drivers/pci-stub/remove_id # ivtv1 echo " 0016"> /sys/bus/pci/drivers/pci-stub/new_id echo ":04:09.0"> /sys/bus/pci/drivers/ivtv/unbind echo ":04:09.0"> /sys/bus/pci/drivers/pci-stub/bind echo " 0016"> /sys/bus/pci/drivers/pci-stub/remove_id I did not try unbinding the usb device so I can also try that. I don'.t understand what is happening with the 0016. I configured the pci card in kvm and I believe kvm does the binding to pci-stub in recent versions. Where is the 0016%oming from? Okay, qemu-kvm might do it today, I don't know - I haven't changed that script for the past year. But are you sure that it's not libvirt/virsh/virt-manager which does that for you? If you use the managed="yes" attribute on the in libvirt XML, then libvirt will automatically do the pcistub bind/unbind, followed by a device reset at guest startup& the reverse at shutdown. If you have conflicting devices on the bus though, libvirt won't attempt to unbind them, unless you had also explicitly assigned all those conflicting devices to the same guest. Daniel I definitely have to try again (right now having some stability problems on the server that I am debugging). The shared IRQs are as follows: 16: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 18: 252995 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb8, ivtv0 19: 58281 0 0 0 0 0 0 0 IO-APIC-fasteoi ata_piix, ata_piix, uhci_hcd:usb5, uhci_hcd:usb7, ivtv1 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4 23:713 6906 0 76919 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6 So I have IRQ sharing with usb1, usb8, usb5, usb7. Uffand your ata HDD controller. I guess i was much luckier than you are, my ivtv0 didn't conflict at all and ivtv1 only conflicted with USB. I have also read that ehci refers to USB 2.0 and uhci to USB 1.1 is that correct? Anyway, how would I now identify the USB PCI devices that I would need to unbind to get rid of the sharing with the USB ports? Play around with: lspci -v lspci -n lsusb -v lsusb -t You can also just start by unbinding the first one and take note when you hit the right ones...once you unbind one, it will disappear from cat /proc/interrupts. When you're down to having only ivtv0 on one interrupt and only ivtv1 on another interrupt, then you're ready to bind with pci-stub and boot your guest. It also doesn't really matter in which slot I put the PVR-500 card because both cards share IRQs with USB in all cases. You also have your conflicting ata contro
Re: network performance between host and guest/iscsi to the rescue
Erik Brakkee wrote: Brian Jackson wrote: On 12/17/2010 4:29 PM, Erik Brakkee wrote: Hi, For a backup of data from a VM to a USB mounted disk I want to circumvent the USB 1.1 limitations on the guest and instead copy the data over to the host using scp/ssh. I have setup a network using virtio and NAT like this: function='0x0'/> What does that equate to in command line options? Check libvirt logs maybe. What version of qemu-kvm? Guest details? Host details? When I now create a 1GB file using dd and copy it over from the guest to the host, I am seeing a performance between 25-30 MB/s. Is it to and from the same disk? If so, maybe you could try a tmpfs in the guest or host so you aren't constantly seeking back and forth on the same disk. Also have you tried something like rsyncd instead of scp? Maybe you are hitting some sort of encryption limitation. My question is if this is normal because I have seen others on the internet achieve far greater speeds. Depends on a lot of factors. Certainly raw bandwidth wise, virtio-net is capable of a lot more than that. With vhost-net here, I can get over 5gbps guest to host. And that's on crappy old first gen cpus (no ept/etc.). In any case the speeds are comparable to current USB 2.0 speeds but I intend on using USB 3.0 so would like to get a little bit more out of it. What would I use to speed this up a bit futher? Cheers Erik -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html You are right. I was indeed not so smart of me. I was assuming that the overhead of SSH would be negligible. However, I am seeing similar transfer speeds when I copy a file on the localhost to the localhost using SSH (on the host, not even on a guest). Netperf tests of the network show much higher speeds (approx 900Mbps) so the network is not the problem. The bottleneck is definitely SSH. I have also experimented with rsh (not really secure) and that gave some improvements but not much. For instance, bandwidth increasing from approx 25MB/s to 35MB/s. Then I thought about using iscsi and this really works great. Since KVM does not allow passing of new storage to a running guest (did not work for me in any case), iscsi is the way out. I simply configured a target on the host and an initiator on the guest and I am achieving native speeds for disk access. When I overwrite a 900MB file (the iscsi target is 1GB) multiple times using 'dd if=/dev/zero bs=1M count=900 of=test.bin' I am seeing a performance of approx. 90MB/s on the iscsi initiator and approx. 75MB/s on the root partition. Both partitions are ext4 file systems and the root file system is using virtio. Also the network interface is using virtio. My test setup is as follows: Sony Vaio F11 laptop, 8GB memory, Intel Core i7 Q720 (1.6GHz), host and guest both run Opensuse 11.3 64bit. I created a 1GB logical volume on the host and exposed this as a target to the guest. The network interface is using virtio with NAT networking. In any case my problem is solved. I intend to mount a SATA backup disk over USB 3.0 on the host and then do a backup from the guest directly onto this backup disk. Currently, I am using USB 2.0 and I am getting speeds of at most 30MB/s. In any case, this promises something for the bandwidth that I can get for backup from a guest in the future. The interesting thing about this is that iscsi over a virtio network seems to be a bit faster in this case than a virtio disk. This makes it tempting to setup an iSCSI boot on the guest if not for the additional complexity this brings (had some experiences before with opensuse 11.1 and it wasn't mature then). Also, I don't know how this would scale over multiple VMs. Alternatively, perhaps the implementation of virtio disks could be (transparently) based on iscsi or on iscsi concepts. I am curious to hear what you think of these results and if you also have similar experiences. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
unbinding PCI device at boot
Hi all, I would like to unbind some PCI devices at boot time to avoid shared IRQ problems with PCI passthrough. One way of doing it is to simply write an init.d script (done that before) which is quite simple. But before I do this, is there another better 'linux' way to do this? In this particular case it is about unbinding some USB PCI devices. Cheers Erik PS. running on opensuse 11.3. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: network performance between host and guest...
Brian Jackson wrote: On 12/17/2010 4:29 PM, Erik Brakkee wrote: Hi, For a backup of data from a VM to a USB mounted disk I want to circumvent the USB 1.1 limitations on the guest and instead copy the data over to the host using scp/ssh. I have setup a network using virtio and NAT like this: function='0x0'/> What does that equate to in command line options? Check libvirt logs maybe. What version of qemu-kvm? Guest details? Host details? When I now create a 1GB file using dd and copy it over from the guest to the host, I am seeing a performance between 25-30 MB/s. Is it to and from the same disk? If so, maybe you could try a tmpfs in the guest or host so you aren't constantly seeking back and forth on the same disk. Also have you tried something like rsyncd instead of scp? Maybe you are hitting some sort of encryption limitation. My question is if this is normal because I have seen others on the internet achieve far greater speeds. Depends on a lot of factors. Certainly raw bandwidth wise, virtio-net is capable of a lot more than that. With vhost-net here, I can get over 5gbps guest to host. And that's on crappy old first gen cpus (no ept/etc.). In any case the speeds are comparable to current USB 2.0 speeds but I intend on using USB 3.0 so would like to get a little bit more out of it. What would I use to speed this up a bit futher? Cheers Erik -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html You are right. I was indeed not so smart of me. I was assuming that the overhead of SSH would be negligible. However, I am seeing similar transfer speeds when I copy a file on the localhost to the localhost using SSH (on the host, not even on a guest). Netperf tests of the network show much higher speeds (approx 900Mbps) so the network is not the problem. The bottleneck is definitely SSH. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
network performance between host and guest...
Hi, For a backup of data from a VM to a USB mounted disk I want to circumvent the USB 1.1 limitations on the guest and instead copy the data over to the host using scp/ssh. I have setup a network using virtio and NAT like this: When I now create a 1GB file using dd and copy it over from the guest to the host, I am seeing a performance between 25-30 MB/s. My question is if this is normal because I have seen others on the internet achieve far greater speeds. In any case the speeds are comparable to current USB 2.0 speeds but I intend on using USB 3.0 so would like to get a little bit more out of it. What would I use to speed this up a bit futher? Cheers Erik -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: USB Passthrough 1.1 performance problem...
Erik Brakkee wrote: 2010/12/14 Erik Brakkee: Daniel P. Berrange wrote: On Tue, Dec 14, 2010 at 12:55:04PM +0100, Kenni Lund wrote: 2010/12/14 Erik Brakkee: From: Kenni Lund 2010/12/14 Erik Brakkee: From: Kenni Lund Does this mean I have a chance now that PCI passthrough of my WinTV PVR-500 might work now? Passthrough of a PVR-500 has been working for a long time. I've been running with passthrough of a PVR-500 in my HTPC, since November/December 2009...so it should work with any recent kernel and any recent version of qemu-kvm you can find today - No patching needed. The only issue I had with the PVR-500 card, was when *I* didn't free up the shared interrupts...once I fixed that, it "just worked". How did you free up those shared interrupts then? I tried different slots but always get conflicts with the USB irqs. I did an unbind of the conflicting device (eg. disabled it). I moved the PVR-500 card around in the different slots and once I got a conflict with the integrated sound card, I left the PVR-500 card in that slot (it's a headless machine, so no need for sound) and configured unbind of the sound card at boot time. On my old system I think it was conflicting with one of the USB controllers as well, but it didn't really matter, as I only lost a few of the ports on the back of the computer for that particular USB controller - I still had plenty of USB ports left and if I really needed more ports, I could just plug in an extra USB PCI card. My /etc/rc.local boot script looks like the following today: -- #Remove HDA conflicting with ivtv1 echo ":00:1b.0"> /sys/bus/pci/drivers/HDA\ Intel/unbind # ivtv0 echo " 0016"> /sys/bus/pci/drivers/pci-stub/new_id echo ":04:08.0"> /sys/bus/pci/drivers/ivtv/unbind echo ":04:08.0"> /sys/bus/pci/drivers/pci-stub/bind echo " 0016"> /sys/bus/pci/drivers/pci-stub/remove_id # ivtv1 echo " 0016"> /sys/bus/pci/drivers/pci-stub/new_id echo ":04:09.0"> /sys/bus/pci/drivers/ivtv/unbind echo ":04:09.0"> /sys/bus/pci/drivers/pci-stub/bind echo " 0016"> /sys/bus/pci/drivers/pci-stub/remove_id I did not try unbinding the usb device so I can also try that. I don'.t understand what is happening with the 0016. I configured the pci card in kvm and I believe kvm does the binding to pci-stub in recent versions. Where is the 0016%oming from? Okay, qemu-kvm might do it today, I don't know - I haven't changed that script for the past year. But are you sure that it's not libvirt/virsh/virt-manager which does that for you? If you use the managed="yes" attribute on the in libvirt XML, then libvirt will automatically do the pcistub bind/unbind, followed by a device reset at guest startup& the reverse at shutdown. If you have conflicting devices on the bus though, libvirt won't attempt to unbind them, unless you had also explicitly assigned all those conflicting devices to the same guest. Daniel I definitely have to try again (right now having some stability problems on the server that I am debugging). The shared IRQs are as follows: 16: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 18: 252995 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb8, ivtv0 19: 58281 0 0 0 0 0 0 0 IO-APIC-fasteoi ata_piix, ata_piix, uhci_hcd:usb5, uhci_hcd:usb7, ivtv1 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4 23:713 6906 0 76919 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6 So I have IRQ sharing with usb1, usb8, usb5, usb7. Uffand your ata HDD controller. I guess i was much luckier than you are, my ivtv0 didn't conflict at all and ivtv1 only conflicted with USB. I have also read that ehci refers to USB 2.0 and uhci to USB 1.1 is that correct? Anyway, how would I now identify the USB PCI devices that I would need to unbind to get rid of the sharing with the USB ports? Play around with: lspci -v lspci -n lsusb -v lsusb -t You can also just start by unbinding the first one and take note when you hit the right ones...once you unbind one, it will disappear from cat /proc/interrupts. When you're down to having only ivtv0 on one interrupt and only ivtv1 on another interrupt, then you're ready to bind with pci-stub and boot your guest. It
Re: USB Passthrough 1.1 performance problem...
> 2010/12/14 Erik Brakkee : >> Daniel P. Berrange wrote: >>> >>> On Tue, Dec 14, 2010 at 12:55:04PM +0100, Kenni Lund wrote: >>> >>>> >>>> 2010/12/14 Erik Brakkee: >>>> >>>>>> >>>>>> From: Kenni Lund >>>>>> 2010/12/14 Erik Brakkee: >>>>>> >>>>>>>> >>>>>>>> From: Kenni Lund >>>>>>>> >>>>>>>>> >>>>>>>>> Does this mean I have a chance now that PCI passthrough of my >>>>>>>>> WinTV >>>>>>>>> PVR-500 >>>>>>>>> might work now? >>>>>>>>> >>>>>>>> >>>>>>>> Passthrough of a PVR-500 has been working for a long time. I've >>>>>>>> been >>>>>>>> running with passthrough of a PVR-500 in my HTPC, since >>>>>>>> November/December 2009...so it should work with any recent kernel >>>>>>>> and >>>>>>>> any recent version of qemu-kvm you can find today - No patching >>>>>>>> needed. The only issue I had with the PVR-500 card, was when *I* >>>>>>>> didn't free up the shared interrupts...once I fixed that, it "just >>>>>>>> worked". >>>>>>>> >>>>>>> >>>>>>> How did you free up those shared interrupts then? I tried different >>>>>>> slots >>>>>>> but always get conflicts with the USB irqs. >>>>>>> >>>>>> >>>>>> I did an unbind of the conflicting device (eg. disabled it). I moved >>>>>> the PVR-500 card around in the different slots and once I got a >>>>>> conflict with the integrated sound card, I left the PVR-500 card in >>>>>> that slot (it's a headless machine, so no need for sound) and >>>>>> configured unbind of the sound card at boot time. On my old system I >>>>>> think it was conflicting with one of the USB controllers as well, >>>>>> but >>>>>> it didn't really matter, as I only lost a few of the ports on the >>>>>> back >>>>>> of the computer for that particular USB controller - I still had >>>>>> plenty of USB ports left and if I really needed more ports, I could >>>>>> just plug in an extra USB PCI card. >>>>>> >>>>>> My /etc/rc.local boot script looks like the following today: >>>>>> -- >>>>>> #Remove HDA conflicting with ivtv1 >>>>>> echo ":00:1b.0"> /sys/bus/pci/drivers/HDA\ Intel/unbind >>>>>> >>>>>> # ivtv0 >>>>>> echo " 0016"> /sys/bus/pci/drivers/pci-stub/new_id >>>>>> echo ":04:08.0"> /sys/bus/pci/drivers/ivtv/unbind >>>>>> echo ":04:08.0"> /sys/bus/pci/drivers/pci-stub/bind >>>>>> echo " 0016"> /sys/bus/pci/drivers/pci-stub/remove_id >>>>>> >>>>>> # ivtv1 >>>>>> echo " 0016"> /sys/bus/pci/drivers/pci-stub/new_id >>>>>> echo ":04:09.0"> /sys/bus/pci/drivers/ivtv/unbind >>>>>> echo ":04:09.0"> /sys/bus/pci/drivers/pci-stub/bind >>>>>> echo " 0016"> /sys/bus/pci/drivers/pci-stub/remove_id >>>>>> >>>>> >>>>> I did not try unbinding the usb device so I can also try that. >>>>> >>>>> I don'.t understand what is happening with the 0016. I >>>>> configured >>>>> the >>>>> pci card in kvm and I believe kvm does the binding to pci-stub in >>>>> recent >>>>> versions. Where is the 0016%oming from? >>>>> >>>> >>>> Okay, qemu-kvm might do it today, I don't know - I haven't changed >>>> that script for the past year. But are you sure that it's not >>>> libvirt/virsh/virt-manager which does that for you? >>>> >>> >>> If you use the managed="yes" attribute on the in libvirt >>> XML, then libvirt will automatically do the pcistub bind/unbind, >>> followed by a device reset
Re: USB Passthrough 1.1 performance problem...
Daniel P. Berrange wrote: On Tue, Dec 14, 2010 at 12:55:04PM +0100, Kenni Lund wrote: 2010/12/14 Erik Brakkee: From: Kenni Lund 2010/12/14 Erik Brakkee: From: Kenni Lund Does this mean I have a chance now that PCI passthrough of my WinTV PVR-500 might work now? Passthrough of a PVR-500 has been working for a long time. I've been running with passthrough of a PVR-500 in my HTPC, since November/December 2009...so it should work with any recent kernel and any recent version of qemu-kvm you can find today - No patching needed. The only issue I had with the PVR-500 card, was when *I* didn't free up the shared interrupts...once I fixed that, it "just worked". How did you free up those shared interrupts then? I tried different slots but always get conflicts with the USB irqs. I did an unbind of the conflicting device (eg. disabled it). I moved the PVR-500 card around in the different slots and once I got a conflict with the integrated sound card, I left the PVR-500 card in that slot (it's a headless machine, so no need for sound) and configured unbind of the sound card at boot time. On my old system I think it was conflicting with one of the USB controllers as well, but it didn't really matter, as I only lost a few of the ports on the back of the computer for that particular USB controller - I still had plenty of USB ports left and if I really needed more ports, I could just plug in an extra USB PCI card. My /etc/rc.local boot script looks like the following today: -- #Remove HDA conflicting with ivtv1 echo ":00:1b.0"> /sys/bus/pci/drivers/HDA\ Intel/unbind # ivtv0 echo " 0016"> /sys/bus/pci/drivers/pci-stub/new_id echo ":04:08.0"> /sys/bus/pci/drivers/ivtv/unbind echo ":04:08.0"> /sys/bus/pci/drivers/pci-stub/bind echo " 0016"> /sys/bus/pci/drivers/pci-stub/remove_id # ivtv1 echo " 0016"> /sys/bus/pci/drivers/pci-stub/new_id echo ":04:09.0"> /sys/bus/pci/drivers/ivtv/unbind echo ":04:09.0"> /sys/bus/pci/drivers/pci-stub/bind echo " 0016"> /sys/bus/pci/drivers/pci-stub/remove_id I did not try unbinding the usb device so I can also try that. I don'.t understand what is happening with the 0016. I configured the pci card in kvm and I believe kvm does the binding to pci-stub in recent versions. Where is the 0016%oming from? Okay, qemu-kvm might do it today, I don't know - I haven't changed that script for the past year. But are you sure that it's not libvirt/virsh/virt-manager which does that for you? If you use the managed="yes" attribute on the in libvirt XML, then libvirt will automatically do the pcistub bind/unbind, followed by a device reset at guest startup& the reverse at shutdown. If you have conflicting devices on the bus though, libvirt won't attempt to unbind them, unless you had also explicitly assigned all those conflicting devices to the same guest. Daniel I definitely have to try again (right now having some stability problems on the server that I am debugging). The shared IRQs are as follows: 16: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 18: 252995 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb8, ivtv0 19: 58281 0 0 0 0 0 0 0 IO-APIC-fasteoi ata_piix, ata_piix, uhci_hcd:usb5, uhci_hcd:usb7, ivtv1 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4 23:713 6906 0 76919 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6 So I have IRQ sharing with usb1, usb8, usb5, usb7. I have also read that ehci refers to USB 2.0 and uhci to USB 1.1 is that correct? Anyway, how would I now identify the USB PCI devices that I would need to unbind to get rid of the sharing with the USB ports? It also doesn't really matter in which slot I put the PVR-500 card because both cards share IRQs with USB in all cases. I have also used an add on USB PCI card but still got these conflicts. I was considering to get a PCIe USB card instead to try out in the hope that that would use different IRQs. Is that a realistic expectation? That way, I could disable all on-board USB (in the BIOS even) and use the add-on USB only. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: USB Passthrough 1.1 performance problem...
Jan Kiszka wrote: Are there some tuning parameters I can use or perhaps even kernel configuration paramters on the host to solve this? Cheers Erik Host:Motherboard Supermicro X8DTi-F, Intel Xeon L5630, 12MB OS: Opensuse 11.3 64 bit Guest: OS: Opensuse 11.3 64 bit I can say now that I am giving up on getting this to work. One alternative was to use PCI passthrough the USB hardware, but that didn't work for the USB that was on the motherboard. So I bought a USB PCI card and tried to use PCI passthrough for that. Unfortunately other problems occured there. For one, the problem with 4K alignment. But I could fix that by using the pci=resource_alignment=... kernel parameter. In my grub/menu.lst it says: kernel /vmlinuz-2.6.34.7-0.5-default root=/dev/hsystem/root quiet showopts intel_iommu=on pci=resource_alignment=01:04.0;01:04.1;01:04.2 noirqdebug vga=0x31a The noirqdebug flas was needed to avoid the host from disabling the IRQ (it was a shared IRQ). Using this, I could configure PCI passthrough and start the VM. Also the USB device showed up there. Only it did not work at all. Here is a summary of my journey up until know: The original approach I wanted to use was to pass my old PCI card (WinTV PVR-500) to a VM. This card is a well supported card and has been doing fine for me. Because of the PCI passthrough problems with the wintv card, I decided to try a USB card instead. This gave me a 'ctrl buffer too small' issue that I could solve by taking the source RPM for kvm and applying a known patch from red hat (increasing buffer size from 2048 to 8192). But then I got jerky video, probably due to USB 1.1 issues. To bypass these I could use PCI passthrough for USB. But with the PCI passthrough of this card I am again running into issues probably related to Shared IRQs. So, after all this I am back to square one. I have now modified my approach so instead of running a separate minimal host with my old server as a guest, I am now running the old server (same install) on the new hardware, using it as a host. I would definitely be interested in trying this out further in the future. I even tried Xen for a brief moment, only to realize that my host and guest felt slower (slower startup and execution) and much more difficult to handle. From the experience of the last two days fulltime trying to get things working I can only conclude that the following two features would be really important to have: * Extended PCI passthrough support o shared IRQ support Addressed by the series I sent out today. Does this mean I have a chance now that PCI passthrough of my WinTV PVR-500 might work now? What version is this and where can I get this for opensuse? I still have the setup I used for testing with the host OS still installed but not running so it would be really easy to try out new releases of KVM (it is not a serious production server after all but mainly used to run some websites and mailing lists). o supporting cases where memory is not aligned on a 4K boundary Hmm, I'm seeing warnings here when passing through one of my EHCIs, but no fatal errors. In my case, the domain just didn't start. Btw. I was using 0.12.5 on opensuse 11.3 but could only find the sources for 0.12.3 on download.opensuse.org (perhaps I looked wrong) and I patched those for th 4K issue. PCI passthrough also did not work with my wintv pci card with KVM 0.12.5. * USB passthrough o support USB 2.0 o support USB 3.0 (but taking one step at a time, 2.0 would also be great). Note that this will not solve any real-time issue (if that is part of your problem). E.g.: While my EHCIs work nicely in PCI-passthrough scenarios, I'm unable to use certain webcams that sooner or later run out of sync. Jan Is your point in this case that USB in a VM based on PCI passthrough will always have problems when it comes to more real-time issues or does this only apply to USB passthrough? I can imagine that PCI passthrough is better since it uses hardware support. By the way, I have seen issues in the past whereby the tv card stopped working because of high load on the server running natively so real-time issues also exist apart from virtualization. Apart from this, I am definitely interested in getting updates for KVM on opensuse. Is there a channel for that? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: USB Passthrough 1.1 performance problem...
Erik Brakkee wrote: Hi, I am using a tv card in a VM and get jerky video.As I understand it, the VM is using USB 1.1. However, when I set the USB controller in the BIOS of my server to Fullspeed (12 Mbit/s) which is the USB 1.1 speed I am able to get perfect results on the host but still on the guest the video is jerky. Are there some tuning parameters I can use or perhaps even kernel configuration paramters on the host to solve this? Cheers Erik Host:Motherboard Supermicro X8DTi-F, Intel Xeon L5630, 12MB OS: Opensuse 11.3 64 bit Guest: OS: Opensuse 11.3 64 bit I can say now that I am giving up on getting this to work. One alternative was to use PCI passthrough the USB hardware, but that didn't work for the USB that was on the motherboard. So I bought a USB PCI card and tried to use PCI passthrough for that. Unfortunately other problems occured there. For one, the problem with 4K alignment. But I could fix that by using the pci=resource_alignment=... kernel parameter. In my grub/menu.lst it says: kernel /vmlinuz-2.6.34.7-0.5-default root=/dev/hsystem/root quiet showopts intel_iommu=on pci=resource_alignment=01:04.0;01:04.1;01:04.2 noirqdebug vga=0x31a The noirqdebug flas was needed to avoid the host from disabling the IRQ (it was a shared IRQ). Using this, I could configure PCI passthrough and start the VM. Also the USB device showed up there. Only it did not work at all. Here is a summary of my journey up until know: The original approach I wanted to use was to pass my old PCI card (WinTV PVR-500) to a VM. This card is a well supported card and has been doing fine for me. Because of the PCI passthrough problems with the wintv card, I decided to try a USB card instead. This gave me a 'ctrl buffer too small' issue that I could solve by taking the source RPM for kvm and applying a known patch from red hat (increasing buffer size from 2048 to 8192). But then I got jerky video, probably due to USB 1.1 issues. To bypass these I could use PCI passthrough for USB. But with the PCI passthrough of this card I am again running into issues probably related to Shared IRQs. So, after all this I am back to square one. I have now modified my approach so instead of running a separate minimal host with my old server as a guest, I am now running the old server (same install) on the new hardware, using it as a host. I would definitely be interested in trying this out further in the future. I even tried Xen for a brief moment, only to realize that my host and guest felt slower (slower startup and execution) and much more difficult to handle. From the experience of the last two days fulltime trying to get things working I can only conclude that the following two features would be really important to have: * Extended PCI passthrough support o shared IRQ support o supporting cases where memory is not aligned on a 4K boundary * USB passthrough o support USB 2.0 o support USB 3.0 (but taking one step at a time, 2.0 would also be great). From the above, PCI passthrough improvements are most important as they will also allow (cheap) workarounds for USB passthrough issues. I am definitely interested in hearing about new releases from KVM. Is there a mailing list I should subscribe to get information about the latest releases without intermediate discussions? I would also be interested in using latest stable versions of KVM for opensuse. Is there any repository I can configure to get those? Cheers Erik -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
USB Passthrough 1.1 performance problem...
Hi, I am using a tv card in a VM and get jerky video.As I understand it, the VM is using USB 1.1. However, when I set the USB controller in the BIOS of my server to Fullspeed (12 Mbit/s) which is the USB 1.1 speed I am able to get perfect results on the host but still on the guest the video is jerky. Are there some tuning parameters I can use or perhaps even kernel configuration paramters on the host to solve this? Cheers Erik Host:Motherboard Supermicro X8DTi-F, Intel Xeon L5630, 12MB OS: Opensuse 11.3 64 bit Guest: OS: Opensuse 11.3 64 bit -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough on Sony Vaio F11 laptop...
Jan Kiszka wrote: Am 15.11.2010 22:55, Erik Brakkee wrote: Jan Kiszka wrote: Am 14.11.2010 14:21, Erik Brakkee wrote: Jan Kiszka wrote: Strange, should work. I would suggest to post your full kernel log, maybe there is some enlightening message hidden. I don't think it is a problem of your kernel version, but I'm able to pass through devices on OpenSUSE 11.3 with kernel-desktop-2.6.36-90.1.x86_64 from their kernel repository. Jan Exactly what server logs do you need. Is this only /var/log/messages or more? And do I need to set specific options there? Any other log files that you need? dmesg> log-file Before, generating these logs I will upgrade to a later kernel. As far as I can tell, that will still be a 2.6.34 kernel. Perhaps I should try the 2.6.36 kernel as well. Do you have the URL for the kernel repository I should use? (cannot find an obvious kernel repository in YAST2). http://download.opensuse.org/repositories/Kernel:/HEAD/openSUSE_11.3 Jan I have attached the logs of /var/log/messages, dmesg, and qemu log (other.log), as well as the kernel config parameters (/proc/config.gz). I did the test with two kernels (see the tar.gz file): one kernel a 2.6.34 and the other a 2.6.36 kernel. Comparing the dmesg with my kernel log, I'm missing messages like [0.023960] DMAR: Host address width 36 [0.023962] DMAR: DRHD base: 0x00fed9 flags: 0x0 [0.023968] IOMMU 0: reg_base_addr fed9 ver 1:0 cap c9008020e30272 ecap 1000 [0.023970] DMAR: DRHD base: 0x00fed93000 flags: 0x1 [0.023974] IOMMU 1: reg_base_addr fed93000 ver 1:0 cap c9008020630272 ecap 1000 [0.023976] DMAR: RMRR base: 0x00bf6e9000 end: 0x00bf6f [0.023978] DMAR: No ATSR found about the Intel DMAR (IOMMU) setup. Are you sure that you have an Intel chipset with the required features? And have you checked that "VT-d" is enabled in the BIOS (or however it may be called there)? Jan I have now done a PCI passthrough test on some real hardware. Supermicro X8DTi-F motherboard (intel 5520 chipset), Intel Xeon L5630 processor. I enabled Intel VT-d in the BIOS and them simply selected one of the network interfaces for passthrough the the VM. And then... it just worksd. Also, I am seeing output similar to what you posted above. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
windows vista enterpise in KVM and multi-core....
Hi all, I got KVM to run a windows vista enterprise installation. The problem that I am seeing now is that windows vista is seeing only two processors when I am in fact passing 8 (quad core hyperthreading). The issue seems to be that the 8 virtual CPUs on my single physical CPU appear as 8 distinct processors on windows vista enterprise and that edition is limited to two physical processors (see for instance http://en.wikipedia.org/wiki/Windows_Vista_editions#Comparison_chart). Is there a way in which I can pass the 8 virtual CPUs to the vista image in such a way that they appear as one CPU with 8 cores? Cheers Erik -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough on Sony Vaio F11 laptop...
Jan Kiszka wrote: Strange, should work. I would suggest to post your full kernel log, maybe there is some enlightening message hidden. I don't think it is a problem of your kernel version, but I'm able to pass through devices on OpenSUSE 11.3 with kernel-desktop-2.6.36-90.1.x86_64 from their kernel repository. Jan Exactly what server logs do you need. Is this only /var/log/messages or more? And do I need to set specific options there? Any other log files that you need? Before, generating these logs I will upgrade to a later kernel. As far as I can tell, that will still be a 2.6.34 kernel. Perhaps I should try the 2.6.36 kernel as well. Do you have the URL for the kernel repository I should use? (cannot find an obvious kernel repository in YAST2). Cheers Erik -- Nonsense and other useful things: http://brakkee.org MountainHoppers: http://mountainhoppers.nl Track Detective: http://trackdetective.com Twitter: http://twitter.com/ErikBrakkee -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: PCI passthrough on Sony Vaio F11 laptop...
Jan Kiszka wrote: What IRQ is the sky2 using when assigned to the host? Is it really a shared IRQ (I bet not as it should be using MSI)? Also, check in the libvirt logs what qemu-kvm reports on the console. Jan The output from 'cat /proc/interrupts' directly after boot is in the sky2.interrupts. The relevant line from the output is 35: 0 0 0 0 1 0 0 0 PCI-MSI-edge s...@pci::04:00.0 So it looks like it is using MSI (although I don't have clue what that means). The log file for my domain (the name is "other") is attached in "other.log" In the log file I see: No IOMMU found. Unable to assign device "hostdev0" Does this mean that I don't have IOMMU available on my laptop? The output from 'dmesg | grep -i IOMMU' showed that Intel-IOMMI was enabled, but perhaps that means only that the IOMMU option is activated but not that it is really functioning. It must be either (1) Intel VT-d available on my laptop and there is some configuration/software problem or (2) The output from dmesg is misleading. What do you think? Cheers Erik -- Nonsense and other useful things: http://brakkee.org MountainHoppers: http://mountainhoppers.nl Track Detective: http://trackdetective.com Twitter: http://twitter.com/ErikBrakkee CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0:879 0 0 145987 0 0 0 0 IO-APIC-edge timer 1:178 0 0 37 0 0 0 0 IO-APIC-edge i8042 8: 0 0 0 15 0 0 0 0 IO-APIC-edge rtc0 9: 1702 0 0133 0 0 0 0 IO-APIC-fasteoi acpi 12: 0 0 0149 0 0 0 0 IO-APIC-edge i8042 16: 0 20448 0 0 0 0 429 0 IO-APIC-fasteoi ehci_hcd:usb1, ohci1394, hda_intel, nvidia 17: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi mmc0 19: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi mmc1 23: 0 0 65 0 0 2113 0 0 IO-APIC-fasteoi ehci_hcd:usb2 24: 140835 0 0 0 0 0 0 0 HPET_MSI-edge hpet2 25: 0 116293 0 0 0 0 0 0 HPET_MSI-edge hpet3 26: 0 0 47034 0 0 0 0 0 HPET_MSI-edge hpet4 27: 0 0 0 109561 0 0 0 0 HPET_MSI-edge hpet5 28: 0 0 0 0 32441 0 0 0 HPET_MSI-edge hpet6 34: 4355 0 0 9520 0 0 0 0 PCI-MSI-edge ahci 35: 0 0 0 0 1 0 0 0 PCI-MSI-edge s...@pci::04:00.0 36: 59335 0 0 0 5302 0 0 0 PCI-MSI-edge iwlagn 37: 0 0 0 0 0207 0 0 PCI-MSI-edge hda_intel NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts LOC: 1613 1517 1427 1337 1247 37604 110821 35565 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 0 0 0 0 Performance monitoring interrupts PND: 0 0 0 0 0 0 0 0 Performance pending work RES:463661442372542520 535507 Rescheduling interrupts CAL: 1361489 1329 1260 1353 1340 1240 1322 Function call interrupts TLB:281265281381800810 779553 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP:
PCI passthrough on Sony Vaio F11 laptop...
Hi all, I have KVM running successfully on opensuse 11.3 (64bit) with a opensuse 11.1 guest. I am using hardware virtualization and want to experiment with PCI passthrough with the (wired) network card on my laptop because I want to repeat similar steps later on a server (with a TV card). The main specs of the laptop are Intel Core I7 Q720 1.6GHz with 8MB of memory. I have followed the instructions on http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization/chap-Virtualization-PCI_passthrough.html. Everything is successful until I start the domain. In that case I get the following messages in the logs: Nov 13 00:23:45 pelican kernel: [ 3302.793043] pci-stub :04:00.0: claimed by stub Nov 13 00:23:45 pelican kernel: [ 3302.793197] pci-stub :04:00.0: claimed by stub Nov 13 00:23:45 pelican kernel: [ 3302.931578] kvm_vm_ioctl_deassign_device: device hasn't been assigned before, so cannot be deassigned Nov 13 00:23:48 pelican libvirtd: 00:23:48.136: error : qemuMonitorOpenUnix:269 : monitor socket did not show up.: Connection refused Nov 13 00:23:48 pelican libvirtd: 00:23:48.136: error : qemuConnectMonitor:1192 : Failed to connect monitor for other To be absolutely certa CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0:371 0 0 203026 0 0 0 0 IO-APIC-edge timer 1: 12870 0 0 39173 0 0 0 IO-APIC-edge i8042 7:259 0 0 0 0 0 0 0 IO-APIC-edge 8: 0 0 0 4 0 0 0 0 IO-APIC-edge rtc0 9: 2823 0 0141 0 0 0 0 IO-APIC-fasteoi acpi 12: 15 0 0155 0 0 0 0 IO-APIC-edge i8042 16: 0 298817 0 0 0 0634 0 IO-APIC-fasteoi ehci_hcd:usb1, ohci1394, hda_intel, nvidia 17: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi mmc0 19: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi mmc1 23: 0 0 99 0 0 95116 0 0 IO-APIC-fasteoi ehci_hcd:usb2 24: 687633 0 0 0 0 0 0 0 HPET_MSI-edge hpet2 25: 0 637204 0 0 0 0 0 0 HPET_MSI-edge hpet3 26: 0 0 199161 0 0 0 0 0 HPET_MSI-edge hpet4 27: 0 0 0 134606 0 0 0 0 HPET_MSI-edge hpet5 28: 0 0 0 0 237717 0 0 0 HPET_MSI-edge hpet6 34: 58784 0 0 7780 0 0 0 0 PCI-MSI-edge ahci 36:939 0 0 0 138686 0 0 0 PCI-MSI-edge iwlagn 37: 0 0 0 0 0 210 0 0 PCI-MSI-edge hda_intel NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts LOC: 1535 1438 1347 1257 1168 396803 120435 79080 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 0 0 0 0 Performance monitoring interrupts PND: 0 0 0 0 0 0 0 0 Performance pending work RES: 4164 4332 2050 1413 3244 2790 1630 1200 Rescheduling interrupts CAL: 1394 1395 1279 1195 1431 1430 1429575 Function call interrupts TLB: 5007 2623 3751 3045 5617 3902 4486 4115 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 14 14 14 14 14 14 14 14 Machine check polls ERR:259 MIS: 0
