RE: Virtio network performance problem
I think it is an unsync tsc problem. First, make sure you pin all of the process threads. There is thread per vcpu + io thread +more non relevant. You can do it by adding the taskset before the cmdline. Second, you said that you use smp guest. So windows also sees unsync tsc. So, either test with UP guest or learn how to pin windows receiving ISR, DPC and the user app. Well, testing on Intel or newer AMD is another option. I tested it again now on Intel with UP guest and there is no such a problem. Hope to test it next week on AMD SMP guest. Regards, Dor I made sure to pin all 5 threads from my guest kvm process to two cores on a single socket, and also tried pinning them all to a single core, but neither made a difference. I then powered down all the cores (using the /sys entries) until only two cores on a single socket were left, which made no improvement. I then powered all cores down except one (and verified that only the one showed up in /proc/cpuinfo). Naturally the guest slowed a bit, but the erratic and negative pings from the guest to the host br0 were still there. If I boot the guest with smp 1 on the kvm command line, the ping times are fine. It's only when I start the guest with 2 VCPUs that the ping times are erratic and some pings are negative. Just to recap, this is on a brand new Dell Poweredge R805 with dual quad-core Opteron 2350s, with KVM-79 built against the CentOS-packaged 2.6.18 kernel. The guest in question is Windows Server 2003 x64 SP2 with 2 VCPUs. I couldn't think of anything else to try, so I loaded up another server, a new Dell Poweredge 2950 with dual quad-core Xeon E5430s. I installed and configured kvm-79 following the exact same process I used to configure it on the AMD host. When I boot the _same_ guest (it's in iscsi) on the PE2950/Intel host, the ping times are perfect. I'm interested to know why this is (before I purchase any more AMD servers) but my priority right now is trying to figure out what I'm doing wrong that is causing virtio networking to be extremely slow (usually around 55Mb/s) on every host I've built it on. I'm sure I'm doing something wrong and I just can't place it. I'll start another thread for that, though, because it seems to be a different problem. In the meantime I'll just keep this guest on my Intel servers, I guess. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Virtio network performance problem
On Wed, Dec 03, 2008 at 11:20:08AM -0800, Chris Wedgwood wrote: TSC instability? Is this an SMP guest? Ok, I tried pinning the kvm process to two cores (0,2) on a single socket, but that didn't seem to make any difference for my virtio network performance. I also tried pinning the process to a single core, which also didn't seem to have any effect. Someone on IRC suggested that it sounded like a clocking issue, since some of my ping times are negative. He suggested trying a different clock source. I tried it with dynticks, rtc, and unix. None of them seem better, although all of them seem different in terms of patterns in the ping times. Sorry if this makes it a long post, but I don't know how to describe it other than to paste an example (below). Not sure if this indicates that it is clock-related or if it is meaningless. In any event, I'm not sure where to go from here. Another suggestion from IRC was that it was due to the age of my host kernel (2.6.18) and the fact that it doesn't support high-res timers. If I can avoid replacing the distro kernel, I'd like to, but I'll do what I have to, I suppose. With dynticks (these are all with -net user, as I had some trouble with my tap interface last night while testing this. The results are roughly the same as when I was using tap before, though): Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=143ms TTL=255 Reply from 10.0.2.2: bytes=32 time=143ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=143ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-139ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-141ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-133ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=143ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 With rtc: Reply from 10.0.2.2: bytes=32 time=-224ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-223ms TTL=255 Reply from 10.0.2.2: bytes=32 time=4ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=225ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-223ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-224ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=225ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=225ms TTL=255 Reply from 10.0.2.2: bytes=32 time=225ms TTL=255 With unix: Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-190ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=192ms TTL=255 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Virtio network performance problem
Adrian Schmitz wrote: On Wed, Dec 03, 2008 at 11:20:08AM -0800, Chris Wedgwood wrote: TSC instability? Is this an SMP guest? Ok, I tried pinning the kvm process to two cores (0,2) on a single socket, but that didn't seem to make any difference for my virtio network performance. I also tried pinning the process to a single core, which also didn't seem to have any effect. I think it is an unsync tsc problem. First, make sure you pin all of the process threads. There is thread per vcpu + io thread +more non relevant. You can do it by adding the taskset before the cmdline. Second, you said that you use smp guest. So windows also sees unsync tsc. So, either test with UP guest or learn how to pin windows receiving ISR, DPC and the user app. Well, testing on Intel or newer AMD is another option. I tested it again now on Intel with UP guest and there is no such a problem. Hope to test it next week on AMD SMP guest. Regards, Dor Someone on IRC suggested that it sounded like a clocking issue, since some of my ping times are negative. He suggested trying a different clock source. I tried it with dynticks, rtc, and unix. None of them seem better, although all of them seem different in terms of patterns in the ping times. Sorry if this makes it a long post, but I don't know how to describe it other than to paste an example (below). Not sure if this indicates that it is clock-related or if it is meaningless. In any event, I'm not sure where to go from here. Another suggestion from IRC was that it was due to the age of my host kernel (2.6.18) and the fact that it doesn't support high-res timers. If I can avoid replacing the distro kernel, I'd like to, but I'll do what I have to, I suppose. With dynticks (these are all with -net user, as I had some trouble with my tap interface last night while testing this. The results are roughly the same as when I was using tap before, though): Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=143ms TTL=255 Reply from 10.0.2.2: bytes=32 time=143ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=143ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-139ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-141ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-133ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=143ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 With rtc: Reply from 10.0.2.2: bytes=32 time=-224ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-223ms TTL=255 Reply from 10.0.2.2: bytes=32 time=4ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=225ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-223ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-224ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=225ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=225ms TTL=255 Reply from 10.0.2.2: bytes=32 time=225ms TTL=255 With unix: Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-190ms TTL=255 Reply from 10.0.2.2: bytes=32 time=-191ms TTL=255 Reply from 10.0.2.2: bytes=32 time=1ms TTL=255 Reply from 10.0.2.2: bytes=32 time=192ms TTL=255 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Virtio network performance problem
I'm having a problem with virtio networking. I mentioned this in a previous thread, but I'm starting a new one since Dor solved the problem I reported in the original thread. This appears to be a new problem. Iperf shows throughput of ~120 Mb/s between a Windows Server 2003 x64 guest and the br0 interface on the host. Also, a ping between the same two interfaces looks pretty bad: Pinging 10.10.10.128 with 32 bytes of data: Reply from 10.10.10.128: bytes=32 time=-389ms TTL=64 Reply from 10.10.10.128: bytes=32 time1ms TTL=64 Reply from 10.10.10.128: bytes=32 time=392ms TTL=64 Reply from 10.10.10.128: bytes=32 time=392ms TTL=64 Reply from 10.10.10.128: bytes=32 time=377ms TTL=64 Reply from 10.10.10.128: bytes=32 time=392ms TTL=64 Reply from 10.10.10.128: bytes=32 time1ms TTL=64 Reply from 10.10.10.128: bytes=32 time=401ms TTL=64 Reply from 10.10.10.128: bytes=32 time=1ms TTL=64 Using e1000 driver on this guest, iperf shows 320+ Mb/s throughput, and latency looks like it should. Host kernel:2.6.18 KVM Version:79 (modules and tools built and installed from source) Guest OS: Win2k3 x64 with all service packs and patches Guest virtio drivers: Version 3 (received from Dor Laor yesterday.. these fixed the guest OS crashes I was having with version 2) Any suggestions are greatly appreciated. If there's additional info I can provide, please let me know. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Virtio network performance problem
On Wed, Dec 03, 2008 at 11:20:08AM -0800, Chris Wedgwood wrote: TSC instability? Is this an SMP guest? Actually, worded badly. A UP guest on an SMP host with unsync'ed TSCs would show this too obviously. As would changes in the TSC speed. The first I guess we can avoid by pinning that latter I'm not so sure about. I don't know enough to tell whether there's a TSC problem or not, to be honest. The host has 2X AMD quad-core CPUs (let me know if you want the /proc/cpuinfo). The guest has 2 VCPUs. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Virtio network performance problem
On Wed, Dec 03, 2008 at 02:11:45PM -0500, Adrian Schmitz wrote: Iperf shows throughput of ~120 Mb/s between a Windows Server 2003 x64 guest and the br0 interface on the host. Also, a ping between the same two interfaces looks pretty bad: Pinging 10.10.10.128 with 32 bytes of data: Reply from 10.10.10.128: bytes=32 time=-389ms TTL=64 TSC instability? Is this an SMP guest? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Virtio network performance problem
On Wed, Dec 03, 2008 at 11:20:08AM -0800, Chris Wedgwood wrote: TSC instability? Is this an SMP guest? Actually, worded badly. A UP guest on an SMP host with unsync'ed TSCs would show this too obviously. As would changes in the TSC speed. The first I guess we can avoid by pinning that latter I'm not so sure about. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Virtio network performance problem
Adrian Schmitz wrote: I'm having a problem with virtio networking. I mentioned this in a previous thread, but I'm starting a new one since Dor solved the problem I reported in the original thread. This appears to be a new problem. On an older kernel, you'll likely see packet corruption with virtio-net because of an uninitialized variable. Please try the patch I just posted (Fix uninitialized offset in virtio-net receive_header) and let me know if it helps. Regards, Anthony Liguori -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
