Re: VM in Xen 4.1; poor packet forwarding performance on xn0
Hi Roger, This is what they provided me: - The Xen 4.x node is running a custom 3.1.1 kernel. - The Xen 3.x node is running a 2.6.18 kernel. They didn't elaborate on the customisations. :( Sean On Sat, Apr 26, 2014 at 1:06 PM, Roger Pau Monné roger@citrix.comwrote: On 26/04/14 13:57, seanr...@gmail.com wrote: Hi Roger, Thanks for the patch -- sadly, it didn't work. No change. Hello, I did have to modify a bit for releng/10.0; for some reason patch refused to apply it cleanly. It looked fairly straightforward but I attached inline the patch I ultimately applied below just in case I got it wrong. Patch looks fine. Are there any other potential differences between Xen 3.4 and 4.1? (my provider migrated my problem VPS to a 3.4 host and the problem evaporated; I am trying this on a new 4.1 VPS that I was able to reproduce the problem on). I'm quite sure this is not related to the underlying Xen version, but to the Linux Dom0 kernel version your provider is using, could you ask them the Linux versions they are using on the 4.1 and 3.4 hosts? Roger. ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
Hi Roger,
Thanks for the patch -- sadly, it didn't work. No change.
I did have to modify a bit for releng/10.0; for some reason patch refused
to apply it cleanly. It looked fairly straightforward but I attached inline
the patch I ultimately applied below just in case I got it wrong.
Are there any other potential differences between Xen 3.4 and 4.1? (my
provider migrated my problem VPS to a 3.4 host and the problem evaporated;
I am trying this on a new 4.1 VPS that I was able to reproduce the problem
on).
Sean
Index: hvm.c
===
--- hvm.c (revision 264963)
+++ hvm.c (working copy)
@@ -626,6 +626,7 @@
xhp.domid = DOMID_SELF;
xhp.index = HVM_PARAM_CALLBACK_IRQ;
+#if 0
if (xen_feature(XENFEAT_hvm_callback_vector) != 0) {
int error;
@@ -638,6 +639,7 @@
printf(Xen HVM callback vector registration failed (%d).
Falling back to emulated device interrupt\n, error);
}
+#endif
xen_vector_callback_enabled = 0;
if (dev == NULL) {
/*
@@ -783,7 +785,7 @@
info.mfn = vtophys(vcpu_info) PAGE_SHIFT;
info.offset = vtophys(vcpu_info) - trunc_page(vtophys(vcpu_info));
- rc = HYPERVISOR_vcpu_op(VCPUOP_register_vcpu_info, cpu, info);
+ rc = 1;
if (rc != 0)
DPCPU_SET(vcpu_info, HYPERVISOR_shared_info-vcpu_info[cpu]);
else
On Tue, Apr 22, 2014 at 3:10 PM, Roger Pau Monné roger@citrix.comwrote:
On 18/04/14 16:57, seanr...@gmail.com wrote:
Hi there freebsd-xen,
I tried first on freebsd-questions@ without success, so I thought to
retry
here.
I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am
experiencing
slow network performance on xn0 that seems to have developed after I
upgraded to FreeBSD 10 (no other changes). I can only achieve about
0.5mbps
through this interface when forwarding packets; packets in a single
direction are fine (e.g; downloading to the VPS or pushing from the VPS)
and clock in at many (10 usually) mbps.
Interestingly, my identical VM (configuration managed centrally) running
on
Xen 3.4 (HVM) does *not* have this issue.
Hello,
The difference between Xen 3.4 and Xen 4.1 is that FreeBSD will make use
of the vector callback, the PV timer and PV IPIs when running on Xen
4.0 (which should provide better performance). I'm attaching a patch
that will make FreeBSD behave the same way when running on either Xen
3.4 or Xen 4.1 (by disabling all this new additions), could you please
give it a try?
I did a little debugging and here's what I've noticed:
- Not related to OpenVPN, repro'd using ssh -d.
- Slow VM has a very low rate of context switches (~250) while
forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using
systat -v.
- I can't repro a context switch limit (tried a limited fork() bomb).
- Tried with *and* without LRO and TSO on xn0 (and all combinations of
LRO and TSO on/off)
I've got the feeling that the issue you are seeing is not related to the
Xen version itself, but the Linux Dom0 kernel version (which I suppose
is different in the Xen 3.4 and Xen 4.1 hosts). Could you ask your
provider which Linux Dom0 kernel are they using on the different hosts?
Roger.
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
On 26/04/14 13:57, seanr...@gmail.com wrote: Hi Roger, Thanks for the patch -- sadly, it didn't work. No change. Hello, I did have to modify a bit for releng/10.0; for some reason patch refused to apply it cleanly. It looked fairly straightforward but I attached inline the patch I ultimately applied below just in case I got it wrong. Patch looks fine. Are there any other potential differences between Xen 3.4 and 4.1? (my provider migrated my problem VPS to a 3.4 host and the problem evaporated; I am trying this on a new 4.1 VPS that I was able to reproduce the problem on). I'm quite sure this is not related to the underlying Xen version, but to the Linux Dom0 kernel version your provider is using, could you ask them the Linux versions they are using on the 4.1 and 3.4 hosts? Roger. ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
On 18/04/14 16:57, seanr...@gmail.com wrote:
Hi there freebsd-xen,
I tried first on freebsd-questions@ without success, so I thought to retry
here.
I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am experiencing
slow network performance on xn0 that seems to have developed after I
upgraded to FreeBSD 10 (no other changes). I can only achieve about 0.5mbps
through this interface when forwarding packets; packets in a single
direction are fine (e.g; downloading to the VPS or pushing from the VPS)
and clock in at many (10 usually) mbps.
Interestingly, my identical VM (configuration managed centrally) running on
Xen 3.4 (HVM) does *not* have this issue.
Hello,
The difference between Xen 3.4 and Xen 4.1 is that FreeBSD will make use
of the vector callback, the PV timer and PV IPIs when running on Xen
4.0 (which should provide better performance). I'm attaching a patch
that will make FreeBSD behave the same way when running on either Xen
3.4 or Xen 4.1 (by disabling all this new additions), could you please
give it a try?
I did a little debugging and here's what I've noticed:
- Not related to OpenVPN, repro'd using ssh -d.
- Slow VM has a very low rate of context switches (~250) while
forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using
systat -v.
- I can't repro a context switch limit (tried a limited fork() bomb).
- Tried with *and* without LRO and TSO on xn0 (and all combinations of
LRO and TSO on/off)
I've got the feeling that the issue you are seeing is not related to the
Xen version itself, but the Linux Dom0 kernel version (which I suppose
is different in the Xen 3.4 and Xen 4.1 hosts). Could you ask your
provider which Linux Dom0 kernel are they using on the different hosts?
Roger.
From a866add6f87d726ae2c6ab5837d1b6eefb721be8 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne roger@citrix.com
Date: Tue, 22 Apr 2014 09:50:09 +0200
Subject: [PATCH]
---
sys/x86/xen/hvm.c |4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/sys/x86/xen/hvm.c b/sys/x86/xen/hvm.c
index 72811dc..8ad0f2d 100644
--- a/sys/x86/xen/hvm.c
+++ b/sys/x86/xen/hvm.c
@@ -626,6 +626,7 @@ xen_hvm_set_callback(device_t dev)
xhp.domid = DOMID_SELF;
xhp.index = HVM_PARAM_CALLBACK_IRQ;
+#if 0
if (xen_feature(XENFEAT_hvm_callback_vector) != 0) {
int error;
@@ -638,6 +639,7 @@ xen_hvm_set_callback(device_t dev)
printf(Xen HVM callback vector registration failed (%d).
Falling back to emulated device interrupt\n, error);
}
+#endif
xen_vector_callback_enabled = 0;
if (dev == NULL) {
/*
@@ -783,7 +785,7 @@ xen_hvm_cpu_init(void)
info.mfn = vtophys(vcpu_info) PAGE_SHIFT;
info.offset = vtophys(vcpu_info) - trunc_page(vtophys(vcpu_info));
- rc = HYPERVISOR_vcpu_op(VCPUOP_register_vcpu_info, cpu, info);
+ rc = 1;
if (rc != 0)
DPCPU_SET(vcpu_info, HYPERVISOR_shared_info-vcpu_info[cpu]);
else
--
1.7.7.5 (Apple Git-26)
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
Em 18/04/2014, às 11:57, seanr...@gmail.com seanr...@gmail.com escreveu: Hi there freebsd-xen, I tried first on freebsd-questions@ without success, so I thought to retry here. I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am experiencing slow network performance on xn0 that seems to have developed after I upgraded to FreeBSD 10 (no other changes). I can only achieve about 0.5mbps through this interface when forwarding packets; packets in a single direction are fine (e.g; downloading to the VPS or pushing from the VPS) and clock in at many (10 usually) mbps. Interestingly, my identical VM (configuration managed centrally) running on Xen 3.4 (HVM) does *not* have this issue. I did a little debugging and here's what I've noticed: - Not related to OpenVPN, repro'd using ssh -d. - Slow VM has a very low rate of context switches (~250) while forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using systat -v. - I can't repro a context switch limit (tried a limited fork() bomb). - Tried with *and* without LRO and TSO on xn0 (and all combinations of LRO and TSO on/off) I started digging into the Xen drivers but I realised I am not equipped to debug this. I'm looking for suggestions on what to check. My provider is offering to move me to an older Xen host but I'd like to see if I can help identify a bug either in my configuration or perhaps in FreeBSD's Xen code. Sean _ Do you use pf? ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
On Sat, Apr 19, 2014 at 1:11 PM, Tiago Ribeiro sha...@gmail.com wrote: Em 18/04/2014, às 11:57, seanr...@gmail.com seanr...@gmail.com escreveu: Hi there freebsd-xen, I tried first on freebsd-questions@ without success, so I thought to retry here. I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am experiencing slow network performance on xn0 that seems to have developed after I upgraded to FreeBSD 10 (no other changes). I can only achieve about 0.5mbps through this interface when forwarding packets; packets in a single direction are fine (e.g; downloading to the VPS or pushing from the VPS) and clock in at many (10 usually) mbps. Interestingly, my identical VM (configuration managed centrally) running on Xen 3.4 (HVM) does *not* have this issue. I did a little debugging and here's what I've noticed: - Not related to OpenVPN, repro'd using ssh -d. - Slow VM has a very low rate of context switches (~250) while forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using systat -v. - I can't repro a context switch limit (tried a limited fork() bomb). - Tried with *and* without LRO and TSO on xn0 (and all combinations of LRO and TSO on/off) I started digging into the Xen drivers but I realised I am not equipped to debug this. I'm looking for suggestions on what to check. My provider is offering to move me to an older Xen host but I'd like to see if I can help identify a bug either in my configuration or perhaps in FreeBSD's Xen code. Sean _ Do you use pf? I do. Fairly simple ruleset: default deny, a few specific port allows, NAT. Sean ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
VM in Xen 4.1; poor packet forwarding performance on xn0
Hi there freebsd-xen, I tried first on freebsd-questions@ without success, so I thought to retry here. I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am experiencing slow network performance on xn0 that seems to have developed after I upgraded to FreeBSD 10 (no other changes). I can only achieve about 0.5mbps through this interface when forwarding packets; packets in a single direction are fine (e.g; downloading to the VPS or pushing from the VPS) and clock in at many (10 usually) mbps. Interestingly, my identical VM (configuration managed centrally) running on Xen 3.4 (HVM) does *not* have this issue. I did a little debugging and here's what I've noticed: - Not related to OpenVPN, repro'd using ssh -d. - Slow VM has a very low rate of context switches (~250) while forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using systat -v. - I can't repro a context switch limit (tried a limited fork() bomb). - Tried with *and* without LRO and TSO on xn0 (and all combinations of LRO and TSO on/off) I started digging into the Xen drivers but I realised I am not equipped to debug this. I'm looking for suggestions on what to check. My provider is offering to move me to an older Xen host but I'd like to see if I can help identify a bug either in my configuration or perhaps in FreeBSD's Xen code. Sean ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to freebsd-xen-unsubscr...@freebsd.org
