Re: VM in Xen 4.1; poor packet forwarding performance on xn0
Hi Roger, This is what they provided me: - The Xen 4.x node is running a custom 3.1.1 kernel. - The Xen 3.x node is running a 2.6.18 kernel. They didn't elaborate on the customisations. :( Sean On Sat, Apr 26, 2014 at 1:06 PM, Roger Pau Monné wrote: > On 26/04/14 13:57, seanr...@gmail.com wrote: > > Hi Roger, > > > > Thanks for the patch -- sadly, it didn't work. No change. > > Hello, > > > > > I did have to modify a bit for releng/10.0; for some reason patch > > refused to apply it cleanly. It looked fairly straightforward but I > > attached inline the patch I ultimately applied below just in case I got > > it wrong. > > Patch looks fine. > > > Are there any other potential differences between Xen 3.4 and 4.1? (my > > provider migrated my problem VPS to a 3.4 host and the problem > > evaporated; I am trying this on a new 4.1 VPS that I was able to > > reproduce the problem on). > > I'm quite sure this is not related to the underlying Xen version, but to > the Linux Dom0 kernel version your provider is using, could you ask them > the Linux versions they are using on the 4.1 and 3.4 hosts? > > Roger. > ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
On 26/04/14 13:57, seanr...@gmail.com wrote: > Hi Roger, > > Thanks for the patch -- sadly, it didn't work. No change. Hello, > > I did have to modify a bit for releng/10.0; for some reason patch > refused to apply it cleanly. It looked fairly straightforward but I > attached inline the patch I ultimately applied below just in case I got > it wrong. Patch looks fine. > Are there any other potential differences between Xen 3.4 and 4.1? (my > provider migrated my problem VPS to a 3.4 host and the problem > evaporated; I am trying this on a new 4.1 VPS that I was able to > reproduce the problem on). I'm quite sure this is not related to the underlying Xen version, but to the Linux Dom0 kernel version your provider is using, could you ask them the Linux versions they are using on the 4.1 and 3.4 hosts? Roger. ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
Hi Roger,
Thanks for the patch -- sadly, it didn't work. No change.
I did have to modify a bit for releng/10.0; for some reason patch refused
to apply it cleanly. It looked fairly straightforward but I attached inline
the patch I ultimately applied below just in case I got it wrong.
Are there any other potential differences between Xen 3.4 and 4.1? (my
provider migrated my problem VPS to a 3.4 host and the problem evaporated;
I am trying this on a new 4.1 VPS that I was able to reproduce the problem
on).
Sean
Index: hvm.c
===
--- hvm.c (revision 264963)
+++ hvm.c (working copy)
@@ -626,6 +626,7 @@
xhp.domid = DOMID_SELF;
xhp.index = HVM_PARAM_CALLBACK_IRQ;
+#if 0
if (xen_feature(XENFEAT_hvm_callback_vector) != 0) {
int error;
@@ -638,6 +639,7 @@
printf("Xen HVM callback vector registration failed (%d). "
"Falling back to emulated device interrupt\n", error);
}
+#endif
xen_vector_callback_enabled = 0;
if (dev == NULL) {
/*
@@ -783,7 +785,7 @@
info.mfn = vtophys(vcpu_info) >> PAGE_SHIFT;
info.offset = vtophys(vcpu_info) - trunc_page(vtophys(vcpu_info));
- rc = HYPERVISOR_vcpu_op(VCPUOP_register_vcpu_info, cpu, &info);
+ rc = 1;
if (rc != 0)
DPCPU_SET(vcpu_info, &HYPERVISOR_shared_info->vcpu_info[cpu]);
else
On Tue, Apr 22, 2014 at 3:10 PM, Roger Pau Monné wrote:
> On 18/04/14 16:57, seanr...@gmail.com wrote:
> > Hi there freebsd-xen,
> >
> > I tried first on freebsd-questions@ without success, so I thought to
> retry
> > here.
> >
> > I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am
> experiencing
> > slow network performance on xn0 that seems to have developed after I
> > upgraded to FreeBSD 10 (no other changes). I can only achieve about
> 0.5mbps
> > through this interface when forwarding packets; packets in a single
> > direction are fine (e.g; downloading to the VPS or pushing from the VPS)
> > and clock in at many (>10 usually) mbps.
> >
> > Interestingly, my identical VM (configuration managed centrally) running
> on
> > Xen 3.4 (HVM) does *not* have this issue.
>
> Hello,
>
> The difference between Xen 3.4 and Xen 4.1 is that FreeBSD will make use
> of the vector callback, the PV timer and PV IPIs when running on Xen >
> 4.0 (which should provide better performance). I'm attaching a patch
> that will make FreeBSD behave the same way when running on either Xen
> 3.4 or Xen 4.1 (by disabling all this new additions), could you please
> give it a try?
>
> >
> > I did a little debugging and here's what I've noticed:
> >- Not related to OpenVPN, repro'd using ssh -d.
> >- Slow VM has a very low rate of context switches (~250) while
> > forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using
> > systat -v.
> > - I can't repro a context switch limit (tried a limited fork() bomb).
> > - Tried with *and* without LRO and TSO on xn0 (and all combinations of
> > LRO and TSO on/off)
>
> I've got the feeling that the issue you are seeing is not related to the
> Xen version itself, but the Linux Dom0 kernel version (which I suppose
> is different in the Xen 3.4 and Xen 4.1 hosts). Could you ask your
> provider which Linux Dom0 kernel are they using on the different hosts?
>
> Roger.
>
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
On 18/04/14 16:57, seanr...@gmail.com wrote:
> Hi there freebsd-xen,
>
> I tried first on freebsd-questions@ without success, so I thought to retry
> here.
>
> I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am experiencing
> slow network performance on xn0 that seems to have developed after I
> upgraded to FreeBSD 10 (no other changes). I can only achieve about 0.5mbps
> through this interface when forwarding packets; packets in a single
> direction are fine (e.g; downloading to the VPS or pushing from the VPS)
> and clock in at many (>10 usually) mbps.
>
> Interestingly, my identical VM (configuration managed centrally) running on
> Xen 3.4 (HVM) does *not* have this issue.
Hello,
The difference between Xen 3.4 and Xen 4.1 is that FreeBSD will make use
of the vector callback, the PV timer and PV IPIs when running on Xen >
4.0 (which should provide better performance). I'm attaching a patch
that will make FreeBSD behave the same way when running on either Xen
3.4 or Xen 4.1 (by disabling all this new additions), could you please
give it a try?
>
> I did a little debugging and here's what I've noticed:
>- Not related to OpenVPN, repro'd using ssh -d.
>- Slow VM has a very low rate of context switches (~250) while
> forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using
> systat -v.
> - I can't repro a context switch limit (tried a limited fork() bomb).
> - Tried with *and* without LRO and TSO on xn0 (and all combinations of
> LRO and TSO on/off)
I've got the feeling that the issue you are seeing is not related to the
Xen version itself, but the Linux Dom0 kernel version (which I suppose
is different in the Xen 3.4 and Xen 4.1 hosts). Could you ask your
provider which Linux Dom0 kernel are they using on the different hosts?
Roger.
>From a866add6f87d726ae2c6ab5837d1b6eefb721be8 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne
Date: Tue, 22 Apr 2014 09:50:09 +0200
Subject: [PATCH]
---
sys/x86/xen/hvm.c |4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/sys/x86/xen/hvm.c b/sys/x86/xen/hvm.c
index 72811dc..8ad0f2d 100644
--- a/sys/x86/xen/hvm.c
+++ b/sys/x86/xen/hvm.c
@@ -626,6 +626,7 @@ xen_hvm_set_callback(device_t dev)
xhp.domid = DOMID_SELF;
xhp.index = HVM_PARAM_CALLBACK_IRQ;
+#if 0
if (xen_feature(XENFEAT_hvm_callback_vector) != 0) {
int error;
@@ -638,6 +639,7 @@ xen_hvm_set_callback(device_t dev)
printf("Xen HVM callback vector registration failed (%d). "
"Falling back to emulated device interrupt\n", error);
}
+#endif
xen_vector_callback_enabled = 0;
if (dev == NULL) {
/*
@@ -783,7 +785,7 @@ xen_hvm_cpu_init(void)
info.mfn = vtophys(vcpu_info) >> PAGE_SHIFT;
info.offset = vtophys(vcpu_info) - trunc_page(vtophys(vcpu_info));
- rc = HYPERVISOR_vcpu_op(VCPUOP_register_vcpu_info, cpu, &info);
+ rc = 1;
if (rc != 0)
DPCPU_SET(vcpu_info, &HYPERVISOR_shared_info->vcpu_info[cpu]);
else
--
1.7.7.5 (Apple Git-26)
___
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
Em 19/04/2014, às 10:17, "seanr...@gmail.com" escreveu: > On Sat, Apr 19, 2014 at 1:11 PM, Tiago Ribeiro wrote: > > > > Em 18/04/2014, às 11:57, "seanr...@gmail.com" escreveu: > > > > Hi there freebsd-xen, > > > > I tried first on freebsd-questions@ without success, so I thought to retry > > here. > > > > I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am experiencing > > slow network performance on xn0 that seems to have developed after I > > upgraded to FreeBSD 10 (no other changes). I can only achieve about 0.5mbps > > through this interface when forwarding packets; packets in a single > > direction are fine (e.g; downloading to the VPS or pushing from the VPS) > > and clock in at many (>10 usually) mbps. > > > > Interestingly, my identical VM (configuration managed centrally) running on > > Xen 3.4 (HVM) does *not* have this issue. > > > > I did a little debugging and here's what I've noticed: > > - Not related to OpenVPN, repro'd using ssh -d. > > - Slow VM has a very low rate of context switches (~250) while > > forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using > > systat -v. > > - I can't repro a context switch limit (tried a limited fork() bomb). > > - Tried with *and* without LRO and TSO on xn0 (and all combinations of > > LRO and TSO on/off) > > > > I started digging into the Xen drivers but I realised I am not equipped to > > debug this. I'm looking for suggestions on what to check. My provider is > > offering to move me to an older Xen host but I'd like to see if I can help > > identify a bug either in my configuration or perhaps in FreeBSD's Xen code. > > > > Sean > > _ > > > Do you use pf? > > I do. Fairly simple ruleset: default deny, a few specific port allows, NAT. > > Sean I can't help you, i'm very interested in this topic. I have case when i use xenserver with freebsd 10, and pf with nat, and works ok, and case with dom0 debian 6 and do not work, and one case with netbsd dom0, and freebsd 9 with kernel compiled for xen, and works fine. ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
On Sat, Apr 19, 2014 at 1:11 PM, Tiago Ribeiro wrote: > > > > Em 18/04/2014, às 11:57, "seanr...@gmail.com" > escreveu: > > > > Hi there freebsd-xen, > > > > I tried first on freebsd-questions@ without success, so I thought to > retry > > here. > > > > I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am > experiencing > > slow network performance on xn0 that seems to have developed after I > > upgraded to FreeBSD 10 (no other changes). I can only achieve about > 0.5mbps > > through this interface when forwarding packets; packets in a single > > direction are fine (e.g; downloading to the VPS or pushing from the VPS) > > and clock in at many (>10 usually) mbps. > > > > Interestingly, my identical VM (configuration managed centrally) running > on > > Xen 3.4 (HVM) does *not* have this issue. > > > > I did a little debugging and here's what I've noticed: > > - Not related to OpenVPN, repro'd using ssh -d. > > - Slow VM has a very low rate of context switches (~250) while > > forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using > > systat -v. > > - I can't repro a context switch limit (tried a limited fork() bomb). > > - Tried with *and* without LRO and TSO on xn0 (and all combinations of > > LRO and TSO on/off) > > > > I started digging into the Xen drivers but I realised I am not equipped > to > > debug this. I'm looking for suggestions on what to check. My provider is > > offering to move me to an older Xen host but I'd like to see if I can > help > > identify a bug either in my configuration or perhaps in FreeBSD's Xen > code. > > > > Sean > > _ > > > Do you use pf? I do. Fairly simple ruleset: default deny, a few specific port allows, NAT. Sean ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: VM in Xen 4.1; poor packet forwarding performance on xn0
> Em 18/04/2014, às 11:57, "seanr...@gmail.com" escreveu: > > Hi there freebsd-xen, > > I tried first on freebsd-questions@ without success, so I thought to retry > here. > > I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am experiencing > slow network performance on xn0 that seems to have developed after I > upgraded to FreeBSD 10 (no other changes). I can only achieve about 0.5mbps > through this interface when forwarding packets; packets in a single > direction are fine (e.g; downloading to the VPS or pushing from the VPS) > and clock in at many (>10 usually) mbps. > > Interestingly, my identical VM (configuration managed centrally) running on > Xen 3.4 (HVM) does *not* have this issue. > > I did a little debugging and here's what I've noticed: > - Not related to OpenVPN, repro'd using ssh -d. > - Slow VM has a very low rate of context switches (~250) while > forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using > systat -v. > - I can't repro a context switch limit (tried a limited fork() bomb). > - Tried with *and* without LRO and TSO on xn0 (and all combinations of > LRO and TSO on/off) > > I started digging into the Xen drivers but I realised I am not equipped to > debug this. I'm looking for suggestions on what to check. My provider is > offering to move me to an older Xen host but I'd like to see if I can help > identify a bug either in my configuration or perhaps in FreeBSD's Xen code. > > Sean > _ Do you use pf? ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
VM in Xen 4.1; poor packet forwarding performance on xn0
Hi there freebsd-xen, I tried first on freebsd-questions@ without success, so I thought to retry here. I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am experiencing slow network performance on xn0 that seems to have developed after I upgraded to FreeBSD 10 (no other changes). I can only achieve about 0.5mbps through this interface when forwarding packets; packets in a single direction are fine (e.g; downloading to the VPS or pushing from the VPS) and clock in at many (>10 usually) mbps. Interestingly, my identical VM (configuration managed centrally) running on Xen 3.4 (HVM) does *not* have this issue. I did a little debugging and here's what I've noticed: - Not related to OpenVPN, repro'd using ssh -d. - Slow VM has a very low rate of context switches (~250) while forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using systat -v. - I can't repro a context switch limit (tried a limited fork() bomb). - Tried with *and* without LRO and TSO on xn0 (and all combinations of LRO and TSO on/off) I started digging into the Xen drivers but I realised I am not equipped to debug this. I'm looking for suggestions on what to check. My provider is offering to move me to an older Xen host but I'd like to see if I can help identify a bug either in my configuration or perhaps in FreeBSD's Xen code. Sean ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
