Re: altq priq Anomaly (Solved)
On 7/22/07, Daniel Melameth <[EMAIL PROTECTED]> wrote: > On 7/22/07, Stuart Henderson <[EMAIL PROTECTED]> wrote: > > On 2007/07/20 15:20, Daniel Melameth wrote: > > > then go back to the broken behavior sometime later. A reboot of the box > > > or > > > removing altq is the only way to resolve the issue, temporarily. I've > > > tried > > > both priq and cbq, adjusting tbrsize, recompiling the kernel with a higher > > > HZ value and using different hardware and different Internet connections, > > > but the issue persists. > > > > Try a snapshot, i386 moved to the new timecounter code after 4.1. > > > > Though I note there is also an XXX about variable cpu frequency (in > > sys/altq/altq_subr.c) which might affect you if you adjust cpu speed > > (manually or by apmd). > > Thanks for taking the time to reply. I can't readily do a snapshot > now, but since I am using apmd, I'll try this avenue first and see > what happens. Thanks to Stuart's review of the altq code, I have, finally, resolved this issue which has affected me for years. Disabling apmd did not resolve the issue, but disabling the Dynamically Switchable setting for CPU Frequency in the BIOS appears to have addressed this problem. I have been using the system for a couple of months now and altq has been very solid.
Re: altq priq Anomaly
On Wed, Aug 01, 2007 at 12:55:15PM +0100, Stuart Henderson wrote: > On 2007/07/30 17:33, Can Erkin Acar wrote: > > > > The problem with this diff is that it assumes an ADSL link. > > While 'vcmux' is obviously ADSL terminology, I assume > > having 'pppoe' or 'bridge' would confuse others trying to > > use non-adsl pppoe connections or even real bridges. > > That's a very fair point. I was aiming for a balance of accuracy > and simplicity with the keyword names, and might have gone too far > in the direction of simplicity (though the listing in the manual > is done in more detail). > > > While altq is not my area, I think the diff would be > > more useful if it allowed an 'overhead' and a 'scale' > > to be set on an interface. You could then document suitable > > values for commonly used setups (pppoe/vcmux/whatever). > > There are a finite number of adaptations which would be really useful, > so I think that adds unnecessary complexity to the configuration > language, and it's easier to see what's going on when this is expressed > directly in C than by listing magic numbers in an already-long manual > page. With the framework in place it's simple to see where additions > should go. I am really not sure about this. I would prefer to pass the 'magic' parameters to the kernel, and put the tables into pfctl. That way the kernel and pfctl do not have to match exactly, and you could fine tune in userland, without changing the kernel. > How would you feel about it with more descriptive keywords? Descriptive keywords would be better, but see the above comment. This way you can do both.
Re: altq priq Anomaly
On 2007/07/30 17:33, Can Erkin Acar wrote: > > The problem with this diff is that it assumes an ADSL link. > While 'vcmux' is obviously ADSL terminology, I assume > having 'pppoe' or 'bridge' would confuse others trying to > use non-adsl pppoe connections or even real bridges. That's a very fair point. I was aiming for a balance of accuracy and simplicity with the keyword names, and might have gone too far in the direction of simplicity (though the listing in the manual is done in more detail). > While altq is not my area, I think the diff would be > more useful if it allowed an 'overhead' and a 'scale' > to be set on an interface. You could then document suitable > values for commonly used setups (pppoe/vcmux/whatever). There are a finite number of adaptations which would be really useful, so I think that adds unnecessary complexity to the configuration language, and it's easier to see what's going on when this is expressed directly in C than by listing magic numbers in an already-long manual page. With the framework in place it's simple to see where additions should go. How would you feel about it with more descriptive keywords?
Re: altq priq Anomaly
On Fri, Jul 27, 2007 at 06:53:27PM -0600, Daniel Melameth wrote: > On 7/22/07, Daniel Melameth <[EMAIL PROTECTED]> wrote: > > On 7/22/07, Stuart Henderson <[EMAIL PROTECTED]> wrote: > > > On 2007/07/20 15:20, Daniel Melameth wrote: > > > > then go back to the broken behavior sometime later. A reboot of the > > > > box or > > > > removing altq is the only way to resolve the issue, temporarily. I've > > > > tried > > > > both priq and cbq, adjusting tbrsize, recompiling the kernel with a > > > > higher > > > > HZ value and using different hardware and different Internet > > > > connections, > > > > but the issue persists. > > > > > > Try a snapshot, i386 moved to the new timecounter code after 4.1. > > > > > > Though I note there is also an XXX about variable cpu frequency (in > > > sys/altq/altq_subr.c) which might affect you if you adjust cpu speed > > > (manually or by apmd). > > > > > > > Full detail on the issue is available from my gnats post at > > > > # Attempt to account for overhead of RFC 1483 bridging, ATM, PPPoE and > > > > TCP > > > > > > For this you might like to try a diff of mine and report back > > > (http://spacehopper.org/openbsd/altq_tbradapt.diff). > > > It won't help the other problem though. > > > > Thanks for taking the time to reply. I can't readily do a snapshot > > now, but since I am using apmd, I'll try this avenue first and see > > what happens. I also went ahead and incorporated your diffs into my > > currently running 4.1-stable and it appears to be working quite well, > > but I'm not certain if I'm piloting these changes the correct way--so > > if there's an ideal way you'd recommend piloting this, let me know. > > Regardless, I'll monitor the result of these diffs this week and give > > you a heads up at the end of the week. > > Stuart, > > I did some simple piloting of your diffs and they work quite nicely. > The details: > > Without altq, I get a fairly consistent roundtrip latency of 58ms to a > known good host when there is no congestion--and when the upstream is > saturated, latency to the same host goes up to ~220ms. If I use altq > without your diffs and specify my maximum upload bandwidth of 896Kb, > as expected the overhead of PPPoE tramples this and I still get ~220ms > under saturation even with a high priority queue. In order to get > decency latency in this case, I have to artificially estimate a > bandwidth of ~716Kb--and this provides for a high priority queue > latency of ~68ms while saturated, which is pretty good. > > With your diffs though, not only do I get ~68ms for a high priority > queue while using a bandwidth of 896Kb on a saturated connection, but > my maximum upload speed is, consistently, a bit higher as well when > compared to using a bandwidth of 716Kb without your diffs. I > continued to pilot your diffs further by seeding and downloading > multiple torrents simultaneously--to more heavily expose small TCP > ACKs to the stream--and I continued to get these consistently good > results. > > I have been using your diffs for almost a week now with nary an issue. > So, my next question is: > > HOW DO WE GET YOUR GREAT DIFFS INTO -CURRRENT OR A SNAPSHOT? The problem with this diff is that it assumes an ADSL link. While 'vcmux' is obviously ADSL terminology, I assume having 'pppoe' or 'bridge' would confuse others trying to use non-adsl pppoe connections or even real bridges. While altq is not my area, I think the diff would be more useful if it allowed an 'overhead' and a 'scale' to be set on an interface. You could then document suitable values for commonly used setups (pppoe/vcmux/whatever).
Re: altq priq Anomaly
On 7/22/07, Daniel Melameth <[EMAIL PROTECTED]> wrote: > On 7/22/07, Stuart Henderson <[EMAIL PROTECTED]> wrote: > > On 2007/07/20 15:20, Daniel Melameth wrote: > > > then go back to the broken behavior sometime later. A reboot of the box > > > or > > > removing altq is the only way to resolve the issue, temporarily. I've > > > tried > > > both priq and cbq, adjusting tbrsize, recompiling the kernel with a higher > > > HZ value and using different hardware and different Internet connections, > > > but the issue persists. > > > > Try a snapshot, i386 moved to the new timecounter code after 4.1. > > > > Though I note there is also an XXX about variable cpu frequency (in > > sys/altq/altq_subr.c) which might affect you if you adjust cpu speed > > (manually or by apmd). > > > > > Full detail on the issue is available from my gnats post at > > > # Attempt to account for overhead of RFC 1483 bridging, ATM, PPPoE and TCP > > > > For this you might like to try a diff of mine and report back > > (http://spacehopper.org/openbsd/altq_tbradapt.diff). > > It won't help the other problem though. > > Thanks for taking the time to reply. I can't readily do a snapshot > now, but since I am using apmd, I'll try this avenue first and see > what happens. I also went ahead and incorporated your diffs into my > currently running 4.1-stable and it appears to be working quite well, > but I'm not certain if I'm piloting these changes the correct way--so > if there's an ideal way you'd recommend piloting this, let me know. > Regardless, I'll monitor the result of these diffs this week and give > you a heads up at the end of the week. Stuart, I did some simple piloting of your diffs and they work quite nicely. The details: Without altq, I get a fairly consistent roundtrip latency of 58ms to a known good host when there is no congestion--and when the upstream is saturated, latency to the same host goes up to ~220ms. If I use altq without your diffs and specify my maximum upload bandwidth of 896Kb, as expected the overhead of PPPoE tramples this and I still get ~220ms under saturation even with a high priority queue. In order to get decency latency in this case, I have to artificially estimate a bandwidth of ~716Kb--and this provides for a high priority queue latency of ~68ms while saturated, which is pretty good. With your diffs though, not only do I get ~68ms for a high priority queue while using a bandwidth of 896Kb on a saturated connection, but my maximum upload speed is, consistently, a bit higher as well when compared to using a bandwidth of 716Kb without your diffs. I continued to pilot your diffs further by seeding and downloading multiple torrents simultaneously--to more heavily expose small TCP ACKs to the stream--and I continued to get these consistently good results. I have been using your diffs for almost a week now with nary an issue. So, my next question is: HOW DO WE GET YOUR GREAT DIFFS INTO -CURRRENT OR A SNAPSHOT?
Re: altq priq Anomaly
On 7/22/07, Stuart Henderson <[EMAIL PROTECTED]> wrote: On 2007/07/20 15:20, Daniel Melameth wrote: > then go back to the broken behavior sometime later. A reboot of the box or > removing altq is the only way to resolve the issue, temporarily. I've tried > both priq and cbq, adjusting tbrsize, recompiling the kernel with a higher > HZ value and using different hardware and different Internet connections, > but the issue persists. Try a snapshot, i386 moved to the new timecounter code after 4.1. Though I note there is also an XXX about variable cpu frequency (in sys/altq/altq_subr.c) which might affect you if you adjust cpu speed (manually or by apmd). > Full detail on the issue is available from my gnats post at > # Attempt to account for overhead of RFC 1483 bridging, ATM, PPPoE and TCP For this you might like to try a diff of mine and report back (http://spacehopper.org/openbsd/altq_tbradapt.diff). It won't help the other problem though. Thanks for taking the time to reply. I can't readily do a snapshot now, but since I am using apmd, I'll try this avenue first and see what happens. I also went ahead and incorporated your diffs into my currently running 4.1-stable and it appears to be working quite well, but I'm not certain if I'm piloting these changes the correct way--so if there's an ideal way you'd recommend piloting this, let me know. Regardless, I'll monitor the result of these diffs this week and give you a heads up at the end of the week. The only breakage I've noticed so far appears to be related to pfstat which reports the following (and I assume this is related to the new headers, but I haven't looked into it yet): ioctl: DIOCGETALTQS: Permission denied pf_query: query_queues() failed Thanks again Stuart.
Re: altq priq Anomaly
A recompile of pfstat has addressed this. On 7/22/07, Daniel Melameth <[EMAIL PROTECTED]> wrote: Thanks for taking the time to reply. I can't readily do a snapshot now, but since I am using apmd, I'll try this avenue first and see what happens. I also went ahead and incorporated your diffs into my currently running 4.1-stable and it appears to be working quite well, but I'm not certain if I'm piloting these changes the correct way--so if there's an ideal way you'd recommend piloting this, let me know. Regardless, I'll monitor the result of these diffs this week and give you a heads up at the end of the week. The only breakage I've noticed so far appears to be related to pfstat which reports the following (and I assume this is related to the new headers, but I haven't looked into it yet): ioctl: DIOCGETALTQS: Permission denied pf_query: query_queues() failed
Re: altq priq Anomaly
On 2007/07/20 15:20, Daniel Melameth wrote: > then go back to the broken behavior sometime later. A reboot of the box or > removing altq is the only way to resolve the issue, temporarily. I've tried > both priq and cbq, adjusting tbrsize, recompiling the kernel with a higher > HZ value and using different hardware and different Internet connections, > but the issue persists. Try a snapshot, i386 moved to the new timecounter code after 4.1. Though I note there is also an XXX about variable cpu frequency (in sys/altq/altq_subr.c) which might affect you if you adjust cpu speed (manually or by apmd). > Full detail on the issue is available from my gnats post at > # Attempt to account for overhead of RFC 1483 bridging, ATM, PPPoE and TCP For this you might like to try a diff of mine and report back (http://spacehopper.org/openbsd/altq_tbradapt.diff). It won't help the other problem though.
RE: altq priq Anomaly?
Amir S Mesry wrote:
> Interesting issue, I have't encountered it. 3MB Down/384KB Up.
>
> altq on $eth0 priq bandwidth 325Kb queue { q_pri, q_def }
> queue q_pri priority 7
> queue q_def priority 1 priq(default
>
> What program are you using to measure it?
Please see my original post... FTP.
RE: altq priq Anomaly?
Stefan Zill wrote: > Jon Hart wrote: > > On Thu, Jun 23, 2005 at 07:39:41AM -0400, Melameth, Daniel D. wrote: > > > The TCP ACKs are not the issue. The issue is I never get more > > > than half of what I set the bandwidth value to. > > > > I've never been able to get exactly the bandwidth I specified in my > > pf.conf altq rules. > > I almost always get, what I specify in the altq-definitions. Yet, > Once I had such an issue. I ran some kind of local proxy, which > connected from my (tun0) to (tun0) and later this data left through > tun0 on a different connection. The first connection, although not > physically passing through the external interface, consumed > altq-bandwidth. Maybe you have a similiar kind of issue, such that > your traffic passes through altq twice. I removed my problem by > simply using (lo0) to (lo0) connections, instead of (tun0) to (tun0). > Maybe this behaviour can be considered a bug, maybe it can be removed > by using better pf-rules. No proxy is in place, but I might look into this further... or maybe it's a bug.
RE: altq priq Anomaly?
Interesting issue, I have't encountered it. 3MB Down/384KB Up.
altq on $eth0 priq bandwidth 325Kb queue { q_pri, q_def }
queue q_pri priority 7
queue q_def priority 1 priq(default
What program are you using to measure it?
Amir Mesry
[EMAIL PROTECTED]
Cadillac Jack, Inc.
http://www.cadillacjack.com/
Network & Systems Administrator
2420 Meadowbrook Parkway
Duluth, GA 30096
770-865-0034
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf
Of Melameth, Daniel D.
Sent: Thursday, June 23, 2005 7:40 AM
To: pf@benzedrine.cx
Subject: RE: altq priq Anomaly?
Ingolf Zeiner Petersen wrote:
> Melameth, Daniel D. wrote:
> > > I implemented altq's priq a while back in the hope of "speeding
> > > up" my overall 'net connection by prioritizing empty TCP ACKs.
> > > However, I noticed that I was never coming close to my 256Kb/s
> > > upload cap since I did this and looked into it a little bit
> > > further today.
> I don't know if you've got ADSL, but for my own case I got 1024/256
> ADSL, and I have actually set the up-limit to 180Kbit/s to get useful
> latency-rates (together with priq queues). Because empty TCP ACKs is
> much about how fast the packets get sent to the destination - this
> becomes relevant.
>
> Try it, and tell us about your result! :-)
The TCP ACKs are not the issue. The issue is I never get more than half
of what I set the bandwidth value to.
RE: altq priq Anomaly?
Jon Hart wrote: > On Thu, Jun 23, 2005 at 07:39:41AM -0400, Melameth, Daniel D. wrote: > > The TCP ACKs are not the issue. The issue is I never get more than > > half of what I set the bandwidth value to. > > I've never been able to get exactly the bandwidth I specified in my > pf.conf altq rules. In trying to figure out what bandwidth > specification in pf.conf would give me the rate I pay for, it really > is about 2x. At home I've got 6M down, 768K up. My altq definitions > specify 12M down and 1.5M up and everything works fine. Well, I guess I'm glad I'm not the only one having the issue. I might file a PR.
Re: altq priq Anomaly?
Jon Hart wrote: On Thu, Jun 23, 2005 at 07:39:41AM -0400, Melameth, Daniel D. wrote: The TCP ACKs are not the issue. The issue is I never get more than half of what I set the bandwidth value to. I've never been able to get exactly the bandwidth I specified in my pf.conf altq rules. I almost always get, what I specify in the altq-definitions. Yet, Once I had such an issue. I ran some kind of local proxy, which connected from my (tun0) to (tun0) and later this data left through tun0 on a different connection. The first connection, although not physically passing through the external interface, consumed altq-bandwidth. Maybe you have a similiar kind of issue, such that your traffic passes through altq twice. I removed my problem by simply using (lo0) to (lo0) connections, instead of (tun0) to (tun0). Maybe this behaviour can be considered a bug, maybe it can be removed by using better pf-rules. HTH Stefan
Re: altq priq Anomaly?
On Thu, Jun 23, 2005 at 07:39:41AM -0400, Melameth, Daniel D. wrote: > The TCP ACKs are not the issue. The issue is I never get more than half > of what I set the bandwidth value to. I've never been able to get exactly the bandwidth I specified in my pf.conf altq rules. In trying to figure out what bandwidth specification in pf.conf would give me the rate I pay for, it really is about 2x. At home I've got 6M down, 768K up. My altq definitions specify 12M down and 1.5M up and everything works fine. -jon
RE: altq priq Anomaly?
Ingolf Zeiner Petersen wrote: > Melameth, Daniel D. wrote: > > > I implemented altq's priq a while back in the hope of "speeding > > > up" my overall 'net connection by prioritizing empty TCP ACKs. > > > However, I noticed that I was never coming close to my 256Kb/s > > > upload cap since I did this and looked into it a little bit > > > further today. > I don't know if you've got ADSL, but for my own case I got 1024/256 > ADSL, and I have actually set the up-limit to 180Kbit/s to get useful > latency-rates (together with priq queues). Because empty TCP ACKs is > much about how fast the packets get sent to the destination - this > becomes relevant. > > Try it, and tell us about your result! :-) The TCP ACKs are not the issue. The issue is I never get more than half of what I set the bandwidth value to.
Re: altq priq Anomaly?
Melameth, Daniel D. wrote: I implemented altq's priq a while back in the hope of "speeding up" my overall 'net connection by prioritizing empty TCP ACKs. However, I noticed that I was never coming close to my 256Kb/s upload cap since I did this and looked into it a little bit further today. I don't know if you've got ADSL, but for my own case I got 1024/256 ADSL, and I have actually set the up-limit to 180Kbit/s to get useful latency-rates (together with priq queues). Because empty TCP ACKs is much about how fast the packets get sent to the destination - this becomes relevant. Try it, and tell us about your result! :-)
RE: altq priq Anomaly?
I sent this email back in March when I was running 3.5 and didn't look
into this further because this was an older release--but now I'm running
3.7 and I have the same issue. Any ideas? No one on misc@ seems to...
Melameth, Daniel D. wrote:
> I sent something similar to this to misc@ with nary a response so I
> hope someone on this specialized list can shed some light on this...
>
>
> I implemented altq's priq a while back in the hope of "speeding up" my
> overall 'net connection by prioritizing empty TCP ACKs. However, I
> noticed that I was never coming close to my 256Kb/s upload cap since I
> did this and looked into it a little bit further today.
>
>
> It appears an expected bandwidth value for altq in this scenario:
>
> altq on $ext_if priq bandwidth 240Kb queue { tcp_ack, default }
>
> ..limits my upload speed to about half (128Kb/s) of what I should get
> and doubling this value:
>
> altq on $ext_if priq bandwidth 480Kb queue { tcp_ack, default }
>
> ..gives me my 256Kb/s back.
>
> Details below... Any thoughts appreciated.
>
>
> $ uname -mrsv
> OpenBSD 3.5 GENERIC#34 i386
>
> Snippet from my normal pf.conf:
>
> altq on $ext_if priq bandwidth 240Kb queue { tcp_ack, default }
> queue tcp_ack priority 7
> queue default priority 1 priq ( default )
> ..
> nat on $ext_if from $int_if:network to ! $modem -> ( $ext_if:0 )
> nat on $ext_if from $int_if:network to $modem -> $ext_alias
> ..
> pass in quick on $ext_if inet proto tcp from ! $int_if:network to
> \ ( $ext_if:0 ) port ssh flags S/SA keep state queue (
> default,
> tcp_ack )
> ..
> pass out quick on $ext_if inet proto tcp to any flags S/SA \
> keep state queue ( default, tcp_ack )
>
> Output of pfctl -vvs queue during a large FTP upload:
>
> $ sudo pfctl -vvs queue
> queue tcp_ack priority 7
> [ pkts: 5 bytes:270 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 0/ 50 ]
> queue default priq( default )
> [ pkts:862 bytes: 754536 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 9/ 50 ]
>
> queue tcp_ack priority 7
> [ pkts: 5 bytes:270 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 0/ 50 ]
> [ measured: 0.0 packets/s, 0 b/s ]
> queue default priq( default )
> [ pkts:952 bytes: 827452 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 11/ 50 ]
> [ measured:18.0 packets/s, 116.67Kb/s ]
>
> queue tcp_ack priority 7
> [ pkts: 8 bytes:432 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 0/ 50 ]
> [ measured: 0.3 packets/s, 129.60 b/s ]
> queue default priq( default )
> [ pkts: 1052 bytes: 907197 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 10/ 50 ]
> [ measured:19.0 packets/s, 122.13Kb/s ]
>
>
> Simply updating my pf.conf with:
>
> altq on $ext_if priq bandwidth 480Kb queue { tcp_ack, default }
>
> ..and reloading my rules with:
>
> $ sudo pfctl -f /etc/pf.conf
>
> ..during the current same FTP upload process gives:
>
> $ sudo pfctl -vvs queue
> queue tcp_ack priority 7
> [ pkts: 0 bytes: 0 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 0/ 50 ]
> queue default priq( default )
> [ pkts: 53 bytes: 54598 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 7/ 50 ]
>
> queue tcp_ack priority 7
> [ pkts: 0 bytes: 0 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 0/ 50 ]
> [ measured: 0.0 packets/s, 0 b/s ]
> queue default priq( default )
> [ pkts:233 bytes: 212130 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 5/ 50 ]
> [ measured:36.0 packets/s, 252.05Kb/s ]
>
> queue tcp_ack priority 7
> [ pkts: 3 bytes:162 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 0/ 50 ]
> [ measured: 0.3 packets/s, 129.60 b/s ]
> queue default priq( default )
> [ pkts:410 bytes: 369754 dropped pkts: 0 bytes:
> 0 ]
> [ qlength: 2/ 50 ]
> [ measured:35.7 packets/s, 252.12Kb/s ]
>
>
> So what's going on? Why is the first altq statement not allowing me
> to get close to my maximum (when there is mostly nothing in the higher
> priority queue)?
