Re: pf ALTQ bandwidth limited to a 32bit value (4294Mb)
On 06.07.2011 18:50, Peter N. M. Hansteen wrote: Calomel Orginfallibilismindefeasibil...@calomel.org writes: ALTQ using hfsc is limited to a maximum parent bandwidth of 4294Mb. This value is 2^32 or 4,294,967,296 bits. If you set the bandwidth any higher, altq will flip back to zero. This bug was found when trying to test 10 gigabit and 40 gigabit bandwidth models. These tests were done on OpenBSD 32bit and 64bit as well as FreeBSD 32bit and 64bit. Nice to hear you've got access to relatively high end gear for testing, I'm sure it will come in handy when the time comes to test any proposed fixes. The obvious workaround in the short term is to do the traffic shaping and filtering a bit closer to the end user, where bandwidth is a bit more scarce. In the slightly longer term, I'm sure a verified bug report (with patches against -current code if feasible) would be much appreciated. - Peter Haha ! I notice that too! Man, I'm shaping traffic for users end which total traffic is about 4Gb , but in my case I just used hfsc, and my mighty firewall generating script just divides the whole bandwidth like total/(no.users+1) for guaranteed speed. 1 stands for default queue. ___ freebsd-pf@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-pf To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
Re: pf ALTQ bandwidth limited to a 32bit value (4294Mb)
On Wed, Jul 6, 2011 at 5:25 PM, Calomel Org
infallibilismindefeasibil...@calomel.org wrote:
ALTQ using hfsc is limited to a maximum parent bandwidth of 4294Mb.
This value is 2^32 or 4,294,967,296 bits. If you set the bandwidth any
higher, altq will flip back to zero. This bug was found when trying
to test 10 gigabit and 40 gigabit bandwidth models. These tests were
done on OpenBSD 32bit and 64bit as well as FreeBSD 32bit and 64bit.
If anyone else can verify this independently and agree with the
results I would be happy to register it as a bug.
How to replicate:
A quick test is setting the bandwidth to 4294Mb and doing a pfctl -sq
to check altq.
altq on $ExtIf bandwidth 4294Mb hfsc queue { ack, web}
queue root_em0 on em0 bandwidth 4.29Gb priority 0 {ack, web}
Now set the bandwidth to 4295Mb and notice altq has flip to zero and
add the 32.70Kb difference.
altq on $ExtIf bandwidth 4295Mb hfsc queue { ack, web }
queue root_em0 on em0 bandwidth 32.70Kb priority 0 {ack, web}
Again, we can set the bandwidth to a multiple of two(2) to 8589Mb.
The bandwidth value flips to zero once and the result is 4.29Gb.
altq on $ExtIf bandwidth 8589Mb hfsc queue { ack, web}
queue root_em0 on em0 bandwidth 4.29Gb priority 0 {ack, web}
If we add one more megabit to 8590Mb the value flips twice and we are
left with 65.41Kb.
altq on $ExtIf bandwidth 8590Mb hfsc queue { ack, web}
queue root_em0 on em0 bandwidth 65.41Kb priority 0 {ack, web}
It is true that there is a limit because of data type used.
Though it cannot be fixed easily on i386 but on amd64 this should work.
Index: sys/contrib/pf/net/pfvar.h
===
--- sys/contrib/pf/net/pfvar.h (revision 223824)
+++ sys/contrib/pf/net/pfvar.h (working copy)
@@ -1491,13 +1491,13 @@
/* scheduler spec */
u_int8_t scheduler; /* scheduler type */
u_int16_ttbrsize; /* tokenbucket regulator size */
- u_int32_tifbandwidth; /* interface bandwidth */
+ u_int64_tifbandwidth; /* interface bandwidth */
/* queue spec */
char qname[PF_QNAME_SIZE]; /* queue name */
char parent[PF_QNAME_SIZE]; /* parent name */
u_int32_tparent_qid;/* parent queue id */
- u_int32_tbandwidth; /* queue bandwidth */
+ u_int64_tbandwidth; /* queue bandwidth */
u_int8_t priority; /* priority */
#ifdef __FreeBSD__
u_int8_t local_flags; /* dynamic interface */
Thanks.
--
Calomel @ https://calomel.org
Open Source Research and Reference
___
freebsd-pf@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
--
Ermal
___
freebsd-pf@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
Re: pf ALTQ bandwidth limited to a 32bit value (4294Mb)
Greg, Pf performance is quite good. The machine averages 9.2 Gbits/sec or 1.15 gigabytes per second (GB/s) in each direction, simultaneously. Interrupt load is about 12%. With a single Linux box on either side of the firewall we see around 161K packets per second and can create 4000 states per second. With more machines on either side on the firewall we are looking to get closer to a million pps and a higher state creation rate. I am not sure what limits we will see then. We published a more detailed write up here under Can we achieve 10 gigabit speeds ? Network Tuning and Performance https://calomel.org/network_performance.html -- Calomel @ https://calomel.org Open Source Research and Reference On Wed, Jul 06, 2011 at 07:09:59PM -0400, Greg Hennessy wrote: ALTQ using hfsc is limited to a maximum parent bandwidth of 4294Mb. This value is 2^32 or 4,294,967,296 bits. If you set the bandwidth any higher, altq will flip back to zero. This bug was found when trying to test 10 gigabit and 40 gigabit bandwidth models. What a problem to have :-) On a side note out of pure curiousity, what's PF performance like up in the stratosphere under those sort of loads/packet rates ? :-) Greg ___ freebsd-pf@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-pf To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org ___ freebsd-pf@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-pf To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
Re: pf ALTQ bandwidth limited to a 32bit value (4294Mb)
Ermal,
Thanks for the diff. When we tried it on FreeBSD 8.2-p2, ALTq would no
long start. We also looked into the source under
/usr/src/sys/contrib/altq/altq. Sadly, most of the changes we made
either broke altq completely or had no effect.
If you have any other ideas we would be happy to try them out.
--
Calomel @ https://calomel.org
Open Source Research and Reference
On Thu, Jul 07, 2011 at 09:28:13AM -0400, Ermal Lu?i wrote:
On Wed, Jul 6, 2011 at 5:25 PM, Calomel Org
infallibilismindefeasibil...@calomel.org wrote:
ALTQ using hfsc is limited to a maximum parent bandwidth of 4294Mb.
This value is 2^32 or 4,294,967,296 bits. If you set the bandwidth any
higher, altq will flip back to zero. This bug was found when trying
to test 10 gigabit and 40 gigabit bandwidth models. These tests were
done on OpenBSD 32bit and 64bit as well as FreeBSD 32bit and 64bit.
If anyone else can verify this independently and agree with the
results I would be happy to register it as a bug.
How to replicate:
A quick test is setting the bandwidth to 4294Mb and doing a pfctl -sq
to check altq.
?altq on $ExtIf bandwidth 4294Mb hfsc queue { ack, web}
?queue root_em0 on em0 bandwidth 4.29Gb priority 0 {ack, web}
Now set the bandwidth to 4295Mb and notice altq has flip to zero and
add the 32.70Kb difference.
?altq on $ExtIf bandwidth 4295Mb hfsc queue { ack, web }
?queue root_em0 on em0 bandwidth 32.70Kb priority 0 {ack, web}
Again, we can set the bandwidth to a multiple of two(2) to 8589Mb.
The bandwidth value flips to zero once and the result is 4.29Gb.
?altq on $ExtIf bandwidth 8589Mb hfsc queue { ack, web}
?queue root_em0 on em0 bandwidth 4.29Gb priority 0 {ack, web}
If we add one more megabit to 8590Mb the value flips twice and we are
left with 65.41Kb.
?altq on $ExtIf bandwidth 8590Mb hfsc queue { ack, web}
?queue root_em0 on em0 bandwidth 65.41Kb priority 0 {ack, web}
It is true that there is a limit because of data type used.
Though it cannot be fixed easily on i386 but on amd64 this should work.
Index: sys/contrib/pf/net/pfvar.h
===
--- sys/contrib/pf/net/pfvar.h (revision 223824)
+++ sys/contrib/pf/net/pfvar.h (working copy)
@@ -1491,13 +1491,13 @@
/* scheduler spec */
u_int8_t scheduler; /* scheduler type */
u_int16_ttbrsize; /* tokenbucket regulator size
*/
- u_int32_tifbandwidth; /* interface bandwidth */
+ u_int64_tifbandwidth; /* interface bandwidth */
/* queue spec */
char qname[PF_QNAME_SIZE]; /* queue name */
char parent[PF_QNAME_SIZE]; /* parent name */
u_int32_tparent_qid;/* parent queue id */
- u_int32_tbandwidth; /* queue bandwidth */
+ u_int64_tbandwidth; /* queue bandwidth */
u_int8_t priority; /* priority */
#ifdef __FreeBSD__
u_int8_t local_flags; /* dynamic interface */
Thanks.
--
? Calomel @ https://calomel.org
? Open Source Research and Reference
___
freebsd-pf@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
--
Ermal
___
freebsd-pf@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
___
freebsd-pf@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
Re: pf ALTQ bandwidth limited to a 32bit value (4294Mb)
Interesting read. Sounds like you need a BreakingPoint to really give it a workout with real world style traffic profiles and volumes. Greg Sent from my Android phone using TouchDown (www.nitrodesk.com) From: Calomel Org Sent: 07 July 2011 19:41:50 To: Greg Hennessy Cc: freebsd-pf@freebsd.org Subject: Re: pf ALTQ bandwidth limited to a 32bit value (4294Mb) Greg, Pf performance is quite good. The machine averages 9.2 Gbits/sec or 1.15 gigabytes per second (GB/s) in each direction, simultaneously. Interrupt load is about 12%. With a single Linux box on either side of the firewall we see around 161K packets per second and can create 4000 states per second. With more machines on either side on the firewall we are looking to get closer to a million pps and a higher state creation rate. I am not sure what limits we will see then. We published a more detailed write up here under Can we achieve 10 gigabit speeds ? Network Tuning and Performance https://calomel.org/network_performance.html -- Calomel @ https://calomel.org Open Source Research and Reference On Wed, Jul 06, 2011 at 07:09:59PM -0400, Greg Hennessy wrote: ALTQ using hfsc is limited to a maximum parent bandwidth of 4294Mb. This value is 2^32 or 4,294,967,296 bits. If you set the bandwidth any higher, altq will flip back to zero. This bug was found when trying to test 10 gigabit and 40 gigabit bandwidth models. What a problem to have :-) On a side note out of pure curiousity, what's PF performance like up in the stratosphere under those sort of loads/packet rates ? :-) Greg ___ freebsd-pf@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-pf To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org ___ freebsd-pf@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-pf To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
Re: pf ALTQ bandwidth limited to a 32bit value (4294Mb)
Calomel Org infallibilismindefeasibil...@calomel.org writes: ALTQ using hfsc is limited to a maximum parent bandwidth of 4294Mb. This value is 2^32 or 4,294,967,296 bits. If you set the bandwidth any higher, altq will flip back to zero. This bug was found when trying to test 10 gigabit and 40 gigabit bandwidth models. These tests were done on OpenBSD 32bit and 64bit as well as FreeBSD 32bit and 64bit. Nice to hear you've got access to relatively high end gear for testing, I'm sure it will come in handy when the time comes to test any proposed fixes. The obvious workaround in the short term is to do the traffic shaping and filtering a bit closer to the end user, where bandwidth is a bit more scarce. In the slightly longer term, I'm sure a verified bug report (with patches against -current code if feasible) would be much appreciated. - Peter -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/ Remember to set the evil bit on all malicious network traffic delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds. ___ freebsd-pf@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-pf To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
RE: pf ALTQ bandwidth limited to a 32bit value (4294Mb)
ALTQ using hfsc is limited to a maximum parent bandwidth of 4294Mb. This value is 2^32 or 4,294,967,296 bits. If you set the bandwidth any higher, altq will flip back to zero. This bug was found when trying to test 10 gigabit and 40 gigabit bandwidth models. What a problem to have :-) On a side note out of pure curiousity, what's PF performance like up in the stratosphere under those sort of loads/packet rates ? :-) Greg ___ freebsd-pf@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-pf To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
