Re: dummynet issues
On Tue, Dec 01, 2009 at 12:48:43PM -0800, Julian Elischer wrote: this should be made an errata item for 8.0 I'm not sure about the procedure, should i contact re@ team? -- Oleg. === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- o...@rinet.ru === ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org
Re: dummynet issues
On Mon, Nov 30, 2009 at 11:58:55PM -0500, Ben Kelly wrote:
I actually have not measured my bandwidth to validate dummynet. I have
simply observed these messages repeating in my log:
dummynet: OUCH! pipe should have been idle!
Under normal conditions I don't really need the dummynet rules to shape
traffic for my configuration to work, so it has not been a high priority for
me yet. Do you see the log messages?
Thanks.
- Ben
It seems i've found the problem. Please test attached patch (it's for R8.0
sources and include r198845). I'm interested in some feedback:
1) does it solve 'OUCH' messages problem?
2) does it solve bandwidth problem (if there was any)?
--
Oleg.
=== Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- o...@rinet.ru ===
Index: sys/netinet/ipfw/ip_dummynet.c
===
RCS file: /home/ncvs/src/sys/netinet/ipfw/ip_dummynet.c,v
retrieving revision 1.5.2.1.2.1
diff -u -r1.5.2.1.2.1 ip_dummynet.c
--- sys/netinet/ipfw/ip_dummynet.c 25 Oct 2009 01:10:29 - 1.5.2.1.2.1
+++ sys/netinet/ipfw/ip_dummynet.c 1 Dec 2009 17:23:45 -
@@ -244,6 +244,17 @@
static int dummynet_io(struct mbuf **, int , struct ip_fw_args *);
/*
+ * Flow queue is idle if:
+ * 1) it's empty for at least 1 tick
+ * 2) it has invalid timestamp (WF2Q case)
+ * 3) parent pipe has no 'exhausted' burst.
+ */
+#define QUEUE_IS_IDLE(q) ((q)-head == NULL (q)-S == (q)-F + 1 \
+ curr_time (q)-idle_time + 1 \
+ ((q)-numbytes + (curr_time - (q)-idle_time - 1) * \
+ (q)-fs-pipe-bandwidth = (q)-fs-pipe-burst))
+
+/*
* Heap management functions.
*
* In the heap, first node is element 0. Children of i are 2i+1 and 2i+2.
@@ -1004,7 +1015,7 @@
fs-last_expired = time_uptime ;
for (i = 0 ; i = fs-rq_size ; i++) /* last one is overflow */
for (prev=NULL, q = fs-rq[i] ; q != NULL ; )
- if (q-head != NULL || q-S != q-F+1) {
+ if (!QUEUE_IS_IDLE(q)) {
prev = q ;
q = q-next ;
} else { /* entry is idle, expire it */
@@ -1134,7 +1145,7 @@
break ; /* found */
/* No match. Check if we can expire the entry */
- if (pipe_expire q-head == NULL q-S == q-F+1 ) {
+ if (pipe_expire QUEUE_IS_IDLE(q)) {
/* entry is idle and not in any heap, expire it */
struct dn_flow_queue *old_q = q ;
@@ -1408,18 +1419,20 @@
if (q-idle_time curr_time) {
/* Calculate available burst size. */
q-numbytes +=
- (curr_time - q-idle_time) * pipe-bandwidth;
+ (curr_time - q-idle_time - 1) * pipe-bandwidth;
if (q-numbytes pipe-burst)
q-numbytes = pipe-burst;
if (io_fast)
q-numbytes += pipe-bandwidth;
}
} else { /* WF2Q. */
- if (pipe-idle_time curr_time) {
+ if (pipe-idle_time curr_time
+ pipe-scheduler_heap.elements == 0
+ pipe-not_eligible_heap.elements == 0) {
/* Calculate available burst size. */
pipe-numbytes +=
- (curr_time - pipe-idle_time) * pipe-bandwidth;
- if (pipe-numbytes pipe-burst)
+ (curr_time - pipe-idle_time - 1) * pipe-bandwidth;
+ if (pipe-numbytes 0 pipe-numbytes pipe-burst)
pipe-numbytes = pipe-burst;
if (io_fast)
pipe-numbytes += pipe-bandwidth;
___
freebsd-ipfw@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org
Re: dummynet issues
Oleg Bulyzhin wrote: On Mon, Nov 30, 2009 at 11:58:55PM -0500, Ben Kelly wrote: I actually have not measured my bandwidth to validate dummynet. I have simply observed these messages repeating in my log: dummynet: OUCH! pipe should have been idle! Under normal conditions I don't really need the dummynet rules to shape traffic for my configuration to work, so it has not been a high priority for me yet. Do you see the log messages? Thanks. - Ben It seems i've found the problem. Please test attached patch (it's for R8.0 sources and include r198845). I'm interested in some feedback: 1) does it solve 'OUCH' messages problem? 2) does it solve bandwidth problem (if there was any)? The patch fixes the problem: now it seems all ok, no more OUCH messages and pipe bandwidth limiting works again. Thank you very much, Oleg!! Best regards, -- Kevin ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org
Re: dummynet issues
Kevin Smith wrote: Oleg Bulyzhin wrote: On Mon, Nov 30, 2009 at 11:58:55PM -0500, Ben Kelly wrote: I actually have not measured my bandwidth to validate dummynet. I have simply observed these messages repeating in my log: dummynet: OUCH! pipe should have been idle! Under normal conditions I don't really need the dummynet rules to shape traffic for my configuration to work, so it has not been a high priority for me yet. Do you see the log messages? Thanks. - Ben It seems i've found the problem. Please test attached patch (it's for R8.0 sources and include r198845). I'm interested in some feedback: 1) does it solve 'OUCH' messages problem? 2) does it solve bandwidth problem (if there was any)? The patch fixes the problem: now it seems all ok, no more OUCH messages and pipe bandwidth limiting works again. Thank you very much, Oleg!! Best regards, this should be made an errata item for 8.0 ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org
Re: dummynet issues
On Mon, Nov 30, 2009 at 08:18:15PM +0100, Kevin Smith wrote: Mailing LIst Member wrote: Kevin Smith wrote: Alex Almeida wrote: Hi, The same happened with me, just by setting: net.inet.ip.fw.one_pass: 0 And stopped the message, however I was using version 6.4. I hope that helps you, Hugs Alex Almeida Kevin Smith escreveu: Hi, I'm experiencing some dummynet issues after upgrading from 7-STABLE to 8.0-RELEASE. My /var/log/messages is full of these logs: Nov 29 15:34:18 stone kernel: dummynet: OUCH! pipe should have been idle! Nov 29 15:34:49 stone last message repeated 409 times Nov 29 15:36:49 stone last message repeated 1595 times Nov 29 15:46:50 stone last message repeated 8162 times Nov 29 15:56:51 stone last message repeated 7099 times Nov 29 16:06:52 stone last message repeated 4771 times Nov 29 16:16:53 stone last message repeated 3859 times Nov 29 16:26:54 stone last message repeated 3493 times Nov 29 16:36:55 stone last message repeated 5874 times Also I noticed that traffic shaping is not working any longer , i.e.: actually outgoing pipes do not limit bandwidth at all. Until 8 Release upgrading the same configuration was working perfectly. This is my uname -a FreeBSD stone.it 8.0-RELEASE FreeBSD 8.0-RELEASE #5: Sat Nov 28 20:22:30 CET 2009 ke...@stone.it:/usr/obj/usr/src/sys/STONE i386 Attached my dmesg.boot and my kernel configuration. Is anybody experiencing same issues? Thank you, regards, -- Kevin ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org Hi, I've already net.inet.ip.fw.one_pass set to 0, and also setting it to 1, even if this is not what I need, does not fix. Thank you anyway, regards, -- Kevin ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org I know this may be a rediculous question, given the audience, however, I will inquire anyhow. Have you verified that your kernel has installed cleanly? I am still on 7.2-Stable, on my production servers, however, when I have attempted hasty upgrades, and reinstalled the kernel after compiling it on the new system, it required a second iteration of cleaning, and recompiling/installing for the kernel to recognize the net options that are being referenced. Hence, when I initially compiled and installed the kernel, I had to repeat the process a second time, to see all firewalling activated correctly.(pipes, and other rules) Of course, as previously indicated, this may be just a new bug for the newer system, however, I would try that first. Also, make sure you port the new version of your rules/config file to the 8-Release branch. I have had trouble going between 6 and 7 with some of these rules not being recognized but the option compiled. I can't remember what that was, for a good example, however, just a few things to investigate. I apologize if these options have already been investigated. Respectfully, Martes Hi, thank you for your answer, but everything seems quite fine with 8-RELEASE except for this issue. I tried to recompile without SMP or PREEMPTION options and also to apply oleg@ 's patch at http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/ipfw/ip_dummynet.c?rev=1.5.2.2;content-type=text%2Fplain committed on RELENG_8 with no luck. Upgrading was fine, I can say that all has been installed cleanly. Thank you, regards, -- Kevin ___ freebsd-curr...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org Please show your pipe/queue configuration commands and your ipfw ruleset. sysctl net.inet.ip.fw sysctl net.inet.ip.dummynet output would not hurt too. -- Oleg. === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- o...@rinet.ru === ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org
Re: dummynet issues
Ben Kelly wrote:
On Nov 30, 2009, at 3:12 PM, Oleg Bulyzhin wrote:
Please show your pipe/queue configuration commands and your ipfw ruleset.
sysctl net.inet.ip.fw sysctl net.inet.ip.dummynet output would not hurt
too.
I've also run into the problem recently on 9-CURRENT (last synced on
11/13/2009). My configuration looks like:
# Configure traffic shaping.
$fw pipe 10 config bw 950Kbit/s
$fw queue 10 config pipe 10 weight 100
$fw queue 20 config pipe 10 weight 1
# Shape traffic to avoid ACK starvation when our upload is saturated.
$fw add 6100 queue 10 tcp from any to any tcpflags ack iplen 0-80 out via $oif
$fw add 6110 queue 10 udp from any to any iplen 0-80 out via $oif
$fw add 6120 queue 20 tcp from any to any \{ not tcpflags ack or not iplen
0-80 \} out via $oif
$fw add 6130 queue 20 udp from any to any not iplen 0-80 out via $oif
The output of the sysctl elements are:
gate# sysctl net.inet.ip.fw
net.inet.ip.fw.dyn_keepalive: 1
net.inet.ip.fw.dyn_short_lifetime: 5
net.inet.ip.fw.dyn_udp_lifetime: 10
net.inet.ip.fw.dyn_rst_lifetime: 1
net.inet.ip.fw.dyn_fin_lifetime: 1
net.inet.ip.fw.dyn_syn_lifetime: 20
net.inet.ip.fw.dyn_ack_lifetime: 300
net.inet.ip.fw.static_count: 42
net.inet.ip.fw.dyn_max: 4096
net.inet.ip.fw.dyn_count: 232
net.inet.ip.fw.curr_dyn_buckets: 256
net.inet.ip.fw.dyn_buckets: 256
net.inet.ip.fw.default_to_accept: 0
net.inet.ip.fw.tables_max: 128
net.inet.ip.fw.default_rule: 65535
net.inet.ip.fw.verbose_limit: 0
net.inet.ip.fw.verbose: 0
net.inet.ip.fw.one_pass: 0
net.inet.ip.fw.autoinc_step: 100
net.inet.ip.fw.enable: 1
gate# sysctl net.inet.ip.dummynet
net.inet.ip.dummynet.debug: 0
net.inet.ip.dummynet.pipe_byte_limit: 1048576
net.inet.ip.dummynet.pipe_slot_limit: 100
net.inet.ip.dummynet.io_pkt_drop: 1601
net.inet.ip.dummynet.io_pkt_fast: 146359
net.inet.ip.dummynet.io_pkt: 26208842
net.inet.ip.dummynet.io_fast: 0
net.inet.ip.dummynet.tick_lost: 0
net.inet.ip.dummynet.tick_diff: 1352176
net.inet.ip.dummynet.tick_adjustment: 239751
net.inet.ip.dummynet.tick_delta_sum: -494
net.inet.ip.dummynet.tick_delta: 1
net.inet.ip.dummynet.red_max_pkt_size: 1500
net.inet.ip.dummynet.red_avg_pkt_size: 512
net.inet.ip.dummynet.red_lookup_depth: 256
net.inet.ip.dummynet.max_chain_len: 16
net.inet.ip.dummynet.expire: 1
net.inet.ip.dummynet.search_steps: 0
net.inet.ip.dummynet.searches: 0
net.inet.ip.dummynet.extract_heap: 16
net.inet.ip.dummynet.ready_heap: 0
net.inet.ip.dummynet.hash_size: 64
Thanks for the help.
- Ben
Hi,
this is my pipe/queue configuration:
/sbin/ipfw pipe 1 config bw 256kbits/s
/sbin/ipfw queue 3 config pipe 1 weight 40 mask all
/sbin/ipfw queue 4 config pipe 1 weight 50 mask all
/sbin/ipfw add queue 3 all from any to any out via tun\? uid asterisk
/sbin/ipfw add queue 3 all from any to any 80 out via tun\?
/sbin/ipfw add queue 3 all from any to any 53 out via tun\?
/sbin/ipfw add queue 3 all from me 4300 to any out via tun\?
/sbin/ipfw add queue 3 all from me 1194 to any out via tun\?
/sbin/ipfw add queue 4 all from any to any out via tun\? tcpflags
\!syn,ack not jail ${MLDONKEYJID:=1}
/sbin/ipfw add queue 4 all from any to any out via tun\? not jail
${MLDONKEYJID:=1}
/sbin/ipfw queue 1 config pipe 1 weight 1 gred 0.8/16/39/1 mask all
/sbin/ipfw queue 2 config pipe 1 weight 2 gred 0.02/3/6/0.06 mask all
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 40
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 41
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 42
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 43
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 44
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 45
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 46
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 47
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 48
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 49
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 50
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 51
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 52
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 53
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 55
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 56
/sbin/ipfw add queue 2 all from any to any out via tun\? iplen 57
/sbin/ipfw add queue 1 all from any to any out via tun\? jail
${MLDONKEYJID:=1}
and these are some system settings:
net.inet.ip.dummynet.debug: 0
net.inet.ip.dummynet.pipe_byte_limit: 1048576
net.inet.ip.dummynet.pipe_slot_limit: 100
net.inet.ip.dummynet.io_pkt_drop: 1316
net.inet.ip.dummynet.io_pkt_fast: 146311
net.inet.ip.dummynet.io_pkt: 3006844
net.inet.ip.dummynet.io_fast: 0
net.inet.ip.dummynet.tick_lost: 0
net.inet.ip.dummynet.tick_diff: 18983852
net.inet.ip.dummynet.tick_adjustment:
Re: dummynet issues
On Mon, Nov 30, 2009 at 03:38:50PM -0500, Ben Kelly wrote: I've also run into the problem recently on 9-CURRENT (last synced on 11/13/2009). My configuration looks like: My quick attempt to reproduce the issue failed. Perhaps i'm missing something. How are you measuring connection bandwidth? -- Oleg. === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- o...@rinet.ru === ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org
Re: dummynet issues
Oleg Bulyzhin wrote: On Mon, Nov 30, 2009 at 03:38:50PM -0500, Ben Kelly wrote: I've also run into the problem recently on 9-CURRENT (last synced on 11/13/2009). My configuration looks like: My quick attempt to reproduce the issue failed. Perhaps i'm missing something. How are you measuring connection bandwidth? Just an aside - queue weights only come into play when queues are full. If you don't specify the queue size in terms of entries as opposed to bytes, you may not get the behavior you expect with small packets. What's your nominal connect rate in each direction? Your pipe should be that size or smaller, of course. Another aside - in practice, bulk traffic is at MTU size, and interactive packets or naked TCP ACKs are small. iplen alone is sufficient for the packets you want to assign a lower weight. -- Michael Sierchio +1 415 378 1182 PO Box 9036 Berkeley CA 94709 US ku...@tenebras.com ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org
Re: dummynet issues
On Nov 30, 2009, at 6:45 PM, Oleg Bulyzhin wrote: On Mon, Nov 30, 2009 at 03:38:50PM -0500, Ben Kelly wrote: I've also run into the problem recently on 9-CURRENT (last synced on 11/13/2009). My configuration looks like: My quick attempt to reproduce the issue failed. Perhaps i'm missing something. How are you measuring connection bandwidth? I actually have not measured my bandwidth to validate dummynet. I have simply observed these messages repeating in my log: dummynet: OUCH! pipe should have been idle! Under normal conditions I don't really need the dummynet rules to shape traffic for my configuration to work, so it has not been a high priority for me yet. Do you see the log messages? Thanks. - Ben___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org
Re: dummynet issues
Hi, The same happened with me, just by setting: net.inet.ip.fw.one_pass: 0 And stopped the message, however I was using version 6.4. I hope that helps you, Hugs Alex Almeida Kevin Smith escreveu: Hi, I'm experiencing some dummynet issues after upgrading from 7-STABLE to 8.0-RELEASE. My /var/log/messages is full of these logs: Nov 29 15:34:18 stone kernel: dummynet: OUCH! pipe should have been idle! Nov 29 15:34:49 stone last message repeated 409 times Nov 29 15:36:49 stone last message repeated 1595 times Nov 29 15:46:50 stone last message repeated 8162 times Nov 29 15:56:51 stone last message repeated 7099 times Nov 29 16:06:52 stone last message repeated 4771 times Nov 29 16:16:53 stone last message repeated 3859 times Nov 29 16:26:54 stone last message repeated 3493 times Nov 29 16:36:55 stone last message repeated 5874 times Also I noticed that traffic shaping is not working any longer , i.e.: actually outgoing pipes do not limit bandwidth at all. Until 8 Release upgrading the same configuration was working perfectly. This is my uname -a FreeBSD stone.it 8.0-RELEASE FreeBSD 8.0-RELEASE #5: Sat Nov 28 20:22:30 CET 2009 ke...@stone.it:/usr/obj/usr/src/sys/STONE i386 Attached my dmesg.boot and my kernel configuration. Is anybody experiencing same issues? Thank you, regards, -- Kevin ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org
Re: dummynet issues
Alex Almeida wrote: Hi, The same happened with me, just by setting: net.inet.ip.fw.one_pass: 0 And stopped the message, however I was using version 6.4. I hope that helps you, Hugs Alex Almeida Kevin Smith escreveu: Hi, I'm experiencing some dummynet issues after upgrading from 7-STABLE to 8.0-RELEASE. My /var/log/messages is full of these logs: Nov 29 15:34:18 stone kernel: dummynet: OUCH! pipe should have been idle! Nov 29 15:34:49 stone last message repeated 409 times Nov 29 15:36:49 stone last message repeated 1595 times Nov 29 15:46:50 stone last message repeated 8162 times Nov 29 15:56:51 stone last message repeated 7099 times Nov 29 16:06:52 stone last message repeated 4771 times Nov 29 16:16:53 stone last message repeated 3859 times Nov 29 16:26:54 stone last message repeated 3493 times Nov 29 16:36:55 stone last message repeated 5874 times Also I noticed that traffic shaping is not working any longer , i.e.: actually outgoing pipes do not limit bandwidth at all. Until 8 Release upgrading the same configuration was working perfectly. This is my uname -a FreeBSD stone.it 8.0-RELEASE FreeBSD 8.0-RELEASE #5: Sat Nov 28 20:22:30 CET 2009 ke...@stone.it:/usr/obj/usr/src/sys/STONE i386 Attached my dmesg.boot and my kernel configuration. Is anybody experiencing same issues? Thank you, regards, -- Kevin ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org Hi, I've already net.inet.ip.fw.one_pass set to 0, and also setting it to 1, even if this is not what I need, does not fix. Thank you anyway, regards, -- Kevin ___ freebsd-ipfw@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to freebsd-ipfw-unsubscr...@freebsd.org
