Re: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated
Jonathan Lynch wrote: Andy, thanks again for your help. Yes, HZ is still 1000 in 2.6.12. I tried your suggestions are here are the results. ASCII diagram (network A) --> (eth1) core router (eth0) --> (network C) (eth2) ^ | | (network B) Looking at the following graphics http://140.203.56.30/~jlynch/htb/core_router.png http://140.203.56.30/~jlynch/htb/voip_stream_23691.png voip_stream_23691.png is a graph of the delay of the voice stream travelling from network A to network C in test 2. Notice from the core router graph that there is only voip traffic passing through the core router until time 07:55 and the delay in voip stream is 0.25 ms until then. After this time tcp traffic is introduced saturating the outgoing interface of the core router (eth0). The delay increases to a maximum of 2.75 ms , which is a considerable improvement on 30ms when I was using the ceil value of 100mbit. But there is a lot of jitter. I suppose you could hope for a bit less jitter 12k burst is about 1ms at 100mbit. There is a tweak you can do for htb which may help - in net/sched/sch_htb.c there is a #define HYSTERESIS 1 - changing it to 0 and recompiling kernel/the module makes things more accurate. With the ceil at 90Mbit, the outgoing bit rate of eth0 has gone from 98mbit to approx 90Mbit as can be seen from the core router graph for eth0 bytes out. Note that with the tcp traffic is all http downloads, so most Ethernet frames will be of maximum size, 1518 bytes, so 98mbits is the maximum throughput possible on a 100mbit card, taking into account the overheads of ethernet such as the interframe gap, preamble and start frame delimiter. Im not sure how to configure some of the htb parameters. The following is my understanding of them and a few questions I have as well. How exactly does the HZ value have a bearing on the ceil value ? How can I calculate a maximum for the ceil value ? It's more to do with burst/cburst than ceil. 12kb is the minimum burst size for a 100 mbit NIC with a timer resolution of 1ms (1000hz) and tc calculates the smallest possible burst when it is not specified, right ?. It seems not, I think hysteresis may be involved again here (but then one of my tcs is hacked about a bit). You can see what htb is using as defaults by doing tc -s -d class ls .. If I do that on similar kernels one with hysteresis 0 and one with 1 I see quite different values. I chose 12k as big enough for the 90mbit test 12000*8*1000=96mbit at ip level and it seemed like a nice multiple of 1500mtu :-) cburst is the number of bytes that can be burst as fast as the interface can transmit them. It is smaller than burst can is ideally one packet size, right ? Ideally 1 packet but not achievable with htb at lan speed and hz 1000, also AIUI the way htb does drr means with mixed packet sizes things aren't packet perfect even at low rates. Saying that I use htb at low rates and can apparently get packet perfect with my traffic mix. I think hfsc can do it perfectly on both counts. quantum determines the ratio at which the classes share their parents bandwidth. Each class is given quantum number of bytes before serving the next class, right ? Yea setting 1500 probably makes no difference for this test. Is there any way I can limit the jitter of the VoIP traffic passing through the htb class ? Try the hysteresis and/or setting the rate for interactive way higher than it's traffic rate. I did a quick test to see how things were for me at 100mbit. Because my other pcs are slow I needed to use two as receivers for netperf. I noticed something I didn't expect with red or the settings you use - one of the pcs is slower and has less memory thus smaller tcp socket size. Using 4 streams two to each unshaped they get about the same, though with txqueuelen = 1000 there are no drops (with window scalng off there is a difference). With red and wscale on, the red really favoured the fast pc - I tried a 40k bfifo so that I got drops, expecting to see the same, but it was still far more even than the red. I couldn't really simulate the voip traffic in theory I should be able to use ping with -i < 1 sec, but using the latest inetutils you get a flood ping if you do that. I reported this about 18 months ago and it's supposedly fixed in the cvs (though I don't know if fixed means it just says invalid argument rather than actually does what's asked, because I have failed to build it so far). So if anyone reading this has a i386 ping that -i 0.5 works on, please mail me the binary :-) Andy. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] wondershaper query
Hello, I am trying wondershaper-1.1a on a friend's pppoe connection on her Linux box. There are a few things I don't understand. 1. She has pppoe connection so should DEV=eth0 or DEV=ppp0 ? 2. Her ISP just says on her payment bill that the speed is 128kbps, but doesn't mention any downlink/uplink speed, so in that case what should be, DOWNLINK= and UPLINK= ? 3. She uses the net in her small office and people mostly to browse the net, send emails sometimes ftp data out and sometimes ssh to other servers to trouble shoot their programs. In such a case is wondershaper helpful? Or is it not required at all? Thanks in advance. With warm regards, Payal ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] multiple uplinks works on linux router but not on clients
Hi, I've been testing the whole time and found some interesting information. Obviously I haven't compiled the kernel very well. I booted the machine in it's old kernel, 2.6.8-2 and now the pc's in my network are using all internet interfaces. I see that with a traceroute. What did I do wrong in compiling and patching the kernel. Probably I missed an important setting in the kernel configuration, but which? It's the first time for me I compiled a kernel and it was very though to have it working. I got a lot of errors during the first tries. Later on, compiling succeeded without any errors, so I thought it was working fine. I can see that the patch applied, because failover works, but there is something else what doesn't work ad it should. Who understand this strange problem? thnx. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] HTB and PRIO qdiscs introducing extra latency when output interface is saturated
Andy, thanks again for your help. Yes, HZ is still 1000 in 2.6.12. I tried your suggestions are here are the results. ASCII diagram (network A) --> (eth1) core router (eth0) --> (network C) (eth2) ^ | | (network B) Looking at the following graphics http://140.203.56.30/~jlynch/htb/core_router.png http://140.203.56.30/~jlynch/htb/voip_stream_23691.png voip_stream_23691.png is a graph of the delay of the voice stream travelling from network A to network C in test 2. Notice from the core router graph that there is only voip traffic passing through the core router until time 07:55 and the delay in voip stream is 0.25 ms until then. After this time tcp traffic is introduced saturating the outgoing interface of the core router (eth0). The delay increases to a maximum of 2.75 ms , which is a considerable improvement on 30ms when I was using the ceil value of 100mbit. But there is a lot of jitter. With the ceil at 90Mbit, the outgoing bit rate of eth0 has gone from 98mbit to approx 90Mbit as can be seen from the core router graph for eth0 bytes out. Note that with the tcp traffic is all http downloads, so most Ethernet frames will be of maximum size, 1518 bytes, so 98mbits is the maximum throughput possible on a 100mbit card, taking into account the overheads of ethernet such as the interframe gap, preamble and start frame delimiter. Im not sure how to configure some of the htb parameters. The following is my understanding of them and a few questions I have as well. How exactly does the HZ value have a bearing on the ceil value ? How can I calculate a maximum for the ceil value ? 12kb is the minimum burst size for a 100 mbit NIC with a timer resolution of 1ms (1000hz) and tc calculates the smallest possible burst when it is not specified, right ?. cburst is the number of bytes that can be burst as fast as the interface can transmit them. It is smaller than burst can is ideally one packet size, right ? quantum determines the ratio at which the classes share their parents bandwidth. Each class is given quantum number of bytes before serving the next class, right ? Is there any way I can limit the jitter of the VoIP traffic passing through the htb class ? Jonathan On Wed, 2005-07-27 at 22:53 +0100, Andy Furniss wrote: > Jonathan Lynch wrote: > > Andy, Many thanks for your reply. Below is some output from the queueing > > disciplines to show that the filters are working correctly and they are > > going to the right classes. > > OK classification looks good then. > > > > > > pass_on means if no class id equal to the result of the filter is found > > then try next filter, which is the BE class in this case. > > Ahh I'll have to play with this dsmark stuff one day :-) > > > > > So back to the main question, could anyone tell me why the delay is so > > high (30ms) for VoIP packets which are treated with the EF phb when the > > outgoing interface of core router to network c is saturated ? > > I would test next with htb setup like (assuming you are HZ=1000 - you > will be under rate if not) - > > ... > > tc class add dev $1 parent 2:0 classid 2:1 htb rate 90Mbit ceil 90Mbit > quantum 1500 burst 12k cburst 12k > > tc class add dev $1 parent 2:1 classid 2:10 htb rate 1500kbit ceil > 90Mbit quantum 1500 burst 12k cburst 12k > ... > > tc class add dev $1 parent 2:1 classid 2:20 htb rate 5Mbit ceil 90Mbit > quantum 1500 burst 12k cburst 12k > > ... > > > If that doesn't make things any better then you could try giving the > 2:10 class a rate alot higher than it needs and see if that helps. > > > Andy. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Routing for multiple uplinks/providers problem.
I think I said something wrong in my last message. You DNAT incoming packets and then SNAT them when they come back if your Linux router has some server behind it. I don't know if this is your case (having servers behind the router). (I needed to top-post here --- maybe not). On 7/28/05, Nelson Castillo <[EMAIL PROTECTED]> wrote: > Hi John. > > On 7/28/05, John McMonagle <[EMAIL PROTECTED]> wrote: > > > Find that if I ping the same site from 2 computers it may work on one > > and fail on the other. > > Also was surprised that some time they are going out different > > interfaces at the same time. > > Same symptoms I had. > > > Have snat on both interfaces > > When you SNAT incoming packets, you need to do something different > from what is in the HOWTO ([4]) because SNAT is done before the > routing desition (check the Kernel Packet Traveling Diagram[5]). > > I had the same problem [1]. The solution is to use conntrack and mark > packets on arrival, and then route them back using the fwmark[2]. > > There's no need to tell you I had a hard time with this. There should > be a warning about this in the HOWTO (in this page [4]). > > The proposed solution I quote in [2] worked for me for the > multiple uplink providers + SNAT problem. > > It is (Using the same variables that are in the HOWTO [4]): > > 1) Mark packages on arrival: > > iptables -t mangle -A PREROUTING -m conntrack --ctorigdst $IP1 -j > MARK --set-mark=1 > iptables -t mangle -A PREROUTING -m conntrack --ctorigdst $IP2 -j > MARK --set-mark=2 > > And then use the mark to route the outgoing packages correctly. > > ip rule add fwmark 1 table T1 > ip rule add fwmark 2 table T2 > > Regards, > Nelson.- > > PD : I solved my problem with IPVS and multiple uplink providers (see [3]). > > [1] http://mailman.ds9a.nl/pipermail/lartc/2005q2/016171.html > [2] http://mailman.ds9a.nl/pipermail/lartc/2005q2/016441.html > [3] http://arhuaco.blogspot.com/2005/07/ipvs-and-conntrack.html > [4] http://lartc.org/howto/lartc.rpdb.multiple-links.html > [5] http://www.docum.org/docum.org/kptd/ > > -- > Homepage : http://geocities.com/arhuaco > > The first principle is that you must not fool yourself > and you are the easiest person to fool. > -- Richard Feynman. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Routing for multiple uplinks/providers problem.
Hi John. On 7/28/05, John McMonagle <[EMAIL PROTECTED]> wrote: > Find that if I ping the same site from 2 computers it may work on one > and fail on the other. > Also was surprised that some time they are going out different > interfaces at the same time. Same symptoms I had. > Have snat on both interfaces When you SNAT incoming packets, you need to do something different from what is in the HOWTO ([4]) because SNAT is done before the routing desition (check the Kernel Packet Traveling Diagram[5]). I had the same problem [1]. The solution is to use conntrack and mark packets on arrival, and then route them back using the fwmark[2]. There's no need to tell you I had a hard time with this. There should be a warning about this in the HOWTO (in this page [4]). The proposed solution I quote in [2] worked for me for the multiple uplink providers + SNAT problem. It is (Using the same variables that are in the HOWTO [4]): 1) Mark packages on arrival: iptables -t mangle -A PREROUTING -m conntrack --ctorigdst $IP1 -j MARK --set-mark=1 iptables -t mangle -A PREROUTING -m conntrack --ctorigdst $IP2 -j MARK --set-mark=2 And then use the mark to route the outgoing packages correctly. ip rule add fwmark 1 table T1 ip rule add fwmark 2 table T2 Regards, Nelson.- PD : I solved my problem with IPVS and multiple uplink providers (see [3]). [1] http://mailman.ds9a.nl/pipermail/lartc/2005q2/016171.html [2] http://mailman.ds9a.nl/pipermail/lartc/2005q2/016441.html [3] http://arhuaco.blogspot.com/2005/07/ipvs-and-conntrack.html [4] http://lartc.org/howto/lartc.rpdb.multiple-links.html [5] http://www.docum.org/docum.org/kptd/ -- Homepage : http://geocities.com/arhuaco The first principle is that you must not fool yourself and you are the easiest person to fool. -- Richard Feynman. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] multiple uplinks works on linux router but not on clients
Hello, I'm trying to achieve the following. 2 ISP's and 1 LAN with failover. So, when 1 internet connection fails the other need to take over. I'm aware of the restrictions. The current sessions are lost. I did this before, one and a half year ago or something. But at that router wasn't failover. Now i've read the nano howto and followed it. I'm using Debian with kernel 2.6.12-3 patched with Julian Anastasov's routes patch. This integrated failover functionality. I need to say. It works great on the machine itself. Everytime I start a traceroute another link will be used. When I unplug an internet link, all of my traceroutes uses the internet connections which are up. When I re-plug the internet link my traceroutes will use the one again. This is how it needs to be, also for the connected pc's. But that's not working. All traceroutes on client machine are going through the same router: 192.168.20.1->192.168.32.1->internet I used following firewall rules, but that shouldn't be the problem: iptables -t filter -N keep_stateiptables -t filter -A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPTiptables -t filter -A keep_state -j RETURN iptables -t nat -N keep_stateiptables -t nat -A keep_state -m state --state RELATED,ESTABLISHED -j ACCEPTiptables -t nat -A keep_state -j RETURN iptables -t nat -A POSTROUTING -o eth1 -s 192.168.20.0/24 -j SNAT --to 192.168.1.240iptables -t nat -A POSTROUTING -o eth2 -s 192.168.20.0/24 -j SNAT --to 10.0.0.240iptables -t nat -A POSTROUTING -o eth3 -s 192.168.20.0/24 -j SNAT --to 192.168.32.240 iptables -t nat -A PREROUTING -j keep_stateiptables -t nat -A POSTROUTING -j keep_stateiptables -t nat -A OUTPUT -j keep_stateiptables -t filter -A INPUT -j keep_stateiptables -t filter -A FORWARD -j keep_stateiptables -t filter -A OUTPUT -j keep_state It looks like the problem comes from my multi route table 222. When I change the order of the devices in the multiroute rule all of my traceroutes on the clients are using the last nexthop via. I'm using three eth's as external interfaces and always the last one in my multihop route will be used by my workstations. Setup: ip rule ls0: from all lookup local50: from all lookup main201: from 192.168.1.0/24 lookup 201202: from 10.0.0.0/24 lookup 202203: from 192.168.32.0/24 lookup 203222: from all lookup 22232766: from all lookup main32767: from all lookup default ip ro s t 222default proto static nexthop via 192.168.1.1 dev eth1 weight 1 nexthop via 10.0.0.138 dev eth2 weight 1 nexthop via 192.168.32.1 dev eth3 weight 1 ip ro s t 201default via 192.168.1.1 dev eth1 proto static src 192.168.1.240prohibit default proto static metric 1 ip ro s t 202default via 10.0.0.138 dev eth2 proto static src 10.0.0.240prohibit default proto static metric 1 ip ro s t 203default via 192.168.32.1 dev eth3 proto static src 192.168.32.240prohibit default proto static metric 1 Well, I think you have all the information. What I want to do is simply routing on multiple interfaces. No queue based routing or something. Linux may just choose which route is going to be used. Thanks for your help in advance. Kind regards, Hans Middelhoek Netherlands ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Routing for multiple uplinks/providers problem.
Been running this for quite a while and noticed that have intermittent problems getting out. Find that if I ping the same site from 2 computers it may work on one and fail on the other. Also was surprised that some time they are going out different interfaces at the same time. Seems to work all the time from the firewall. Running 2.6.10 kernel with the multipath routing patches on a debian sarge system. # ip rule 0: from all lookup local 60: from all lookup main 200:from all lookup 200 201:from 216.170.136.0/24 lookup isp1 201:from 24.196.120.28/30 lookup isp2 222:from all lookup multi 222:from all lookup multi 32766: from all lookup main 32767: from all lookup default cat /etc/iproute2/rt_tables # # reserved values # 255 local 254 main 253 default 0 unspec # # local # 1 inr.ruhep 201 isp1 202 isp2 222 multi [EMAIL PROTECTED]:~# ip route list table 200 192.168.0.0/16 via 192.168.2.254 dev eth0 [EMAIL PROTECTED]:~# ip route list table 201 default via 216.170.136.1 dev eth1 proto static src 216.170.136.82 prohibit default proto static metric 1 [EMAIL PROTECTED]:~# ip route list table 202 default via 24.196.120.29 dev eth2 proto static src 24.196.120.30 prohibit default proto static metric 1 [EMAIL PROTECTED]:~# ip route list table 222 default proto static nexthop via 216.170.136.1 dev eth1 weight 1 nexthop via 24.196.120.29 dev eth2 weight 4 using shorewall to setup rules. iptable -L Chain PREROUTING (policy ACCEPT) target prot opt source destination net_dnat all -- anywhere anywhere net_dnat all -- anywhere anywhere loc_dnat all -- anywhere anywhere Chain POSTROUTING (policy ACCEPT) target prot opt source destination eth1_masq all -- anywhere anywhere eth2_masq all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain eth1_masq (1 references) target prot opt source destination masq2 all -- 192.168.2.0/24 anywhere Chain eth2_masq (1 references) target prot opt source destination masq1 all -- 192.168.2.0/24 anywhere Chain loc_dnat (1 references) target prot opt source destination REDIRECT tcp -- anywhere anywheretcp dpt:www redir ports 3128 Chain masq1 (1 references) target prot opt source destination RETURN all -- anywhere 192.168.0.0/16 RETURN all -- fonroute.advocap.org anywhere SNAT all -- anywhere anywhere to:24.196.120.30 Chain masq2 (1 references) target prot opt source destination RETURN all -- anywhere 192.168.0.0/16 RETURN all -- fonroute.advocap.org anywhere SNAT all -- anywhere anywhere to:216.170.136.73 Chain net_dnat (2 references) target prot opt source destination DNAT tcp -- !192.168.0.0/16 anywheremultiport dports ssh,www to:192.168.2.1 DNAT tcp -- !192.168.0.0/16 anywheremultiport dports smtp,imaps,https to:192.168.2.10 DNAT tcp -- !192.168.0.0/16 anywheretcp dpt:2525 to:192.168.2.10:25 DNAT tcp -- !192.168.0.0/16 anywheretcp dpt:8000 to:192.168.2.12:443 DNAT tcp -- !192.168.0.0/16 anywheretcp dpt:9000 to:192.168.2.12:22 REDIRECT tcp -- anywhere anywheretcp dpt:1022 redir ports 22 Have snat on both interfaces Have rules to keep vpn traffic from getting snated. Any solution? Any way to troubleshoot? John begin:vcard fn:John McMonagle n:McMonagle;John org:Advocap Inc adr;dom:;;2929 Harrison St;Oshkosh;WI;54936 email;internet:[EMAIL PROTECTED] title:IT Manager tel;work:920-426-0150 x-mozilla-html:FALSE url:http://www.advocap.org version:2.1 end:vcard ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
