[LARTC] Re: tc n00b
Hi Nikolay, Thanks very much for your help - the script is now working. The downlink shaping works as expected, but the uplink shaping seems to give 4 times more bandwidth than it ought to - so I've just divided the number by 4 and it is satisfactory. However, I've now discovered that pings from one of my shaped NAT clients, to another LAN machine that usually take 1ms, now take ~3500ms when I am using the bandwidth. How can I avoid these enormous queues? I have cc'd the list in case anyone else has an idea. Have a good weekend everyone! Cheers, Jonathan Jonathan Gazeley Wireless VPN Team Information Systems Computing University of Bristol Nikolay Kichukov wrote: Hey, sorry for delay. I had some issues with my primary internet connection and had to change the primary mail host as well. I looked at the script. Looks totally fine to me. If you are not sure where exactly the problem lies, try adding ONLY the ingress rules. Test it, see if it works. Then tweak if needed the script you sent to me. tc qdisc add dev $LAN ingress handle : tc filter add dev eth0 parent : protocol ip prio 2 u32 match ip src IPHERE police rate ${UPLINK}kbit burst 16kb drop flowid : Cheers, -Nik ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: tc n00b
Hello, You need to recompile your kernel and include the appropriate modules for htb to work. The other idea I have is to use policer to filter how much traffic PCs in the LAN upload. This is done on the LAN interface. Eliminates the need to mark packets, etc. You just drop all the packets that are coming in too fast. And presumably your LAN can do at least 100mbps, so the delay of packet retransmission can be neglected. HTH, -Nikolay Martin Milata wrote: On Mon, Jul 30, 2007 at 02:58:00PM +0100, Jonathan Gazeley wrote: [...] 137.222.235.125 RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel [...] Hint: If you run your script as bash -x script_name (or use #!/bin/sh -x as shabang), you will be able to see which exact command caused the error message. Regards, -MM ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: tc n00b
Hi Nikolay, Thanks for your help - this looks useful. Is it possible to apply a police filter invidiually to each IP behind the NAT? Thanks, Jonathan Nikolay Kichukov wrote: Hello, You need to recompile your kernel and include the appropriate modules for htb to work. The other idea I have is to use policer to filter how much traffic PCs in the LAN upload. This is done on the LAN interface. Eliminates the need to mark packets, etc. You just drop all the packets that are coming in too fast. And presumably your LAN can do at least 100mbps, so the delay of packet retransmission can be neglected. HTH, -Nikolay Martin Milata wrote: On Mon, Jul 30, 2007 at 02:58:00PM +0100, Jonathan Gazeley wrote: [...] 137.222.235.125 RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel [...] Hint: If you run your script as bash -x script_name (or use #!/bin/sh -x as shabang), you will be able to see which exact command caused the error message. Regards, -MM ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -- Jonathan Gazeley Wireless VPN Team Information Systems Computing University of Bristol ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: tc n00b
Hello Jonathan, Indeed. I have tested with limited number of IPs though. Not sure how that scheme will behave if you apply it to a huge network. Cheers, -Nikolay Jonathan Gazeley wrote: Hi Nikolay, Thanks for your help - this looks useful. Is it possible to apply a police filter invidiually to each IP behind the NAT? Thanks, Jonathan Nikolay Kichukov wrote: Hello, You need to recompile your kernel and include the appropriate modules for htb to work. The other idea I have is to use policer to filter how much traffic PCs in the LAN upload. This is done on the LAN interface. Eliminates the need to mark packets, etc. You just drop all the packets that are coming in too fast. And presumably your LAN can do at least 100mbps, so the delay of packet retransmission can be neglected. HTH, -Nikolay Martin Milata wrote: On Mon, Jul 30, 2007 at 02:58:00PM +0100, Jonathan Gazeley wrote: [...] 137.222.235.125 RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel [...] Hint: If you run your script as bash -x script_name (or use #!/bin/sh -x as shabang), you will be able to see which exact command caused the error message. Regards, -MM ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: tc n00b
Hi Nikolay, How might this be implemented? I have used a shell script that loops around with a new IP address each time, and then my police line looks like this: tc filter add dev $LAN parent 1: protocol ip prio 50 u32 match ip src 137.222.$j.$i police rate ${UPLINK}kbit burst 10k drop flowid :1 However my clients still have unlimited uplink. The other day, someone told me that then the tc box is also NATing, the source IP is rewritten before the police filter is applied - meaning that you cannot match on source IP. How did you overcome this problem? Thanks for your help, Jonathan Nikolay Kichukov wrote: Hello Jonathan, Indeed. I have tested with limited number of IPs though. Not sure how that scheme will behave if you apply it to a huge network. Cheers, -Nikolay Jonathan Gazeley wrote: Hi Nikolay, Thanks for your help - this looks useful. Is it possible to apply a police filter invidiually to each IP behind the NAT? Thanks, Jonathan Nikolay Kichukov wrote: Hello, You need to recompile your kernel and include the appropriate modules for htb to work. The other idea I have is to use policer to filter how much traffic PCs in the LAN upload. This is done on the LAN interface. Eliminates the need to mark packets, etc. You just drop all the packets that are coming in too fast. And presumably your LAN can do at least 100mbps, so the delay of packet retransmission can be neglected. HTH, -Nikolay Martin Milata wrote: On Mon, Jul 30, 2007 at 02:58:00PM +0100, Jonathan Gazeley wrote: [...] 137.222.235.125 RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel [...] Hint: If you run your script as bash -x script_name (or use #!/bin/sh -x as shabang), you will be able to see which exact command caused the error message. Regards, -MM ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -- Jonathan Gazeley Wireless VPN Team Information Systems Computing University of Bristol ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: tc n00b
Hello, The policer is not 1: but :, not engress(root) but ingress. Let me give you an example: tc qdisc add dev eth0 ingress handle : TC_FILTER=tc filter add dev eth0 parent : protocol ip $TC_FILTER prio 2 u32 match ip src 192.168.0.6/32 police rate 32kbit burst 16kb drop flowid : $TC_FILTER prio 2 u32 match ip src 192.168.0.4/32 police rate 128kbit burst 32kb drop flowid : $TC_FILTER prio 2 u32 match ip src 192.168.0.2/32 police rate 128kbit burst 32kb drop flowid : $TC_FILTER prio 2 u32 match ip src 192.168.0.5/32 police rate 128kbit burst 32kb drop flowid : eth0 is the LAN interface which the 192.168.0.0/24 IPs are connected to. The rest is self explanatory. Let me know if I can help you with anything else. Cheers, -Nik Jonathan Gazeley wrote: Hi Nikolay, How might this be implemented? I have used a shell script that loops around with a new IP address each time, and then my police line looks like this: tc filter add dev $LAN parent 1: protocol ip prio 50 u32 match ip src 137.222.$j.$i police rate ${UPLINK}kbit burst 10k drop flowid :1 However my clients still have unlimited uplink. The other day, someone told me that then the tc box is also NATing, the source IP is rewritten before the police filter is applied - meaning that you cannot match on source IP. How did you overcome this problem? Thanks for your help, Jonathan Nikolay Kichukov wrote: Hello Jonathan, Indeed. I have tested with limited number of IPs though. Not sure how that scheme will behave if you apply it to a huge network. Cheers, -Nikolay Jonathan Gazeley wrote: Hi Nikolay, Thanks for your help - this looks useful. Is it possible to apply a police filter invidiually to each IP behind the NAT? Thanks, Jonathan Nikolay Kichukov wrote: Hello, You need to recompile your kernel and include the appropriate modules for htb to work. The other idea I have is to use policer to filter how much traffic PCs in the LAN upload. This is done on the LAN interface. Eliminates the need to mark packets, etc. You just drop all the packets that are coming in too fast. And presumably your LAN can do at least 100mbps, so the delay of packet retransmission can be neglected. HTH, -Nikolay Martin Milata wrote: On Mon, Jul 30, 2007 at 02:58:00PM +0100, Jonathan Gazeley wrote: [...] 137.222.235.125 RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel [...] Hint: If you run your script as bash -x script_name (or use #!/bin/sh -x as shabang), you will be able to see which exact command caused the error message. Regards, -MM ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: tc n00b
Hello Jonathan, The scenario works perfectly well on a NAT router. See, you drop excess of bits on the interface where the packets arrive. Which is before nating. Maybe we speak about different scenarios here? What I describe limits the maximum upload speed for ip in the LAN. Let me know the packet flow with the interfaces and IP addresses. Cheers, -Nikolay p.s. I am also CCing the lartc mailing list in case someone else can help. Jonathan Gazeley wrote: Hi Nikolay, Thanks for this. I tried using the code below, but it did not work for me. Is your server running tc also a NAT box? The reason I think my code isn't working is because NAT and tc are on the same server, meaning that the source IP of an outgoing packet is rewritten _before_ it gets to tc -- meaning that it is not possible to match packets by source IP. Cheers, Jonathan Nikolay Kichukov wrote: Hello, The policer is not 1: but :, not engress(root) but ingress. Let me give you an example: tc qdisc add dev eth0 ingress handle : TC_FILTER=tc filter add dev eth0 parent : protocol ip $TC_FILTER prio 2 u32 match ip src 192.168.0.6/32 police rate 32kbit burst 16kb drop flowid : $TC_FILTER prio 2 u32 match ip src 192.168.0.4/32 police rate 128kbit burst 32kb drop flowid : $TC_FILTER prio 2 u32 match ip src 192.168.0.2/32 police rate 128kbit burst 32kb drop flowid : $TC_FILTER prio 2 u32 match ip src 192.168.0.5/32 police rate 128kbit burst 32kb drop flowid : eth0 is the LAN interface which the 192.168.0.0/24 IPs are connected to. The rest is self explanatory. Let me know if I can help you with anything else. Cheers, -Nik Jonathan Gazeley wrote: Hi Nikolay, How might this be implemented? I have used a shell script that loops around with a new IP address each time, and then my police line looks like this: tc filter add dev $LAN parent 1: protocol ip prio 50 u32 match ip src 137.222.$j.$i police rate ${UPLINK}kbit burst 10k drop flowid :1 However my clients still have unlimited uplink. The other day, someone told me that then the tc box is also NATing, the source IP is rewritten before the police filter is applied - meaning that you cannot match on source IP. How did you overcome this problem? Thanks for your help, Jonathan Nikolay Kichukov wrote: Hello Jonathan, Indeed. I have tested with limited number of IPs though. Not sure how that scheme will behave if you apply it to a huge network. Cheers, -Nikolay Jonathan Gazeley wrote: Hi Nikolay, Thanks for your help - this looks useful. Is it possible to apply a police filter invidiually to each IP behind the NAT? Thanks, Jonathan Nikolay Kichukov wrote: Hello, You need to recompile your kernel and include the appropriate modules for htb to work. The other idea I have is to use policer to filter how much traffic PCs in the LAN upload. This is done on the LAN interface. Eliminates the need to mark packets, etc. You just drop all the packets that are coming in too fast. And presumably your LAN can do at least 100mbps, so the delay of packet retransmission can be neglected. HTH, -Nikolay Martin Milata wrote: On Mon, Jul 30, 2007 at 02:58:00PM +0100, Jonathan Gazeley wrote: [...] 137.222.235.125 RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel [...] Hint: If you run your script as bash -x script_name (or use #!/bin/sh -x as shabang), you will be able to see which exact command caused the error message. Regards, -MM ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Re: tc n00b
Hello Jonathan. At 2007-07-30 12:40:00 +0100, [EMAIL PROTECTED] wrote: So far I have managed to get the download limits working. However I need to shape on both interfaces so I recycled the same code to apply to uploads but it didn't work and I can't figure out why That's not really enough information to try to debug your problem, but I can think of one problem you might encounter. Since you're doing NAT for your clients, you should be aware that the source address is rewritten (i.e. in nat/POSTROUTING) _before_ egress QoS processing. So if you're trying to classify outgoing traffic based on their source IP address, it won't work. One alternative is to mark packets from the internal network (i.e. use -J MARK --set-mark N in mangle/PREROUTING), and write a filter on the outgoing interface that assigns traffic to classes based on how it's marked (u32 match mark ...). (If you want more details, ask.) (If anyone has other suggestions, I would be interested in them too.) -- ams ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Re: tc n00b
Hi Abhijit, Thanks a lot for your advice - I didn't realise that the source IP was re-written before the traffic was shaped. I have attached the script I wrote. As I said before, the download limit does successfully work and each client (I am using 2 test clients) gets 512kbit each. However the upload is still unlimited. But I don't believe this is currently due to the source IP being re-written - tc itself doesn't like my commands. They were literally copied and pasted from the download commands and altered as appropriate, as you see in the script. When I run this script, for each iteration of lines 48-49 produces the following error: 137.222.235.125 Error: Qdisc tbf is classless. Error: Qdisc 1: is classless. Unknown filter 1:, hence option protocol is unparsable I don't really understand that error - especially as the identical code does work for the download limits. If you can offer any more help, I'd be most grateful. Cheers, Jonathan Abhijit Menon-Sen wrote: Hello Jonathan. At 2007-07-30 12:40:00 +0100, [EMAIL PROTECTED] wrote: So far I have managed to get the download limits working. However I need to shape on both interfaces so I recycled the same code to apply to uploads but it didn't work and I can't figure out why That's not really enough information to try to debug your problem, but I can think of one problem you might encounter. Since you're doing NAT for your clients, you should be aware that the source address is rewritten (i.e. in nat/POSTROUTING) _before_ egress QoS processing. So if you're trying to classify outgoing traffic based on their source IP address, it won't work. One alternative is to mark packets from the internal network (i.e. use -J MARK --set-mark N in mangle/PREROUTING), and write a filter on the outgoing interface that assigns traffic to classes based on how it's marked (u32 match mark ...). (If you want more details, ask.) (If anyone has other suggestions, I would be interested in them too.) -- ams -- Jonathan Gazeley Wireless VPN Team Information Systems Computing University of Bristol #!/bin/sh ## JONATHAN'S TC SCRIPT # LAN interfaces LAN=eth0 WAN=eth1 # Maximum global uplink and downlink in mbit/s GLOBAL_DOWN=100 GLOBAL_UP=100 # Maximum per-user download upload speed in kbit/s DOWNLINK=512 UPLINK=256 # Subnets to be stamped down upon, delimited by spaces SUBNETS='235' # IP range in each subnet LOW_IP=1 HIGH_IP=125 #-Don't mess with stuff below---| #-this line or you'll break it--| # Flush existing rules tc qdisc del dev $LAN root tc qdisc del dev $WAN root # Create root class for 100mbit interface - total traffic can't exceed this tc qdisc add dev $LAN root handle 1: cbq avpkt 1000 bandwidth ${GLOBAL_DOWN}mbit tc qdisc add dev $WAN root handle 1: cbq avpkt 1000 bandwidth ${GLOBAL_UP}mbit # Set useful counters jcount=1 icount=1 total=0 # Apply rules for all included subnets for j in $SUBNETS do for i in `seq $LOW_IP $HIGH_IP` do total=$((total+1)) echo 137.222.$j.$i tc class add dev $LAN parent 1: classid 1:$total tbf rate ${DOWNLINK}kbit allot 1500 prio 5 bounded isolated tc filter add dev $LAN parent 1: protocol ip prio 16 u32 match ip dst 137.222.$j.$i flowid 1:$total tc class add dev $wAN parent 1: classid 1:$total tbf rate ${UPLINK}kbit allot 1500 prio 5 bounded isolated tc filter add dev $wAN parent 1: protocol ip prio 16 u32 match ip src 137.222.$j.$i flowid 1:$total i=i+1 done j=j+1 done echo echo $total miscreants were stamped down upon. Good work Pokey! echo Their IP addresses were in the following ranges: for j in $SUBNETS do echo 137.222.$j.$LOW_IP-$HIGH_IP done ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Re: tc n00b
At 2007-07-30 14:16:22 +0100, [EMAIL PROTECTED] wrote: I don't really understand that error - especially as the identical code does work for the download limits. I think it's only that you define $WAN and later use $wAN, so tc thinks it's missing an argument, and gets confused. -- ams ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Re: tc n00b
Eck, how embarrassing. Thanks for that - now fixed. I still get errors though: 137.222.235.125 Error: Qdisc tbf is classless. Error: Qdisc tbf is classless. Any ideas what's broken? I'm not so hot on classful queueing disciplines! Cheers, Jonathan Jonathan Gazeley Wireless VPN Team Information Systems Computing University of Bristol Abhijit Menon-Sen wrote: At 2007-07-30 14:16:22 +0100, [EMAIL PROTECTED] wrote: I don't really understand that error - especially as the identical code does work for the download limits. I think it's only that you define $WAN and later use $wAN, so tc thinks it's missing an argument, and gets confused. -- ams ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Re: tc n00b
At 2007-07-30 14:36:03 +0100, [EMAIL PROTECTED] wrote: 137.222.235.125 Error: Qdisc tbf is classless. Error: Qdisc tbf is classless. One of these is from the $LAN line, and one from the $WAN one, right? Any ideas what's broken? I'm not so hot on classful queueing disciplines! It's not really clear to me what you want, but I'm guessing you want to add a CBQ (not TBF) class, and then add a TBF qdisc (with tc qdisc add) under that class. But I don't know why you would want to do that. (I'd recommend using HTB instead of CBQ, and attaching a prio qdisc to each HTB class.) -- ams ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Re: tc n00b
As far as I'm concerned, it doesn't matter what I use, so long as I get the result - I just need to have each user alloted a certain upload and download speed. Nothing too fancy. I tried switching to HTB. I amended my commands but I don't know if my kernel supports it. I've got CentOS 5.0 with kernel 2.6.18 but I now get errors like these: 137.222.235.125 RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel Any clues? (Sorry to ask so many favours, and thanks for your time) Jonathan Abhijit Menon-Sen wrote: At 2007-07-30 14:36:03 +0100, [EMAIL PROTECTED] wrote: 137.222.235.125 Error: Qdisc tbf is classless. Error: Qdisc tbf is classless. One of these is from the $LAN line, and one from the $WAN one, right? Any ideas what's broken? I'm not so hot on classful queueing disciplines! It's not really clear to me what you want, but I'm guessing you want to add a CBQ (not TBF) class, and then add a TBF qdisc (with tc qdisc add) under that class. But I don't know why you would want to do that. (I'd recommend using HTB instead of CBQ, and attaching a prio qdisc to each HTB class.) -- ams -- Jonathan Gazeley Wireless VPN Team Information Systems Computing University of Bristol ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Re: tc n00b
On Mon, Jul 30, 2007 at 02:58:00PM +0100, Jonathan Gazeley wrote: [...] 137.222.235.125 RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel RTNETLINK answers: No such file or directory RTNETLINK answers: Invalid argument We have an error talking to the kernel [...] Hint: If you run your script as bash -x script_name (or use #!/bin/sh -x as shabang), you will be able to see which exact command caused the error message. Regards, -MM ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc