Re: [LARTC] Theory test
On 12/6/05, Andy Furniss [EMAIL PROTECTED] wrote: Kenneth Kalmer wrote: Are we talking about ingress or egress? Egress, all my ingress experiments worked 100% (mostly prioritization, that's all) That's handy I was expecting you to say ingress. Outbound should be totally under your control. How much bandwidth do you have (and how much are you sacrificing)? Sadly, only 512kbits but upgrading to 1024kbits in Feb You have 512kbit upstream - symmetrical dsl - what rate do you set htb? ADSL, 512kbps down and 256kbps up. Parent for the internet traffic is set at 500kbps, to make sure it becomes the bottleneck... What are the lengths of queues for each user? Pardon me? It's less relevant for egress - for ingress shaping the length of the queues if too long can mess things up, and unless you use esfq it would have been hard when shaping 200 users to reduce it/them. I attach an esfq to each child HTB, but as you say it would be less relevenat for egress... What is the link DSL/other? ADSL, African style... Not sdsl then - is your egress 512? Do you know what type of connection you have eg pppoa/e or bridged ip etc. I assume whatever it is ends up as atm cells? Barely, as said above it's 512/256 VPN. Underneath the VPN it runs PPPoE, but the service simulates a leased line, static ip's, the works... Andy. -- Kenneth Kalmer [EMAIL PROTECTED] [EMAIL PROTECTED] stats http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpageusername=kenneth%2Ekalmer ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
Kenneth Kalmer wrote: ADSL, 512kbps down and 256kbps up. Parent for the internet traffic is set at 500kbps, to make sure it becomes the bottleneck... I used to use 400 when I had 512 ingress, so I am amazed that works - but then you say ingress not the problem. I attach an esfq to each child HTB, but as you say it would be less relevenat for egress... Were it ingress I woud say have just one class with esfq for sharing out bulk traffic per user. Do you know what type of connection you have eg pppoa/e or bridged ip etc. I assume whatever it is ends up as atm cells? Barely, as said above it's 512/256 VPN. Underneath the VPN it runs PPPoE, but the service simulates a leased line, static ip's, the works... I bet there are alot of overheads on that - and if you are pushing the rate close to limit like you are on ingress I suspect you are going overlimits. Even if you test with an upload and find a rate that seems OK it will all fall apart when the traffic consists of small packets. You have real ips aswell - so all your students can become p2p nodes = lots of small packets. I would consider using htb's mpu and overhead on each rate/ceil mpu with pppoe/atm is going to be 106 bytes - overhead I am not sure as it's not normal dsl - if it were you could patch tc/htb to do it perfectly. Often your atm level sync rate will be a bit higher than the advertised rate. If you can get your kit to tell you what that is it will be helpful. Andy. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
On 12/6/05, Andy Furniss [EMAIL PROTECTED] wrote: Kenneth Kalmer wrote: ADSL, 512kbps down and 256kbps up. Parent for the internet traffic is set at 500kbps, to make sure it becomes the bottleneck... I used to use 400 when I had 512 ingress, so I am amazed that works - but then you say ingress not the problem. I'll tone it down to see if it makes a difference, but I need to keep it as close as possible to the 512 because the line gets very congested... I attach an esfq to each child HTB, but as you say it would be less relevenat for egress... Were it ingress I woud say have just one class with esfq for sharing out bulk traffic per user. I meant on traffic going to the network (egress) I attached an esfq to each users' limit Do you know what type of connection you have eg pppoa/e or bridged ip etc. I assume whatever it is ends up as atm cells? Barely, as said above it's 512/256 VPN. Underneath the VPN it runs PPPoE, but the service simulates a leased line, static ip's, the works... I bet there are alot of overheads on that - and if you are pushing the rate close to limit like you are on ingress I suspect you are going overlimits. Even if you test with an upload and find a rate that seems OK it will all fall apart when the traffic consists of small packets. Amazingly not, we have the same line in the office, no shaping, and we often sustain 110% capacity for very long periods of time. I believe the provider uses very heavy compression on the line. Still, it's blazingly fast compared to the traditional ADSL offerings available here. You have real ips aswell - so all your students can become p2p nodes = lots of small packets. I would consider using htb's mpu and overhead on each rate/ceil mpu with pppoe/atm is going to be 106 bytes - overhead I am not sure as it's not normal dsl - if it were you could patch tc/htb to do it perfectly. Often your atm level sync rate will be a bit higher than the advertised rate. If you can get your kit to tell you what that is it will be helpful. The students are NATted, and firewalled to hell and back, so filesharing is not a problem. They try, but who wouldn't... Andy. Thanks -- Kenneth Kalmer [EMAIL PROTECTED] [EMAIL PROTECTED] stats http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpageusername=kenneth%2Ekalmer ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
Kenneth Kalmer wrote: On 12/6/05, Andy Furniss [EMAIL PROTECTED] wrote: Kenneth Kalmer wrote: ADSL, 512kbps down and 256kbps up. Parent for the internet traffic is set at 500kbps, to make sure it becomes the bottleneck... I used to use 400 when I had 512 ingress, so I am amazed that works - but then you say ingress not the problem. I'll tone it down to see if it makes a difference, but I need to keep it as close as possible to the 512 because the line gets very congested... Yes 400 may be a bit far I used it because latency was important to me. Remember queueing traffic on the wrong end of a bottleneck is not ideal - if you want to run at 99% then the queue will only grow 1 packet for every 100 it passes - useless really. You need to be the one that decides which packets get dropped to have some influence over the sharing out of the bandwidth - too close to the limit and the dropping will just happen at the isp/teleco FIFO. It's for the same reason that I am concerned about queue length - if it/they are too long then the other end won't know about the tail drop quickly enough as it will take time for the packets infront to clear. (I hacked esfq to head drop for this reason - though there are issues with it to do with imq that I haven't sorted yet. It seems to work and is stable for me, but could do with a bit more rigorous testing). I attach an esfq to each child HTB, but as you say it would be less relevenat for egress... Were it ingress I woud say have just one class with esfq for sharing out bulk traffic per user. I meant on traffic going to the network (egress) I attached an esfq to each users' limit Hmm - maybe our ingresses and egresses are getting confused - when I say ingress I refer to traffic coming in on the bottleneck (internet) link - even if you are shaping it as egress on your lan facing interface. So to be clear the Problem is actually what I call ingress - the 512kbit traffic - yes? Do you know what type of connection you have eg pppoa/e or bridged ip etc. I assume whatever it is ends up as atm cells? Barely, as said above it's 512/256 VPN. Underneath the VPN it runs PPPoE, but the service simulates a leased line, static ip's, the works... I bet there are alot of overheads on that - and if you are pushing the rate close to limit like you are on ingress I suspect you are going overlimits. Even if you test with an upload and find a rate that seems OK it will all fall apart when the traffic consists of small packets. Amazingly not, we have the same line in the office, no shaping, and we often sustain 110% capacity for very long periods of time. I believe the provider uses very heavy compression on the line. Still, it's blazingly fast compared to the traditional ADSL offerings available here. Nice in some ways, but if compression is being used you effectivly have variable bandwidth depending on traffic type, which is going to be a pain to set a rate for. Are you sure it's compression and not a more generous sync rate - your modem should tell you - it would be interesting to know. You have real ips aswell The students are NATted, and firewalled to hell and back, so filesharing is not a problem. They try, but who wouldn't... Ahh OK. Andy. PS - I am getting undeliverable mails when replying in this thread. Failed to deliver to '[EMAIL PROTECTED]' SMTP module(domain terminator.swip.net) reports: host terminator.swip.net says: 550 Recipient not allowed to receive email (GOT_TO) ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
On 12/6/05, Andy Furniss [EMAIL PROTECTED] wrote: Kenneth Kalmer wrote: On 12/6/05, Andy Furniss [EMAIL PROTECTED] wrote: Kenneth Kalmer wrote: ADSL, 512kbps down and 256kbps up. Parent for the internet traffic is set at 500kbps, to make sure it becomes the bottleneck... I used to use 400 when I had 512 ingress, so I am amazed that works - but then you say ingress not the problem. I'll tone it down to see if it makes a difference, but I need to keep it as close as possible to the 512 because the line gets very congested... Yes 400 may be a bit far I used it because latency was important to me. Remember queueing traffic on the wrong end of a bottleneck is not ideal - if you want to run at 99% then the queue will only grow 1 packet for every 100 it passes - useless really. You need to be the one that decides which packets get dropped to have some influence over the sharing out of the bandwidth - too close to the limit and the dropping will just happen at the isp/teleco FIFO. Understood, I didn't think the difference needs to be that big though... It's for the same reason that I am concerned about queue length - if it/they are too long then the other end won't know about the tail drop quickly enough as it will take time for the packets infront to clear. (I hacked esfq to head drop for this reason - though there are issues with it to do with imq that I haven't sorted yet. It seems to work and is stable for me, but could do with a bit more rigorous testing). I attach an esfq to each child HTB, but as you say it would be less relevenat for egress... Were it ingress I woud say have just one class with esfq for sharing out bulk traffic per user. I meant on traffic going to the network (egress) I attached an esfq to each users' limit Hmm - maybe our ingresses and egresses are getting confused - when I say ingress I refer to traffic coming in on the bottleneck (internet) link - even if you are shaping it as egress on your lan facing interface. So to be clear the Problem is actually what I call ingress - the 512kbit traffic - yes? Yip, I meant that I shape the internet link as network egress so to speak... On the ingress side (IMQ) I'm playing with protocol prioritization for things like DNS, SSL TCP/ACK. Do you know what type of connection you have eg pppoa/e or bridged ip etc. I assume whatever it is ends up as atm cells? Barely, as said above it's 512/256 VPN. Underneath the VPN it runs PPPoE, but the service simulates a leased line, static ip's, the works... I bet there are alot of overheads on that - and if you are pushing the rate close to limit like you are on ingress I suspect you are going overlimits. Even if you test with an upload and find a rate that seems OK it will all fall apart when the traffic consists of small packets. Amazingly not, we have the same line in the office, no shaping, and we often sustain 110% capacity for very long periods of time. I believe the provider uses very heavy compression on the line. Still, it's blazingly fast compared to the traditional ADSL offerings available here. Nice in some ways, but if compression is being used you effectivly have variable bandwidth depending on traffic type, which is going to be a pain to set a rate for. Are you sure it's compression and not a more generous sync rate - your modem should tell you - it would be interesting to know. I don't have access to the modem, it's a Cisco router that they control. However, I know the physical capacity of the link from the telco is 512kbps downstream, and I shape according to that. You have real ips aswell The students are NATted, and firewalled to hell and back, so filesharing is not a problem. They try, but who wouldn't... Ahh OK. :) Andy. PS - I am getting undeliverable mails when replying in this thread. Same here, might be a list member... Failed to deliver to '[EMAIL PROTECTED]' SMTP module(domain terminator.swip.net) reports: host terminator.swip.net says: 550 Recipient not allowed to receive email (GOT_TO) -- Kenneth Kalmer [EMAIL PROTECTED] [EMAIL PROTECTED] stats http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpageusername=kenneth%2Ekalmer ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
On 12/5/05, Peter Surda [EMAIL PROTECTED] wrote: On Mon, 5 Dec 2005 19:42:09 +0200 Kenneth Kalmer [EMAIL PROTECTED] wrote: Guys Hi, Sharing a link between 200 users Been there, done that (1400 users even). -= WRR =- My favourite, but with the most disappointment at the moment... I can see the weights are adjusted, and our trials have shown that the link gets shared equally. However, in implementation it doesn't work that way. The abusers can still go mad, and now at link capacity (X), no longer at Z. This has caused some serious problems for non-abusive users. You probably need to use bigger incr and decr in wrr. Check out shurdix' tc script: http://docs.shurdix.org/shurdix:learn , it calculates the parameters automatically, you only need to set incoming and outgoing bandwidth. Hmm, hacked it up so much that I can bearly recognize it... IIRC you had two variables, one tfor the time it takes to go from MAX to MIN, and another for the opposite. I changed them so that the MAX to MIN will now be 15 seconds, and the later 450 seconds. Did I miss the plot here? We can't tolerate abuse around here... Kenneth Kalmer [EMAIL PROTECTED] Yours sincerely, Peter Best -- Kenneth Kalmer [EMAIL PROTECTED] [EMAIL PROTECTED] stats http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpageusername=kenneth%2Ekalmer ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
On Mon, 5 Dec 2005 19:42:09 +0200 Kenneth Kalmer [EMAIL PROTECTED] wrote: Guys Hi, Sharing a link between 200 users Been there, done that (1400 users even). -= WRR =- My favourite, but with the most disappointment at the moment... I can see the weights are adjusted, and our trials have shown that the link gets shared equally. However, in implementation it doesn't work that way. The abusers can still go mad, and now at link capacity (X), no longer at Z. This has caused some serious problems for non-abusive users. You probably need to use bigger incr and decr in wrr. Check out shurdix' tc script: http://docs.shurdix.org/shurdix:learn , it calculates the parameters automatically, you only need to set incoming and outgoing bandwidth. Kenneth Kalmer [EMAIL PROTECTED] Yours sincerely, Peter -- http://www.shurdix.org - Linux distribution for routers and firewalls ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
On Monday 05 December 2005 18:42, Kenneth Kalmer wrote: -= HTB =- Set the parent class for internet traffic to X, with 200 children. Each child has a rate of Y, their totals equal X. Each child also has a ceil of Z. This means that Z * 200 X, hence the over subscription. I'm using pretty much the same with success, although not for 200 users, just 5. However, the bandwidth is considerably slower than what you are likely to have (128kbit), so it may be just as critical. What happens here is that several people download at Z, but their speed does not decrease when more people start accessing the internet. They stay at Z, which is a problem. This is a serious problem which should not happen. So assuming that there is no error in your configuration, it's likely to be a HTB bug which should be fixed. -= HFSC =- Tried playing around with the curves, but a lack of knowledge and resources has hampered me from figuring out this one. Is there still no documentation for HFSC around? -= WRR =- Sounds very interesting, unfortunately I didn't have the time to try this one out. So I can't comment on the problems you have with it either. :-( Can anyone advise me on how to get this done properly, please. Somewhere I must be missing something small, and I don't want to paste millions of lines of scripts in here until I know I've got the theory right. The over subscription is the big problem, pure rate limiting works like a charm in my other experiments. The theory sounds fine to me. About the sum of ceils being bigger than the rate of the parent class, that's not really over subscription, but the whole point of the ceil parameter. Over subscription to my understanding would be the case if the guaranteed rates of classes would exceed the parent class rate. But since you say that it sums up to X, the theory sounds just fine to me. I'd like to have a look at your HTB setup, since that is the scheduler I'm most familiar with. If the script is too long, you could upload it somewhere and post the URL to it. Or just cut out the repetitive parts if you aren't using functions for that already anyway (I'm assuming that the 200 user classes are all created the same way). Regards, Andreas Klauer ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
Kenneth Kalmer wrote: Guys Considering the festive season is upon us, thanks to everyone contributing to the list and helping all the readers with your great input! I don't want to mention names, I'll most certainly leave someone out. With this mail I'd like to test some theory on bandwidth management, with my own successes and failures during the past year. Sharing a link between 200 users This has probably been my worst headache this year, since all the trials go well but the implementation doesn't run as expected. So here goes. We have one connection that is shared by 200 users. Mostly students, so abuse is rampant. Each user should have an upper-limit for speed, but the upper limit times the number of users exceed the link capacity (over subscription). The speed must degrade as more and more users are online, so that in peak times the link must still be usable for each and every user, even if dreadfully slow. Here is the implementation in theory: Total internet capacity: X Total number of users: 200 Minimum transfer rates: Y = X / 200 Maximum transfer rates: Z = 8 * Y Over subscription rate: 1:50 Are we talking about ingress or egress? How much bandwidth do you have (and how much are you sacrificing)? What are the lengths of queues for each user? What is the link DSL/other? Andy. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
On 12/5/05, Peter Surda [EMAIL PROTECTED] wrote: On Mon, 5 Dec 2005 19:42:09 +0200 Kenneth Kalmer [EMAIL PROTECTED] wrote: Guys Hi, Sharing a link between 200 users Been there, done that (1400 users even). -= WRR =- My favourite, but with the most disappointment at the moment... I can see the weights are adjusted, and our trials have shown that the link gets shared equally. However, in implementation it doesn't work that way. The abusers can still go mad, and now at link capacity (X), no longer at Z. This has caused some serious problems for non-abusive users. You probably need to use bigger incr and decr in wrr. Check out shurdix' tc script: http://docs.shurdix.org/shurdix:learn , it calculates the parameters automatically, you only need to set incoming and outgoing bandwidth. I'll check this out. By hacking shurdix's tc.sh to sh*t is how I learned to configure WRR properly... Thanks :) Kenneth Kalmer [EMAIL PROTECTED] Yours sincerely, Peter -- http://www.shurdix.org - Linux distribution for routers and firewalls -- Kenneth Kalmer [EMAIL PROTECTED] [EMAIL PROTECTED] stats http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpageusername=kenneth%2Ekalmer ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
On 12/5/05, Andreas Klauer [EMAIL PROTECTED] wrote: On Monday 05 December 2005 18:42, Kenneth Kalmer wrote: -= HTB =- Set the parent class for internet traffic to X, with 200 children. Each child has a rate of Y, their totals equal X. Each child also has a ceil of Z. This means that Z * 200 X, hence the over subscription. I'm using pretty much the same with success, although not for 200 users, just 5. However, the bandwidth is considerably slower than what you are likely to have (128kbit), so it may be just as critical. This is the scary part, I only have 512kbit now, will be upgrading to 1024kbit in Feb... What happens here is that several people download at Z, but their speed does not decrease when more people start accessing the internet. They stay at Z, which is a problem. This is a serious problem which should not happen. So assuming that there is no error in your configuration, it's likely to be a HTB bug which should be fixed. Doubt it, the problem will be on my side... I've seen so many guys here go HTB not working!, and be proved fatally wrong... -= HFSC =- Tried playing around with the curves, but a lack of knowledge and resources has hampered me from figuring out this one. Is there still no documentation for HFSC around? There is, still very cryptic. The best is scattered through the pf mailing lists... -= WRR =- Sounds very interesting, unfortunately I didn't have the time to try this one out. So I can't comment on the problems you have with it either. :-( Can anyone advise me on how to get this done properly, please. Somewhere I must be missing something small, and I don't want to paste millions of lines of scripts in here until I know I've got the theory right. The over subscription is the big problem, pure rate limiting works like a charm in my other experiments. The theory sounds fine to me. About the sum of ceils being bigger than the rate of the parent class, that's not really over subscription, but the whole point of the ceil parameter. Over subscription to my understanding would be the case if the guaranteed rates of classes would exceed the parent class rate. But since you say that it sums up to X, the theory sounds just fine to me. I'd like to have a look at your HTB setup, since that is the scheduler I'm most familiar with. If the script is too long, you could upload it somewhere and post the URL to it. Or just cut out the repetitive parts if you aren't using functions for that already anyway (I'm assuming that the 200 user classes are all created the same way). Correct, right now I'm going to bed. But I'll rip apart my wrr, hfsc and htb scripts and place them in a message for everyone to check out. They're all populated from data in text files, so the scripts themselves are pretty light. Regards, Andreas Klauer ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -- Kenneth Kalmer [EMAIL PROTECTED] [EMAIL PROTECTED] stats http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpageusername=kenneth%2Ekalmer ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
On 12/5/05, Andy Furniss [EMAIL PROTECTED] wrote: Kenneth Kalmer wrote: Guys Considering the festive season is upon us, thanks to everyone contributing to the list and helping all the readers with your great input! I don't want to mention names, I'll most certainly leave someone out. With this mail I'd like to test some theory on bandwidth management, with my own successes and failures during the past year. Sharing a link between 200 users This has probably been my worst headache this year, since all the trials go well but the implementation doesn't run as expected. So here goes. We have one connection that is shared by 200 users. Mostly students, so abuse is rampant. Each user should have an upper-limit for speed, but the upper limit times the number of users exceed the link capacity (over subscription). The speed must degrade as more and more users are online, so that in peak times the link must still be usable for each and every user, even if dreadfully slow. Here is the implementation in theory: Total internet capacity: X Total number of users: 200 Minimum transfer rates: Y = X / 200 Maximum transfer rates: Z = 8 * Y Over subscription rate: 1:50 Are we talking about ingress or egress? Egress, all my ingress experiments worked 100% (mostly prioritization, that's all) How much bandwidth do you have (and how much are you sacrificing)? Sadly, only 512kbits but upgrading to 1024kbits in Feb What are the lengths of queues for each user? Pardon me? What is the link DSL/other? ADSL, African style... -- Kenneth Kalmer [EMAIL PROTECTED] [EMAIL PROTECTED] stats http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpageusername=kenneth%2Ekalmer ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Theory test
Kenneth Kalmer wrote: Are we talking about ingress or egress? Egress, all my ingress experiments worked 100% (mostly prioritization, that's all) That's handy I was expecting you to say ingress. Outbound should be totally under your control. How much bandwidth do you have (and how much are you sacrificing)? Sadly, only 512kbits but upgrading to 1024kbits in Feb You have 512kbit upstream - symmetrical dsl - what rate do you set htb? What are the lengths of queues for each user? Pardon me? It's less relevant for egress - for ingress shaping the length of the queues if too long can mess things up, and unless you use esfq it would have been hard when shaping 200 users to reduce it/them. What is the link DSL/other? ADSL, African style... Not sdsl then - is your egress 512? Do you know what type of connection you have eg pppoa/e or bridged ip etc. I assume whatever it is ends up as atm cells? Andy. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
