Re: help needed - state of california needs a benchmark - beware bufferbloat
On 01/29/2011 01:00 PM, Mike wrote: Hello, My company is small clec / broadband provider serving rural communities in northern California, and we are the recipient of a small grant from the state thru our public utilities commission. We went out to 'middle of nowhere' and deployed adsl2+ in fact (chalk one up for the good guys!), and now that we're done, our state puc wants to gather performance data to evaluate the result of our project and ensure we delivered what we said we were going to. Bigger picture, our state is actively attempting to map broadband availability and service levels available and this data will factor into this overall picture, to be used for future grant/loan programs and other support mechanisms, so this really is going to touch every provider who serves end users in the state. The rub is, that they want to legislate that web based 'speedtest.com' is the ONLY and MOST AUTHORITATIVE metric that trumps all other considerations and that the provider is %100 at fault and responsible for making fraudulent claims if speedtest.com doesn't agree. No discussion is allowed or permitted about sync rates, packet loss, internet congestion, provider route diversity, end user computer performance problems, far end congestion issues, far end server issues or cpu loading, latency/rtt, or the like. They are going to decide that the quality of any provider service, is solely and exclusively resting on the numbers returned from 'speedtest.com' alone, period. All of you in this audience, I think, probably immediately understand the various problems with such an assertion. Its one of these situations where - to the uninitiated - it SEEMS LIKE this is the right way to do this, and it SEEMS LIKE there's some validity to whats going on - but in practice, we engineering types know it's a far different animal and should not be used for real live benchmarking of any kind where there is a demand for statistical validity. My feeling is that - if there is a need for the state to do benchmarking, then it outta be using statistically significant methodologies for same along the same lines as any other benchmark or test done by other government agencies and national standards bodies that are reproducible and dependable. The question is, as a hotbutton issue, how do we go about getting 'the message' across, how do we go about engineering something that could be considered statistically relevant, and most importantly, how do we get this to be accepted by non-technical legislators and regulators? Mike, For general tests of most things an ISP does, ICSI's netalyzr tests can't be beat. http://netalyzr.icsi.berkeley.edu/ There are also tests at m-lab that may be useful: http://www.measurementlab.net/ As in all pieces of software, these may have bugs; netalyzr was under detecting bufferbloat on high bandwidth links until recently; this should be fixed now, I hope. And SamKnows is doing the FCC broadband tests. The speedtest.net tests (and pingtest.net) are good as far as they go (and you can host them someplace yourself; as others have noted, having and endpoint at someplace you control is wise); but they don't tell the whole story: they miss a vital issue that has been hidden. Here's the rub: Most tests have focussed on bandwidth (now misnamed speed by marketing, which it isn't). Some tests have tested latency. But there have been precious few that test latency under load, which is how we've gotten into a world of hurt on broadband over the last decade, where we now have a situation where a large fraction of broadband has latencies under load measured in *seconds*. (See: http://gettys.wordpress.com/ and bufferbloat.net). These both make for fuming retail customers, as well as lots of service calls (I know, I generated quite a few myself over the years). This is a killer for lots of applications, VOIP, teleconferencing, gaming, remote desktop hosting, etc. Netalyzr tries to test for excessive buffering, as does at least one of the mlabs tests. Dave Clark and I have been talking to SamKnows and Ookla to try to get latency under load tests added to the mix. I think we've been having some traction at getting such tests added, but it's slightly too soon to tell. We also need tests to identify ISP's failing to run queue management internal to their networks, as there is both research and anecdotal data that shows that that is also much more common than it should be. Some ISP's do a wonderful job, and others don't; Van Jacobson believes this is because Sally Floyd and his classic RED algorithm is buggy, and tuning it has scared many operators off; I believe his explanation. So far, so bad. Then there is the home router/host disaster: As soon as you move the bottleneck link from the broadband hop to the 802.11 link usually beyond it these days (by higher broadband bandwidth, or by having several chimneys in your house as I do, dropping the wireless bandwidth), you
Re: help needed - state of california needs a benchmark
On 29/01/11 11:36 AM, Roy wrote: On 1/29/2011 10:00 AM, Mike wrote: The rub is, that they want to legislate that web based 'speedtest.com' is the ONLY and MOST AUTHORITATIVE metric that trumps all other considerations You took the state's money so you are stuck with their dumb rules. As I read the OP's post, it's not a rule *YET* (they want to legislate...) and he's asking for help to convince them to adopt a better rule, which seems like a perfectly reasonable objective. jc
Re: help needed - state of california needs a benchmark
On 29/01/11 10:00 -0800, Mike wrote: The rub is, that they want to legislate that web based 'speedtest.com' is the ONLY and MOST AUTHORITATIVE metric that trumps all other considerations and that the provider is %100 at fault and responsible for making fraudulent claims if speedtest.com doesn't agree. No discussion is allowed or permitted about sync rates, packet loss, internet congestion, provider route diversity, end user computer performance problems, far end congestion issues, far end server issues or cpu loading, latency/rtt, or the like. They are going to decide that the quality of any provider service, is solely and exclusively resting on the numbers returned from 'speedtest.com' alone, period. If you license the software with Ookla, you can install it on a local server and, with your permission, be listed on the speedtest.net site. When your customers visit speedtest.net, your server is, or is close to, the default server that your customers land at. You could try to convince the state that their metric is suboptimal and X is superior, but if your *customers* are anything like ours, it's even harder to educate them why remote speed tests aren't always an accurate measurement of the service you're providing. We've learned to pick our fights, and this isn't one of them. -- Dan White
Re: help needed - state of california needs a benchmark
Mike, nothing is perfect, so let's just start with that. What the FCC has done to measure this is to partner with Sam Knows and then have friendly DSL subs for the participating telcos to run modified CPE firmware to test against their servers. We have been collecting data for this for the past couple of months, actually. More can be found here: http://www.samknows.com/broadband/fcc_and_samknows While even that I have issues with, it certainly is better than hitting that speedtest site where anything at all problematic on the customer LAN side of the CPE can cause erroneous results. Good luck, -Jeff On Jan 29, 2011, at 10:00 AM, Mike wrote: Hello, My company is small clec / broadband provider serving rural communities in northern California, and we are the recipient of a small grant from the state thru our public utilities commission. We went out to 'middle of nowhere' and deployed adsl2+ in fact (chalk one up for the good guys!), and now that we're done, our state puc wants to gather performance data to evaluate the result of our project and ensure we delivered what we said we were going to. Bigger picture, our state is actively attempting to map broadband availability and service levels available and this data will factor into this overall picture, to be used for future grant/loan programs and other support mechanisms, so this really is going to touch every provider who serves end users in the state. The rub is, that they want to legislate that web based 'speedtest.com' is the ONLY and MOST AUTHORITATIVE metric that trumps all other considerations and that the provider is %100 at fault and responsible for making fraudulent claims if speedtest.com doesn't agree. No discussion is allowed or permitted about sync rates, packet loss, internet congestion, provider route diversity, end user computer performance problems, far end congestion issues, far end server issues or cpu loading, latency/rtt, or the like. They are going to decide that the quality of any provider service, is solely and exclusively resting on the numbers returned from 'speedtest.com' alone, period. All of you in this audience, I think, probably immediately understand the various problems with such an assertion. Its one of these situations where - to the uninitiated - it SEEMS LIKE this is the right way to do this, and it SEEMS LIKE there's some validity to whats going on - but in practice, we engineering types know it's a far different animal and should not be used for real live benchmarking of any kind where there is a demand for statistical validity. My feeling is that - if there is a need for the state to do benchmarking, then it outta be using statistically significant methodologies for same along the same lines as any other benchmark or test done by other government agencies and national standards bodies that are reproducible and dependable. The question is, as a hotbutton issue, how do we go about getting 'the message' across, how do we go about engineering something that could be considered statistically relevant, and most importantly, how do we get this to be accepted by non-technical legislators and regulators? Mike-
Re: help needed - state of california needs a benchmark
I think the big deal here is the 100% thing. If Speedtest is one of many tests, then I don't particularly see the problem. It shouldn't be any more difficult to convince politicians that any system (testing or otherwise) can have problems than it is to convince them of any other hard fact. (IOW: Nearly impossible, but you have to try. :) -- TTFN, patrick On Jan 29, 2011, at 1:29 PM, Jeff Richmond wrote: Mike, nothing is perfect, so let's just start with that. What the FCC has done to measure this is to partner with Sam Knows and then have friendly DSL subs for the participating telcos to run modified CPE firmware to test against their servers. We have been collecting data for this for the past couple of months, actually. More can be found here: http://www.samknows.com/broadband/fcc_and_samknows While even that I have issues with, it certainly is better than hitting that speedtest site where anything at all problematic on the customer LAN side of the CPE can cause erroneous results. Good luck, -Jeff On Jan 29, 2011, at 10:00 AM, Mike wrote: Hello, My company is small clec / broadband provider serving rural communities in northern California, and we are the recipient of a small grant from the state thru our public utilities commission. We went out to 'middle of nowhere' and deployed adsl2+ in fact (chalk one up for the good guys!), and now that we're done, our state puc wants to gather performance data to evaluate the result of our project and ensure we delivered what we said we were going to. Bigger picture, our state is actively attempting to map broadband availability and service levels available and this data will factor into this overall picture, to be used for future grant/loan programs and other support mechanisms, so this really is going to touch every provider who serves end users in the state. The rub is, that they want to legislate that web based 'speedtest.com' is the ONLY and MOST AUTHORITATIVE metric that trumps all other considerations and that the provider is %100 at fault and responsible for making fraudulent claims if speedtest.com doesn't agree. No discussion is allowed or permitted about sync rates, packet loss, internet congestion, provider route diversity, end user computer performance problems, far end congestion issues, far end server issues or cpu loading, latency/rtt, or the like. They are going to decide that the quality of any provider service, is solely and exclusively resting on the numbers returned from 'speedtest.com' alone, period. All of you in this audience, I think, probably immediately understand the various problems with such an assertion. Its one of these situations where - to the uninitiated - it SEEMS LIKE this is the right way to do this, and it SEEMS LIKE there's some validity to whats going on - but in practice, we engineering types know it's a far different animal and should not be used for real live benchmarking of any kind where there is a demand for statistical validity. My feeling is that - if there is a need for the state to do benchmarking, then it outta be using statistically significant methodologies for same along the same lines as any other benchmark or test done by other government agencies and national standards bodies that are reproducible and dependable. The question is, as a hotbutton issue, how do we go about getting 'the message' across, how do we go about engineering something that could be considered statistically relevant, and most importantly, how do we get this to be accepted by non-technical legislators and regulators? Mike-
RE: help needed - state of california needs a benchmark
We've learned to pick our fights, and this isn't one of them. -- Dan White The most effective mechanism I've seen for explaining the problem is latency and VOIP. Set up an artificially latency-ridden, high bandwidth connection, then connect to a PBX using a softphone. One call is generally sufficient proof of the issue. Ookla does offer another metric, at http://www.pingtest.net/, which provides some valuable additional information. You can therefore infer an argument by speedtest.net: Gov: Speedtest.net is an authorative location for all testing. Speedtest.net: Anyone can host our test application, so that is clearly false. Gov: The only important factor in certification is bandwidth to speedtest.net. Speedtest.net: We offer other connection quality tests that don't rely on bandwidth. I often find that statements people make rely on half-truths gleaned from other people, and that generally, the fastest way to conclude an argument is to go to the source and extract the complete truth, and then present in contrast. It is difficult to argue with your own source. :-) Nathan
Re: help needed - state of california needs a benchmark
On 1/29/2011 10:00 AM, Mike wrote: Hello, My company is small clec / broadband provider serving rural communities in northern California, and we are the recipient of a small grant from the state thru our public utilities commission. We went out to 'middle of nowhere' and deployed adsl2+ in fact (chalk one up for the good guys!), and now that we're done, our state puc wants to gather performance data to evaluate the result of our project and ensure we delivered what we said we were going to. Bigger picture, our state is actively attempting to map broadband availability and service levels available and this data will factor into this overall picture, to be used for future grant/loan programs and other support mechanisms, so this really is going to touch every provider who serves end users in the state. The rub is, that they want to legislate that web based 'speedtest.com' is the ONLY and MOST AUTHORITATIVE metric that trumps all other considerations and that the provider is %100 at fault and responsible for making fraudulent claims if speedtest.com doesn't agree. No discussion is allowed or permitted about sync rates, packet loss, internet congestion, provider route diversity, end user computer performance problems, far end congestion issues, far end server issues or cpu loading, latency/rtt, or the like. They are going to decide that the quality of any provider service, is solely and exclusively resting on the numbers returned from 'speedtest.com' alone, period. All of you in this audience, I think, probably immediately understand the various problems with such an assertion. Its one of these situations where - to the uninitiated - it SEEMS LIKE this is the right way to do this, and it SEEMS LIKE there's some validity to whats going on - but in practice, we engineering types know it's a far different animal and should not be used for real live benchmarking of any kind where there is a demand for statistical validity. My feeling is that - if there is a need for the state to do benchmarking, then it outta be using statistically significant methodologies for same along the same lines as any other benchmark or test done by other government agencies and national standards bodies that are reproducible and dependable. The question is, as a hotbutton issue, how do we go about getting 'the message' across, how do we go about engineering something that could be considered statistically relevant, and most importantly, how do we get this to be accepted by non-technical legislators and regulators? Mike- You took the state's money so you are stuck with their dumb rules. Furthermore the CPUC people aren't stupid. They have highly paid consultants as well as professors from colleges in California that are advising them. Unless you have some plan for a very inexpensive alternative, don't think you are going to make any headway
Re: help needed - state of california needs a benchmark
Mike wrote: The rub is, that they want to legislate that web based 'speedtest.com' is the ONLY and MOST AUTHORITATIVE metric that trumps all other considerations and that the provider is %100 at fault and responsible for making fraudulent claims if speedtest.com doesn't agree. speedtest.net?
Re: help needed - state of california needs a benchmark
On Sat, Jan 29, 2011 at 10:00:36AM -0800, Mike wrote: issue, how do we go about getting 'the message' across, how do we go about engineering something that could be considered statistically relevant, and most importantly, how do we get this to be accepted by non-technical legislators and regulators? How about this analogy: Using speedtest.com as the sole benchmark is like trying to test the speed and throughput of the city streets in Sacramento by seeing how long it takes to drive to New York City and back. Oh, and why should we be responsible for the speeds on the Interstate portions of that route when we only control the city streets and local secondary roads?
Re: help needed - state of california needs a benchmark
On Sat, Jan 29, 2011 at 1:29 PM, Jeff Richmond jeff.richm...@gmail.com wrote: Mike, nothing is perfect, so let's just start with that. What the FCC has done to measure this is to partner with Sam Knows and then have friendly DSL subs for the participating telcos to run modified CPE firmware to test against their servers. We have been collecting data for this for the past couple of months, actually. More can be found here: http://www.samknows.com/broadband/fcc_and_samknows note that samknows has some deficiencies in their platform, at least: o no ip v6 support o the home-routers/gateways/aps randomly reboot with completely non-functional setups their customer support ... isn't really available, ever. While even that I have issues with, it certainly is better than hitting that speedtest site where anything at all problematic on the customer LAN side of the CPE can cause erroneous results. the above aside, how about using the network test suite from Mlabs? http://www.measurementlab.net/measurement-lab-tools Or ask the swedish folk who've deployed 'speedtest' gear for the swedish isp/users to tst against? (common test infrastructure). -chris
Re: help needed - state of california needs a benchmark
Morning Mike, The *New Zealand Government* don't use speedtest.net as a benchmark. Our Government uses a consulting company to provide a range of tests that address the issues you're talking about and benchmarks are published each year. http://www.comcom.govt.nz/broadband-reports The user and network communities are not 100% happy with the way this testing is done either. http://www.geekzone.co.nz/forums.asp?forumid=49topicid=73698 Some providers are know to fudge the results by putting QoS on the test paths. http://weathermap.karen.net.nz/ is a New Zealand academic project that shows their network performance in real time. This is a very useful site for demonstrating the sort of tools that Governments should be looking for when doing performance measuring. Recent work done by Jared Kells, in Australia, on consumer level network performance shows a very interesting picture (pictures are best for political people). http://forums.whirlpool.net.au/forum-replies.cfm?t=1579142 Kells demonstrates that providers deliver very different results for national and international sites. Kells provides a set of Open Source tools to do your own testing. http://www.truenet.co.nz - John Butt - is a commercial start up providing another range of testing metrics which the user community at www.geekzone.co.nz seem to be much happier with as a proper indication of network performance. I have talked with John personally and can attest that the testing is fairly robust and addresses issues that you've raised. http://www.truenet.co.nz/how-does-it-work The recent upgrades of www.telstraclear.co.nz HFC network from DOCIS2.0 (25/2 max) to DOCIS3.0 (100/10 testing introduction speed) presented a range of challenges for John's testing. http ramp up speeds to 100mbit cause impact on test results, so John had to change the way they were testing to get a better performance presentation. Internode in Australia have learnt the hard way recently that consumer expectation of their new NBN FTTH network needs to be managed carefully. As a result of some very poor media press over the performance of an education site recently installed in Tasmania, they have engaged in quite a bit of consumer education around network performance. http://www.theaustralian.com.au/news/nation/governments-broadband-not-up-to-speed-at-tasmanian-school/story-e6frg6nf-1225961150410 - http://whrl.pl/Rcyrhz - http://forums.whirlpool.net.au/user/6258Simon Hackett - Internode CEO responds. *Speedtest.net* will only provide a BIR/PIR measure, and not CIR, which is not an indicator of service quality. In New Zealand SpeedTest.net is used extensively with a number of hosting servers. The information is fundamentally flawed as you have no control over what testing the end user performs. In my case I can product three different tests from a 15/2 HFC service and get 3 different results. http://www.speedtest.net/result/1133639492.png - Test 1 - The application has identified that I am located in Christchurch New Zealand so has selected a Christchurch based server for testing (www.snap.co.nz). As you can see the results show ~7.5/2.1mbits/s. http://www.speedtest.net/result/1133642520.png - Test 2 - This time I've chosen the CityLink (www.citylink.co.nz) server in Wellington New Zealand. ~6.2/1.97bits/s. http://www.speedtest.net/result/1052386636.png - Test 3 - from 12/12/10 shows ~15.1/2.15. This was tested to an Auckland, New Zealand server. I did run a set of tests this morning to the Auckland servers as well, however they are all being limited to the same numbers as the Christchurch test (1) now. None of the servers are on my providers network and performance is governed by the peering/hand overs between the networks. Christchurch - Wellington - 320km - Christchurch - Auckland - 750km straight line distances according to Google Earth. The HFC service I'm using will deliver a through put of 15/2 for some time even at peek usage times when pulling content off the providers own network. Ok, that's enough for now. I hope this helps and let me know if you need any more assistance. Cheers Don
RE: help needed - state of california needs a benchmark
Configure your DNS server so that speedtest.net and every variation to point to the Speedtest that you host... Frank -Original Message- From: Mike [mailto:mike-na...@tiedyenetworks.com] Sent: Saturday, January 29, 2011 12:01 PM To: NANOG list Subject: help needed - state of california needs a benchmark Hello, My company is small clec / broadband provider serving rural communities in northern California, and we are the recipient of a small grant from the state thru our public utilities commission. We went out to 'middle of nowhere' and deployed adsl2+ in fact (chalk one up for the good guys!), and now that we're done, our state puc wants to gather performance data to evaluate the result of our project and ensure we delivered what we said we were going to. Bigger picture, our state is actively attempting to map broadband availability and service levels available and this data will factor into this overall picture, to be used for future grant/loan programs and other support mechanisms, so this really is going to touch every provider who serves end users in the state. The rub is, that they want to legislate that web based 'speedtest.com' is the ONLY and MOST AUTHORITATIVE metric that trumps all other considerations and that the provider is %100 at fault and responsible for making fraudulent claims if speedtest.com doesn't agree. No discussion is allowed or permitted about sync rates, packet loss, internet congestion, provider route diversity, end user computer performance problems, far end congestion issues, far end server issues or cpu loading, latency/rtt, or the like. They are going to decide that the quality of any provider service, is solely and exclusively resting on the numbers returned from 'speedtest.com' alone, period. All of you in this audience, I think, probably immediately understand the various problems with such an assertion. Its one of these situations where - to the uninitiated - it SEEMS LIKE this is the right way to do this, and it SEEMS LIKE there's some validity to whats going on - but in practice, we engineering types know it's a far different animal and should not be used for real live benchmarking of any kind where there is a demand for statistical validity. My feeling is that - if there is a need for the state to do benchmarking, then it outta be using statistically significant methodologies for same along the same lines as any other benchmark or test done by other government agencies and national standards bodies that are reproducible and dependable. The question is, as a hotbutton issue, how do we go about getting 'the message' across, how do we go about engineering something that could be considered statistically relevant, and most importantly, how do we get this to be accepted by non-technical legislators and regulators? Mike-
Re: help needed - state of california needs a benchmark
On Sun, 30 Jan 2011, Don Gould wrote: Ok, that's enough for now. I hope this helps and let me know if you need any more assistance. In Sweden, Bredbandskollen.se (translates to Broadband check) rules supreme. It uses two parallell TCP sessions to measure speed, and the whole industry has agreed (mostly product managers were involved, little attention was given to technical arguments) to set a minimum standard for each service and let people cancel contracts or downgrade if they didn't get that level. For instance, an ADSL2+ connection sold as up to 24 megabit/s now let's you cancel or downgrade if you don't get 12 megabit/s TCP throughput. For 100/10 service the lower limit is 40 megabit/s. There is a requirement for the user to not use wireless and to put their computer directly into the ethernet jack (ETTH) without the use of a NAT router, because these can heavily affect service speed. It also features a guide to how to diagnose why you're getting a lower than expected speed. Customer complaints are down so generally this seems to increase customer satisfaction. My biggest objection is that with a 100/10 service download speeds can vary a lot depending what browser vendor and version one is using, TCP settings of course also play a role. The upside is that the sevice is extremely easy to use. -- Mikael Abrahamssonemail: swm...@swm.pp.se
