Re: disregard, test

[Shrdlu]

On 7/9/2013 1:38 AM, Saku Ytti wrote:

https://i.chzbgr.com/maxW500/7644490752/h49306FE3/

many complain that they've not seen emails from nanog in few days (since
5th day of 'The Cidr Report')


Next time? Please consider just examining the archives, so that you may
verify that indeed, a miracle has occurred, and that indeed, no one has
anything in particular to say. I admit that I checked the archives
myself, when it seemed to quiet.

http://mailman.nanog.org/pipermail/nanog/

Here's hoping for a quiet week, with no operational (or other) issues
that need NANOG participants' attention.

Re: disregard, test

[JP]

On Tue, Jul 09, 2013 at 11:38:37AM +0300, Saku Ytti wrote:
> https://i.chzbgr.com/maxW500/7644490752/h49306FE3/
> 
> many complain that they've not seen emails from nanog in few days (since
> 5th day of 'The Cidr Report')
 
Test win condition not specified.

-J

disregard, test

[Saku Ytti]

https://i.chzbgr.com/maxW500/7644490752/h49306FE3/

many complain that they've not seen emails from nanog in few days (since
5th day of 'The Cidr Report')

-- 
  ++ytti

RE: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Frank Bulk]

Here's a 39-page report that might differ with your perspective: 
http://mitas.csail.mit.edu/papers/Bauer_Clark_Lehr_Broadband_Speed_Measurements.pdf

And another report: 
http://www.netforecast.com/Reports/NFR5103_comScore_ISP_Speed_Test_Accuracy.pdf

Frank

-Original Message-
From: Mike [mailto:mike-na...@tiedyenetworks.com] 
Sent: Wednesday, April 03, 2013 4:08 PM
To: nanog@nanog.org
Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test



These speedtests are pure unscientific bs and I'd love to see them 
called out on the carpet for it.

Mike-

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Srikanth Sundaresan]

[Plug alert]

For longer term monitoring, Project BISmark provides an easy-to-use
system. It's an open source, customizable OpenWRT-based home router that
runs periodic network measurements (latency, throughput, packetloss,
jitter, etc) to nearby MLab servers.

It uses netperf (single and multiple TCP threads), and shaperprobe,
(UDP) for throughput measurements.

Although its original target audience is home users, it can also be used
as a monitoring tool in bigger networks.

More information here:
http://projectbismark.net/

Slides from the talk at NANOG 53:
https://www.nanog.org/meetings/nanog53/presentations/Monday/Sundaresan.pdf

- Srikanth

On 04/04/2013 06:45 AM, Steve Haavik wrote:
>> It'd be nice to know if NDT was not accurate as well.  Anyone tested it?
> 
> We've been using it for a few years. On my laptop that runs linux I get
> fairly consistent results (around 935Mb/s up and down right now) over a
> 1Gig routed link (a couple routers and a firewall in between.) On the
> Windows boxes I usually see a 100 to 200 Mb/s drop on the upload side.
> The last time I checked, you can compile a commandline version of the
> client. I seem to remember the commandline client not taking quite as
> bad a hit on the tests compared to running it on linux, but it's been a
> while since I tried it.
> 
> For us it's been way more accurate than the various speedtest servers
> our customers insist on trying. A while back I switched from compiling
> my own kernel and NDT to using perfSONAR-PS
> (http://psps.perfsonar.net/). I like that they've got live-cd and
> net-install versions. If nothing else it's useful for pointing out the
> difference between a local network issue and Internet Suckage.
> 
> 

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Valdis . Kletnieks]

On Thu, 04 Apr 2013 17:29:40 +0200, Mikael Abrahamsson said:
> On Thu, 4 Apr 2013, valdis.kletni...@vt.edu wrote:
>
> > RFC4989 TCP Extended Statistics MIB. M. Mathis, J. Heffner, R.
> > Raghunarayan. May 2007. (Format: TXT=153768 bytes) (Status: PROPOSED
> > STANDARD)
> >
> > Looks like a taker to me.  Also, see the work the Web10G group is doing for
> > Linux: http://www.web10g.org
>
> RFC 4989 doesn't seem to officially exist. Ah, it's 4898.

Bargh. How did I get a typo in there? :)

> Yes, RFC4898 seems to contain a lot of interesting information, question
> is how to destill this down to something the user can understand and that
> is of interest for a support engineer who might be trying to diagnose the
> customer problem.
>
> I agree web10g seems to be of interest as well. I'm going to read through
> their documents tomorrow.

I recently got the web10g folks and the Linux kernel and networking folks
talking to each other, it may get upstreamed in the reasonably near future.
I'll make sure somebody keeps this list informed


pgp912if8_sFU.pgp
Description: PGP signature

RE: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Dennis Burgess]

The MT speed test is a multi-connection test, think 20 streams or connections 
at once.Most web based tests are single stream.  Now you get into 802.11N 
speedtests where they are optimized for many connections MIMO operations, 
hence, a single connection don't show good results, where a MT test at 20 
streams would.  

Dennis Burgess, Mikrotik Certified Trainer Author of "Learn RouterOS- Second 
Edition" 
 Link Technologies, Inc -- Mikrotik & WISP Support Services 
   
 Office: 314-735-0270 Website: http://www.linktechs.net - Skype: linktechs  
   
 -- Create Wireless Coverage's with www.towercoverage.com - 900Mhz - LTE - 3G - 
3.65 - TV Whitespace  


-Original Message-
From: Lorell Hathcock [mailto:lor...@hathcock.org] 
Sent: Monday, April 1, 2013 7:19 PM
To: nanog@nanog.org
Cc: Nathan Hathcock
Subject: Speedtest Results speedtest.net vs Mikrotik bandwidth test

All:

 

I am having some speedtest results that are difficult to interpret.

 

I am a small WISP multi-homed with Cogent and Level 3 in Houston, TX.  I am 
running BGP with each with 100 Mbps+ on each link.

 

Some of my customers have begun complaining that they are not getting the 
proper speeds.  They are using speedtest.net and/or speakeasy.net to test the 
results.

 

My network is Mikrotik based and as such, I have access to Mikrotik's built-in 
bandwidth testing.

 

With a laptop on site, running against speedtest.net (which kicked me over to 
the Comcast speedtest server instance) I can only get 4 Mbps up and 1.5 Mbps 
down.  That is consistent on their desktops too.  We eliminated their routing 
equipment and other consumers of the bandwidth and tested and got similar 
results.

 

But when  we run the Mikrotik bandwidth tests (even to off-net Mikrotik devices 
in Hawaii and Mission, TX) we get 25+ Mbps synchronous.

 

We have run traceroutes to various traceroute servers and they go through 
Cogent and/or Level 3.  For the most part it does not seem to matter which path 
it takes, the bandwidth seems to be about the same going both routes.

 

When we run the laptop-based btest.exe against Mikrotik bandwidth test servers, 
the laptop got significantly better results (14 Mbps) , but not 25+ Mbps.

 

It is almost like there is a Java based problem with speedtest.net.

 

Thoughts?

 

Thanks,

 

Lorell Hathcock

 

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Mikael Abrahamsson]

On Thu, 4 Apr 2013, valdis.kletni...@vt.edu wrote:


RFC4989 TCP Extended Statistics MIB. M. Mathis, J. Heffner, R.
Raghunarayan. May 2007. (Format: TXT=153768 bytes) (Status: PROPOSED
STANDARD)

Looks like a taker to me.  Also, see the work the Web10G group is doing for
Linux: http://www.web10g.org


RFC 4989 doesn't seem to officially exist. Ah, it's 4898.

Yes, RFC4898 seems to contain a lot of interesting information, question 
is how to destill this down to something the user can understand and that 
is of interest for a support engineer who might be trying to diagnose the 
customer problem.


I agree web10g seems to be of interest as well. I'm going to read through 
their documents tomorrow.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Valdis . Kletnieks]

On Thu, 04 Apr 2013 06:18:34 +0200, Mikael Abrahamsson said:

> I have pitched the idea in the IETF to have TCP stacks themselves report
> IP performance indicators (aggregate) and that a standard for this to be
> standardised. No takers so far.

RFC4989 TCP Extended Statistics MIB. M. Mathis, J. Heffner, R.
 Raghunarayan. May 2007. (Format: TXT=153768 bytes) (Status: PROPOSED
 STANDARD)

Looks like a taker to me.  Also, see the work the Web10G group is doing for
Linux: http://www.web10g.org


pgpdTw7aMkzcF.pgp
Description: PGP signature

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Steve Haavik]

It'd be nice to know if NDT was not accurate as well.  Anyone tested it?


We've been using it for a few years. On my laptop that runs linux I get 
fairly consistent results (around 935Mb/s up and down right now) over a 
1Gig routed link (a couple routers and a firewall in between.) On the Windows 
boxes I usually see a 100 to 200 Mb/s drop on the upload side. The last 
time I checked, you can compile a commandline version of the client. I 
seem to remember the commandline client not taking quite as bad a hit on 
the tests compared to running it on linux, but it's been a while since I 
tried it.


For us it's been way more accurate than the various speedtest servers our 
customers insist on trying. A while back I switched from compiling my own 
kernel and NDT to using perfSONAR-PS (http://psps.perfsonar.net/). I like 
that they've got live-cd and net-install versions. If nothing else it's 
useful for pointing out the difference between a local network issue and 
Internet Suckage.

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Jason Hellenthal]

When is speed ever ensured past someone else's edge/border ?

You may pass through your upstream that fast but once you are out in the open 
range you are free game to all the lions, tigers & bears..,

There is always going to be something eating you. Best off letting it be the 
Spanish queasiness from the night before than the results from speedtest.net

 

-- 

 Jason Hellenthal
 JJH448-ARIN
 - (2^(N-1))


On Apr 4, 2013, at 4:14, Mike  wrote:

> On 04/03/2013 02:48 PM, valdis.kletni...@vt.edu wrote:
>> On Wed, 03 Apr 2013 14:07:48 -0700, Mike said:
>> 
>>> These speedtests are pure unscientific bs and I'd love to see them
>>> called out on the carpet for it.
>> 
>> As far as I know, it's possible for the end-to-end reported values to be
>> lower than your immediate upstream due to issues further upstream.
>> 
>> But if it reports 20MBbits/sec down and 5MBits/sec up, then the link is
>> able to go *at least* that fast.
>> 
>> (If anybody's got evidence of it reporting more than the link is technically
>> capable of, feel free to correct me...)
> 
> 
> 
> Yeah, I do... I've had T1 lines reported at 4.7mbps down and 2.8mbps up.
> 
> These tests are hogwash.
> 
> Mike-
> 

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Mike]

On 04/03/2013 02:48 PM, valdis.kletni...@vt.edu wrote:

On Wed, 03 Apr 2013 14:07:48 -0700, Mike said:


These speedtests are pure unscientific bs and I'd love to see them
called out on the carpet for it.


As far as I know, it's possible for the end-to-end reported values to be
lower than your immediate upstream due to issues further upstream.

But if it reports 20MBbits/sec down and 5MBits/sec up, then the link is
able to go *at least* that fast.

(If anybody's got evidence of it reporting more than the link is technically
capable of, feel free to correct me...)




Yeah, I do... I've had T1 lines reported at 4.7mbps down and 2.8mbps up.

These tests are hogwash.

Mike-

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Andy Warner]

The only reliable way to really test performance is to saturate the
pipe (Iperf) and have a sufficiently well provisioned target. NDT does
a good job using short non-saturation tests, but it is susceptible to
slow start and other challenges. In general, NDT results will be more
conservative than best case, whereas a lot of other tests are very
optimistic best cases.

FWIW, the actively maintained code has moved to:
https://code.google.com/p/ndt/ and 3.6.4 is a bit more stable and
flexible on some platforms than 3.6.5.

You can either standup your own test server or point at the public
sites run by a few universities and MeasurementLab
(http://www.measurementlab.net/mlab_sites), which are not as widely
distributed as the Ookla / speedtest.net targets, but they tend to be
better provisioned and the result data for the Mlab targets is made
available to the public.

Once you've compiled the client, you can run again the closest host via:
$ web100clt -n ndt.iupui.donar.measurement-lab.org

(default install will put the test client in /usr/local/bin)

If anybody want to host Mlab collection servers, they're always
looking for more hosts (http://measurementlab.net/getinvolved).

--
Andy

On Thu, Apr 4, 2013 at 8:28 AM, Scott Weeks  wrote:
>
>
> --- n...@foobar.org wrote:
> From: Nick Hilliard 
>
>>> They may do some magic with bandwidth delay products.. If that was the case,
>>> they may have written it for a standard latency versus something that is
>>> unreasonable by interweb standards.
>
> I don't know how they calculate bandwidth, but I was surprised that their 
> system
> gave such wrong results under what were effectively lab conditions.
> --
>
>
> It'd be nice to know if NDT was not accurate as well.  Anyone tested it?
>
> scott
>

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Carl Rosevear]

I have paid the ransom.  Actually we pay it on a recurring basis even.  ;)
As for what it peaks at, good question.  The infrastructure we run it on is
going to be the problem at some point, although currently has not proven to
be a limiting factor to the best of my knowledge.  Our customers see valid
results...  I mean obviously it's not showing their link speed, it is
showing the characteristics of their connectivity to our speed test server.
 We use a couple of threads on the download test and if I take results,
divide by number of threads, look at the connection characteristics and do
the math to estimate throughput, there is at least usable parity there.
 But it's really useful for our support team when a customer is complaining
about some kind of bandwidth/latency issue into our cloud.  We have some
people in far places with 300+ms latency and 30+ms jitter, etc, trying to
use interactive sessions.  Oh and to be more correct, we actually have the
whole Ookla Line Quality package.  Very useful for us.  Also, customers
seem to love the whole flash animation thing.  Its what web users expect
these days...  it's really been a great experience for everyone... no
complaints on our end, aside from price, but I am always complaining about
that.  For those trying to just jam bits through a pipe to see if their
last mile is performing, slightly less useful unless there is one at their
ISP, but that is not our use case.

-Carl



On Wed, Apr 3, 2013 at 8:02 PM, Warren Bailey <
wbai...@satelliteintelligencegroup.com> wrote:

> I guess the Speedtest servers near metro areas do probably get pretty beat
> up. Has anyone paid the Ookla ransom for their own public server? I'd be
> really curious to see what they peak at.
>
>
> Sent from my T-Mobile 4G LTE Device
>

-- 
*Carl Rosevear*
Manager of Operations
*Skytap, Inc. | The Intuitive Enterprise Cloud*

crosev...@skytap.com | O: 206-588-8899 | F: 206-624-2214

Follow us: Blog <http://blog.skytap.com/> |
Twitter<http://twitter.com/#!/skytap>
 | LinkedIn <http://www.linkedin.com/company/skytap>

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Mikael Abrahamsson]

On Wed, 3 Apr 2013, joel jaeggli wrote:

Telling people to get by with even less instrumentation then they have 
already doesn't win you any friends. The solution to bad instruments is 
better instruments not breaking flow meter off the well.


I have pitched the idea in the IETF to have TCP stacks themselves report 
IP performance indicators (aggregate) and that a standard for this to be 
standardised. No takers so far.


I hate test traffic, I want to know how the real traffic is doing instead. 
In my opinion, people are way too happy to inject a lot of "useless" test 
traffic.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[joel jaeggli]

On 4/3/13 3:20 PM, Warren Bailey wrote:

Try it with upwards of 900ms of variable latency.

on linux

tc qdisc add dev eth0 root netem delay 900ms 150msdistribution normal

and then you can slowly test the internet to your hearts content.


Sent from my T-Mobile 4G LTE Device



 Original message 
From: Nick Hilliard 
Date: 04/03/2013 3:04 PM (GMT-08:00)
To: valdis.kletni...@vt.edu
Cc: nanog@nanog.org
Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test


On 3 Apr 2013, at 22:48, valdis.kletni...@vt.edu wrote:

(If anybody's got evidence of it reporting more than the link is technically
capable of, feel free to correct me...)

I've seen speedtest.net give results significantly greater than the physical bw 
of the client's network link.

Nick

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[joel jaeggli]

On 4/3/13 6:25 PM, Warren Bailey wrote:

I'm shocked Ookla hasn't been eaten by some major ISP. Speed tests are the root 
of most complaints. Your link is congested (oversubed) and you then attempt to 
completely saturate your bandwidth to tell your provider what a suck job they 
are doing. I can't imagine wireless isps or those with limited  bandwidth 
haven't black holed those kind of performance tools. My world (satellite) is 
plagued by people who are speed testing very narrow band connections and 
expecting 15mbps down. They don't realize Speedtest is not an accurate 
representation of your connection as you cannot influence your bandwidth 
upstream. Ds3 from you to your 56k modem type of scenario comes to mind. It may 
*not* be your provider who is responsible for your issues (some people 
Speedtest just to call their provider to complain for service credits etc).

Telling people to get by with even less instrumentation then they have 
already doesn't win you any friends. The solution to bad instruments is 
better instruments not breaking flow meter off the well.

Sent from my T-Mobile 4G LTE Device



 Original message 
From: Seth Mattinen 
Date: 04/03/2013 6:13 PM (GMT-08:00)
To: nanog@nanog.org
Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test


On 4/3/13 2:52 PM, Paul Stewart wrote:

We host one of the gazillion speed test sites and for networks that are
close to us we find it "reasonably accurate" .. a good benchmark at least ..



The speedtest.net that's hosted on one of my directly connected transits
is consistently wrong, which is always fun. But the customers cling to
those results like it's the word of God.

~Seth

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Warren Bailey]

I guess the Speedtest servers near metro areas do probably get pretty beat up. 
Has anyone paid the Ookla ransom for their own public server? I'd be really 
curious to see what they peak at.


Sent from my T-Mobile 4G LTE Device



 Original message 
From: Seth Mattinen 
Date: 04/03/2013 6:36 PM (GMT-08:00)
To: nanog@nanog.org
Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test


On 4/3/13 6:25 PM, Warren Bailey wrote:
> I'm shocked Ookla hasn't been eaten by some major ISP. Speed tests are
> the root of most complaints. Your link is congested (oversubed) and you
> then attempt to completely saturate your bandwidth to tell your provider
> what a suck job they are doing. I can't imagine wireless isps or those
> with limited  bandwidth haven't black holed those kind of performance
> tools. My world (satellite) is plagued by people who are speed testing
> very narrow band connections and expecting 15mbps down. They don't
> realize Speedtest is not an accurate representation of your connection
> as you cannot influence your bandwidth upstream. Ds3 from you to your
> 56k modem type of scenario comes to mind. It may *not* be your provider
> who is responsible for your issues (some people Speedtest just to call
> their provider to complain for service credits etc).
>

In my case I know the gig connection between me and that transit is
nowhere near saturated and works OK, so I have to assume the server
they're hosting speedtest.net on is either constantly hosed or uses a
10Base-T interface, possibly token ring.

~Seth

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Seth Mattinen]

On 4/3/13 6:25 PM, Warren Bailey wrote:
> I'm shocked Ookla hasn't been eaten by some major ISP. Speed tests are
> the root of most complaints. Your link is congested (oversubed) and you
> then attempt to completely saturate your bandwidth to tell your provider
> what a suck job they are doing. I can't imagine wireless isps or those
> with limited  bandwidth haven't black holed those kind of performance
> tools. My world (satellite) is plagued by people who are speed testing
> very narrow band connections and expecting 15mbps down. They don't
> realize Speedtest is not an accurate representation of your connection
> as you cannot influence your bandwidth upstream. Ds3 from you to your
> 56k modem type of scenario comes to mind. It may *not* be your provider
> who is responsible for your issues (some people Speedtest just to call
> their provider to complain for service credits etc). 
> 

In my case I know the gig connection between me and that transit is
nowhere near saturated and works OK, so I have to assume the server
they're hosting speedtest.net on is either constantly hosed or uses a
10Base-T interface, possibly token ring.

~Seth

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Warren Bailey]

I'm shocked Ookla hasn't been eaten by some major ISP. Speed tests are the root 
of most complaints. Your link is congested (oversubed) and you then attempt to 
completely saturate your bandwidth to tell your provider what a suck job they 
are doing. I can't imagine wireless isps or those with limited  bandwidth 
haven't black holed those kind of performance tools. My world (satellite) is 
plagued by people who are speed testing very narrow band connections and 
expecting 15mbps down. They don't realize Speedtest is not an accurate 
representation of your connection as you cannot influence your bandwidth 
upstream. Ds3 from you to your 56k modem type of scenario comes to mind. It may 
*not* be your provider who is responsible for your issues (some people 
Speedtest just to call their provider to complain for service credits etc).


Sent from my T-Mobile 4G LTE Device



 Original message 
From: Seth Mattinen 
Date: 04/03/2013 6:13 PM (GMT-08:00)
To: nanog@nanog.org
Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test


On 4/3/13 2:52 PM, Paul Stewart wrote:
> We host one of the gazillion speed test sites and for networks that are
> close to us we find it "reasonably accurate" .. a good benchmark at least ..
>


The speedtest.net that's hosted on one of my directly connected transits
is consistently wrong, which is always fun. But the customers cling to
those results like it's the word of God.

~Seth

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Owen DeLong]

(a) may be valid.
(b) is fishy

(a) may be valid because it may be that your ISP has a better set of peering
relationships towards your VPN server and your company's ISP has better
peering relationships towards the Speedtest server than your ISP has
towards the Speedtest server.

I'm not saying that IS the case, but I have seen instances where such results
were, in fact, perfectly legitimate for such reasons.

Owen

On Apr 3, 2013, at 17:20 , Chris Hindy  wrote:

> I can run two speedtest.net session side by side on my home network on one
> laptop, and over VPN to my employer's Long Island locale on a second,
> pointed at the same speedtest server, over the same wifi and ADSL and have
> the VPN connection report speeds that are (a) 50% better on VPN than not;
> and, (b) exceed my ADSL's hard cap by 10+ mbps.  That smells a bit fishy
> to me, all in all.
> 
> -c
> 
> On 03-04-2013 18:02 , "Nick Hilliard"  wrote:
> 
>> On 3 Apr 2013, at 22:48, valdis.kletni...@vt.edu wrote:
>>> (If anybody's got evidence of it reporting more than the link is
>>> technically
>>> capable of, feel free to correct me...)
>> 
>> I've seen speedtest.net give results significantly greater than the
>> physical bw of the client's network link.
>> 
>> Nick
>> 
>> 
> 

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Seth Mattinen]

On 4/3/13 2:52 PM, Paul Stewart wrote:
> We host one of the gazillion speed test sites and for networks that are
> close to us we find it "reasonably accurate" .. a good benchmark at least ..
> 


The speedtest.net that's hosted on one of my directly connected transits
is consistently wrong, which is always fun. But the customers cling to
those results like it's the word of God.

~Seth

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Scott Weeks]

--- n...@foobar.org wrote:
From: Nick Hilliard 

>> They may do some magic with bandwidth delay products.. If that was the case, 
>> they may have written it for a standard latency versus something that is 
>> unreasonable by interweb standards. 

I don't know how they calculate bandwidth, but I was surprised that their 
system 
gave such wrong results under what were effectively lab conditions. 
--


It'd be nice to know if NDT was not accurate as well.  Anyone tested it?

scott

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Chris Hindy]

I can run two speedtest.net session side by side on my home network on one
laptop, and over VPN to my employer's Long Island locale on a second,
pointed at the same speedtest server, over the same wifi and ADSL and have
the VPN connection report speeds that are (a) 50% better on VPN than not;
and, (b) exceed my ADSL's hard cap by 10+ mbps.  That smells a bit fishy
to me, all in all.

-c

On 03-04-2013 18:02 , "Nick Hilliard"  wrote:

>On 3 Apr 2013, at 22:48, valdis.kletni...@vt.edu wrote:
>> (If anybody's got evidence of it reporting more than the link is
>>technically
>> capable of, feel free to correct me...)
>
>I've seen speedtest.net give results significantly greater than the
>physical bw of the client's network link.
>
>Nick
>
>

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Nick Hilliard]

On 3 Apr 2013, at 23:41, Warren Bailey  
wrote:
> They may do some magic with bandwidth delay products.. If that was the case, 
> they may have written it for a standard latency versus something that is 
> unreasonable by interweb standards. 

I don't know how they calculate bandwidth, but I was surprised that their 
system gave such wrong results under what were effectively lab conditions. 

Nick

> 
> 
> Sent from my T-Mobile 4G LTE Device
> 
> 
> 
>  Original message 
> From: Nick Hilliard  
> Date: 04/03/2013 3:35 PM (GMT-08:00) 
> To: Warren Bailey  
> Cc: valdis.kletni...@vt.edu,nanog@nanog.org 
> Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test 
> 
> 
> On 3 Apr 2013, at 23:20, Warren Bailey 
>  wrote:
>> Try it with upwards of 900ms of variable latency. 
> 
> The last crazy result I got was 146mbit/s on a hardwired 100 mbit link  and 
> 1-2ms latency to the speedtest.net server I was using at the time (same data 
> centre).  Testing this sort of thing with high latency and jitter is 
> understandably hard, but I didn't see a good reason at the time why it should 
> have been so badly out with good underlying network characteristics.
> 
> Nick
> 
> 
> 
>> 
>> 
>> Sent from my T-Mobile 4G LTE Device
>> 
>> 
>> 
>>  Original message 
>> From: Nick Hilliard  
>> Date: 04/03/2013 3:04 PM (GMT-08:00) 
>> To: valdis.kletni...@vt.edu 
>> Cc: nanog@nanog.org 
>> Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test 
>> 
>> 
>> On 3 Apr 2013, at 22:48, valdis.kletni...@vt.edu wrote:
>> > (If anybody's got evidence of it reporting more than the link is 
>> > technically
>> > capable of, feel free to correct me...)
>> 
>> I've seen speedtest.net give results significantly greater than the physical 
>> bw of the client's network link.
>> 
>> Nick

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Warren Bailey]

They may do some magic with bandwidth delay products.. If that was the case, 
they may have written it for a standard latency versus something that is 
unreasonable by interweb standards.


Sent from my T-Mobile 4G LTE Device



 Original message 
From: Nick Hilliard 
Date: 04/03/2013 3:35 PM (GMT-08:00)
To: Warren Bailey 
Cc: valdis.kletni...@vt.edu,nanog@nanog.org
Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test


On 3 Apr 2013, at 23:20, Warren Bailey 
mailto:wbai...@satelliteintelligencegroup.com>>
 wrote:
Try it with upwards of 900ms of variable latency.

The last crazy result I got was 146mbit/s on a hardwired 100 mbit link  and 
1-2ms latency to the speedtest.net<http://speedtest.net> server I was using at 
the time (same data centre).  Testing this sort of thing with high latency and 
jitter is understandably hard, but I didn't see a good reason at the time why 
it should have been so badly out with good underlying network characteristics.

Nick





Sent from my T-Mobile 4G LTE Device



 Original message 
From: Nick Hilliard mailto:n...@foobar.org>>
Date: 04/03/2013 3:04 PM (GMT-08:00)
To: valdis.kletni...@vt.edu<mailto:valdis.kletni...@vt.edu>
Cc: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: Speedtest Results speedtest.net<http://speedtest.net> vs Mikrotik 
bandwidth test


On 3 Apr 2013, at 22:48, 
valdis.kletni...@vt.edu<mailto:valdis.kletni...@vt.edu> wrote:
> (If anybody's got evidence of it reporting more than the link is technically
> capable of, feel free to correct me...)

I've seen speedtest.net<http://speedtest.net> give results significantly 
greater than the physical bw of the client's network link.

Nick

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Nick Hilliard]

On 3 Apr 2013, at 23:20, Warren Bailey  
wrote:
> Try it with upwards of 900ms of variable latency. 

The last crazy result I got was 146mbit/s on a hardwired 100 mbit link  and 
1-2ms latency to the speedtest.net server I was using at the time (same data 
centre).  Testing this sort of thing with high latency and jitter is 
understandably hard, but I didn't see a good reason at the time why it should 
have been so badly out with good underlying network characteristics.

Nick



> 
> 
> Sent from my T-Mobile 4G LTE Device
> 
> 
> 
>  Original message 
> From: Nick Hilliard  
> Date: 04/03/2013 3:04 PM (GMT-08:00) 
> To: valdis.kletni...@vt.edu 
> Cc: nanog@nanog.org 
> Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test 
> 
> 
> On 3 Apr 2013, at 22:48, valdis.kletni...@vt.edu wrote:
> > (If anybody's got evidence of it reporting more than the link is technically
> > capable of, feel free to correct me...)
> 
> I've seen speedtest.net give results significantly greater than the physical 
> bw of the client's network link.
> 
> Nick
> 
> 
> 

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Warren Bailey]

Try it with upwards of 900ms of variable latency.


Sent from my T-Mobile 4G LTE Device



 Original message 
From: Nick Hilliard 
Date: 04/03/2013 3:04 PM (GMT-08:00)
To: valdis.kletni...@vt.edu
Cc: nanog@nanog.org
Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test


On 3 Apr 2013, at 22:48, valdis.kletni...@vt.edu wrote:
> (If anybody's got evidence of it reporting more than the link is technically
> capable of, feel free to correct me...)

I've seen speedtest.net give results significantly greater than the physical bw 
of the client's network link.

Nick

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Nick Hilliard]

On 3 Apr 2013, at 22:48, valdis.kletni...@vt.edu wrote:
> (If anybody's got evidence of it reporting more than the link is technically
> capable of, feel free to correct me...)

I've seen speedtest.net give results significantly greater than the physical bw 
of the client's network link.

Nick

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Ben Aitchison]

On Wed, Apr 03, 2013 at 05:48:00PM -0400, valdis.kletni...@vt.edu wrote:
> On Wed, 03 Apr 2013 14:07:48 -0700, Mike said:
> 
> > These speedtests are pure unscientific bs and I'd love to see them
> > called out on the carpet for it.
> 
> As far as I know, it's possible for the end-to-end reported values to be
> lower than your immediate upstream due to issues further upstream.
> 
> But if it reports 20MBbits/sec down and 5MBits/sec up, then the link is
> able to go *at least* that fast.
> 
> (If anybody's got evidence of it reporting more than the link is technically
> capable of, feel free to correct me...)

I've had speedtest.net report above ADSL sync rate on ADSL connection.  

Also from my testing, speedtest.net usually under-represents upload speed on 
fast
upload connections.  And for some reason ping shows higher in chrome than in 
internet
explorer.

It also tends to underrepresent far away connections by using too small file 
sizes. If
you use curl on the speedtest random.jpg files and grab the 4000x4000.jpg it'll 
give a
more representive test of download speed.

Ben.

RE: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Paul Stewart]

We host one of the gazillion speed test sites and for networks that are
close to us we find it "reasonably accurate" .. a good benchmark at least ..


Even our installers in the field use it as a "reference point"  YMMV
obviously

Paul


-Original Message-
From: valdis.kletni...@vt.edu [mailto:valdis.kletni...@vt.edu] 
Sent: April-03-13 5:48 PM
To: nanog@nanog.org
Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

On Wed, 03 Apr 2013 14:07:48 -0700, Mike said:

> These speedtests are pure unscientific bs and I'd love to see them 
> called out on the carpet for it.

As far as I know, it's possible for the end-to-end reported values to be
lower than your immediate upstream due to issues further upstream.

But if it reports 20MBbits/sec down and 5MBits/sec up, then the link is able
to go *at least* that fast.

(If anybody's got evidence of it reporting more than the link is technically
capable of, feel free to correct me...)

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Valdis . Kletnieks]

On Wed, 03 Apr 2013 14:07:48 -0700, Mike said:

> These speedtests are pure unscientific bs and I'd love to see them
> called out on the carpet for it.

As far as I know, it's possible for the end-to-end reported values to be
lower than your immediate upstream due to issues further upstream.

But if it reports 20MBbits/sec down and 5MBits/sec up, then the link is
able to go *at least* that fast.

(If anybody's got evidence of it reporting more than the link is technically
capable of, feel free to correct me...)


pgpwYdShrXzg6.pgp
Description: PGP signature

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Mike]

On 04/02/2013 10:13 PM, Seth Mattinen wrote:

On 4/2/13 2:24 PM, Carlos Alcantar wrote:

You might want to consider putting up a speedtest server internal to your
network.  I know there is a fee but well worth it I believe.  You will
still need to take the results with a grain a salt but you will have the
best results as well.



The speedtest.net mini version is free. Same test methodology and brand
recognition for the customers to be satisfied. Paid version if you need
branding or whatever.

~Seth





These speedtests are pure unscientific bs and I'd love to see them 
called out on the carpet for it.


Mike-

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Seth Mattinen]

On 4/2/13 2:24 PM, Carlos Alcantar wrote:
> You might want to consider putting up a speedtest server internal to your
> network.  I know there is a fee but well worth it I believe.  You will
> still need to take the results with a grain a salt but you will have the
> best results as well.
> 

The speedtest.net mini version is free. Same test methodology and brand
recognition for the customers to be satisfied. Paid version if you need
branding or whatever.

~Seth

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Alex Pressé]

The speedtest.net site has a free mini edition
(http://www.speedtest.net/mini.php) you can download and extract to
some http available path (asp, php, jsp all supported). It's a flash
applet, easy to wrap into your own page. Transfers one of ten large
JPG files of random noise (largest is 31MB). IIRC, it somehow does a
pretest to select a file that should take > 10 seconds.

If you're connected at >100Mbit to the hosting server then the results
are rather bogus (not enough time in flight to get any meaningful
averages).

Demos (found via google): https://test.kems.net/
http://speedtest.qualitynet.net/ http://speedtest.fsr.com/

Pros: not a java applet
Cons: adobe flash applet

On Tue, Apr 2, 2013 at 4:37 PM, Scott Weeks  wrote:
>
>
> 
> You might want to consider putting up a speedtest server internal to your
> network.  I know there is a fee but well worth it I believe.  You will
> 
>
>
> I would consider NDT as well: www.internet2.edu/performance/ndt
>
> Last I checked, about 3 years ago, speedtest sent only latin text in
> large packets.  NDT tests much more.  The customers just use a web
> browser and the only caveat is they need to have Java working.
>
> Here's one to get a feeling of what your customers will see:
> http://ndt.anl.gov:7123
>
> scott
>



-- 
Alex Presse
"How much net work could a network work if a network could net work?"

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Scott Weeks]

You might want to consider putting up a speedtest server internal to your
network.  I know there is a fee but well worth it I believe.  You will



I would consider NDT as well: www.internet2.edu/performance/ndt

Last I checked, about 3 years ago, speedtest sent only latin text in
large packets.  NDT tests much more.  The customers just use a web
browser and the only caveat is they need to have Java working.

Here's one to get a feeling of what your customers will see:
http://ndt.anl.gov:7123

scott

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Carlos Alcantar]

You might want to consider putting up a speedtest server internal to your
network.  I know there is a fee but well worth it I believe.  You will
still need to take the results with a grain a salt but you will have the
best results as well.

Carlos Alcantar
Race Communications / Race Team Member
1325 Howard Ave. #604, Burlingame, CA. 94010
Phone: +1 415 376 3314 / car...@race.com / http://www.race.com





-Original Message-
From: Lorell Hathcock 
Date: Tuesday, April 2, 2013 12:54 PM
To: "nanog@nanog.org" 
Subject: RE: Speedtest Results speedtest.net vs Mikrotik bandwidth test

Thanks for the many helpful suggestions I received offline.

One thing that I was able to deduce was that one of the radios along the
path had Ethernet auto negotiate turned on.  I turned it off and the TCP
speeds went way up.  It seems that UDP was not affected by this setting
while TCP was.

Thanks again!

Lorell



-Original Message-
From: Justin M. Streiner [mailto:strei...@cluebyfour.org]
Sent: Monday, April 01, 2013 7:27 PM
To: nanog@nanog.org
Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

On Mon, 1 Apr 2013, Lorell Hathcock wrote:

> I am having some speedtest results that are difficult to interpret.
>
> Some of my customers have begun complaining that they are not getting
> the proper speeds.  They are using speedtest.net and/or speakeasy.net
> to test the results.

Take the speedtest results with a grain of salt.  Once traffic leaves your
network, you no longer have (much) control over how packets flow across the
'rest of the internet'.

Did the customers report when the issue started?
Are they seeing other performance problems (latency/jitter/packet loss)?
Are you sure no internal links/routers are being saturated, even for brief
periods of time?

jms

RE: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Lorell Hathcock]

Thanks for the many helpful suggestions I received offline.

One thing that I was able to deduce was that one of the radios along the
path had Ethernet auto negotiate turned on.  I turned it off and the TCP
speeds went way up.  It seems that UDP was not affected by this setting
while TCP was.

Thanks again!

Lorell



-Original Message-
From: Justin M. Streiner [mailto:strei...@cluebyfour.org] 
Sent: Monday, April 01, 2013 7:27 PM
To: nanog@nanog.org
Subject: Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

On Mon, 1 Apr 2013, Lorell Hathcock wrote:

> I am having some speedtest results that are difficult to interpret.
>
> Some of my customers have begun complaining that they are not getting 
> the proper speeds.  They are using speedtest.net and/or speakeasy.net 
> to test the results.

Take the speedtest results with a grain of salt.  Once traffic leaves your
network, you no longer have (much) control over how packets flow across the
'rest of the internet'.

Did the customers report when the issue started?
Are they seeing other performance problems (latency/jitter/packet loss)?
Are you sure no internal links/routers are being saturated, even for brief
periods of time?

jms

Re: Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Justin M. Streiner]

On Mon, 1 Apr 2013, Lorell Hathcock wrote:


I am having some speedtest results that are difficult to interpret.

Some of my customers have begun complaining that they are not getting the
proper speeds.  They are using speedtest.net and/or speakeasy.net to test
the results.


Take the speedtest results with a grain of salt.  Once traffic leaves your 
network, you no longer have (much) control over how packets flow across 
the 'rest of the internet'.


Did the customers report when the issue started?
Are they seeing other performance problems (latency/jitter/packet loss)?
Are you sure no internal links/routers are being saturated, even for brief 
periods of time?


jms

Speedtest Results speedtest.net vs Mikrotik bandwidth test

[Lorell Hathcock]

All:

 

I am having some speedtest results that are difficult to interpret.

 

I am a small WISP multi-homed with Cogent and Level 3 in Houston, TX.  I am
running BGP with each with 100 Mbps+ on each link.

 

Some of my customers have begun complaining that they are not getting the
proper speeds.  They are using speedtest.net and/or speakeasy.net to test
the results.

 

My network is Mikrotik based and as such, I have access to Mikrotik's
built-in bandwidth testing.

 

With a laptop on site, running against speedtest.net (which kicked me over
to the Comcast speedtest server instance) I can only get 4 Mbps up and 1.5
Mbps down.  That is consistent on their desktops too.  We eliminated their
routing equipment and other consumers of the bandwidth and tested and got
similar results.

 

But when  we run the Mikrotik bandwidth tests (even to off-net Mikrotik
devices in Hawaii and Mission, TX) we get 25+ Mbps synchronous.

 

We have run traceroutes to various traceroute servers and they go through
Cogent and/or Level 3.  For the most part it does not seem to matter which
path it takes, the bandwidth seems to be about the same going both routes.

 

When we run the laptop-based btest.exe against Mikrotik bandwidth test
servers, the laptop got significantly better results (14 Mbps) , but not 25+
Mbps.

 

It is almost like there is a Java based problem with speedtest.net.

 

Thoughts?

 

Thanks,

 

Lorell Hathcock

 

Test: Please Delete Me

[Keith Medcalf]

If this gets delivered please delete me.  Somehow I seem to have MX requests 
for nanog.org failing ...


---
()  ascii ribbon campaign against html e-mail
/\  www.asciiribbon.org

RE: Wanted: Asia bandwidth test files

[David Wilde]

Hi Micah,

> From: micah anderson [mailto:mi...@riseup.net]
> 
> Thanks for the suggestion. Do you know what their bandwidth is? I can
> easily pull a .iso or similar from there to do some tests.
> 

There's some info at http://mirror.aarnet.edu.au/indexabout.html  - it's 
connected at 10Gbps.  

Naturally what throughput you're able to get will depend on how you arrive on 
the AARNet network - there is unfortunately less bandwidth into the west of 
Australia from Asia is than there is into the east coast from the United States.

David

Re: Wanted: Asia bandwidth test files

[Jason Leschnik]

I find the mirrors here are generally beefy

https://launchpad.net/ubuntu/+archivemirrors

Thanks.

On Tuesday, August 7, 2012, Aftab Siddiqui wrote:

> Hi Micah
>
> > Does anyone have any machines in Japan, S. Korea, or other asian
> locations with good bandwidth. where they can host a 100mbit file so I can
> attempt to download it to test this?
> >
>
> you may try downloading from stingray.cyber.net.pk
> It's in Karachi (Pakistan) with GigE limits. Use rsync.
>
> Regards,
>
> Aftab A. Siddiqui.
>
> --
> Regards,
>
> Aftab A. Siddiqui
>


-- 
Regards,
Jason Leschnik.

[m] 0432 35 4224
[w@] jason dot leschnik  ansto dot gov dot au
[U@] jml...@uow.edu.au

Re: Wanted: Asia bandwidth test files

[Aftab Siddiqui]

Hi Micah

> Does anyone have any machines in Japan, S. Korea, or other asian
locations with good bandwidth. where they can host a 100mbit file so I can
attempt to download it to test this?
>

you may try downloading from stingray.cyber.net.pk
It's in Karachi (Pakistan) with GigE limits. Use rsync.

Regards,

Aftab A. Siddiqui.

-- 
Regards,

Aftab A. Siddiqui

RE: Wanted: Asia bandwidth test files

[David Wilde]

Hi Micah,

You could try mirror.aarnet.edu.au, if Australia is sufficiently Asian for 
you...

David


-Original Message-
From: Micah Anderson [mailto:mi...@riseup.net] 
Sent: Friday, 3 August 2012 4:00
To: nanog@nanog.org
Subject: Wanted: Asia bandwidth test files


Hi,

I'm sitting on what is advertised as a 100mbit/sec connection in Cambodia. I 
have been trying to verify that, because I do not believe it is valid.

I did iperf tests from a number of network locations, and at one point I did 
get 71mbit/sec (most of the results were around 20-25mbit/sec or less). But I 
dont think 30 second iperf tests are particularly revealing when the bandwith 
rate might change drastically over the day. I considered doing a 3 day iperf 
test, but somehow this seems not how the tool was designed.

Someone suggested I find test files from various Asian locations to download 
via wget. I found a bunch of 100mb test files for various providers in N. 
America and Europe on webhostingtalk, which were interesting, but I never got 
more than around 5mbit/sec with them.

Does anyone have any machines in Japan, S. Korea, or other asian locations with 
good bandwidth. where they can host a 100mbit file so I can attempt to download 
it to test this?

Other suggestions for reliable tests would also be welcome! Please, dont 
suggest some flash garbage :)

thanks!
micah

-- 

Re: Wanted: Asia bandwidth test files

[PC]

If you can, I suggest finding other well connected hosts and using IPERF in
UDP mode for your testing.  Separating TCP long-fat pipe and slow start
issues from true packet delivery/loss rates at a given bitrate are
beneficial.  Use Linux as most iperf windows builds are based on cygwin and
have issues at higher bitrates.


On Mon, Aug 6, 2012 at 4:10 PM, Shishio Tsuchiya  wrote:

> Hi
> I think RING project NLNOG has potential to help your effort.
> https://ring.nlnog.net/
> At least they have location in tokyo.
>
> And I talked with Seichi Kawamura who is leader of JANOG about method of
> quality verification among the world wide.
> They are using host of Softlayer, amazon and OVH which could select the
> location.
>
> Job and Mucho ccing.
>
> Regards,
> -Shishio
>
>
> (2012/08/03 2:59), Micah Anderson wrote:
> >
> > Hi,
> >
> > I'm sitting on what is advertised as a 100mbit/sec connection in
> > Cambodia. I have been trying to verify that, because I do not believe it
> > is valid.
> >
> > I did iperf tests from a number of network locations, and at one point I
> > did get 71mbit/sec (most of the results were around 20-25mbit/sec or
> > less). But I dont think 30 second iperf tests are particularly revealing
> > when the bandwith rate might change drastically over the day. I
> > considered doing a 3 day iperf test, but somehow this seems not how the
> > tool was designed.
> >
> > Someone suggested I find test files from various Asian locations to
> > download via wget. I found a bunch of 100mb test files for various
> > providers in N. America and Europe on webhostingtalk, which were
> > interesting, but I never got more than around 5mbit/sec with them.
> >
> > Does anyone have any machines in Japan, S. Korea, or other asian
> > locations with good bandwidth. where they can host a 100mbit file so I
> > can attempt to download it to test this?
> >
> > Other suggestions for reliable tests would also be welcome! Please, dont
> > suggest some flash garbage :)
> >
> > thanks!
> > micah
> >
>
>
>
>

Re: Wanted: Asia bandwidth test files

[Shishio Tsuchiya]

Hi
I think RING project NLNOG has potential to help your effort.
https://ring.nlnog.net/
At least they have location in tokyo.

And I talked with Seichi Kawamura who is leader of JANOG about method of 
quality　verification among the world wide.
They are using host of Softlayer, amazon and OVH which could select the 
location.

Job and Mucho ccing.

Regards,
-Shishio


(2012/08/03 2:59), Micah Anderson wrote:
> 
> Hi,
> 
> I'm sitting on what is advertised as a 100mbit/sec connection in
> Cambodia. I have been trying to verify that, because I do not believe it
> is valid.
> 
> I did iperf tests from a number of network locations, and at one point I
> did get 71mbit/sec (most of the results were around 20-25mbit/sec or
> less). But I dont think 30 second iperf tests are particularly revealing
> when the bandwith rate might change drastically over the day. I
> considered doing a 3 day iperf test, but somehow this seems not how the
> tool was designed.
> 
> Someone suggested I find test files from various Asian locations to
> download via wget. I found a bunch of 100mb test files for various
> providers in N. America and Europe on webhostingtalk, which were
> interesting, but I never got more than around 5mbit/sec with them.
> 
> Does anyone have any machines in Japan, S. Korea, or other asian
> locations with good bandwidth. where they can host a 100mbit file so I
> can attempt to download it to test this?
> 
> Other suggestions for reliable tests would also be welcome! Please, dont
> suggest some flash garbage :)
> 
> thanks!
> micah
> 

Re: Wanted: Asia bandwidth test files

[Sadiq Saif]

Linode hosts one to test their Tokyo location -
http://speedtest.tokyo.linode.com/100MB-tokyo.bin

Source - http://www.linode.com/speedtest/

On Thu, Aug 2, 2012 at 1:59 PM, Micah Anderson  wrote:
>
> Hi,
>
> I'm sitting on what is advertised as a 100mbit/sec connection in
> Cambodia. I have been trying to verify that, because I do not believe it
> is valid.
>
> I did iperf tests from a number of network locations, and at one point I
> did get 71mbit/sec (most of the results were around 20-25mbit/sec or
> less). But I dont think 30 second iperf tests are particularly revealing
> when the bandwith rate might change drastically over the day. I
> considered doing a 3 day iperf test, but somehow this seems not how the
> tool was designed.
>
> Someone suggested I find test files from various Asian locations to
> download via wget. I found a bunch of 100mb test files for various
> providers in N. America and Europe on webhostingtalk, which were
> interesting, but I never got more than around 5mbit/sec with them.
>
> Does anyone have any machines in Japan, S. Korea, or other asian
> locations with good bandwidth. where they can host a 100mbit file so I
> can attempt to download it to test this?
>
> Other suggestions for reliable tests would also be welcome! Please, dont
> suggest some flash garbage :)
>
> thanks!
> micah
>
> --
>
>



-- 
Sadiq S
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

Wanted: Asia bandwidth test files

[Micah Anderson]

Hi,

I'm sitting on what is advertised as a 100mbit/sec connection in
Cambodia. I have been trying to verify that, because I do not believe it
is valid.

I did iperf tests from a number of network locations, and at one point I
did get 71mbit/sec (most of the results were around 20-25mbit/sec or
less). But I dont think 30 second iperf tests are particularly revealing
when the bandwith rate might change drastically over the day. I
considered doing a 3 day iperf test, but somehow this seems not how the
tool was designed.

Someone suggested I find test files from various Asian locations to
download via wget. I found a bunch of 100mb test files for various
providers in N. America and Europe on webhostingtalk, which were
interesting, but I never got more than around 5mbit/sec with them.

Does anyone have any machines in Japan, S. Korea, or other asian
locations with good bandwidth. where they can host a 100mbit file so I
can attempt to download it to test this?

Other suggestions for reliable tests would also be welcome! Please, dont
suggest some flash garbage :)

thanks!
micah

-- 

Re: Penetration Test Assistance

[dennis]

Tim,

In the past I've used high level diagrams to illustrate the overall network 
topology with individual tabs (drill down) per data center or POP.
The first step to assessing risk is to identify your assets.  I'd suggest 
performing a discovery of your network.  Keep in mind Pen tests are 
typically inconclusive of availability based threats DOS/DDOS (a very high 
risk today) and in fact specifically avoid tests which might cause 
degradation of service.   I'd suggest including volumetric network (tcp, 
udp), application floods (http get, post, etc. /dns query floods, etc.) and 
slow and low attacks.


Best of Luck,

Dennis

--
From: "Baklarz, Ron" 
Sent: Tuesday, June 05, 2012 12:41 PM
To: "Green, Timothy" 
Cc: 
Subject: RE: Penetration Test Assistance

Not discounting the need for network diagrams, there are also differing 
approaches to pen testing.  One alternative is a sort of black-box 
approach where the pen testers are given little or no advanced knowledge 
of the network. It is up to them to 'discover' what they can through open 
source means and commence their attacks from what they glean from their 
intelligence gathering.  This way they are realistically mimicking the 
hacker methodology.


Ron Baklarz C|CISO, CISSP, CISA, CISM, NSA-IAM/IEM
Chief Information Security Officer
Export Control Compliance Officer
National Passenger Railroad Corporation (AMTRAK)
10 G Street, NE  Office 6E606
Washington, DC 20002
bakl...@amtrak.com

-Original Message-
From: Green, Timothy [mailto:timothy.gr...@mantech.com]
Sent: Tuesday, June 05, 2012 10:53 AM
To: nanog@nanog.org
Subject: Penetration Test Assistance

Howdy all,

I'm a Security Manager of a large network, we are conducting a Pentest 
next month and the testers are demanding a complete network diagram of the 
entire network.  We don't have a "complete" network diagram that shows 
everything and everywhere we are.  At most we have a bunch of network 
diagrams that show what we have in various areas throughout the country. 
I've been asking the network engineers for over a month and they seem to 
be too lazy to put it together or they have no idea where everything is.


I've never been in this situation before.  Should I be honest to the 
testers and tell them here is what we have, we aren't sure if it's 
accurate;  find everything else?  How would they access those areas that 
we haven't identified?   How can I give them access to stuff that I didn't 
know existed?


What do you all do with your large networks?  One huge network diagram, a 
bunch of network diagrams separated by region, or both?  Any pentest 
horror stories?


Thanks,

Tim


This e-mail and any attachments are intended only for the use of the 
addressee(s) named herein and may contain proprietary information. If you 
are not the intended recipient of this e-mail or believe that you received 
this email in error, please take immediate action to notify the sender of 
the apparent error by reply e-mail; permanently delete the e-mail and any 
attachments from your computer; and do not disseminate, distribute, use, 
or copy this message and any attachments.

Re: Penetration Test Assistance

[Brett Watson]

On Jun 5, 2012, at 11:34 AM, Darden, Patrick S. wrote:

> 
> I'm with Barry--a network diagram showing everything from the pov of the pen 
> team should be part of the end report.

Maybe, maybe not. It all depends on the scope of the engagement. I've had 
customers ask for very specific pen test of a group of servers, or specific 
applications, wherein they provide all the topology, system, and network info, 
and just want me to look at one specific area.

Then of course others want a "black box" assessment, wherein they don't tell 
you anything, and expect you to discover whatever you can discover.

I'm personally very specific about scoping, and just give the customer exactly 
what they want but you've got to "interview" each other to figure all of that 
out. And totally agree with a previous poster, you should always get a redacted 
or sample report to see what kind of quality you can expect in the finished 
product.

-b

Re: Penetration Test Assistance

[Peter Kristolaitis]

On 12-06-05 03:48 PM, Brett Watson wrote:

On Jun 5, 2012, at 9:52 AM, Peter Kristolaitis wrote:


As far as horror stories... yeah.   My most memorable experience was a guy 
(with a CISSP designation, working for a company who came highly recommended) 
who:
- Spent a day trying to get his Backtrack CD to "work properly".  When I looked at it, it was 
just a color depth issue in X that took about 45 seconds from "why is this broken?" to "hey 
look, I fixed it!".
- Completely missed the honeypot machine I set up for the test.  I had logs 
from the machine showing that his scanning had hit the machine and had found 
several of the vulnerabilities, but the entire machine was absent from the 
report.
- Called us complaining that a certain behavior that "he'd never seen before" was 
happening when he tried to nmap our network.  The "certain behavior" was a firewall with 
some IPS functionality, along with him not knowing how to read nmap output.
- Completely messed up the report -- three times.  His report had the wrong 
ports&  vulnerabilities listed on the wrong IPs, so according to the report, we 
apparently had FreeBSD boxes running IOS or MS SQL...
- Stopped taking our calls when we asked why the honeypot machine was 
completely missing from the report.

In general, my experience with most "pen testers" is a severe disappointment, and isn't anything that 
couldn't be done in-house by taking the person in your department who has the most ingrained hacker/geek personality, 
giving them Nessus/Metasploit/nmap/etc, pizza and a big ass pot of coffee, and saying "Find stuff we don't know 
about. Go.".   There is the occasional pen tester who is absolutely phenomenal and does the job properly (i.e. the 
guys who actually write their own shellcode, etc), but the vast majority of "pen testers" just use automated 
tools and call it a day.  Like everything else in IT, security has been "commercialized" to the point where 
finding really good vendors/people is hard, because everyone and their mom has CEH, CISSP, and whatever other alphabet 
soup certifications you can imagine.

I agree with a lot of what you've said, but there are absolutely good security 
guys (pen tester, vulnerability assessors, etc) that use both open source and 
commercial automated tools, but still do a fantastic job because they 
understand the underlying technologies and protocols.

I used to do a lot of this in the past, had lots of automated tools, and only 
occasionally wrote some assessment modules or exploit code if necessary.

But again, a person in that position has to understand technology holistically 
(network, systems, software, protocols, etc).

-b


I completely agree.   I didn't mean to imply that using automated tools 
is a bad thing -- simply that running an automated tool to pump out a 
report with no further investigation isn't really a useful pen test.  
I've seen vendors whose "comprehensive penetration testing" was 
basically "We'll run Nessus against your network, write up an executive 
summary and email you the scan results.  Quite the bargain for $20K!"


Automated tools are definitely good to provide a first pass over a 
network, but even then multiple tools should be used, and an experienced 
eye should review the results for anomalies (whether that's a 
vulnerability that has a chance for false positives, discrepancies 
between the results of two or more automated tools, etc).   That kind of 
work, along with more aggressive pen tests and exploit development, need 
a "guru meditation"-level understanding of the involved technologies, 
protocols, etc, as you mentioned.


Like everything else IT, the specific tools used are more or less 
immaterial to an excellent practitioner -- a good programmer can hack 
code in any language, a good network engineer can use any brand of 
network equipment, etc -- because these types of people truly understand 
the systems they're dealing with, and use tools to accomplish a specific 
task which fits into part of the "big picture" they have in their 
heads.   Poor practitioners in a field use tools for the sake of using 
the tool ("I'm scanning a network with Nessus because that's what the 
certification course told me to do") without that deep level of 
understanding, and therefore don't provide any real value to the process.


- Pete

Re: Penetration Test Assistance

[Bacon Zombie]

You should have a look at the Pentest Standards page, it was created
by some very skilled Pen Testers how are trying to create a minimum
standard for all tests and reporting.

http://www.pentest-standard.org/index.php/Main_Page

Also you should just have to give them your external net-block
allocation that is in scope unless it is a more forced test and not a
general external test.

On 5 June 2012 20:48, Brett Watson  wrote:
>
> On Jun 5, 2012, at 9:52 AM, Peter Kristolaitis wrote:
>
>>
>> As far as horror stories... yeah.   My most memorable experience was a guy 
>> (with a CISSP designation, working for a company who came highly 
>> recommended) who:
>>    - Spent a day trying to get his Backtrack CD to "work properly".  When I 
>> looked at it, it was just a color depth issue in X that took about 45 
>> seconds from "why is this broken?" to "hey look, I fixed it!".
>>    - Completely missed the honeypot machine I set up for the test.  I had 
>> logs from the machine showing that his scanning had hit the machine and had 
>> found several of the vulnerabilities, but the entire machine was absent from 
>> the report.
>>    - Called us complaining that a certain behavior that "he'd never seen 
>> before" was happening when he tried to nmap our network.  The "certain 
>> behavior" was a firewall with some IPS functionality, along with him not 
>> knowing how to read nmap output.
>>    - Completely messed up the report -- three times.  His report had the 
>> wrong ports & vulnerabilities listed on the wrong IPs, so according to the 
>> report, we apparently had FreeBSD boxes running IOS or MS SQL...
>>    - Stopped taking our calls when we asked why the honeypot machine was 
>> completely missing from the report.
>>
>> In general, my experience with most "pen testers" is a severe 
>> disappointment, and isn't anything that couldn't be done in-house by taking 
>> the person in your department who has the most ingrained hacker/geek 
>> personality, giving them Nessus/Metasploit/nmap/etc, pizza and a big ass pot 
>> of coffee, and saying "Find stuff we don't know about. Go.".   There is the 
>> occasional pen tester who is absolutely phenomenal and does the job properly 
>> (i.e. the guys who actually write their own shellcode, etc), but the vast 
>> majority of "pen testers" just use automated tools and call it a day.  Like 
>> everything else in IT, security has been "commercialized" to the point where 
>> finding really good vendors/people is hard, because everyone and their mom 
>> has CEH, CISSP, and whatever other alphabet soup certifications you can 
>> imagine.
>
> I agree with a lot of what you've said, but there are absolutely good 
> security guys (pen tester, vulnerability assessors, etc) that use both open 
> source and commercial automated tools, but still do a fantastic job because 
> they understand the underlying technologies and protocols.
>
> I used to do a lot of this in the past, had lots of automated tools, and only 
> occasionally wrote some assessment modules or exploit code if necessary.
>
> But again, a person in that position has to understand technology 
> holistically (network, systems, software, protocols, etc).
>
> -b



-- 
BaconZombie

LOAD "*",8,1

Re: Penetration Test Assistance

[Brett Watson]

On Jun 5, 2012, at 9:52 AM, Peter Kristolaitis wrote:

> 
> As far as horror stories... yeah.   My most memorable experience was a guy 
> (with a CISSP designation, working for a company who came highly recommended) 
> who:
>- Spent a day trying to get his Backtrack CD to "work properly".  When I 
> looked at it, it was just a color depth issue in X that took about 45 seconds 
> from "why is this broken?" to "hey look, I fixed it!".
>- Completely missed the honeypot machine I set up for the test.  I had 
> logs from the machine showing that his scanning had hit the machine and had 
> found several of the vulnerabilities, but the entire machine was absent from 
> the report.
>- Called us complaining that a certain behavior that "he'd never seen 
> before" was happening when he tried to nmap our network.  The "certain 
> behavior" was a firewall with some IPS functionality, along with him not 
> knowing how to read nmap output.
>- Completely messed up the report -- three times.  His report had the 
> wrong ports & vulnerabilities listed on the wrong IPs, so according to the 
> report, we apparently had FreeBSD boxes running IOS or MS SQL...
>- Stopped taking our calls when we asked why the honeypot machine was 
> completely missing from the report.
> 
> In general, my experience with most "pen testers" is a severe disappointment, 
> and isn't anything that couldn't be done in-house by taking the person in 
> your department who has the most ingrained hacker/geek personality, giving 
> them Nessus/Metasploit/nmap/etc, pizza and a big ass pot of coffee, and 
> saying "Find stuff we don't know about. Go.".   There is the occasional pen 
> tester who is absolutely phenomenal and does the job properly (i.e. the guys 
> who actually write their own shellcode, etc), but the vast majority of "pen 
> testers" just use automated tools and call it a day.  Like everything else in 
> IT, security has been "commercialized" to the point where finding really good 
> vendors/people is hard, because everyone and their mom has CEH, CISSP, and 
> whatever other alphabet soup certifications you can imagine.

I agree with a lot of what you've said, but there are absolutely good security 
guys (pen tester, vulnerability assessors, etc) that use both open source and 
commercial automated tools, but still do a fantastic job because they 
understand the underlying technologies and protocols.

I used to do a lot of this in the past, had lots of automated tools, and only 
occasionally wrote some assessment modules or exploit code if necessary.

But again, a person in that position has to understand technology holistically 
(network, systems, software, protocols, etc).

-b

Re: Penetration Test Assistance

[Leo Bicknell]

The bit of information that's missing here is what are you trying
to pentest, and by extension how much do you want to pay your pentest
firm?

For some folks a pentest means starting with zero information and
trying to get IP packets passed a firewall or IDS's undetected.
Basically pentesting layer 3 infrastructure.

For other folks a pentest is purely an application level exercise,
you give the pentester an account on your customer portal for
instance, a full network diagram, and let them try things like SQL
injection or cross site scripting at the applications layer.

Your pentest firm can start with zero information and work all the
way up to an application level attack, but that's costly and time
consuming.  Providing them some information is a way to short circuit
the process.

If you (or appropriate company representative) haven't already
discussed the pros and cons with your pentest firm you're off on the
wrong foot.

-- 
   Leo Bicknell - bickn...@ufp.org - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/


pgpdRgOJhzM8j.pgp
Description: PGP signature

Re: Penetration Test Assistance

[Harry Hoffman]

There are lots of reasons why a pentester would want a network diagram.

The foremost being a point to which they can say, these are the networks 
that I was given as a point of reference to pentest.


This is often a CYA policy for when people start complaining about the 
scanning that is going to occur and potentially break their systems.


Cheers,
Harry

On 06/05/2012 02:34 PM, Darden, Patrick S. wrote:


I'm with Barry--a network diagram showing everything from the pov of the pen 
team should be part of the end report.

--p

-Original Message-
From: Barry Greene [mailto:bgre...@senki.org]

Hi Tim,

A _good_ pen test team would not need a network diagram. Their first round of 
penetration test would have them build their own network diagram from their 
analysis of your network.

Barry

RE: Penetration Test Assistance

[Darden, Patrick S.]

I'm with Barry--a network diagram showing everything from the pov of the pen 
team should be part of the end report.

--p

-Original Message-
From: Barry Greene [mailto:bgre...@senki.org]

Hi Tim,

A _good_ pen test team would not need a network diagram. Their first round of 
penetration test would have them build their own network diagram from their 
analysis of your network. 

Barry

RE: Penetration Test Assistance

[Darden, Patrick S.]

Seriously.

--p


-Original Message-
From: Aled Morris [mailto:al...@qix.co.uk]

I'd treat this as the first of their pen tests - a social engineering
attack to obtain secret information about the network, and refuse.

Aled

Re: Penetration Test Assistance

[Barry Greene]

Hi Tim,

A _good_ pen test team would not need a network diagram. Their first round of 
penetration test would have them build their own network diagram from their 
analysis of your network. 

Barry


On Jun 5, 2012, at 7:52 AM, Green, Timothy wrote:

> Howdy all,
> 
> I'm a Security Manager of a large network, we are conducting a Pentest next 
> month and the testers are demanding a complete network diagram of the entire 
> network.  We don't have a "complete" network diagram that shows everything 
> and everywhere we are.  At most we have a bunch of network diagrams that show 
> what we have in various areas throughout the country. I've been asking the 
> network engineers for over a month and they seem to be too lazy to put it 
> together or they have no idea where everything is.
> 
> I've never been in this situation before.  Should I be honest to the testers 
> and tell them here is what we have, we aren't sure if it's accurate;  find 
> everything else?  How would they access those areas that we haven't 
> identified?   How can I give them access to stuff that I didn't know existed?
> 
> What do you all do with your large networks?  One huge network diagram, a 
> bunch of network diagrams separated by region, or both?  Any pentest horror 
> stories?
> 
> Thanks,
> 
> Tim
> 
> 
> This e-mail and any attachments are intended only for the use of the 
> addressee(s) named herein and may contain proprietary information. If you are 
> not the intended recipient of this e-mail or believe that you received this 
> email in error, please take immediate action to notify the sender of the 
> apparent error by reply e-mail; permanently delete the e-mail and any 
> attachments from your computer; and do not disseminate, distribute, use, or 
> copy this message and any attachments.

Re: Penetration Test Assistance

[Jason 'XenoPhage' Frisvold]

On Jun 5, 2012, at 12:52 PM, Peter Kristolaitis  wrote:
> In general, my experience with most "pen testers" is a severe disappointment, 
> and isn't anything that couldn't be done in-house by taking the person in 
> your department who has the most ingrained hacker/geek personality, giving 
> them Nessus/Metasploit/nmap/etc, pizza and a big ass pot of coffee, and 
> saying "Find stuff we don't know about. Go.".   There is the occasional pen 
> tester who is absolutely phenomenal and does the job properly (i.e. the guys 
> who actually write their own shellcode, etc), but the vast majority of "pen 
> testers" just use automated tools and call it a day.  Like everything else in 
> IT, security has been "commercialized" to the point where finding really good 
> vendors/people is hard, because everyone and their mom has CEH, CISSP, and 
> whatever other alphabet soup certifications you can imagine.

There are definitely a number of incredible pen-testers out there.  But I agree 
with Peter… If you end up with a "report" that's nothing more than an executive 
statement pasted at the top of a Nessus report, then you've wasted your money.  
To be honest, I'd recommend getting a sample report from the company and quiz 
them on it before committing to a contract with them.

---
Jason 'XenoPhage' Frisvold
xenoph...@godshell.com
---
"Any sufficiently advanced magic is indistinguishable from technology."
- Niven's Inverse of Clarke's Third Law

Re: Penetration Test Assistance

[Aled Morris]

On 5 June 2012 15:52, Green, Timothy  wrote:

> Howdy all,
>
> I'm a Security Manager of a large network, we are conducting a Pentest
> next month and the testers are demanding a complete network diagram of the
> entire network.
>
>
I'd treat this as the first of their pen tests - a social engineering
attack to obtain secret information about the network, and refuse.

Aled

Re: Penetration Test Assistance

[William Herrin]

On 6/5/12, Green, Timothy  wrote:
> I'm a Security Manager of a large network, we are conducting a Pentest next
> month and the testers are demanding a complete network diagram of the entire
> network.  We don't have a "complete" network diagram that shows everything
> and everywhere we are.  At most we have a bunch of network diagrams that
> show what we have in various areas throughout the country. I've been asking
> the network engineers for over a month and they seem to be too lazy to put
> it together or they have no idea where everything is.
>
> I've never been in this situation before.  Should I be honest to the testers
> and tell them here is what we have, we aren't sure if it's accurate;  find
> everything else?

Tim,

Your system is what it is, including any defects in configuration
management. Provide the testers with what you have, give them contact
info for the engineers so they can ask questions and specify that you
expect strengths and weaknesses in configuration management which
impact system security to be reflected in their report.

Regards,
Bill Herrin



-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: Penetration Test Assistance

[Peter Kristolaitis]

On 12-06-05 11:32 AM, Andrew Latham wrote:

On Tue, Jun 5, 2012 at 10:52 AM, Green, Timothy
  wrote:

Howdy all,

I'm a Security Manager of a large network, we are conducting a Pentest next month and the 
testers are demanding a complete network diagram of the entire network.  We don't have a 
"complete" network diagram that shows everything and everywhere we are.  At 
most we have a bunch of network diagrams that show what we have in various areas 
throughout the country. I've been asking the network engineers for over a month and they 
seem to be too lazy to put it together or they have no idea where everything is.

I've never been in this situation before.  Should I be honest to the testers 
and tell them here is what we have, we aren't sure if it's accurate;  find 
everything else?  How would they access those areas that we haven't identified? 
  How can I give them access to stuff that I didn't know existed?

What do you all do with your large networks?  One huge network diagram, a bunch 
of network diagrams separated by region, or both?  Any pentest horror stories?

Thanks,

Tim

Any penetration test should only require your networks and masks.  As
far as a diagram it is of value to keep a staff member with the
singular task of documentation and auditing or an optional contract
basis.  Small things like typographical errors can cause great
confusion in emergency situations.  Take the time and do it right.  I
personally prefer the flexibility and ease of use that Mediawiki
offers but other free and pay solutions exist.



Yup, a list of subnets in use on your network is all I've ever needed to 
provide to pen testers in the past on the few occasions I've worked with 
them.  A good pen test should scan everything on your network anyways, 
with a reasonable chance of figuring out what everything is.


As far as horror stories... yeah.   My most memorable experience was a 
guy (with a CISSP designation, working for a company who came highly 
recommended) who:
- Spent a day trying to get his Backtrack CD to "work properly".  
When I looked at it, it was just a color depth issue in X that took 
about 45 seconds from "why is this broken?" to "hey look, I fixed it!".
- Completely missed the honeypot machine I set up for the test.  I 
had logs from the machine showing that his scanning had hit the machine 
and had found several of the vulnerabilities, but the entire machine was 
absent from the report.
- Called us complaining that a certain behavior that "he'd never 
seen before" was happening when he tried to nmap our network.  The 
"certain behavior" was a firewall with some IPS functionality, along 
with him not knowing how to read nmap output.
- Completely messed up the report -- three times.  His report had 
the wrong ports & vulnerabilities listed on the wrong IPs, so according 
to the report, we apparently had FreeBSD boxes running IOS or MS SQL...
- Stopped taking our calls when we asked why the honeypot machine 
was completely missing from the report.


In general, my experience with most "pen testers" is a severe 
disappointment, and isn't anything that couldn't be done in-house by 
taking the person in your department who has the most ingrained 
hacker/geek personality, giving them Nessus/Metasploit/nmap/etc, pizza 
and a big ass pot of coffee, and saying "Find stuff we don't know about. 
Go.".   There is the occasional pen tester who is absolutely phenomenal 
and does the job properly (i.e. the guys who actually write their own 
shellcode, etc), but the vast majority of "pen testers" just use 
automated tools and call it a day.  Like everything else in IT, security 
has been "commercialized" to the point where finding really good 
vendors/people is hard, because everyone and their mom has CEH, CISSP, 
and whatever other alphabet soup certifications you can imagine.

RE: Penetration Test Assistance

[Baklarz, Ron]

Not discounting the need for network diagrams, there are also differing 
approaches to pen testing.  One alternative is a sort of black-box approach 
where the pen testers are given little or no advanced knowledge of the network. 
It is up to them to 'discover' what they can through open source means and 
commence their attacks from what they glean from their intelligence gathering.  
This way they are realistically mimicking the hacker methodology. 

Ron Baklarz C|CISO, CISSP, CISA, CISM, NSA-IAM/IEM 
Chief Information Security Officer
Export Control Compliance Officer
National Passenger Railroad Corporation (AMTRAK)
10 G Street, NE  Office 6E606 
Washington, DC 20002   
bakl...@amtrak.com

-Original Message-
From: Green, Timothy [mailto:timothy.gr...@mantech.com] 
Sent: Tuesday, June 05, 2012 10:53 AM
To: nanog@nanog.org
Subject: Penetration Test Assistance

Howdy all,

I'm a Security Manager of a large network, we are conducting a Pentest next 
month and the testers are demanding a complete network diagram of the entire 
network.  We don't have a "complete" network diagram that shows everything and 
everywhere we are.  At most we have a bunch of network diagrams that show what 
we have in various areas throughout the country. I've been asking the network 
engineers for over a month and they seem to be too lazy to put it together or 
they have no idea where everything is.

I've never been in this situation before.  Should I be honest to the testers 
and tell them here is what we have, we aren't sure if it's accurate;  find 
everything else?  How would they access those areas that we haven't identified? 
  How can I give them access to stuff that I didn't know existed?

What do you all do with your large networks?  One huge network diagram, a bunch 
of network diagrams separated by region, or both?  Any pentest horror stories?

Thanks,

Tim


This e-mail and any attachments are intended only for the use of the 
addressee(s) named herein and may contain proprietary information. If you are 
not the intended recipient of this e-mail or believe that you received this 
email in error, please take immediate action to notify the sender of the 
apparent error by reply e-mail; permanently delete the e-mail and any 
attachments from your computer; and do not disseminate, distribute, use, or 
copy this message and any attachments.

Re: Penetration Test Assistance

[Quinn Kuzmich]

It's not much of a penetration test, imho, if the "attackers" have detailed
knowledge of your network and systems before the attack.  You should
determine what kind of a scenario you are trying to simulate, and how the
results will be used to improve security.  Is this a "black box" situation,
where you want to see what potential attackers can discover about your
systems without insider information?  Or will this be a step by step,
examine each part of the system and then step back to see what's going on
from a high level scenario?

If you're trying to both reduce vulnerabilities and your attack profile, I
would go for the black box approach and see what your pentesters can come
up with themselves.  Man is a resourceful creature, and you never know what
they could turn up.

Q

On Tue, Jun 5, 2012 at 8:52 AM, Green, Timothy wrote:

> Howdy all,
>
> I'm a Security Manager of a large network, we are conducting a Pentest
> next month and the testers are demanding a complete network diagram of the
> entire network.  We don't have a "complete" network diagram that shows
> everything and everywhere we are.  At most we have a bunch of network
> diagrams that show what we have in various areas throughout the country.
> I've been asking the network engineers for over a month and they seem to be
> too lazy to put it together or they have no idea where everything is.
>
> I've never been in this situation before.  Should I be honest to the
> testers and tell them here is what we have, we aren't sure if it's
> accurate;  find everything else?  How would they access those areas that we
> haven't identified?   How can I give them access to stuff that I didn't
> know existed?
>
> What do you all do with your large networks?  One huge network diagram, a
> bunch of network diagrams separated by region, or both?  Any pentest horror
> stories?
>
> Thanks,
>
> Tim
>
> 
> This e-mail and any attachments are intended only for the use of the
> addressee(s) named herein and may contain proprietary information. If you
> are not the intended recipient of this e-mail or believe that you received
> this email in error, please take immediate action to notify the sender of
> the apparent error by reply e-mail; permanently delete the e-mail and any
> attachments from your computer; and do not disseminate, distribute, use, or
> copy this message and any attachments.
>

Re: Penetration Test Assistance

[Joel jaeggli]

On 6/5/12 07:52 , Green, Timothy wrote:
> Howdy all,
> 
> I'm a Security Manager of a large network, we are conducting a
> Pentest next month and the testers are demanding a complete network
> diagram of the entire network.  We don't have a "complete" network
> diagram that shows everything and everywhere we are.  At most we have
> a bunch of network diagrams that show what we have in various areas
> throughout the country. I've been asking the network engineers for
> over a month and they seem to be too lazy to put it together or they
> have no idea where everything is.
> 
> I've never been in this situation before.  Should I be honest to the
> testers and tell them here is what we have, we aren't sure if it's
> accurate;  find everything else?  How would they access those areas
> that we haven't identified?   How can I give them access to stuff
> that I didn't know existed?
> 
> What do you all do with your large networks?  One huge network
> diagram, a bunch of network diagrams separated by region, or both?
> Any pentest horror stories?

Logical diagrams tend to elide the information consider unnecessary for
them to be suitably informative.

An ethernet switch with 560 network segments radiating out from it may
be accurate but not all that easy to parse or use.

Documentation needs to be sufficiently accurate and appropiate to the
tasks at hand, so it may be that you don't have what you need or perhaps
you do.

> Thanks,
> 
> Tim
> 
>  This e-mail and any attachments are
> intended only for the use of the addressee(s) named herein and may
> contain proprietary information. If you are not the intended
> recipient of this e-mail or believe that you received this email in
> error, please take immediate action to notify the sender of the
> apparent error by reply e-mail; permanently delete the e-mail and any
> attachments from your computer; and do not disseminate, distribute,
> use, or copy this message and any attachments.
> 

Re: Penetration Test Assistance

[jim deleskie]

A complete diagram makes their life easier, may make for a more
complete test, but they are working for you, so if you don't have it,
you don't have.  I'm not a big fan of having  a single diagram with
everything laid out anyway, but I'm from the old shcool.

-jim

On Tue, Jun 5, 2012 at 11:52 AM, Green, Timothy
 wrote:
> Howdy all,
>
> I'm a Security Manager of a large network, we are conducting a Pentest next 
> month and the testers are demanding a complete network diagram of the entire 
> network.  We don't have a "complete" network diagram that shows everything 
> and everywhere we are.  At most we have a bunch of network diagrams that show 
> what we have in various areas throughout the country. I've been asking the 
> network engineers for over a month and they seem to be too lazy to put it 
> together or they have no idea where everything is.
>
> I've never been in this situation before.  Should I be honest to the testers 
> and tell them here is what we have, we aren't sure if it's accurate;  find 
> everything else?  How would they access those areas that we haven't 
> identified?   How can I give them access to stuff that I didn't know existed?
>
> What do you all do with your large networks?  One huge network diagram, a 
> bunch of network diagrams separated by region, or both?  Any pentest horror 
> stories?
>
> Thanks,
>
> Tim
>
> 
> This e-mail and any attachments are intended only for the use of the 
> addressee(s) named herein and may contain proprietary information. If you are 
> not the intended recipient of this e-mail or believe that you received this 
> email in error, please take immediate action to notify the sender of the 
> apparent error by reply e-mail; permanently delete the e-mail and any 
> attachments from your computer; and do not disseminate, distribute, use, or 
> copy this message and any attachments.

Re: Penetration Test Assistance

[Justin M. Streiner]

On Tue, 5 Jun 2012, Green, Timothy wrote:

I'm a Security Manager of a large network, we are conducting a Pentest 
next month and the testers are demanding a complete network diagram of 
the entire network.  We don't have a "complete" network diagram that 
shows everything and everywhere we are.  At most we have a bunch of 
network diagrams that show what we have in various areas throughout the 
country. I've been asking the network engineers for over a month and 
they seem to be too lazy to put it together or they have no idea where 
everything is.


As someone who is charged with both engineering and maintaining the 
records and diagrams of a large network, I take exception to the word 
'lazy' ;)  Network engineers tend to be an over-worked lot, and their work 
is often interrupt-driven, so large blocks of time to work on a single 
task are often a rarity.


The issue is that if they haven't kept their diagrams up to date (many 
people don't, unfortunately), then getting them up to date turns into a 
much more labor-intensive job.  If they have kept the diagrams up to date 
and they're just not getting them to you, then take the issue up with 
their manager.


There might also be the question of how much information they are allowed 
to release to third parties, even if it is for a pentest.  This could mean 
that some information might need to be removed or redacted from the 
diagrams.  Again, the engineering manager/director/CIO/CTO might be able 
to provide clarification on this.


I've never been in this situation before.  Should I be honest to the 
testers and tell them here is what we have, we aren't sure if it's 
accurate;  find everything else?  How would they access those areas that 
we haven't identified?   How can I give them access to stuff that I 
didn't know existed?


From what I've seen, in-depth pentests are often done in coordination with 
other groups, such as engineering/ops.  In a large network, that's often 
done out of necessity,  if for no other reason than dealing with issues 
like the ones you've raised (logistics, communication, etc...).


What do you all do with your large networks?  One huge network diagram, 
a bunch of network diagrams separated by region, or both?  Any pentest 
horror stories?


I don't have any pentest horror stories, but sometimes large network 
diagrams have to be broken up into pieces, to maintain some degree of 
readability.  Large diagrams can get cluttered very quickly if you try to 
put every minute piece of detail on them.  I tend to treat the main 
diagram as a high-level view of the network, and then either break out 
sections that need more detail as a separate drawing, or as a link to our 
internal knowledge base that can go into very high detail, including 
pictures, access information, etc.


There is no right way to diagram every network.  It depends on what best 
suits your needs, and what established proceures are already in place.


jms

Re: Penetration Test Assistance

[Andrew Latham]

On Tue, Jun 5, 2012 at 10:52 AM, Green, Timothy
 wrote:
> Howdy all,
>
> I'm a Security Manager of a large network, we are conducting a Pentest next 
> month and the testers are demanding a complete network diagram of the entire 
> network.  We don't have a "complete" network diagram that shows everything 
> and everywhere we are.  At most we have a bunch of network diagrams that show 
> what we have in various areas throughout the country. I've been asking the 
> network engineers for over a month and they seem to be too lazy to put it 
> together or they have no idea where everything is.
>
> I've never been in this situation before.  Should I be honest to the testers 
> and tell them here is what we have, we aren't sure if it's accurate;  find 
> everything else?  How would they access those areas that we haven't 
> identified?   How can I give them access to stuff that I didn't know existed?
>
> What do you all do with your large networks?  One huge network diagram, a 
> bunch of network diagrams separated by region, or both?  Any pentest horror 
> stories?
>
> Thanks,
>
> Tim

Any penetration test should only require your networks and masks.  As
far as a diagram it is of value to keep a staff member with the
singular task of documentation and auditing or an optional contract
basis.  Small things like typographical errors can cause great
confusion in emergency situations.  Take the time and do it right.  I
personally prefer the flexibility and ease of use that Mediawiki
offers but other free and pay solutions exist.


-- 
~ Andrew "lathama" Latham lath...@gmail.com http://lathama.net ~

Penetration Test Assistance

[Green, Timothy]

Howdy all,

I'm a Security Manager of a large network, we are conducting a Pentest next 
month and the testers are demanding a complete network diagram of the entire 
network.  We don't have a "complete" network diagram that shows everything and 
everywhere we are.  At most we have a bunch of network diagrams that show what 
we have in various areas throughout the country. I've been asking the network 
engineers for over a month and they seem to be too lazy to put it together or 
they have no idea where everything is.

I've never been in this situation before.  Should I be honest to the testers 
and tell them here is what we have, we aren't sure if it's accurate;  find 
everything else?  How would they access those areas that we haven't identified? 
  How can I give them access to stuff that I didn't know existed?

What do you all do with your large networks?  One huge network diagram, a bunch 
of network diagrams separated by region, or both?  Any pentest horror stories?

Thanks,

Tim


This e-mail and any attachments are intended only for the use of the 
addressee(s) named herein and may contain proprietary information. If you are 
not the intended recipient of this e-mail or believe that you received this 
email in error, please take immediate action to notify the sender of the 
apparent error by reply e-mail; permanently delete the e-mail and any 
attachments from your computer; and do not disseminate, distribute, use, or 
copy this message and any attachments.

Re: test-ipv6.com / omgipv6day.com down

[Matthew Luckie]

> What's really needed is a service that looks up a given web page
> over IPv6 from behind a 1280 byte MTU link and reports if all the
> elements load or not.   It dumps a list of elements with success/fail.
>
> This would be useful to send the idiots that block ICMPv6 PTB yet
> send packets bigger than 1280 bytes out too.

http://www.wand.net.nz/scamper/

Works on MacOS X and FreeBSD.  It uses IPFW and rules 1-500 as
necessary.  Example below, showing a website sending > 1280 but
ignoring PTBs sent to it.

$ sudo scamper -F ipfw -I "tbit -t pmtud -u 'http://www.sapo.pt/'
2001:8a0:2104:ff:213:13:146:140"
tbit from 2001:470:d:4de:21f:3cff:fe20:bf4e to 2001:8a0:2104:ff:213:13:146:140
 server-mss 1460, result: pmtud-fail
 app: http, url: http://www.sapo.pt/
 [  0.048] TX SYN 64  seq = 0:0 
 [  0.254] RX SYN/ACK 64  seq = 0:1 
 [  0.255] TX 60  seq = 1:1 
 [  0.255] TX230  seq = 1:1(170)
 [  0.450] RX 60  seq = 1:171   
 [  0.469] RX   1460  seq = 1:171(1400) 
 [  0.469] TX PTB   1280  mtu = 1280
 [  0.470] RX   1460  seq = 1401:171(1400)  
 [  3.467] RX   1460  seq = 1:171(1400) 
 [  3.467] TX PTB   1280  mtu = 1280
 [  9.468] RX   1460  seq = 1:171(1400) 
 [  9.468] TX PTB   1280  mtu = 1280
 [ 21.471] RX   1460  seq = 1:171(1400) 
 [ 21.471] TX PTB   1280  mtu = 1280
 [ 31.933] RX RST 60  seq = 1:4294923802   

Re: test-ipv6.com / omgipv6day.com down

[Mark Andrews]

http://ipv6chicken.com/ tests the path to me.  It doesn't check the
path back to the sites I want to reach though it does provide a
independent third party if there is complainst that PTB's are not
being generated.  It would be useful if it reported the MTU that
was eventually used.  Most OS's have a hook to retrieve this.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: test-ipv6.com / omgipv6day.com down

[Owen DeLong]

My bad... It's .com not .net.

http://www.ipv6chicken.com

Owen

On Jun 4, 2012, at 5:14 PM, Jeroen Massar wrote:

> On 2012-06-04 16:58, Owen DeLong wrote:
>> http://ipv6chicken.net
> 
> $ dig -t any ipv6chicken.net
> 
> ; <<>> DiG 9.8.1-P1 <<>> -t any ipv6chicken.net
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 16935
> 
> The chicken cannot cross the road as the chicken does not exist.
> 
> Greets,
> Jeroen

Re: test-ipv6.com / omgipv6day.com down

[Bryan Irvine]

's/net/com'



On Mon, Jun 4, 2012 at 5:15 PM, Mark Andrews  wrote:
>
> In message , Owen DeLong 
> writes:
>> http://ipv6chicken.net
>>
>> Owen
>
> doesn't exist.
>
> ; <<>> DiG 9.9.1 <<>> ipv6chicken.net
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 5059
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4096
> ;; QUESTION SECTION:
> ;ipv6chicken.net.               IN      A
>
> ;; AUTHORITY SECTION:
> net.                    879     IN      SOA     a.gtld-servers.net. 
> nstld.verisign-grs.com. 1338855235 1800 900 604800 86400
>
> ;; Query time: 0 msec
> ;; SERVER: 127.0.0.1#53(127.0.0.1)
> ;; WHEN: Tue Jun  5 10:14:40 2012
> ;; MSG SIZE  rcvd: 117
>
>>
>> On Jun 4, 2012, at 4:54 PM, Mark Andrews wrote:
>>
>> >
>> > What's really needed is a service that looks up a given web page
>> > over IPv6 from behind a 1280 byte MTU link and reports if all the
>> > elements load or not.   It dumps a list of elements with success/fail.
>> >
>> > This would be useful to send the idiots that block ICMPv6 PTB yet
>> > send packets bigger than 1280 bytes out too.
>> >
>> > Mark
>> > --
>> > Mark Andrews, ISC
>> > 1 Seymour St., Dundas Valley, NSW 2117, Australia
>> > PHONE: +61 2 9871 4742                 INTERNET: ma...@isc.org
>>
> --
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742                 INTERNET: ma...@isc.org
>

Re: test-ipv6.com / omgipv6day.com down

[Mark Andrews]

In message , Owen DeLong 
writes:
> http://ipv6chicken.net
> 
> Owen

doesn't exist.

; <<>> DiG 9.9.1 <<>> ipv6chicken.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 5059
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ipv6chicken.net.   IN  A

;; AUTHORITY SECTION:
net.879 IN  SOA a.gtld-servers.net. 
nstld.verisign-grs.com. 1338855235 1800 900 604800 86400

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jun  5 10:14:40 2012
;; MSG SIZE  rcvd: 117

> 
> On Jun 4, 2012, at 4:54 PM, Mark Andrews wrote:
> 
> > 
> > What's really needed is a service that looks up a given web page
> > over IPv6 from behind a 1280 byte MTU link and reports if all the
> > elements load or not.   It dumps a list of elements with success/fail.
> > 
> > This would be useful to send the idiots that block ICMPv6 PTB yet
> > send packets bigger than 1280 bytes out too.
> > 
> > Mark
> > -- 
> > Mark Andrews, ISC
> > 1 Seymour St., Dundas Valley, NSW 2117, Australia
> > PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
> 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: test-ipv6.com / omgipv6day.com down

[Jeroen Massar]

On 2012-06-04 16:58, Owen DeLong wrote:
> http://ipv6chicken.net

$ dig -t any ipv6chicken.net

; <<>> DiG 9.8.1-P1 <<>> -t any ipv6chicken.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 16935

The chicken cannot cross the road as the chicken does not exist.

Greets,
 Jeroen

Re: test-ipv6.com / omgipv6day.com down

[Owen DeLong]

http://ipv6chicken.net

Owen

On Jun 4, 2012, at 4:54 PM, Mark Andrews wrote:

> 
> What's really needed is a service that looks up a given web page
> over IPv6 from behind a 1280 byte MTU link and reports if all the
> elements load or not.   It dumps a list of elements with success/fail.
> 
> This would be useful to send the idiots that block ICMPv6 PTB yet
> send packets bigger than 1280 bytes out too.
> 
> Mark
> -- 
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

RE: test-ipv6.com / omgipv6day.com down

[Frank Bulk]

Much of that can be found here: http://www.wand.net.nz/pmtud/

Frank

-Original Message-
From: Mark Andrews [mailto:ma...@isc.org] 
Sent: Monday, June 04, 2012 6:54 PM
To: Jeroen Massar
Cc: nanog@nanog.org
Subject: Re: test-ipv6.com / omgipv6day.com down


What's really needed is a service that looks up a given web page
over IPv6 from behind a 1280 byte MTU link and reports if all the
elements load or not.   It dumps a list of elements with success/fail.

This would be useful to send the idiots that block ICMPv6 PTB yet
send packets bigger than 1280 bytes out too.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: test-ipv6.com / omgipv6day.com down

[Mark Andrews]

What's really needed is a service that looks up a given web page
over IPv6 from behind a 1280 byte MTU link and reports if all the
elements load or not.   It dumps a list of elements with success/fail.

This would be useful to send the idiots that block ICMPv6 PTB yet
send packets bigger than 1280 bytes out too.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org

Re: test-ipv6.com / omgipv6day.com down

[Jeroen Massar]

On 2012-06-04 08:13, Jason Fesler wrote:
> 
> On Jun 4, 2012, at 7:09 AM, Jeroen Massar wrote:
> 
>> You got a bunch of mirrors for it right? Should not be to tricky to
>> get someone to let their act as the real thing for a bit.
> 
> I've got redirects up now to spread the load across VMs.   For the
> next couple of days, I don't expect a single VM to handle the load.

I am actually not expecting that much of the hype to come out, just like
last year it will easily be forgotten unless somebody is able to spin
that PR engine really really really hard.

> Thanks to all who've sent me a response; and thanks to Host Virtual
> and to Network Design GmbH, for taking the immediate load.
> 
> Once we're stable, and I get my *official* day job requirements met
> for World IPV6 Launch, I"ll come back to getting the original gear
> replaced.  I've got a couple hardware offers in (Alex, Mark, thank
> you), and this might just be the reason to flat out refresh the
> hardware if ixSystems has something suitable already built.

Awesome!

Greets,
 Jeroen

Re: test-ipv6.com / omgipv6day.com down

[Jason Fesler]

On Jun 4, 2012, at 7:09 AM, Jeroen Massar wrote:

> You got a bunch of mirrors for it right? Should not be to tricky to get 
> someone to let their act as the real thing for a bit.

I've got redirects up now to spread the load across VMs.   For the next couple 
of days, I don't expect a single VM to handle the load.

Thanks to all who've sent me a response; and thanks to Host Virtual and to 
Network Design GmbH, for taking the immediate load.

Once we're stable, and I get my *official* day job requirements met for World 
IPV6 Launch, I"ll come back to getting the original gear replaced.  I've got a 
couple hardware offers in (Alex, Mark, thank you), and this might just be the 
reason to flat out refresh the hardware if ixSystems has something suitable 
already built.


-jason

Re: test-ipv6.com / omgipv6day.com down

[Jeroen Massar]

On 4 Jun 2012, at 06:50, Jason Fesler  wrote:

> I know a lot of people are using / pointing to test-ipv6.com .  The hardware 
> picked a bad week to quit sniffing glue.

You got a bunch of mirrors for it right? Should not be to tricky to get someone 
to let their act as the real thing for a bit.

Greets,
 Jeroen

test-ipv6.com / omgipv6day.com down

[Jason Fesler]

I know a lot of people are using / pointing to test-ipv6.com .  The hardware 
picked a bad week to quit sniffing glue.

I"ll be working on trying to get it back up today, I need to source hardware.  
Also looking at borrowing a VM for short term.

(speaking only for @test-ipv6.com, not for $employer  - my personal mail 
address is down too).

Re: Speed Test Results

[Joel Maslak]

On Fri, Dec 23, 2011 at 10:13 AM, Livingood, Jason
 wrote:
> If you want to understand the issue in detail, check out the report from
> MIT this year, written by Steve Bauer and available at
> http://mitas.csail.mit.edu/papers/Bauer_Clark_Lehr_Broadband_Speed_Measurem
> ents.pdf.

They should have put a date on their paper, including when the
measurements were done.  It appears to me to have been done sometime
after or around June of 2010.

Re: Speed Test Results

[Sean Harlow]

Basically it's a CYA statement on the part of Ookla/speedtest.net, since their 
test sites are of varying quality.  The Radnor, OH test site sometimes can't 
even properly test a 10mbit SOHO broadband connection, where the Toledo site is 
consistently able to flood every available bit of capacity on my 50/5 home 
connection.

It's just another tool that needs to be used intelligently.  If I'm testing out 
a new ISP or a new speed level I've never had before, I wouldn't immediately 
complain if I didn't get the expected result on a public speed test site as it 
may be something outside of my ISP's control.  On the other hand if things 
start dragging on my home connection or anywhere else that I know I can expect 
a certain result speedtest.net is usually my first stop.
--
Sean Harlow
s...@seanharlow.info

On Dec 25, 2011, at 9:43 PM, Grant Ridder wrote:

> Even though the faq's say they are only good for residential usage, i have
> had no problems with it at school.  My college has 2x 100 Mb circuits from
> TW.  When i run speed tests (I use speedtest.net) with the campus empty, i
> can get around 95Mb up.  The bottleneck is the school's 100Mb switches.
> When the campus is filled (during the week) i can normally get close to 40
> Mb down on a test.
> 
> -Grant

Re: Speed Test Results

[Grant Ridder]

Even though the faq's say they are only good for residential usage, i have
had no problems with it at school.  My college has 2x 100 Mb circuits from
TW.  When i run speed tests (I use speedtest.net) with the campus empty, i
can get around 95Mb up.  The bottleneck is the school's 100Mb switches.
 When the campus is filled (during the week) i can normally get close to 40
Mb down on a test.

-Grant

On Sun, Dec 25, 2011 at 8:10 PM, Scott Berkman  wrote:

> The MIT article is good read, thanks for sharing that.
>
> One thing to watch out for is if the last mile provider is the one hosting
> the speedtest site, that's another variable removed from the equation.  In
> some cases that is a good thing, in others it's not, depending on what you
> are trying to measure.  It's also theoretically possible (and in my opinion
> not only likely but probably fairly common) for some large residential
> ISP's
> to not rate-limit these on-net test sites (either by design or as a side
> result of at what point in the network they apply the rate limiting),
> thereby showing much higher results than the end user could ever possibly
> see in a real world scenario.
>
> Also, when using some of the popular public Ookla/speedtest.net sites,
> their
> FAQ clearly states that the tests are not suitable for certain connection
> types like high speed services and non-residential services in general.
>  One
> good example is Speakeasy's site, which in my personal experience has been
> the one most commonly used by end users (especially those contacting us
> about "speed problems"):
>
> http://www.speakeasy.net/speedtest/issues.php
>
> "Our speed test is tuned to measure residential broadband services up to 20
> Mbps over HTTP. It takes a very customized installation to be able to
> accurately measure up to 100 Mbps over HTTP."
>
> -Scott
>
> -Original Message-
> From: Frank Bulk [mailto:frnk...@iname.com]
> Sent: Sunday, December 25, 2011 8:28 PM
> To: 'Michael Holstein'; jacob miller
> Cc: nanog@nanog.org
> Subject: RE: Speed Test Results
>
> We host an Ookla Speedtest server onsite and find it a very reliable means
> to identify throughput issues.  The source of any performance issues may or
> may not be ours, but if a customer says things are slow we can usually
> identify whether it's their PC or network (browsing is slow but speed test
> runs fine) or a local or regional network issue (speed test runs slow).
>
> If a customer gets less than 90% of the advertised throughput, we follow up
> on it.
>
> Frank
>
> -Original Message-
> From: Michael Holstein [mailto:michael.holst...@csuohio.edu]
> Sent: Friday, December 23, 2011 1:27 PM
> To: jacob miller
> Cc: nanog@nanog.org
> Subject: Re: Speed Test Results
>
>
> > Am having a debate on the results of speed tests sites.
> >
> > Am interested in knowing the thoughts of different individuals in regards
> to this.
> >
> >
>
> They are excellent tools for generating user complaints.
>
> (just like the "do traceroute and count the hops" advice from gamer mags
> of old).
>
> (my $0.02)
>
> Michael Holstein
> Cleveland State University
>
>
>
>
>
>
>

RE: Speed Test Results

[Scott Berkman]

The MIT article is good read, thanks for sharing that.

One thing to watch out for is if the last mile provider is the one hosting
the speedtest site, that's another variable removed from the equation.  In
some cases that is a good thing, in others it's not, depending on what you
are trying to measure.  It's also theoretically possible (and in my opinion
not only likely but probably fairly common) for some large residential ISP's
to not rate-limit these on-net test sites (either by design or as a side
result of at what point in the network they apply the rate limiting),
thereby showing much higher results than the end user could ever possibly
see in a real world scenario.

Also, when using some of the popular public Ookla/speedtest.net sites, their
FAQ clearly states that the tests are not suitable for certain connection
types like high speed services and non-residential services in general.  One
good example is Speakeasy's site, which in my personal experience has been
the one most commonly used by end users (especially those contacting us
about "speed problems"):

http://www.speakeasy.net/speedtest/issues.php

"Our speed test is tuned to measure residential broadband services up to 20
Mbps over HTTP. It takes a very customized installation to be able to
accurately measure up to 100 Mbps over HTTP."

-Scott

-Original Message-
From: Frank Bulk [mailto:frnk...@iname.com] 
Sent: Sunday, December 25, 2011 8:28 PM
To: 'Michael Holstein'; jacob miller
Cc: nanog@nanog.org
Subject: RE: Speed Test Results

We host an Ookla Speedtest server onsite and find it a very reliable means
to identify throughput issues.  The source of any performance issues may or
may not be ours, but if a customer says things are slow we can usually
identify whether it's their PC or network (browsing is slow but speed test
runs fine) or a local or regional network issue (speed test runs slow).

If a customer gets less than 90% of the advertised throughput, we follow up
on it.

Frank

-Original Message-
From: Michael Holstein [mailto:michael.holst...@csuohio.edu]
Sent: Friday, December 23, 2011 1:27 PM
To: jacob miller
Cc: nanog@nanog.org
Subject: Re: Speed Test Results


> Am having a debate on the results of speed tests sites.
>
> Am interested in knowing the thoughts of different individuals in regards
to this.
>
>   

They are excellent tools for generating user complaints.

(just like the "do traceroute and count the hops" advice from gamer mags
of old).

(my $0.02)

Michael Holstein
Cleveland State University

RE: Speed Test Results

[Frank Bulk]

We host an Ookla Speedtest server onsite and find it a very reliable means
to identify throughput issues.  The source of any performance issues may or
may not be ours, but if a customer says things are slow we can usually
identify whether it's their PC or network (browsing is slow but speed test
runs fine) or a local or regional network issue (speed test runs slow).

If a customer gets less than 90% of the advertised throughput, we follow up
on it.

Frank

-Original Message-
From: Michael Holstein [mailto:michael.holst...@csuohio.edu] 
Sent: Friday, December 23, 2011 1:27 PM
To: jacob miller
Cc: nanog@nanog.org
Subject: Re: Speed Test Results


> Am having a debate on the results of speed tests sites.
>
> Am interested in knowing the thoughts of different individuals in regards
to this.
>
>   

They are excellent tools for generating user complaints.

(just like the "do traceroute and count the hops" advice from gamer mags
of old).

(my $0.02)

Michael Holstein
Cleveland State University

Re: Speed Test Results

[Joe Hamelin]

On Fri, Dec 23, 2011 at 9:35 PM, Graham Beneke  wrote:


> That said - people get fixated on the numbers. 80% of the purchased speed
> on non-CIR services is cause for a complaint.
>
> Our biggest issue is people doing tests to destinations 300+ ms away that
> only last for a few seconds and then complaining about poor performance. As
> soon as you mention things like bandwidth delay product the eyes glaze
> over. Heavy use of lossy WISP access network providers doesn't help.


Or that most ADSL lines have about 20% ATM cell "tax" on them.

I did get caught up on a speed test today.  I was turning up a GBLX 100Mb
circuit.  I got the /30 and all the pings were good to the router.  I then
pinged some known hosts in the Westin (about a block away where GBLX's
router was) and saw some not so nice ping times.  I then ran a speedtest
and only got about 2Mb/s.  Come to find out that this was going to be an
MPLS path to the company's California office. Since it hadn't been setup
fully the router had found some path through it's management network to
ping the world through the tester's DSL line on the other side.

So, know the path you are testing.

--
Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474

Re: Speed Test Results

[Graham Beneke]

On 23/12/2011 21:26, Michael Holstein wrote:

They are excellent tools for generating user complaints.


I find that they are useful for filtering out some of the completely 
bogus complaints. We encourage customers to include some test results 
when they contact our NOC to avoid being ignored when they send an "its 
slow" complaint.


That said - people get fixated on the numbers. 80% of the purchased 
speed on non-CIR services is cause for a complaint.


Our biggest issue is people doing tests to destinations 300+ ms away 
that only last for a few seconds and then complaining about poor 
performance. As soon as you mention things like bandwidth delay product 
the eyes glaze over. Heavy use of lossy WISP access network providers 
doesn't help.


--
Graham Beneke

Re: Speed Test Results

[Landon Stewart]

Just a note on this subject although not directly related to the original
question - There some interesting tests available here:
http://www.measurementlab.net/

-- 
Landon Stewart 
Manager of Systems and Engineering
Superb Internet Corp - 888-354-6128 x 4199
Web hosting and more "Ahead of the Rest": http://www.superbhosting.net

Re: Speed Test Results

[Joel jaeggli]

On 12/23/11 11:16 , Joel Maslak wrote:
> On Fri, Dec 23, 2011 at 2:18 AM, jacob miller  wrote:
> 
>> Am having a debate on the results of speed tests sites.
>>
>> Am interested in knowing the thoughts of different individuals in regards to 
>> this.
> 
> It's one data point of many.
> 
> Depending on the speed test site, the protocols it uses, where the
> test is located, any local networking gear (I've seen transparent
> proxies get great speedtest ratings!), etc, they can be useful,
> particularly in verifying that a provider's off-net interconnects and
> partners are doing well.
> 
> However, they are susceptible to things like wireless network issues,
> TCP limitations (one stream vs. many streams), and misconfiguration of
> devices at the customer location.  And the speed test box isn't
> necessarily configured/speced correctly either.

I don't imagine it accounts for l3 emcp either... To be clear, what one
is I assume generally looking for from a speed test is usable throughput
from the vantage point of the end-user running it.

> I second the thoughts on NDT and I like the ICSI Netalyzer.  But I
> wouldn't necessarily put either tool in most end users' hands (I think
> they are too complex for most end users to interpret the results
> properly).
> 

Re: Speed Test Results

[Valdis . Kletnieks]

On Fri, 23 Dec 2011 12:16:38 MST, Joel Maslak said:

> However, they are susceptible to things like wireless network issues,
> TCP limitations (one stream vs. many streams), and misconfiguration of
> devices at the customer location.  And the speed test box isn't
> necessarily configured/speced correctly either.

I have seen some surreal results reported by some of the speed test sites
if you have a sufficiently fat pipe.  Near as I could tell, every single hop was
gigE or better all the way, the speedtest site then tried to apply a correction
for the bottleneck it knew about on its local gigE nterface, and basically 
decided
that the *rest* of the path must be near-infinite speed. ;)



pgpog8XkLzv90.pgp
Description: PGP signature

Re: Speed Test Results

[Michael Holstein]

> Am having a debate on the results of speed tests sites.
>
> Am interested in knowing the thoughts of different individuals in regards to 
> this.
>
>   

They are excellent tools for generating user complaints.

(just like the "do traceroute and count the hops" advice from gamer mags
of old).

(my $0.02)

Michael Holstein
Cleveland State University

Re: Speed Test Results

[Joel Maslak]

On Fri, Dec 23, 2011 at 2:18 AM, jacob miller  wrote:

> Am having a debate on the results of speed tests sites.
>
> Am interested in knowing the thoughts of different individuals in regards to 
> this.

It's one data point of many.

Depending on the speed test site, the protocols it uses, where the
test is located, any local networking gear (I've seen transparent
proxies get great speedtest ratings!), etc, they can be useful,
particularly in verifying that a provider's off-net interconnects and
partners are doing well.

However, they are susceptible to things like wireless network issues,
TCP limitations (one stream vs. many streams), and misconfiguration of
devices at the customer location.  And the speed test box isn't
necessarily configured/speced correctly either.

I second the thoughts on NDT and I like the ICSI Netalyzer.  But I
wouldn't necessarily put either tool in most end users' hands (I think
they are too complex for most end users to interpret the results
properly).

Re: Speed Test Results

[Livingood, Jason]

If you want to understand the issue in detail, check out the report from
MIT this year, written by Steve Bauer and available at
http://mitas.csail.mit.edu/papers/Bauer_Clark_Lehr_Broadband_Speed_Measurem
ents.pdf. 

- Jason



On 12/23/11 4:18 AM, "jacob miller"  wrote:

>Hi,
>
>Am having a debate on the results of speed tests sites.
>
>Am interested in knowing the thoughts of different individuals in regards
>to this.
>
>Regards,
>Jacob
>
>

Re: Speed Test Results

[Alex Brooks]

Hello,

On Fri, Dec 23, 2011 at 4:19 PM, Octavio Alvarez
 wrote:
>
> On Fri, 23 Dec 2011 01:18:40 -0800, jacob miller  wrote:
>
>> Am having a debate on the results of speed tests sites.
>>
>> Am interested in knowing the thoughts of different individuals in regards to 
>> this.
>
>
> They are just a measurement, which need to be correctly used and
> interpreted (that's the difficult part).
>
> Reading bad numbers is not necessarily an indication of a link problem.
>
> Reading "good enough" numbers is only meaningful for the duration of the
> test.
>
> To me, the big problem is that they don't state all the details of the
> tests (for example, how exactly to they do the transfer). Geographical
> location is good, but sometimes not enough. Do they use http, https, ftp
> or their own JS implementation of whatever weird protocol they though of?
> How do I know if I'm hitting my firewall, web cache or ALG?
>

I agree.  But one that is fairly clear in what (and how) it tests (but
to be fair isn't really a 'speed test') that I've come across is ICSI
Netalyzr.  It's pretty useful to give a first impression to a tech of
what's going on with a link.

Take a look at an example report (from a dodgy connection) I dug up:
http://netalyzr.icsi.berkeley.edu/restore/id=43ca208a-28820-e88f1efc-a129-4c92-8968

More info and examples are at http://netalyzr.icsi.berkeley.edu/

I also think that sometimes having a 'speed test' or similar hosted on
a network you are trying to connect to can be useful to find out if a
link is congested, or other problems getting from you to that network.
 An example of this is The BBC's iPlayer diagnostic at
http://www.bbc.co.uk/iplayer/diagnostics (think Hulu, but in the UK).
It tests to all their CDNs (Akami, Limelight etc) using different
streaming methods and gives the results.  Only useful as an overview,
but a decent first guide nevertheless
.
>
> I only use them to get a generic overview of the link.
>

Heck yes!

Alex

Re: Speed Test Results

[Octavio Alvarez]

On Fri, 23 Dec 2011 01:18:40 -0800, jacob miller  wrote:


Am having a debate on the results of speed tests sites.

Am interested in knowing the thoughts of different individuals in  
regards to this.


They are just a measurement, which need to be correctly used and
interpreted (that's the difficult part).

Reading bad numbers is not necessarily an indication of a link problem.

Reading "good enough" numbers is only meaningful for the duration of the
test.

To me, the big problem is that they don't state all the details of the
tests (for example, how exactly to they do the transfer). Geographical
location is good, but sometimes not enough. Do they use http, https, ftp
or their own JS implementation of whatever weird protocol they though of?
How do I know if I'm hitting my firewall, web cache or ALG?

I only use them to get a generic overview of the link.

--
Octavio.

Twitter: @alvarezp2000 -- Identi.ca: @alvarezp

RE: Speed Test Results

[Frank A. Coluccio]

RE: Speed Test Results

[Brandon Kim]

I love using speedtest. My FIOS at home is 25/25. And speedtest consistently 
hits that mark
so I know FIOS is giving me what I paid for.

When Verizon was having internet issues last week my numbers were bad. 

Like someone else said, I would not use it much more for quick gauge. To get 
more granular info
you should be using other tools



> Subject: Re: Speed Test Results
> From: james.cut...@consultant.com
> Date: Fri, 23 Dec 2011 09:02:01 -0500
> To: nanog@nanog.org
> 
> 
> On Dec 23, 2011, at 8:07 AM, Paul Stewart wrote:
> 
> > In my opinion they are only "somewhat reliable" if they are on your network
> > or very close to your network -we operate one of the speedtest.net sites and
> > for our own eyeball traffic find it to be a "reasonable indicator" of what
> > kind of speeds the customer is getting.
> > 
> > To put it a different way, if a customer is getting 20X1 Internet service
> > and the speedtest shows 17 X 0.8 then case closed - if they are getting a
> > speedtest result of 5 X 0.5 then our helpdesk will take a further look -
> > this is really in rough terms...
> > 
> > Paul
> 
> From the consumer viewpoint:
> 
> No single data point should be extrapolated to infinity, but comparing 
> problematic behavior with "normal" behavior is a standard process across all 
> fields.
> 
> Speed tests from several locations done regularly give a baseline for 
> performance.  Major departure from expected numbers from a set of speed test 
> sites can be regarded as an indicator of local loop problems. Did you know 
> that local loops suffer from backhoe fade?  And, DSLAMS fail.
> 
> In my home office, speed tests are just another useful diagnostic helping to 
> locate problem areas - just like in Paul's example.  DSLReports line 
> monitoring service is a similarly useful tool.
> 
> James R. Cutler
> james.cut...@consultant.com
> 
> 
> 
> 
>