RE: Learning about the internet

[Dustin Melancon]

Yeap, that's what I was typing up. 

More than likely what we're seeing is just the far of the link from IAD to 
Paris. This is evident from the fact that the next hop to Frankfurt is only an 
additional 3ms beyond what is reported as IAD.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Brian Loveland
Sent: Monday, November 3, 2014 4:33 PM
To: William Herrin
Cc: nanog@nanog.org
Subject: Re: Learning about the internet

Isn't this most likely a side effect of MPLS tunneling and the 93ms jump there 
is actually the trans-atlantic segment?  All the european hops following it are 
very close in latency to hop 11.

On Mon, Nov 3, 2014 at 5:14 PM, William Herrin  wrote:

> On Mon, Nov 3, 2014 at 4:15 PM, Paige Thompson 
> wrote:
> > I
> > would be able to find an answer as to why the latency between here 
> > in greece and Los Angeles is roughly ~250ms.
>
> > 10.|-- be2171.mpd22.dca01.atlas.cogentco.com  0.0%10   62.6  62.7
> >  62.4  63.3   0.0
> > 11.|-- be2112.ccr41.iad02.atlas.cogentco.com  0.0%10  155.5 155.8
> > 155.5 156.1   0.0
>
> ^^ There's the largest part part of your answer. IAD and DCA are both 
> Northern VA airports; there should be perhaps 4ms of latency between 
> these locations.
>
>
> --
> William Herrin  her...@dirtside.com  b...@herrin.us 
> Owner, Dirtside Systems . Web: <http://www.dirtside.com/> May 
> I solve your unusual networking challenges?
>

Re: Learning about the internet

[Brian Loveland]

Isn't this most likely a side effect of MPLS tunneling and the 93ms jump
there is actually the trans-atlantic segment?  All the european hops
following it are very close in latency to hop 11.

On Mon, Nov 3, 2014 at 5:14 PM, William Herrin  wrote:

> On Mon, Nov 3, 2014 at 4:15 PM, Paige Thompson 
> wrote:
> > I
> > would be able to find an answer as to why the latency between here in
> > greece and Los Angeles is roughly ~250ms.
>
> > 10.|-- be2171.mpd22.dca01.atlas.cogentco.com  0.0%10   62.6  62.7
> >  62.4  63.3   0.0
> > 11.|-- be2112.ccr41.iad02.atlas.cogentco.com  0.0%10  155.5 155.8
> > 155.5 156.1   0.0
>
> ^^ There's the largest part part of your answer. IAD and DCA are both
> Northern VA airports; there should be perhaps 4ms of latency between these
> locations.
>
>
> --
> William Herrin  her...@dirtside.com  b...@herrin.us
> Owner, Dirtside Systems . Web: 
> May I solve your unusual networking challenges?
>

Re: Learning about the internet

[Dustin Melancon]

Hey Paige,

That¹s going to be extremely common when traversing transatlantic cable.
Hopefully I can help explain.

Most of your latency comes from simple speed of light limitations through
optical fiber (~35% slower than vacuum) combined with the fact that the
fiber must take a somewhat indirect path to your destination.
Additionally, a small percentage of latency will be added for all of the
switching/routing/regeneration nodes that must convert the fiber signals
from optical back to electrical (for processing / regeneration) then back
to optical. In your case, your packets are taking the following path:

Los Angeles (Arp Networks to Level3 to Cogent) -> Houston -> Atlanta ->
Washington DC -> Paris -> Frankfurt -> Munich -> Austria -> Bulgaria ->
Athens -> Hellas

Essentially it¹s taking 60ms round trip to get to DC, then another 95ms to
get to Paris, plus 60ms to get to Hellas, then another 20ms to traverse
the local ³last mile² access network to your destination.

If you had a *direct* fiber from LAX to Hellas Greece, that would be
~22,000km of round trip fiber. Working out some simple math, that would
yield approximately 110ms of round trip latency no matter how much
bandwidth you have. Now, add in all of the indirect fiber paths through
major cities plus all of the routing/switching hops and you can see why
you¹re getting 200ms+ latency.

Hope this helps!


Thanks,
Dustin Melancon, Sr. Network Engineer
225-214-3864 Direct ::: 866-978-3698 Toll-Free
­
VENYU : Your Data Made Invincible : www.venyu.com 
­
Read our latest blog post: blog.venyu.com 






On 11/3/14, 3:15 PM, "Paige Thompson"  wrote:

>Hi,
>
>I was just reading about transatlantic cabling in some hopes that I
>would be able to find an answer as to why the latency between here in
>greece and Los Angeles is roughly ~250ms. This seems to be a really
>common thing, although I'd like to understand why and the articles on
>transatlantic cabling as near as I can tell indicate that I am getting
>screwed if anything (not enough information?)
>
>(from Los Angeles to my house)
>Konsole output
>
>Konsole output
>gw~ #mtr --report-wide xxx.access.hol.gr
>Start: Mon Nov  3 13:04:02 2014
>HOST: gwLoss%   Snt   Last   Avg
> Best  Wrst StDev
> 1.|-- 208.79.92.65  10.0%101.5   3.6
>  1.2  15.5   4.6
> 2.|-- s7.lax.arpnetworks.com 0.0%100.8  10.9
>  0.8  54.2  20.7
> 3.|-- vlan953.car2.LosAngeles1.Level3.net   30.0%10   10.5  10.3
> 10.1  10.8   0.0
> 4.|-- ae-27-27.edge6.LosAngeles1.Level3.net 30.0%10   21.8  16.2
>  8.6  47.6  14.7
> 5.|-- ae-4-90.edge1.LosAngeles6.Level3.net  80.0%109.0   8.9
>  8.7   9.0   0.0
> 6.|-- be3036.ccr21.lax04.atlas.cogentco.com 10.0%101.7   2.1
>  1.4   4.3   0.7
> 7.|-- be2076.mpd22.lax01.atlas.cogentco.com 10.0%101.6   1.9
>  1.6   3.2   0.0
> 8.|-- be2068.ccr22.iah01.atlas.cogentco.com  0.0%10   37.7  37.7
> 37.3  39.0   0.3
> 9.|-- be2173.ccr42.atl01.atlas.cogentco.com  0.0%10   51.6  52.4
> 51.5  57.5   1.7
>10.|-- be2171.mpd22.dca01.atlas.cogentco.com  0.0%10   62.6  62.7
> 62.4  63.3   0.0
>11.|-- be2112.ccr41.iad02.atlas.cogentco.com  0.0%10  155.5 155.8
>155.5 156.1   0.0
>12.|-- be2268.ccr42.par01.atlas.cogentco.com  0.0%10  152.6 152.7
>152.5 153.5   0.0
>13.|-- be2278.ccr42.fra03.atlas.cogentco.com  0.0%10  155.3 155.4
>155.1 155.5   0.0
>14.|-- be2229.ccr22.muc01.atlas.cogentco.com  0.0%10  161.2 161.1
>160.9 161.3   0.0
>15.|-- be2223.ccr21.vie01.atlas.cogentco.com  0.0%10  164.9 165.1
>164.9 165.2   0.0
>16.|-- be2046.ccr21.sof02.atlas.cogentco.com  0.0%10  189.5 189.4
>189.3 189.9   0.0
>17.|-- be2118.rcr11.ath01.atlas.cogentco.com  0.0%10  197.5 197.6
>197.4 197.7   0.0
>18.|-- 149.11.120.38  0.0%10  202.7 202.2
>200.3 204.2   1.4
>19.|-- 62.38.97.113  80.0%10  208.5 209.8
>208.5 211.1   1.7
>20.|-- gigaeth04-13.krs00.ar.hol.gr  60.0%10  211.3 213.0
>211.2 218.2   3.4
>21.|-- ???   100.0100.0   0.0
>  0.0   0.0   0.0
>22.|-- .access.hol.gr 40.0%10  231.3 231.4
>231.2 231.7   0.0
>gw~ #
>
>
>
>And to be more clear: I am hoping to learn about the complex trials that
>these packets are going through and how time is being lost if the
>latency across the transatlantic cable is really capable of less the
>60ms of latency? Sure over capacity (3.2Tbits/s wow jeez) is one answer,
>but what are some other possibilities for loss of time?
>
>Also it seems with my VPN (OpenVPN) tunnel I get the most reliable
>connection (fewest drops) with:
>
>Konsole output
>mssfix 576
>fragment 576
>
>Although this could be a false positive as it only *seems* to help with
>reliability since I chang

Re: Learning about the internet

[William Herrin]

On Mon, Nov 3, 2014 at 4:15 PM, Paige Thompson  wrote:
> I
> would be able to find an answer as to why the latency between here in
> greece and Los Angeles is roughly ~250ms.

> 10.|-- be2171.mpd22.dca01.atlas.cogentco.com  0.0%10   62.6  62.7
>  62.4  63.3   0.0
> 11.|-- be2112.ccr41.iad02.atlas.cogentco.com  0.0%10  155.5 155.8
> 155.5 156.1   0.0

^^ There's the largest part part of your answer. IAD and DCA are both
Northern VA airports; there should be perhaps 4ms of latency between these
locations.

A more smug answer is, "Cogent." Google "cogent congestion," "Cogent
peering," and similar search terms.


Other parts of the answer include:

1. The routers are mostly doing store and forward which means each must
receive your entire packet before it can forward it onward to the next
router. This will take more or less time depending on the link speed.

2. There are layer 2 devices doing the same thing in between some of the
routers. Layer 2 is invisible to the layer 3 traceroute. There may also be
MPLS systems in there with the same impact.

3. Greece and LA are more than 11,000 km apart as the crow flies. This
means round-trip packets travel something like 25,000 km. There is some
signal propagation delay involved. U.S. East Coast to Hawaii (about 80% of
that distance) is typically 120 ms.

Regards,
Bill Herrin


--
William Herrin  her...@dirtside.com  b...@herrin.us
Owner, Dirtside Systems . Web: 
May I solve your unusual networking challenges?

Re: Learning about the internet

[joel jaeggli]

looks about right in the neighborhood of 9k miles...

from lax or therebouts.

  Upstream   Intf  Nexthop  Sent   LossMinAvg
 MaxDev
  cogentx   x  10   0.0%194.814
210.255240.989
 16.518
  comcast   x   x  10   0.0%201.723213.942 
239.47
13.205
  l3x   x  10   0.0% 195.51
208.189226.971 10.455
  telia x   x  10   0.0%194.552
207.392225.792 10.321

On 11/3/14 1:15 PM, Paige Thompson wrote:
> Hi,
> 
> I was just reading about transatlantic cabling in some hopes that I
> would be able to find an answer as to why the latency between here in
> greece and Los Angeles is roughly ~250ms. This seems to be a really
> common thing, although I'd like to understand why and the articles on
> transatlantic cabling as near as I can tell indicate that I am getting
> screwed if anything (not enough information?)
> 
> (from Los Angeles to my house)
> Konsole output
> 
> Konsole output
> gw~ #mtr --report-wide xxx.access.hol.gr
> Start: Mon Nov  3 13:04:02 2014
> HOST: gwLoss%   Snt   Last   Avg
>  Best  Wrst StDev
>  1.|-- 208.79.92.65  10.0%101.5   3.6
>   1.2  15.5   4.6
>  2.|-- s7.lax.arpnetworks.com 0.0%100.8  10.9
>   0.8  54.2  20.7
>  3.|-- vlan953.car2.LosAngeles1.Level3.net   30.0%10   10.5  10.3
>  10.1  10.8   0.0
>  4.|-- ae-27-27.edge6.LosAngeles1.Level3.net 30.0%10   21.8  16.2
>   8.6  47.6  14.7
>  5.|-- ae-4-90.edge1.LosAngeles6.Level3.net  80.0%109.0   8.9
>   8.7   9.0   0.0
>  6.|-- be3036.ccr21.lax04.atlas.cogentco.com 10.0%101.7   2.1
>   1.4   4.3   0.7
>  7.|-- be2076.mpd22.lax01.atlas.cogentco.com 10.0%101.6   1.9
>   1.6   3.2   0.0
>  8.|-- be2068.ccr22.iah01.atlas.cogentco.com  0.0%10   37.7  37.7
>  37.3  39.0   0.3
>  9.|-- be2173.ccr42.atl01.atlas.cogentco.com  0.0%10   51.6  52.4
>  51.5  57.5   1.7
> 10.|-- be2171.mpd22.dca01.atlas.cogentco.com  0.0%10   62.6  62.7
>  62.4  63.3   0.0
> 11.|-- be2112.ccr41.iad02.atlas.cogentco.com  0.0%10  155.5 155.8
> 155.5 156.1   0.0
> 12.|-- be2268.ccr42.par01.atlas.cogentco.com  0.0%10  152.6 152.7
> 152.5 153.5   0.0
> 13.|-- be2278.ccr42.fra03.atlas.cogentco.com  0.0%10  155.3 155.4
> 155.1 155.5   0.0
> 14.|-- be2229.ccr22.muc01.atlas.cogentco.com  0.0%10  161.2 161.1
> 160.9 161.3   0.0
> 15.|-- be2223.ccr21.vie01.atlas.cogentco.com  0.0%10  164.9 165.1
> 164.9 165.2   0.0
> 16.|-- be2046.ccr21.sof02.atlas.cogentco.com  0.0%10  189.5 189.4
> 189.3 189.9   0.0
> 17.|-- be2118.rcr11.ath01.atlas.cogentco.com  0.0%10  197.5 197.6
> 197.4 197.7   0.0
> 18.|-- 149.11.120.38  0.0%10  202.7 202.2
> 200.3 204.2   1.4
> 19.|-- 62.38.97.113  80.0%10  208.5 209.8
> 208.5 211.1   1.7
> 20.|-- gigaeth04-13.krs00.ar.hol.gr  60.0%10  211.3 213.0
> 211.2 218.2   3.4
> 21.|-- ???   100.0100.0   0.0
>   0.0   0.0   0.0
> 22.|-- .access.hol.gr 40.0%10  231.3 231.4
> 231.2 231.7   0.0
> gw~ #
> 
> 
> 
> And to be more clear: I am hoping to learn about the complex trials that
> these packets are going through and how time is being lost if the
> latency across the transatlantic cable is really capable of less the
> 60ms of latency? Sure over capacity (3.2Tbits/s wow jeez) is one answer,
> but what are some other possibilities for loss of time?
> 
> Also it seems with my VPN (OpenVPN) tunnel I get the most reliable
> connection (fewest drops) with:
> 
> Konsole output
> mssfix 576
> fragment 576
> 
> Although this could be a false positive as it only *seems* to help with
> reliability since I changed it. Even then but less often than before I
> still experience drops but I want to believe that's possibly due to my
> ISP at that point.. but assuming my ISP was absolutely perfect and never
> a problem what else there to consider?
> 
> Any and all insight is appreciated.
> 
> -Paige
> 
> 




signature.asc
Description: OpenPGP digital signature

Learning about the internet

[Paige Thompson]

Hi,

I was just reading about transatlantic cabling in some hopes that I
would be able to find an answer as to why the latency between here in
greece and Los Angeles is roughly ~250ms. This seems to be a really
common thing, although I'd like to understand why and the articles on
transatlantic cabling as near as I can tell indicate that I am getting
screwed if anything (not enough information?)

(from Los Angeles to my house)
Konsole output

Konsole output
gw~ #mtr --report-wide xxx.access.hol.gr
Start: Mon Nov  3 13:04:02 2014
HOST: gwLoss%   Snt   Last   Avg
 Best  Wrst StDev
 1.|-- 208.79.92.65  10.0%101.5   3.6
  1.2  15.5   4.6
 2.|-- s7.lax.arpnetworks.com 0.0%100.8  10.9
  0.8  54.2  20.7
 3.|-- vlan953.car2.LosAngeles1.Level3.net   30.0%10   10.5  10.3
 10.1  10.8   0.0
 4.|-- ae-27-27.edge6.LosAngeles1.Level3.net 30.0%10   21.8  16.2
  8.6  47.6  14.7
 5.|-- ae-4-90.edge1.LosAngeles6.Level3.net  80.0%109.0   8.9
  8.7   9.0   0.0
 6.|-- be3036.ccr21.lax04.atlas.cogentco.com 10.0%101.7   2.1
  1.4   4.3   0.7
 7.|-- be2076.mpd22.lax01.atlas.cogentco.com 10.0%101.6   1.9
  1.6   3.2   0.0
 8.|-- be2068.ccr22.iah01.atlas.cogentco.com  0.0%10   37.7  37.7
 37.3  39.0   0.3
 9.|-- be2173.ccr42.atl01.atlas.cogentco.com  0.0%10   51.6  52.4
 51.5  57.5   1.7
10.|-- be2171.mpd22.dca01.atlas.cogentco.com  0.0%10   62.6  62.7
 62.4  63.3   0.0
11.|-- be2112.ccr41.iad02.atlas.cogentco.com  0.0%10  155.5 155.8
155.5 156.1   0.0
12.|-- be2268.ccr42.par01.atlas.cogentco.com  0.0%10  152.6 152.7
152.5 153.5   0.0
13.|-- be2278.ccr42.fra03.atlas.cogentco.com  0.0%10  155.3 155.4
155.1 155.5   0.0
14.|-- be2229.ccr22.muc01.atlas.cogentco.com  0.0%10  161.2 161.1
160.9 161.3   0.0
15.|-- be2223.ccr21.vie01.atlas.cogentco.com  0.0%10  164.9 165.1
164.9 165.2   0.0
16.|-- be2046.ccr21.sof02.atlas.cogentco.com  0.0%10  189.5 189.4
189.3 189.9   0.0
17.|-- be2118.rcr11.ath01.atlas.cogentco.com  0.0%10  197.5 197.6
197.4 197.7   0.0
18.|-- 149.11.120.38  0.0%10  202.7 202.2
200.3 204.2   1.4
19.|-- 62.38.97.113  80.0%10  208.5 209.8
208.5 211.1   1.7
20.|-- gigaeth04-13.krs00.ar.hol.gr  60.0%10  211.3 213.0
211.2 218.2   3.4
21.|-- ???   100.0100.0   0.0
  0.0   0.0   0.0
22.|-- .access.hol.gr 40.0%10  231.3 231.4
231.2 231.7   0.0
gw~ #



And to be more clear: I am hoping to learn about the complex trials that
these packets are going through and how time is being lost if the
latency across the transatlantic cable is really capable of less the
60ms of latency? Sure over capacity (3.2Tbits/s wow jeez) is one answer,
but what are some other possibilities for loss of time?

Also it seems with my VPN (OpenVPN) tunnel I get the most reliable
connection (fewest drops) with:

Konsole output
mssfix 576
fragment 576

Although this could be a false positive as it only *seems* to help with
reliability since I changed it. Even then but less often than before I
still experience drops but I want to believe that's possibly due to my
ISP at that point.. but assuming my ISP was absolutely perfect and never
a problem what else there to consider?

Any and all insight is appreciated.

-Paige