Re: 60 ms cross-continent

[Tim Požár]

Did you not read my posting on Quora?

Tim

On 6/20/20 10:49 AM, Wayne Bouchard wrote:

And thus far, no one has mentioned switching speed and other
electronic overhead such as the transceivers (that's the big one,
IIRC.)

I also don't recall if anyone mentioned that the 30ms is as the
photon flies, not fiber distance.

-Wayne

On Sat, Jun 20, 2020 at 05:32:30PM +, Mel Beckman wrote:

An intriguing development in fiber optic media is hollow core optical fiber, 
which achieves 99.7% of the speed of light in a vacuum.

https://www.extremetech.com/computing/151498-researchers-create-fiber-network-that-operates-at-99-7-speed-of-light-smashes-speed-and-latency-records

-mel

On Jun 20, 2020, at 10:14 AM, Dave Cohen  wrote:

??? Doing some rough back of the napkin math, an ultra low-latency path from, 
say, the Westin to 1275 K in Seattle will be in the 59 ms range. This is 
considerably longer than the I-90 driving distance would suggest because:
- Best case optical distance is more like 5500 km, in part because the path 
actually will go Chicago-NJ-WDC and in part because a distance of 5000 km by 
right-of-way will be more like 5500 km when you account for things like 
maintenance coils, in-building wiring, etc.
- You???ll need (at least) three OEO regens on that distance, since there???s 
no value in spending 5x to deploy an optical system that wouldn???t need to 
(like the ones that would manage that distance subsea). This is in addition to 
~60 in-line amplification nodes, although that adds significantly less latency 
even in aggregate

Some of that is simply due to cost savings. In theory, you could probably spend 
a boatload of money to build a route that cuts off some of the distance 
inefficiency and gets you closer to 4500 km optical distance with minimal slack 
coil, and maybe no regens, so you get a real-world performance of 46 ms. But 
there are no algo trading sites of importance in DC, and for everybody else 
there???s not enough money in the difference between 46 and 59 ms for someone 
to go invest in that type of deployment.

Dave Cohen
craetd...@gmail.com

On Jun 20, 2020, at 12:44 PM, Tim Durack  wrote:

???
And of course in your more realistic example:

2742 miles = 4412 km ~ 44 ms optical rtt with no OEO in the path

On Sat, Jun 20, 2020 at 12:36 PM Tim Durack 
mailto:tdur...@gmail.com>> wrote:
Speed of light in glass ~200 km/s

100 km rtt = 1ms

Coast-to-coast ~6000 km ~60ms

Tim:>

On Sat, Jun 20, 2020 at 12:27 PM William Herrin 
mailto:b...@herrin.us>> wrote:
Howdy,

Why is latency between the east and west coasts so bad? Speed of light
accounts for about 15ms each direction for a 30ms round trip. Where
does the other 30ms come from and why haven't we gotten rid of it?

c = 186,282 miles/second
2742 miles from Seattle to Washington DC mainly driving I-90

2742/186282 ~= 0.015 seconds

Thanks,
Bill Herrin

--
William Herrin
b...@herrin.us
https://bill.herrin.us/


--
Tim:>


--
Tim:>


---
Wayne Bouchard
w...@typo.org
Network Dude
http://www.typo.org/~web/

Re: 60 ms cross-continent

[Tim Požár]

Besides the refractive index of glass that makes like go about 2/3rds it 
can in a vacuum, "Stuff" also includes many other things like 
modulation/demodulation, buffers, etc.  I did a quora answer on this you 
can find at:


https://www.quora.com/How-can-one-describe-the-delay-characteristics-of-ping-and-traceroute-commands/answer/Tim-Pozar

On 6/20/20 9:29 AM, Joe Greco wrote:

On Sat, Jun 20, 2020 at 09:24:11AM -0700, William Herrin wrote:

Howdy,

Why is latency between the east and west coasts so bad? Speed of light
accounts for about 15ms each direction for a 30ms round trip. Where
does the other 30ms come from and why haven't we gotten rid of it?

c = 186,282 miles/second
2742 miles from Seattle to Washington DC mainly driving I-90

2742/186282 ~= 0.015 seconds


Speed of light in a fiber is more like 124K miles per second.  It
depends on the refractive index.  And of course amplifiers and stuff.

... JG

Re: An appeal for more bandwidth to the Internet Archive

[Tim Požár]

Internet Archive primary office is located at 300 Funston in San 
Francisco.  This was a Christian Science church so it has the roman 
columns you would expect for a church / library.  You can see it on 
Google Street Views at:


https://www.google.com/maps/place/300+Funston+Ave,+San+Francisco,+CA+94118

Although they serve content out of this site, their primary site for 
bandwidth is at 2512 Florida Ave, Richmond, CA.


IA does have satellite offices around the world for scanning, etc., the 
public facing servers are location in these two locations.


Tim

On 5/12/20 9:24 PM, Terrence Koeman wrote:
Aren't they in a former church or something? I vaguely remember their 
location to be significant for some reason or another. So location may 
weigh heavily.



--
Regards,
    Terrence Koeman, PhD/MTh/BPsy
      Darkness Reigns (Holding) B.V.

Please quote relevant replies.
Spelling errors courtesy of my 'smart'phone.

*From:* David Hubbard 
*Sent:* Wednesday, 13 May 2020 06:02
*To:* nanog@nanog.org
*Subject:* Re: An appeal for more bandwidth to the Internet Archive

Could the operation be moved out of California to achieve
dramatically reduced operating costs and perhaps solve some problems
via cost savings vs increased donation?  I have to imagine with the
storage and processing requirements that the footprint and power
usage in SFO is quite costly.  I have equipment in a few California
colo's and it's easily 3x what I pay for similar in Nevada, before
even getting into tax abatement advantages.



On 5/12/20, 1:33 PM, "NANOG on behalf of colin johnston"
 wrote:

     Is the increased usage due to more users or more existing users
having higher bandwidth at home to request faster ?
     Would be interested if IPS configured firewall used to block
out invalid traffic/spam traffic and if such traffic increased when
back end network capacity increased ?
     What countries are requesting the most data and does this
analysis throw up questions as to why ?
     Are there high network usage hitters which raise question as to
why asking for so much data time and time again and is this valid
traffic use ?

     Colin


         > On 12 May 2020, at 17:33, Tim Požár  wrote:
     >
     > Jared...
     >
     > Thanks for sharing this.  I was the first Director of
Operations from '96 to '98, at was was then Internet Archive/Alex. 
I was the network architect back then got them their ASN and

original address space. Folks may also know, I help start SFMIX with
Matt Peterson.
     >
     > A bit more detail in this...  Some of this I got from Jonah
Edwards who is the current Network Architect at IA.  Yes, the bottle
neck was the line cards.  They have upgraded and that has certainly
helped the bandwidth of late.
     >
     > Peering would be a big help for IA. At this point they have
two 10Gb LAG interfaces that show up on SFMIX that was turned up
last February. Looking at the last couple of weeks the 95th
percentile on this 20Gb LAG is 3 Gb.  As they just turned up on
SFMIX, they are just starting to get peers turned up there. Eyeball
networks that show up on SFMIX are highly encouraged to start
peering with them.  Alas, they are v4 only at this point.
     >
     > Additionally, if folks do have some fat pipes that can donate
bandwidth at 200 Paul, I am sure Jonah won't turn it down.
     >
     > Tim
     >
     > On 5/12/20 4:45 AM, Jared Brown wrote:
     >> Hello all!
     >> Last week the Internet Archive upgraded their bandwidth 30%
from 47 Gbps to 62 Gbps. It was all gobbled up immediately. There's
a lovely solid green graph showing how usage grows vertically as
each interface comes online until it too is 100% saturated. Looking
at the graph legend you can see that their usage for the past 24
hours averages 49.76G on their 50G of transport.
     >> To see the pretty pictures follow the below link:
     >>

https://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/

     >> Relevant parts from the blog post:
     >> "A year ago, usage was 30Gbits/sec. At the beginning of this
year, we were at 40Gbits/sec, and we were handling it. ...
     >> Then Covid-19 hit and demand rocketed to 50Gbits/sec and
overran our network infrastructure’s ability to handle it.  So much
so, our network statistics probes had difficulty collecting data
(hence the white spots in the graphs).
     >> We bought a second router with new line cards, and got it
installed and running (and none of this is easy during a pandemic),

Re: An appeal for more bandwidth to the Internet Archive

[Tim Požár]

Heh.  I just did this recently as I started a low power FM station 
(KPEA-LP).  I did a bulk download of about 10,000 cuts of IA's 78 
collection from 1923 and earlier as those recordings are all in the 
public domain.


Tim

On 5/12/20 11:50 AM, Tom Hayward wrote:

On Tue, May 12, 2020 at 10:32 AM colin johnston  wrote:


Is the increased usage due to more users or more existing users having higher 
bandwidth at home to request faster ?
Would be interested if IPS configured firewall used to block out invalid 
traffic/spam traffic and if such traffic increased when back end network 
capacity increased ?
What countries are requesting the most data and does this analysis throw up 
questions as to why ?
Are there high network usage hitters which raise question as to why asking for 
so much data time and time again and is this valid traffic use ?


Amongst a certain group of users, bulk downloading of the archive is popular:
https://www.reddit.com/r/DataHoarder/comments/an8srw/is_there_anyway_to_bulk_download_collections_from/

Tom

Re: An appeal for more bandwidth to the Internet Archive

[Tim Požár]

Jared...

Thanks for sharing this.  I was the first Director of Operations from 
'96 to '98, at was was then Internet Archive/Alex.  I was the network 
architect back then got them their ASN and original address space. 
Folks may also know, I help start SFMIX with Matt Peterson.


A bit more detail in this...  Some of this I got from Jonah Edwards who 
is the current Network Architect at IA.  Yes, the bottle neck was the 
line cards.  They have upgraded and that has certainly helped the 
bandwidth of late.


Peering would be a big help for IA. At this point they have two 10Gb LAG 
interfaces that show up on SFMIX that was turned up last February. 
Looking at the last couple of weeks the 95th percentile on this 20Gb LAG 
is 3 Gb.  As they just turned up on SFMIX, they are just starting to get 
peers turned up there. Eyeball networks that show up on SFMIX are highly 
encouraged to start peering with them.  Alas, they are v4 only at this 
point.


Additionally, if folks do have some fat pipes that can donate bandwidth 
at 200 Paul, I am sure Jonah won't turn it down.


Tim

On 5/12/20 4:45 AM, Jared Brown wrote:

Hello all!

Last week the Internet Archive upgraded their bandwidth 30% from 47 Gbps to 62 
Gbps. It was all gobbled up immediately. There's a lovely solid green graph 
showing how usage grows vertically as each interface comes online until it too 
is 100% saturated. Looking at the graph legend you can see that their usage for 
the past 24 hours averages 49.76G on their 50G of transport.

To see the pretty pictures follow the below link:
https://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/

Relevant parts from the blog post:
"A year ago, usage was 30Gbits/sec. At the beginning of this year, we were at 
40Gbits/sec, and we were handling it. ...

Then Covid-19 hit and demand rocketed to 50Gbits/sec and overran our network 
infrastructure’s ability to handle it.  So much so, our network statistics 
probes had difficulty collecting data (hence the white spots in the graphs).

We bought a second router with new line cards, and got it installed and running (and 
none of this is easy during a pandemic), and increased our capacity from 47Gbits/sec 
peak to 62Gbits/sec peak.   And we are handling it better, but it is still 
consumed."

It is obvious that the Internet Archive needs more bandwidth to power the 
Wayback machine and to fulfill its mission of being the Internet library and 
the historic archive of our times.

The Internet Archive is present at Digital Realty SFO (200 Paul) and a member 
of the San Francisco Metropolitan Internet Exchange (SFMIX).
I appeal to all list members present or capable of getting to these facilities 
to peer with and/or donate bandwidth to the Internet Archive.
I appeal to all vendors and others with equipment that they can donate to the 
Internet Archive to contact them so that they can scale their services and 
sustain their growth.

The Internet Archive is currently running 10G equipment. If you can help them 
gain 100G connectivity, 100G routing, 100G switching and/or 100G DWDM 
capabilities, please reach out to them. They have the infrastructure and dark 
fiber to transition to 100G, but lack the equipment. You can find the Internet 
Archive's contact information below or you can contact Jonah at the Archive Org 
directly either by email or via the contact information available on his 
Twitter profile @jonahedwards.

You can also donate at https://archive.org/donate/
The Internet Archive is a 501(c)(3) non-profit. Donations are  tax-deductible.


Contact information:
https://archive.org/about/contact.php

Volunteering:
https://archive.org/about/volunteerpositions.php


Disclaimer: I am not affiliated with the Internet Archive. Nobody asked me to 
write this post. If something angers you about this post, be angry at me. I 
merely think that the Internet Archive is a good thing and deserves our support.

Jared

Re: CISA critical infrastructure letters

[Tim Požár]

They are so open ended, they are really useless.  Not sure why they 
didn't issue this with a company affiliation, etc to nail it down to say 
credentials that the person may have with them.


Back in my Broadcast Engineering days, I would get passes issued by the 
local LE such as the SF Police department or as a "Registered Disaster 
Service Worker" issued by the State of California.  Each of these would 
have my name, photo etc.  These were respected and got me through 
numerous police lines in the past.


https://www.lns.com/house/pozar/laminates/

On 3/25/20 11:20 AM, Sean Donelan wrote:
The CISA critical infrastructure letters are a courtesy request letter. 
If people abuse its purpose, local officials do not need to extend any 
courtesy and can deny access.


The CISA letter is only for "providing emergency communications 
sustainment and restoration support to critical communications 
infrastructure facilities."


It is NOT a general purpose, ignore anything or go anywhere letter.

Do NOT abuse the courtesy or no one will extend the courtesy.

Re: DHS letters for fuel and facility access

[Tim Požár]

As an ex-broadcaster, I have never seen one of these letters.  Even 
during our 1989 earthquake.   In fact, I knew of one station that 
ordered a genset as power was down after the earthquake for several days 
and it was commandeered by LE as they were being driven to the 
transmitter site, three times!  Three different gensets!




On 3/16/20 1:20 PM, Sean Donelan wrote:


On some other mailing lists, FCC licensed operators are reporting they 
have received letters from the Department of Homeland Security 
authorizing "access" and "fuel" priority.


Occasionally, DHS issues these letters after natural disasters such as 
hurricanes for hospitals and critical facilities.  I haven't heard of 
them issued for pandemics.

Re: Hi-Rise Building Fiber Suggestions

[Tim Požár]

Also, Juniper switches will stack over fiber.  I have deployed Virtual
Chassis over multiple IDFs.  The VC ports can be (and highly suggested)
to be in a ring.

https://www.juniper.net/documentation/en_US/junos/topics/concept/virtual-chassis-ex4200-overview.html

https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/virtual-chassis-ex4300-configuring.html

On 2/25/20 6:32 PM, Norman Jester wrote:
> I’m in the process of choosing hardware
> for a 30 story building. If anyone has experience with this I’d appreciate 
> any tips.
> 
> There are two fiber pairs running up the building riser. I need to put a POE 
> switch on each floor using this fiber. 
> 
> The idea is to cut the fiber at each floor and insert a switch and daisy 
> chain the switches together using one pair, and using the other pair as the 
> failover side of the ring going back to the source so if one device fails it 
> doesn’t take the whole string down.
> 
> The problem here is how many switches can be strung together and I would not 
> try more than 3 to 5. This is not something I typically do (stacking 
> switches). I have fears of STP and/or RSTP issue stacking past Ethernet 
> switch to switch limits (if they still exist??)
> 
> Is there a device with a similar protocol as the old 3com (now HP IDF) 
> stacking capability via fiber? 
> 
> I’d like to use something inexpensive as its to power ubiquiti wifi on each 
> floor.  Ideally if you know something I don’t about ubiquiti switches that 
> can do this I’d appreciate knowing.
> 
> Norman
> 

Re: Hi-Rise Building Fiber Suggestions

[Tim Požár]

If you are limited on fiber runs, how about using 10Gb BiDi optics to
limit a ring to say two sets of 15 switches.

Tim

On 2/25/20 8:21 PM, Bradley Burch wrote:
> Should consider DWDM or GPON and in those look at passive optical 
> technologies that can benefit the project.
> 
>> On Feb 25, 2020, at 9:33 PM, Norman Jester  wrote:
>>
>> I’m in the process of choosing hardware
>> for a 30 story building. If anyone has experience with this I’d appreciate 
>> any tips.
>>
>> There are two fiber pairs running up the building riser. I need to put a POE 
>> switch on each floor using this fiber. 
>>
>> The idea is to cut the fiber at each floor and insert a switch and daisy 
>> chain the switches together using one pair, and using the other pair as the 
>> failover side of the ring going back to the source so if one device fails it 
>> doesn’t take the whole string down.
>>
>> The problem here is how many switches can be strung together and I would not 
>> try more than 3 to 5. This is not something I typically do (stacking 
>> switches). I have fears of STP and/or RSTP issue stacking past Ethernet 
>> switch to switch limits (if they still exist??)
>>
>> Is there a device with a similar protocol as the old 3com (now HP IDF) 
>> stacking capability via fiber? 
>>
>> I’d like to use something inexpensive as its to power ubiquiti wifi on each 
>> floor.  Ideally if you know something I don’t about ubiquiti switches that 
>> can do this I’d appreciate knowing.
>>
>> Norman
>>

Re: DDoS attack

[Tim Požár]

This is lame.  They should be able to view NAT translation tables or
better yet have some method of watching flows.

Tim

On 12/9/19 12:11 PM, Christopher Morrow wrote:
> I'd note that: "what prefixes?" isn't answered here... like: "what is
> the thing on your network which is being attacked?"
> 
> On Mon, Dec 9, 2019 at 3:08 PM ahmed.dala...@hrins.net
>  wrote:
>>
>> Dear All,
>>
>> My network is being flooded with UDP packets, Denial of Service attack, 
>> soucing from Cloud flare and Google IP Addresses, with 200-300 mbps minimum 
>> traffic, the destination in my network are IP prefixes that is currnetly not 
>> used but still getting traffic with high volume.
>> The traffic is being generated with high intervals between 10-30 Minutes for 
>> each time, maxing to 800 mbps
>> When reached out cloudflare support, they mentioned that there services are 
>> running on Nat so they can’t pin out which server is attacking based on ip 
>> address alone, as a single IP has more than 5000 server behind it, providing 
>> 1 source IP and UDP source port, didn’t help either
>> Any suggestions?
>>
>> Regards,
>> Ahmed Dala Ali

Re: California network infrastructure report (FCC)

[Tim Požár]

That lines up with my experience living in Mill Valley and trying to
tether here.  Either cell service is down, or it is up but not data.
Finally gave up and drove into SF to steal some electrons and wifi at a
dive bar on Geary.

Tim

On 10/27/19 5:21 PM, Sean Donelan wrote:
> 
> According to reporting to the FCC:
> https://www.fcc.gov/document/ca-power-shutoff-communications-status-report-oct-27-2019
> 
> 
> Cell sites out of service: overall 2.4% (630 out of 25,893)
> 
> Marin County: 49.6% out of service (105 out of 270)
> Lake County: 19.3% out of service (11 out of 57)
> Calaveras County: 18.6% out of service (8 out of 43)
> Sonoma County: 17.1% out of service (73 out of 427)
> Santa Cruz County: 16.4% out of service (35 out of 213)
> 
> Cable and Wireline systems (combined)
> 
> 393,735 subscribers out of service. Only Outside Plant and Hub
> locations. Likely does not include loss of Customer Premise Equipment
> (CPE) power.
> 
> 
> Broadcast
> 
> 4 FM radio stations out of service, 2 FM stations with programming sent
> to another station.
>