Re: UDP/123 policers & status

[Damian Menscher via NANOG]

On Wed, Mar 18, 2020 at 7:05 PM Harlan Stenn  wrote:

> On 3/18/2020 4:46 PM, Damian Menscher via NANOG wrote:
> > On Wed, Mar 18, 2020 at 8:45 AM Steven Sommars
> > mailto:stevesommars...@gmail.com>> wrote:
> >
> > The various NTP filters (rate limits, packet size limits) are
> > negatively affecting the NTP Pool, the new secure NTP protocol
> > (Network Time Security) and other clients.  NTP filters were
> > deployed several years ago to solve serious DDoS issues, I'm not
> > second guessing those decisions.  Changing the filters to instead
> > block NTP mode 7, which cover monlist and other diagnostics, would
> > improve NTP usability.
> >
> >
> http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf
>
> >
> >
> > I've advocated a throttle (not a hard block) on udp/123 packets with 468
> > Bytes/packet (the size of a full monlist response).  In your paper you
> > mention NTS extensions can be 200+ bytes.  How large do those packets
> > typically get, in practice?  And how significant is packet loss for them
> > (if there's high packet loss during the occasional attack, does that
> > pose a problem)?
>
> I expect to see NTP UDP packets that would approach the MTU limit, in
> some cases.
>
> If a packet is "too big" for some pathway, then are we talking about a
> fractional packet loss or are we talking about 100% packet loss (dropped
> mid-way due to size)?
>

I implement it as a throttle... but that effectively means "0% packet loss
most of the time, and significant packet loss when there's a large
attack".  Doing it as a complete block (which Sommars showed some networks
do) is unnecessary.

So my question is whether occasional bursts of hard-block are a problem.
If you only need large packets during the initialization phase, but the
ongoing communication is done with smaller packets, then my approach may be
acceptable, and we just need to convince other network operators to
throttle rather than hard-block the large packets.  But if you need large
packets all the time, and occasional breakage of large packets causes
problems, then I'll need to re-think my approach.

Damian

Re: COVID-19 vs. our Networks

[Scott Weeks]

We do about 70-80Gbps at peak over the external 
BGP links we have and I am not seeing a large 
increase nor am I seeing it spread out over time.  
We're an eyeball network plus some really large 
customers.

Anyone else seeing something different?  We're
now into the 3rd day, so I thought I'd see
something change by now.

scott

Re: UDP/123 policers & status

[Harlan Stenn]

On 3/18/2020 4:46 PM, Damian Menscher via NANOG wrote:
> On Wed, Mar 18, 2020 at 8:45 AM Steven Sommars
> mailto:stevesommars...@gmail.com>> wrote:
> 
> The various NTP filters (rate limits, packet size limits) are
> negatively affecting the NTP Pool, the new secure NTP protocol
> (Network Time Security) and other clients.  NTP filters were
> deployed several years ago to solve serious DDoS issues, I'm not
> second guessing those decisions.  Changing the filters to instead
> block NTP mode 7, which cover monlist and other diagnostics, would
> improve NTP usability.
> 
> 
> http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf  
> 
> 
> I've advocated a throttle (not a hard block) on udp/123 packets with 468
> Bytes/packet (the size of a full monlist response).  In your paper you
> mention NTS extensions can be 200+ bytes.  How large do those packets
> typically get, in practice?  And how significant is packet loss for them
> (if there's high packet loss during the occasional attack, does that
> pose a problem)?

I expect to see NTP UDP packets that would approach the MTU limit, in
some cases.

If a packet is "too big" for some pathway, then are we talking about a
fractional packet loss or are we talking about 100% packet loss (dropped
mid-way due to size)?

> Damian

-- 
Harlan Stenn 
http://networktimefoundation.org - be a member!

Re: UDP/123 policers & status

[Damian Menscher via NANOG]

On Wed, Mar 18, 2020 at 8:45 AM Steven Sommars 
wrote:

> The various NTP filters (rate limits, packet size limits) are negatively
> affecting the NTP Pool, the new secure NTP protocol (Network Time Security)
> and other clients.  NTP filters were deployed several years ago to solve
> serious DDoS issues, I'm not second guessing those decisions.  Changing the
> filters to instead block NTP mode 7, which cover monlist and other
> diagnostics, would improve NTP usability.
>
> http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf
>
>

I've advocated a throttle (not a hard block) on udp/123 packets with 468
Bytes/packet (the size of a full monlist response).  In your paper you
mention NTS extensions can be 200+ bytes.  How large do those packets
typically get, in practice?  And how significant is packet loss for them
(if there's high packet loss during the occasional attack, does that pose a
problem)?

Damian

Re: Quagga for production?

[Mark Tinka]

On 18/Mar/20 22:22, Nick Hilliard wrote:

> Yeah.  I was thinking more for the case of customer-facing anycast
> resolvers, in which case BGP down means that the network is down, and
> if the network is down it doesn't matter than DNS is also down because
> their shared fate means that when BGP is back up, DNS will start
> working again.

As much as possible, I'll always choose to have the most basic
infrastructure available under abnormal conditions, regardless of the
service.

ME3600X's and ASR920's, for example, will install 0/0 and ::/0 in FIB
last. If your access to the core depends entirely on BGP in such
scenarios, you will be unable to access it for as much as 10 minutes.

Mark.

Re: Quagga for production?

[Mark Tinka]

On 18/Mar/20 22:22, Nick Hilliard wrote:

>
> Yeah.  I was thinking more for the case of customer-facing anycast
> resolvers, in which case BGP down means that the network is down, and
> if the network is down it doesn't matter than DNS is also down because
> their shared fate means that when BGP is back up, DNS will start
> working again.

As much as possible, I'll always choose to have the most basic
infrastructure available under abnormal conditions, regardless of the
service.

ME3600X's and ASR920's, for example, will install 0/0 and ::/0 in FIB
last. If your access to the core depends entirely on BGP in such
scenarios, you will be unable to access the it for as much as 10 minutes.

Mark.

Fwd: Internet operations during pandemics

[Christopher Morrow]

Did other folk on nanog-l see the nLnog-l note copied here?
I wonder how folk are planning for things (noted in the slides)
  o  supply chain for parts/equipment
 Wait, I can't get me a new shiny shipped because what??

  o ongoing rollout of new equipment
 I'm deploying next week in KIX, I'm currently in LAX how do I get
there? equipment arrives.. in between...oops!

  o noc/etc support staff
omg.. wait, I can't have my noc staff in the same room? our 'wfh'
solution is ... wait, where is that?
how do i get their phone queue sent to them? omg :( 

  o services capacity crunches
I love my shiny new dns service.. .wait, why is there a smoking
hole where my dns servers were?

I think some of this has been discussed (shifts in peaks, leveling of peaks)
Some hasn't really...  I expect that at least sharing some 'err, our
WFH changed now we do: X, Y , Z and use M to get N solved'
could be super cool to discuss/share and iterate for better solutions
for all of our users.

thoughts? :)

thanks!
-chris
(note all the hard work in this message is not mine... thanks Job!)

-- Forwarded message -
From: Job Snijders 
Date: Wed, Mar 18, 2020 at 6:02 PM
Subject: Internet operations during pandemics
To: 


Dear all,

I threw together a slidedeck today on the potential impact and second
order effects of COVID-19 on Internet network operations.

http://instituut.net/~job/netops_during_pandemics.pdf

I hope we together over time can add and extend projections in the deck
on what will happen and how we can mitigate the negative effects on
Internet operations.

We have to answer questions such as:

1) what problems already exist today because of a few weeks of C19?
2) What problems are still coming? Will those be localized or globally?
3) What possible workarounds can we plan for those problems?

I would appreciate feedback, comments, corrections or whatever you want
to tell me. None of us have been in this situation before, so my guess
is as good as yours.

Kind regards,

Job

Re: Quagga for production?

[Nick Hilliard]

Mark Tinka wrote on 18/03/2020 17:02:

I prefer to have a number of core systems accessible in the IGP, because
BGP can sometimes get hosed for one reason or another.

BGP always needs IGP to work. The reverse is not true, and reduces us to
absolute basics when it hits the fan (which it has, a few times before).


Yeah.  I was thinking more for the case of customer-facing anycast 
resolvers, in which case BGP down means that the network is down, and if 
the network is down it doesn't matter than DNS is also down because 
their shared fate means that when BGP is back up, DNS will start working 
again.


Nick

SD-WAN Operators Group

[Hiers, David]

Hi,
If you’re interested in SD-WAN, I’ve started a NANOG-knockoff over on groups.io.

https://groups.io/g/sdwanoperators

it has all the usual SMTP controls:

  *   Post: sdwanoperat...@groups.io
  *   Subscribe: 
sdwanoperators+subscr...@groups.io
  *   Unsubscribe: 
sdwanoperators+unsubscr...@groups.io
  *   Group Owner: 
sdwanoperators+ow...@groups.io
  *   Help: sdwanoperators+h...@groups.io



Regards,

David


--
This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, notify the sender immediately by return email and delete the message 
and any attachments from your system.

NetEase contact

[Jack Leung (R* NYC)]

Hi all

Does anyone here have a contact at NetEase, an online service provider out of 
China.  We have a number of customers in Asia who have them as their 
provider/carrier and are hitting outdated DNS entries so I'm trying to get to 
the bottom of it.  Support tickets have gone unanswered.

Thanks!

Jack

Re: DHS letters for fuel and facility access

[Ben Cannon]

It flabbergasts me to no end that nobody simulated the actual incident they are 
guarding against.

But I guess that’s why we run telecom companies.

Diesel piston generators need to be run for 30min every 30 (absent engineer 
calcs permitting lower, but, why).

You should also consider a pull and re-strike on that breaker 3 times.

Most transmission level circuit breakers will auto-retry 3x then quit if they 
trip each time. 

Your ATS should smooth this, but that function needs to get tested too.

Things you learn in heavy civil construction that you don’t necessarily learn 
in telecom even.

-Ben

> On Mar 18, 2020, at 9:58 AM, Paul Nash  wrote:
> 
> You just have to make sure that you test the right thing.
> 
> In a former life I was an electrical engineer. My first job was with a 
> consulting engineering firm; out biggest customer was the biggest supermarket 
> chain in South Africa.  One of my tasks was to travel to one of their stores 
> each Saturday after closing (those were the days when they closed at noon on 
> a Saturday until Monday morning) and test their stand generators.
> 
> The manager’s idea was usually to press the start button, check that the big 
> diesel started, then shut down and go home.  My idea was to pull the main 
> incoming breaker.  9 times out of 10 on first visit, the diesel would start, 
> and then die as soon as the load kicked in because of carbon buildup in the 
> cylinders.
> 
> After discussions with the supermarket management, they decided to (a) have 
> all the diesels serviced ASAP, and (b) adopt my protocol of start diesel, 
> wait for it to come under load, run for at least 30 minutes to get up to heat 
> and clear the carbon deposits.
> 
> I use a similar technique for failover tests on servers, routers, firewalls — 
> pull the power cord and see what happens, pull the incoming network and see 
> what happens.
> 
> This was stymied by a recent network outage where the ISP network was up and 
> running, connected back to their local PoP and thence to their backbone, but 
> connectivity from that network to the critical servers was down.  So now we 
> test end-to-end that the server is reachable, and let the network fail over 
> if not.
> 
>paul
> 
>> On Mar 18, 2020, at 11:56 AM, Karl Auer  wrote:
>> 
>> An untested emergency system has to be regarded as a non-existent
>> emergency system.
>> 
>> No matter how painful it is to test, no matter how expensive it is to
>> test, the pain and the expense are nothing compared to the pain and
>> expense of having an actual emergency and discovering that the
>> emergency system doesn't work...
>> 
>> Multiplied by infinity if it costs lives.
>> 
>> Regards, K.
>> 
>> -- 
>> ~~~
>> Karl Auer (ka...@biplane.com.au)
>> http://www.biplane.com.au/kauer
>> http://twitter.com/kauer389
>> 
>> GPG fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170
>> Old fingerprint: 8D08 9CAA 649A AFEF E862 062A 2E97 42D4 A2A0 616D
>> 
>> 
> 

Re: COVID-19 vs. our Networks

[Alexandre Petrescu]

I saw on TV official requests to police radio to remove masks (Idont 
know why, because they dont have any anyways)


Le 18/03/2020 à 16:29, Mark Tinka a écrit :


On 17/Mar/20 20:54, Dan White wrote:

  


Attackers taking advantage of this situation is a serious concern.

In South Africa, we have people claiming to be from the Department of
Health and one other reputable medical care group, going door-to-door
offering Coronavirus testing:


https://www.iol.co.za/dailynews/news/kwazulu-natal/south-africans-warned-about-door-to-door-coronavirus-test-scam-44958885


Be alert; the scammers are.

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 18/Mar/20 18:09, Jeff Shultz wrote:

> Is it so difficult to put an "override, but keep counting" button on a
> device like this?

Where's the money in that?

Mark.

Re: Quagga for production?

[Mark Tinka]

On 18/Mar/20 18:01, Nick Hilliard wrote:
 
>
> I used to use ISIS for this, but more recently moved to ebgp with
> 1s/3s timers.  The convergence characteristics are reasonable and as
> the only routing protocol dependence is bgp, we can use bird which in
> turn allow us to automate provisioning via saltstack.  Automating
> quagga and frr is hacky.

I prefer to have a number of core systems accessible in the IGP, because
BGP can sometimes get hosed for one reason or another.

BGP always needs IGP to work. The reverse is not true, and reduces us to
absolute basics when it hits the fan (which it has, a few times before).

Mark.

Re: DHS letters for fuel and facility access

[Paul Nash]

You just have to make sure that you test the right thing.

In a former life I was an electrical engineer. My first job was with a 
consulting engineering firm; out biggest customer was the biggest supermarket 
chain in South Africa.  One of my tasks was to travel to one of their stores 
each Saturday after closing (those were the days when they closed at noon on a 
Saturday until Monday morning) and test their stand generators.

The manager’s idea was usually to press the start button, check that the big 
diesel started, then shut down and go home.  My idea was to pull the main 
incoming breaker.  9 times out of 10 on first visit, the diesel would start, 
and then die as soon as the load kicked in because of carbon buildup in the 
cylinders.

After discussions with the supermarket management, they decided to (a) have all 
the diesels serviced ASAP, and (b) adopt my protocol of start diesel, wait for 
it to come under load, run for at least 30 minutes to get up to heat and clear 
the carbon deposits.

I use a similar technique for failover tests on servers, routers, firewalls — 
pull the power cord and see what happens, pull the incoming network and see 
what happens.

This was stymied by a recent network outage where the ISP network was up and 
running, connected back to their local PoP and thence to their backbone, but 
connectivity from that network to the critical servers was down.  So now we 
test end-to-end that the server is reachable, and let the network fail over if 
not.

paul

> On Mar 18, 2020, at 11:56 AM, Karl Auer  wrote:
> 
> An untested emergency system has to be regarded as a non-existent
> emergency system.
> 
> No matter how painful it is to test, no matter how expensive it is to
> test, the pain and the expense are nothing compared to the pain and
> expense of having an actual emergency and discovering that the
> emergency system doesn't work...
> 
> Multiplied by infinity if it costs lives.
> 
> Regards, K.
> 
> -- 
> ~~~
> Karl Auer (ka...@biplane.com.au)
> http://www.biplane.com.au/kauer
> http://twitter.com/kauer389
> 
> GPG fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170
> Old fingerprint: 8D08 9CAA 649A AFEF E862 062A 2E97 42D4 A2A0 616D
> 
> 

Re: Google Fiber (KC) NOC contact

[Louie Lee via NANOG]

Hey Blake,

Thanks for reaching out. I’m the IP Address Manager. Since this is more a
matter of the CPE or local gateway router configuration, I’ve referred the
matter to our Operation team to follow up on.

FYI, our frontline call center does escalate matters rather promptly after
a report is taken. They have very clear guidelines to escalate everything
that they cannot correct on their own.

Louie

P.S. Yes, I know I CC'ed the NANOG list. I trust y'all not to abuse things.

-- 

Louie Lee, 李景雲

Peering Coordinator (AS16591 )

Network Capacity Manager

IP Numbers Administrator

Google Fiber

lou...@google.com

(650) 253-2847

*There are 10 types of people in the world: Those who understand binary,
and those who don't.*


On Wed, Mar 18, 2020 at 7:51 AM Blake Hudson  wrote:

> Does someone from Google Fiber hang out on this list? I've contacted
> arin-cont...@google.com (the WHOIS tech and admin contact), but not
> gotten any response and I suspect contacting a frontline callcenter
> would be fruitless. It appears that some portion of customers in KC are
> being provided with a lame/broken NTP server address and it's causing
> some VPN/Telephony problems for those affected. If someone has a better
> POC please let me know so we can get this cleared up for us and other
> businesses in the KC area.
>
> --Blake
>

traffic sag last night (early this morning)

[Aaron Gould]

At 00:49 minutes past midnight today I saw a bit of a traffic sag across all
3 of my different upstream providers.  All in Texas.  Anyone else see that ?

 

-Aaron

Re: UDP/123 policers & status

[Saku Ytti]

On Wed, 18 Mar 2020 at 18:05, Ca By  wrote:

> Yeh, not changing ipv4 filters, Sorry pool. Burned once, twice shy.

On many edge routers from Juniper, Nokia and Cisco you can create
offset based bit-matches. I'm NTP illiterate, but isn't NTP mode in
fixed offset after UDP header? So it should be possible to do this in
hardware at linerate just the same as matching UDP port on typical
edge router.

-- 
  ++ytti

Re: COVID-19 vs. our Networks

[Jeff Shultz]

Is it so difficult to put an "override, but keep counting" button on a
device like this?

On Wed, Mar 18, 2020 at 8:04 AM Mark Tinka  wrote:
>
>
>
> On 17/Mar/20 20:06, Owen DeLong wrote:
>
>
> > I don’t get this… X-Ray machines (and other critical medical equipment) 
> > should operate in a fail-safe mode where a license screw up doesn’t prevent 
> > the machine from operating.
> >
> > If the hospital hasn’t paid up, find a way to go after the hospital, but 
> > don’t kill patients to collect your fee.
>
> For my very simple 1+1 mind, I totally agree.
>
> Perhaps, it's far easier to collect (overdue) fees with a gun to your
> head, if I don't actually need to point one at you.
>
>
>
> > Why should there be a license server at all? Why should an X-ray machine 
> > have an external dependency like that in the first place, even if it’s a 
> > local server?
>
> My Google OnHub wireless AP is completely unmanageable if I (against
> Google's advice) run it in Bridged mode. If I want to be able to reach
> it and manage it with an app or a web site, it needs to run as a router,
> even if all I want from it is to be an AP. You can guess who long mine
> have gone without a software update, then...
>
> Who knows why people come up with the BS they do?
>
> Mark.



-- 
Jeff Shultz

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: Quagga for production?

[Nick Hilliard]

Mark Tinka wrote on 18/03/2020 14:25:
At the moment, I run Quagga with OSPF and export that into my IS-IS core 
to drive Anycast services.


I used to use ISIS for this, but more recently moved to ebgp with 1s/3s 
timers.  The convergence characteristics are reasonable and as the only 
routing protocol dependence is bgp, we can use bird which in turn allow 
us to automate provisioning via saltstack.  Automating quagga and frr is 
hacky.


Nick

Re: UDP/123 policers & status

[Ca By]

On Wed, Mar 18, 2020 at 8:46 AM Steven Sommars 
wrote:

> The various NTP filters (rate limits, packet size limits) are negatively
> affecting the NTP Pool, the new secure NTP protocol (Network Time Security)
> and other clients.  NTP filters were deployed several years ago to solve
> serious DDoS issues, I'm not second guessing those decisions.  Changing the
> filters to instead block NTP mode 7, which cover monlist and other
> diagnostics, would improve NTP usability.
>
> http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf
>
>

Yeh, not changing ipv4 filters, Sorry pool. Burned once, twice shy.

There is no simple way to do router filters based on ntp app modes.

I suggest people be aware of time.google.com

And  time.cloudflare.com

CB


> On Tue, Mar 17, 2020 at 11:17 AM Mark Tinka  wrote:
>
>>
>>
>> On 17/Mar/20 18:05, Ca By wrote:
>>
>>
>>
>>
>> +1 , still see, still have policers
>>
>> Fyi, ipv6 ntp / udp tends to have a much higher success rate getting
>> through cgn / policers / ...
>>
>>
>> For those that have come in as attacks toward customers, we've "scrubbed"
>> them where there has been interest.
>>
>> Mark.
>>
>

Re: DHS letters for fuel and facility access

[Karl Auer]

An untested emergency system has to be regarded as a non-existent
emergency system.

No matter how painful it is to test, no matter how expensive it is to
test, the pain and the expense are nothing compared to the pain and
expense of having an actual emergency and discovering that the
emergency system doesn't work...

Multiplied by infinity if it costs lives.

Regards, K.

-- 
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer
http://twitter.com/kauer389

GPG fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170
Old fingerprint: 8D08 9CAA 649A AFEF E862 062A 2E97 42D4 A2A0 616D

Re: COVID-19 vs. our Networks

[Dan White]

On 03/18/20 09:29 -0500, Blake Hudson wrote:



On 3/17/2020 1:54 PM, Dan White wrote:

On 03/17/20 14:38 -0400, Rich Kulawiec wrote:

On Tue, Mar 17, 2020 at 08:38:28AM -0700, Mike Bolitho wrote:

That's the good news.   Here's the bad news: in about 2-3 weeks, when
our health care systems are stretched to the breaking point, there will
be a window of opportunity for adversaries to maximize the damage.


On a slightly tangential topic, we had a dictionary attack against 
customer voice accounts over night, presumably to implement toll fraud.

We were in the middle of working out work-from-home plans and were quite
distracted with other things. We managed to get on top of it quickly
once someone noticed.

Attackers taking advantage of this situation is a serious concern.

Dan, we're aware of another telco that ran into a similar fraud 
situation last week. They stood up some more restrictive ACLs to 
combat the fraud, but broke VoIP RTP in the process. 'Hit em while 
they're occupied' type of attacks I guess should be expected right 
now. As my grandmother would say: an ounce of prevention is worth a 
pound of cure.


Hey Blake,

I appreciate that. We've got two tendencies going on here at the moment:

1) Man the ship of operations. Stay alert and fix the problems that arise.
Be totally reactive, and be the "hero".

2) Increase visibility and focus on network design. Move planned upgrades
up a few weeks/months. Be proactive.

Option 2 is the better long term option, but the risk is that any change,
while short staffed is going to run the risk of unintended consequences. Basically it's 
"If it's not broke, don't fix it." and "Be very paranoid about what you touch.

--
Dan White
Network Admin Lead

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 18/Mar/20 16:35, Seth Mattinen wrote:

>
>
> Do all the SLA's in the world even matter if the contract has a force
> majeure clause?

Feel-good-tick-in-the-box type-thing... like that time a network
operator is asked if any part of their network/service touches any
equipment manufactured by a well-known Chinese OEM :-).

Tick in the box :-)...

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 18/Mar/20 13:24, Rich Kulawiec wrote:

> The use of "you/your" here and throughout is misplaced and inappropriate.
>
> Also: this not an isolated or unique experience.  It's this way pretty
> much everywhere in the US now.  And I can disapprove of it, you can
> disapprove of it, we can all disapprove of it, but like I said, until
> money is completely removed from the calculation, this is how it will be.
> Critiques of process and role and organization and everything else are
> interesting, maybe even correct --  but will change nothing.

Not just the U.S., mate. Not just the U.S.

Some families in Ethiopia and Indonesia would be happy to be proof.

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 18/Mar/20 11:43, Keith Medcalf wrote:

> No.  One simply has to assign a "cost" to "suitability for use".  For
> example, if you put out an RFQ for a CT Machine and someone bids a bag
> of peanuts for $1.50, that is probably the lowest bid, and that is what
> you will get if you choose based entirely on the lowest bid.  However,
> if you also require that the purchased machine also actually be capable
> of performing Computed Tomography then clearly that $1.50 bid will be
> rejected.
>
> You simply have to define what you want to achieve, then do it.

If only it were that simple, with 2020 corporate life.

What I can say tends to work is:

    "You simply have to define what you want to achieve, scream, yell and
 shout, then do it, then scream, yell and shout some more, until you
can't
    tell whether you'll leave the job from being fed up or being asked
to walk".
   
That has a slightly better chance of succeeding more than failing.

I'm old now - I ask once, maybe twice if I've had a beer. Then I carry
on, and we meet in 12 months when it all goes to hell :-).

Mark.

Re: UDP/123 policers & status

[Steven Sommars]

The various NTP filters (rate limits, packet size limits) are negatively
affecting the NTP Pool, the new secure NTP protocol (Network Time Security)
and other clients.  NTP filters were deployed several years ago to solve
serious DDoS issues, I'm not second guessing those decisions.  Changing the
filters to instead block NTP mode 7, which cover monlist and other
diagnostics, would improve NTP usability.

http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf

On Tue, Mar 17, 2020 at 11:17 AM Mark Tinka  wrote:

>
>
> On 17/Mar/20 18:05, Ca By wrote:
>
>
>
>
> +1 , still see, still have policers
>
> Fyi, ipv6 ntp / udp tends to have a much higher success rate getting
> through cgn / policers / ...
>
>
> For those that have come in as attacks toward customers, we've "scrubbed"
> them where there has been interest.
>
> Mark.
>

RE: COVID-19 vs. our Networks

[Keith Medcalf]

On Wednesday, 18 March, 2020 05:24, Rich Kulawiec  wrote:

>On Wed, Mar 18, 2020 at 03:43:37AM -0600, Keith Medcalf wrote:

>> So you failed because you did not require the person making the
>> decision to take responsibility for their decision.  That is, your
>> organization has a severely flawed process wherein the "R" for
>> making the decision is not the same person as has the "R" for
>> the repercussions.

>The use of "you/your" here and throughout is misplaced and
inappropriate.

It is the "Royal You".  However, you can replace that with generics if
y'all wish.

The point is that the root of the problem is the failure of the
organizational decision maker to take responsibility for their decision.

As a business person (now retired) once told me about the things he
sells in his shop, "I will not sell that here because in my opinion it
is crap.  If you want that, you can go to the shop next door.  They will
be quite willing to sell that crap to you, but don't come complaining to
me when your failure to take my advice comes back to bite you in the
ass."

>Also: this not an isolated or unique experience.  It's this way pretty
>much everywhere in the US now.  And I can disapprove of it, you can
>disapprove of it, we can all disapprove of it, but like I said, until
>money is completely removed from the calculation, this is how it will
be.
>Critiques of process and role and organization and everything else are
>interesting, maybe even correct --  but will change nothing.

Yes, it is generally an USian problem.  While I cannot speak to its
prevelance in the US I can attest to the fact that USians try to bring
this philosophy with them were ever they go and that such thinking has
to be repelled with large bats.

I have had to deal with such things several times and my response is
quite simple:  My name will not be associated in any way with that
stupidity other than complete opposition to it.  If you want me to sign
off on it, then I will not.  And if you decide to do it anyway then do
not ask me to have anything to do with the mess that ensues because the
only action you will get from me is "told you so -- you made your bed
now go sleep in it".

Generally the encroachment of ill-conceived plans is staved off until
the resistant retire leaving the inmates in charge of the asylum.

--
The fact that there's a Highway to Hell but only a Stairway to Heaven
says a lot about anticipated traffic volume.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 23:47, Rich Kulawiec wrote:

>
> Decisions are no longer based on the greater good or on anticipating worst
> case scenarios or on maximizing preparedness or anything that we might
> hope they're based on.  They're based, coldly and calculatingly, on money.
>
> If you want this to change -- and I sure would like it to change --
> then money needs to be entirely removed from that calculation.  That is
> a problem whose solution lies outside the scope of NANOG.
>


I've been saying, quietly to friends for a while now, that this system
of money is stretching all of us to the limit, because we are all
chasing it.

In this age of Coronavirus, what good is money if the supermarket
shelves are empty? What good is money if people are dead? Also, money
works because not all of us have it - and yet the goal of any society is
to put it in everyone's hands.

So then, what's the real value [of money] if everyone has the ability to
pay everyone?

Between the Coronavirus and the love spats between Russia and Saudi
Arabia, "artificial" markets have lost 20% of their value. Some
economies with less advanced markets are finding value in other "less
artificial" things, so they can get their toilet paper, hehe :-).


Seriously though, I echo Rich's comments - this isn't a problem NANOG
can solve, but if the Coronavirus has taught us anything, it's that we
seriously need to re-evaluate this "modern humanity" thing we are all
trying to build, and live.

Mark.

Re: COVID-19 vs. our Networks

[Anne P. Mitchell, Esq.]

> On Mar 18, 2020, at 9:24 AM, Mark Tinka  wrote:
> 
> 
> 
> On 17/Mar/20 20:35, Owen DeLong wrote:
> 
>> Step one:
>>  Consumers _AND_ especially mission critical consumers must start 
>> refusing to purchase devices which have inherent dependency on a 
>> vendor-cloud (or any cloud for that matter).
> 
> Good advice for mission-critical consumers. 

>> Stop treating things you don’t own and things that aren’t hosted locally as 
>> “reliable” and make sure that they are not in the mission critical chain of 
>> urgent patient care.

We have told our readers (and, really, anyone who will listen) for years that 
'the cloud' is just another term for 'somebody else's computer'.  Sometimes 
(often) people really need to hear it in such simple terms.

Anne

--
Anne P. Mitchell, Attorney at Law
Dean of Cyberlaw & Cybersecurity, Lincoln Law School
Policy Drafting and Review for Businesses
Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal anti-spam law)
Legislative Consultant, GDPR, CCPA (CA) & CCDPA (CO) Compliance Consultant
Board of Directors, Denver Internet Exchange

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 20:54, Dan White wrote:

>  
>
> Attackers taking advantage of this situation is a serious concern.

In South Africa, we have people claiming to be from the Department of
Health and one other reputable medical care group, going door-to-door
offering Coronavirus testing:

   
https://www.iol.co.za/dailynews/news/kwazulu-natal/south-africans-warned-about-door-to-door-coronavirus-test-scam-44958885

Be alert; the scammers are.

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 20:35, Owen DeLong wrote:

> Step one:
> Consumers _AND_ especially mission critical consumers must start
> refusing to purchase devices which have inherent dependency on a
> vendor-cloud (or any cloud for that matter).

Good advice for mission-critical consumers.

But the kids don't care how the information gets to them, as long as it
gets to them.


>
> Stop treating things you don’t own and things that aren’t hosted
> locally as “reliable” and make sure that they are not in the mission
> critical chain of urgent patient care.
>
> Anything in the healthcare vertical that is outside of the medical
> providers control/ownership is a result of the medical provider buying
> into that model on some level. STOP DOING THAT.
> (How am I suddenly reminded of the old adage “Doctor, doctor, it hurts
> when I do this!”…)
>
> I understand how the allure of lower costs and the frustration of
> “every vendor does this, we can’t find one who doesn’t” plays out.
> However, the only way “every vendor does it” will continue is if every
> vendor continues to be able to make sales without changing.

Product-based mindset from an industrial era is very hard to shake, even
though as consumers of commoditized information in 2020, ourselves, we
actually employ a value-based mindset for our personal consumption,
which we are unable to use to convert our own businesses into. Funny
that, eh...

Your competitor is no longer the shop down the road. It's anyone,
anywhere, with an Internet connection and an idea. No one is immune from
this, not even healthcare providers or the OEM's they choose to buy from.

Cutting costs is not how you stay relevant. But, it's what product-based
businesses know to do, because the alternative is simply too daring to
consider :-).

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 20:33, Emille Blanc wrote:

> In a world where you can license device performance by the megabit/sec/day, 
> or even have to purchase per-use factory reset keys since the manufacture has 
> stripped product owners of that right too, this doesn't totally surprise me.
>
> There would have to be a flip side to that coin - I would have to guess 
> (read: guess) it's a 'n' x-rays/day to "cut costs to the end user." Great 
> practice on paper for little guys, but beyond that...

In the industrial era, it was "knowledge & expertise". In the digital
era, it's "curiosity and creativity".

Access to information is ubiquitous and exponential. Knowledge has been
commoditized. How does that hurt the medical industry, you wonder? Well,
a webachondriac will use the Internet to easily self-diagnose, use an
app to order medication online, and have it delivered, all without ever
leaving his/her house.

How many businesses have lost out on his dime in that process? The local
GP down the corner. The local pharmacy up the corner. And all the supply
chain in between.

If traditional businesses don't adapt, they will become irrelevant.

Cutting costs (as the hospitals and, pretty much, any industry is doing)
is the first path to staving off the death spiral. And then the massacre
follows.

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 20:26, Shane Ronan wrote:

>  Because the hospitals don't own the machines and the companies that
> do, charge the hospital per x-ray. The hospitals moved to this model
> to reduce their costs during "quiet" periods. And by doing so, put
> their patients in jeopardy.

Can be said of, pretty much, any industry in 2020 that sells products
(and not value).

Remember that plane that was designed -MAX? Wonder how that happened.

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 20:06, Owen DeLong wrote:


> I don’t get this… X-Ray machines (and other critical medical equipment) 
> should operate in a fail-safe mode where a license screw up doesn’t prevent 
> the machine from operating.
>
> If the hospital hasn’t paid up, find a way to go after the hospital, but 
> don’t kill patients to collect your fee.

For my very simple 1+1 mind, I totally agree.

Perhaps, it's far easier to collect (overdue) fees with a gun to your
head, if I don't actually need to point one at you.



> Why should there be a license server at all? Why should an X-ray machine have 
> an external dependency like that in the first place, even if it’s a local 
> server?

My Google OnHub wireless AP is completely unmanageable if I (against
Google's advice) run it in Bridged mode. If I want to be able to reach
it and manage it with an app or a web site, it needs to run as a router,
even if all I want from it is to be an AP. You can guess who long mine
have gone without a software update, then...

Who knows why people come up with the BS they do?

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 19:56, Alexandre Petrescu wrote:

>  
>
> I buy newspaper every Saturday and every Tuesday since some time now. 
> In addition to local news and The Economist, I include NYTimes
> International edition because thats the only USA thing in my very
> small local news stand in small city.  Different places in the world
> have different options for USA newspapers .

Good for you.

For the rest of us who'd rather trade memes about whether a president is
telling the truth or not, with each tweet, we put a newspaper,
somewhere, out of business.

For better or worse, can't blame the Twitter :-\...

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 19:46, Mike Bolitho wrote:

>
> I totally agree and 99.999% of the time, congestion on the Internet is
> a nuisance, not a critical problem. I'm not sitting here complaining
> that my public internet circuits don't have SLAs or that we run into
> some packet loss and latency here and there under normal operations.
> That's obviously to be expected. But this whole topic is around what
> to do when a once in a lifetime pandemic hits and we're faced with
> unseen levels of congestion across the country's infrastructure. I
> mean the thread is titled COVID-19 Vs Our Networks. That's why I
> brought up the possible application of TSP to tell some of the big
> CDNs that maybe they should limit 4K streaming or big DLCs during a
> pandemic. That's it. And yet I'm getting chastised (not necessarily by
> you) for suggesting that hospitals, governments, water treatment
> plants, power plants, first responders, etc are actually more
> important during times like this.

To me, sounds like a potential business case for an existing or new CDN
provider focused squarely on healthcare, and other such critical
services :-).

As is always the case with invention, "I didn't like what I found, so I
built a better one".

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 19:43, Keith Medcalf wrote:

>
> If by "device" you mean "computer", then you are correct.

"A computer? What's that?" said the kids :-).


> Never in 57 years.

You caught it early :-).


> Never because I don't have any.  But I don't either.  Babbling idiots don't 
> do anything for me.
>
> And before you ask, I get "important news" directly.  If the building next 
> door falls over, I notice.  Otherwise I don't think there *IS* such a thing 
> as *important news*, or I can only think of a couple of "important news" that 
> have happened in my entire lifetime on one hand.  In no case was a babbling 
> idiot or propaganda purveyor of any particular use.

Even if you're a generation or two behind, you're not that far off from
how the kids treat acquisition of information in 2020 :-).


> Never used any of those.  They are just hangouts for yet more babbling 
> idiots.  Some of them are even named appropriately -- like Twitter -- which 
> as I understand it is the place where all the twits congregate.

And it's what's driving all this madness. Sadly, if we want to keep
eating, we have no choice but to follow it and engage with the kids.


> Correct.  No value there.  Just more babbling idiots.

Again, just what the kids feel. Are you sure you're 57 :-)?


> I have an e-mail app on my phone that is connected to my (not someone else's) 
> e-mail server that handles e-mail, contacts, and calendaring in a distributed 
> fashion that is the same on every "device" I own.  If a device will not work 
> with my e-mail server, does not function as I need it to function, or is not 
> safe and secure to my requirements, I do not buy that device (that means that 
> the list of devices that I refuse to buy and will not permit in the same room 
> as me is VERY VERY VERY long).  Most of the other rubbish has been banished 
> because it is nothing more than yet more piles of babbling idiots.

In that way, you're on your own.

Most kids aren't into e-mail; perhaps only because they need one to
install and use Instagram :-).


> Send e-mail.  Or provide an e-mail list.  I will not fiddle faddle with going 
> to websites chock full of malicious websites nor will I let any Tom Dickhead 
> send their malicious crap to me.  By the time the malicious crap infestation 
> is filtered out, there is nothing left.
>
> Then again I am an old fart.

You may very well be, but your perception of value (although in the
alternate universe) mirrors how the kids treat the networks we build for
them today. They don't care about your products. They just want to
achieve their value.

Telco's (and ISP's) are very product-based organizations. "Here's a list
of products, they each cost that, tell me which one you want", said the
Head of Sales. But all the kids want to do is share a bunch of
champagne-popping videos on Instagram, force a bank manager to return
wrongly-billed fees via Twitter, and show the world how badly a police
chase ended on WhatsApp. They don't care whose network they use to share
what gives them "value", and the first chance they get, they will switch
away from your sacred 5G to some random wi-fi hot spot.

What's even wilder is that as consumers of these apps ourselves, we
emulate a value-based need rather than one based on products. But
somehow, as service providers and businesses, we do not seem to know how
to offer value in lieu of product. It fascinates me.

Mark.

Google Fiber (KC) NOC contact

[Blake Hudson]

Does someone from Google Fiber hang out on this list? I've contacted 
arin-cont...@google.com (the WHOIS tech and admin contact), but not 
gotten any response and I suspect contacting a frontline callcenter 
would be fruitless. It appears that some portion of customers in KC are 
being provided with a lame/broken NTP server address and it's causing 
some VPN/Telephony problems for those affected. If someone has a better 
POC please let me know so we can get this cleared up for us and other 
businesses in the KC area.


--Blake

Re: COVID-19 vs. our Networks

[Tom Beecher]

Depends on the verbiage of the clause.

On Wed, Mar 18, 2020 at 10:41 AM Seth Mattinen  wrote:

> On 3/17/20 10:03 AM, Mike Bolitho wrote:
> >
> > We have two redundant private lines out of each hospital connecting back
> > to primary and DR DCs and a metro connecting everything together in each
> > region. But for things we do not own that are not hosted locally, what
> > are we supposed to do? We have to go out DIA to get there. Everything we
> > own is connected via fully SLAed private lines. We have zero issues
> > there. I think people vastly underestimate just how much in the
> > healthcare vertical is outside of a medical providers control/ownership.
> >
>
>
> Do all the SLA's in the world even matter if the contract has a force
> majeure clause?
>

Re: Quagga for production?

[Jens Link]

Mark Tinka  writes:

> On 17/Mar/20 19:39, Jens Link wrote:
>
>>
>> Jens, using frr for quite some time now without any problems
>
> IS-IS, per chance?

Sorry, only BGP for now.

Jens
-- 

| Delbrueckstr. 41| 12051 Berlin, Germany   | +49-151-18721264 |
| http://blog.quux.de | jabber: jensl...@quux.de| ---  | 

Re: COVID-19 vs. our Networks

[Seth Mattinen]

On 3/17/20 10:03 AM, Mike Bolitho wrote:


We have two redundant private lines out of each hospital connecting back 
to primary and DR DCs and a metro connecting everything together in each 
region. But for things we do not own that are not hosted locally, what 
are we supposed to do? We have to go out DIA to get there. Everything we 
own is connected via fully SLAed private lines. We have zero issues 
there. I think people vastly underestimate just how much in the 
healthcare vertical is outside of a medical providers control/ownership.





Do all the SLA's in the world even matter if the contract has a force 
majeure clause?

Re: Quagga for production?

[Mark Tinka]

On 17/Mar/20 19:39, Jens Link wrote:

>
> Jens, using frr for quite some time now without any problems

IS-IS, per chance?

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 19:35, Tom Beecher wrote:
> You're facing essentially the same issue as many in non-healthcare do
> ; how to best talk to applications in Magic Cloud Land. Reaching the
> major cloud providers does not require DIA ; they all have presences
> on the major IXes, and direct peering could be an option too depending
> on your needs and traffic. 
>
> I don't mean to be dismissive of the issues you face, I apologize if
> that's how it comes off. What you describe is certainly challenging,
> but I think that you will have better success with some of the options
> that are out there already than hoping for any resolution of
> intermittent congestion issues in the wild west of the DFZ.

Sounds like a use-case for the cloud providers' so-called "Express
Route" services. But that will only be really successful if the majority
of the services that the hospitals need are hosted on the cloud
platforms that offer these Express Route things.

Then again, the "private" link between a cloud provider and their
customer is typically an MPLS-based one, which is subject to emergent
issues that may impact the operators' backbone.

If the service is not on a cloud provider (or one that can offer an
Express Route thing), then we're back to square one.

Mark.

Re: COVID-19 vs. our Networks

[Blake Hudson]

On 3/17/2020 1:54 PM, Dan White wrote:

On 03/17/20 14:38 -0400, Rich Kulawiec wrote:

On Tue, Mar 17, 2020 at 08:38:28AM -0700, Mike Bolitho wrote:

Anybody who works in the healthcare vertical will tell you just how
bad medical devices are to work with from an IT perspective.


Medical devices are appallingly bad to work with from an IT perspective.

They're designed and built to work in idealized environments that don't
exist, they make unduly optimistic assumptions, they completely fail to
account for hostile actors, and whenever possible they are gratuitously
incompatible to ensure vendor lock-in.

That's the good news.   Here's the bad news: in about 2-3 weeks, when
our health care systems are stretched to the breaking point, there will
be a window of opportunity for adversaries to maximize the damage.


On a slightly tangential topic, we had a dictionary attack against 
customer

voice accounts over night, presumably to implement toll fraud. We were in
the middle of working out work-from-home plans and were quite distracted
with other things. We managed to get on top of it quickly once someone
noticed.

Attackers taking advantage of this situation is a serious concern.

Dan, we're aware of another telco that ran into a similar fraud 
situation last week. They stood up some more restrictive ACLs to combat 
the fraud, but broke VoIP RTP in the process. 'Hit em while they're 
occupied' type of attacks I guess should be expected right now. As my 
grandmother would say: an ounce of prevention is worth a pound of cure.

Re: Quagga for production?

[Mark Tinka]

On 17/Mar/20 19:18, Hiers, David wrote:
>
> Quagga is built into one of our core products, works great.   That
> particular vendor a sponsor of frr, and is replacing quagga with frr soon.
>
>  
>
> Maybe look at the vendor/partner list for quagga and frr, and decide
> which project has better long-term prospects.
>

I've been meaning to test FRR for a year or so now, so I can get proper
native IS-IS support.

At the moment, I run Quagga with OSPF and export that into my IS-IS core
to drive Anycast services.

Anyone on FRR happy with dual-stack IS-IS there, talking to Cisco and
Juniper implementations?

Mark.

Re: DHS letters for fuel and facility access

[Mark Tinka]

On 17/Mar/20 19:08, Warren Kumari wrote:

>
> We had specified that the transfer pump be on the generator feed,
> there was a schematic showing at is being on the generator feed, there
> was even a breaker with a cable marked  "Transfer Pump (HP4,5)" ---
> but it turned out to just be a ~3ft piece of cable stuffed into a
> conduit, and not actually, you know, running all the way down to the
> basement and connected to the transfer pump.

It's the things you rarely think about.

The Fukushima Daiichi nuclear power plant backup generator situation
comes to mind as well.

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 17/Mar/20 19:03, Mike Bolitho wrote:

> I keep seeing this over and over again in this long thread. What's
> your suggestion? How does a hospital, with dozens of third party
> applications/devices across multiple cloud platforms do this?
>
> We have two redundant private lines out of each hospital connecting
> back to primary and DR DCs and a metro connecting everything together
> in each region. But for things we do not own that are not hosted
> locally, what are we supposed to do? We have to go out DIA to get
> there. Everything we own is connected via fully SLAed private lines.
> We have zero issues there. I think people vastly underestimate just
> how much in the healthcare vertical is outside of a medical providers
> control/ownership.

On my WhatsApp profile, one of my tag lines is "Hater of the 'What Do
You Do?' culture and WhatsApp Calling". I detest both equally.

But focusing on the latter, if you call me on WhatsApp, I'll cut you off
and call you back via GSM. The only time I'll entertain any WhatsApp
calls is if GSM coverage is poor, or if I know you are traveling and
can't roam, but have wi-fi.

I'd never blame my mobile providers for poor quality WhatsApp calls, nor
would I do the same to my ISP. I have zero patience for WhatsApp voice
calls to sort themselves out when initiated, and yet plenty of people
enjoy using it for whatever reasons, mostly to "save money". Personally,
wasting time exchanging "Can you hear me now?" is more costly than
having a short and concise call over GSM. If we are going to talk for
hours, let's have a beer.

The point is, as much as some "critical" conversations (want to) take
place on WhatsApp, Facebook have zero control of the quality of that
experience once the bits leave their data centre. I don't know if they
will ever fix that given all the variables that exist thousands of miles
from where the service is hosted, but you might not be forgiven for
thinking you can run a voice-based business on WhatsApp. In fact,
recording a voice note and sending it via WhatsApp is like two-way
walkie-talkie radio, but perhaps more reliable :-).

I really don't know how to fix this for hospitals relying on best-effort
infrastructure to deliver critical, priority services to their patients.

Mark.

Re: COVID-19 vs. our Networks

[Rich Kulawiec]

On Wed, Mar 18, 2020 at 03:43:37AM -0600, Keith Medcalf wrote:
> So you failed because you did not require the person making the decision
> to take responsibility for their decision.  That is, your organization
> has a severely flawed process wherein the "R" for making the decision is
> not the same person as has the "R" for the repercussions.

The use of "you/your" here and throughout is misplaced and inappropriate.

Also: this not an isolated or unique experience.  It's this way pretty
much everywhere in the US now.  And I can disapprove of it, you can
disapprove of it, we can all disapprove of it, but like I said, until
money is completely removed from the calculation, this is how it will be.
Critiques of process and role and organization and everything else are
interesting, maybe even correct --  but will change nothing.

---rsk

RE: COVID-19 vs. our Networks

[Keith Medcalf]

On Tuesday, 17 March, 2020 15:48, Rich Kulawiec  wrote:

>On Tue, Mar 17, 2020 at 11:35:59AM -0700, Owen DeLong wrote:

>> Anything in the healthcare vertical that is outside of the medical
>> providers control/ownership is a result of the medical provider
>> buying into that model on some level. STOP DOING THAT.  (How am I
>> suddenly reminded of the old adage ???Doctor, doctor, it hurts when I
>> do this!??)

>> I understand how the allure of lower costs and the frustration of
>> ???every vendor does this, we can???t find one who doesn???t???
>> plays out.
>>> However, the only way ???every vendor does it??? will continue
>> is if every vendor continues to be able to make sales without
>> changing.

>Fought this battle, lost this battle.

>Why?

>Because the people with the authority to make purchasing decisions are
>not the people who will be on the phone to some vendor's tech support
at
>3 AM on a Sunday morning, frantically pleading with them to fix a
problem
>because they really need that piece of equipment to work right now.

So you failed because you did not require the person making the decision
to take responsibility for their decision.  That is, your organization
has a severely flawed process wherein the "R" for making the decision is
not the same person as has the "R" for the repercussions.

>Decisions are no longer based on the greater good or on anticipating
>worst case scenarios or on maximizing preparedness or anything that
>we might hope they're based on.  They're based, coldly and
calculatingly,
>on money.

No, they are based on whatever the specification for making decisions
happens to be.  If you have chosen that basis to be "cheapest bidder",
then that is what you can expect to receive.

>If you want this to change -- and I sure would like it to change --
>then money needs to be entirely removed from that calculation.  That is
>a problem whose solution lies outside the scope of NANOG.

No.  One simply has to assign a "cost" to "suitability for use".  For
example, if you put out an RFQ for a CT Machine and someone bids a bag
of peanuts for $1.50, that is probably the lowest bid, and that is what
you will get if you choose based entirely on the lowest bid.  However,
if you also require that the purchased machine also actually be capable
of performing Computed Tomography then clearly that $1.50 bid will be
rejected.

You simply have to define what you want to achieve, then do it.

--
The fact that there's a Highway to Hell but only a Stairway to Heaven
says a lot about anticipated traffic volume.

Re: DHS letters for fuel and facility access

[Mark Tinka]

On 17/Mar/20 18:44, Paul Nash wrote:

> September 2001.  Just after the 9/11 attacks, all of lower Manhattan was shut 
> down.  Out link (IIRC) was to a satellite farm on Staten island, across the 
> bay to 60 Hudson.  Power went off, diesels kicked in, fuel trucks was not 
> allowed in, and a few days later we lost all international connectivity.
>
> Lots of important people lost power as well, so the feds decided to let the 
> diesel tankers in after a few days’ deliberations.

Ah, okay. That must have been unique to South Africa, I imagine?

My recollection during that month was we still had connectivity. We
routed our services via PanAmSat's PAS-3R (which later became Intelsat
3R after the acquisition) and landed in their Atlanta teleport. We
weren't impacted, in Uganda, at the time, nor I recall any other outages
in Kenya, Tanzania or Rwanda either.

I'm reminded how, from Uganda, PAS-3R was such a shallow bird, ±6° :-).

Mark.