Re: AS hijacking (Philosophy, rants, GeoMind)

[Sriram, Kotikalapudi (Fed) via NANOG]

Mike,

>As our canned Email stated, AS2 (and many low digit AS') get hijacked and
>often go on to hijack someone's prefix.  AS2 (proper) is rarely changed and
>the chances of an actual prefix hijack from it is extremely low.
>
>So as I've asked our peers, I'll ask here: What is expected of us to be good
>"Net Citizens" with these hijacks?

Thoughts on AS hijack prevention:
With RPKI-based route origin validation (ROV), it turns out that incremental 
solution for prefix hijacking is also an incremental solution for AS hijacking. 
For example -- assuming Invalid routes will be dropped -- if 70% of the 
announced prefixes are protected by ROAs, then those 70% prefixes cannot be 
hijacked with a hijacked AS. (Note: An exception to this is -- a deliberate 
hijacker can still perform what is called forged-origin hijack [1], i.e., the 
attacker matches the hijacked prefix with a hijacked AS such that they both 
belong to the same ROA.)  So, the community should keep pushing ahead with ROA 
and RPKI-based ROV deployments to achieve 100% ROA coverage for announced 
prefixes and also 100% dropping of Invalid. 

The above can also be said about “trusted” IRR-based (or IRR+RPKI based) ROV 
[1]. However, priority should be given to speedup the RPKI/ROA deployment 
towards full adoption.

FYI... Worldwide ROA coverage is currently at 20% for globally routed prefixes.
https://rpki-monitor.antd.nist.gov/

Security guidance regarding route objects in IRR, ROAs in RPKI, and ROV 
deployment can be found here:
[1] “Resilient Interdomain Traffic Exchange: BGP Security and DDoS Mitigation,” 
NIST Special Publication, NIST SP 800-189, December 2019. 
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-189.pdf  
Also, look up:
[2] MANRS: https://www.manrs.org/ 

Thank you.

Regards,
Sriram

Re: Network card with relay in case of power failure

[William Herrin]

On Wed, Jun 17, 2020 at 1:15 PM Dovid Bender  wrote:
> I am sorry if this is off topic.I was once demoed a network device
> that had two interfaces. The traffic would go through the device.
> If there was a power cut or some other malfunction there would be
> a relay that would physically bridge the two network interfaces so the
> traffic would flow as if it was just a network cable. Is anyone aware of
> such a network card or device?

You can also do this with any routing protocol by including a bypass
link and declaring the bypass higher cost than the one through your
potentially malfunctioning device. This has the benefit of not having
to care about whether the nature of the failure will affirmatively
trigger the bypass since it'd negatively triggered on the absence of
packets.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Quality of the internet

[Ben Cannon]

For safety!

Reminds me of bonding channels in an ISDN line.   We had to keep them all 
apart. 

For their own protection.

-Ben

> On Jun 18, 2020, at 6:18 AM, Mark Tinka  wrote:
> 
> 
> 
>> On 18/Jun/20 14:49, Bill Woodcock wrote:
>> 
>> What time was that?
> 
> Back when a 12000 GSR chassis had one line card in slot 0 for the public
> Internet, and another in slot 5 for the MPLS backbone. They had to be
> that far apart, for safety :-)...
> 
> Mark.
> 

Re: Network card with relay in case of power failure

[Warren Kumari]

On Wed, Jun 17, 2020 at 4:15 PM Dovid Bender  wrote:
>
> Hi,
>
> I am sorry if this is off topic.I was once demoed a network device that had 
> two interfaces. The traffic would go through the device. If there was a power 
> cut or some other malfunction there would be a relay that would physically 
> bridge the two network interfaces so the traffic would flow as if it was just 
> a network cable. Is anyone aware of such a network card or device?

Yup -- the Peribit devices used to do this - one of them died, so I
decided to repurpose it into a webserver.
Their implementation seemed to just be a PCI card and some relays -
Ethernet went into RJ-45 jacks on the relay card, and PCI voltage held
the relays "on", routing the Ethernet to an internal NIC. When the
device lost power the relays would close, and bypass the device. The
PCI card also had some chips on it, but I never bothered trying to
figure out if they were for heartbeat, or just power conditioning,
etc.

(I was going to reply to this thread yesterday and include photos, but
I think I've thrown the cards out... )
W

>
> TIA.
>
>


-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

Re: Router Suggestions

[Mark Tinka]

On 18/Jun/20 15:26, Warren Kumari wrote:

> Ah, because, if you word / negotiate your contract carefully, the
> failure to meet the 24/7 SLO can be converted into credit -- either
> actual discounts or simply a big stick when negotiating new stuff.
> Many years ago I worked for a company who repeatedly tripped over
> their own feet - but the one good thing that they actually managed was
> to have a good clause in their support contract -- we got free 24/7
> support for 3 years in a row because the contact had a "if you miss
> the response time more than N% of the time, we ain't gonna pay"
> clause. We had many locations in the USA, but also in Bangalore,
> Chennai, Hyderabad, Paris, London, Mumbai, and Marseille - for some
> reason Marseille was almost always the winner in terms of missing the
> SLO.

Conniving... I like it :-).

What was the saying... "Huge telco's are law firms masquerading as
connectivity providers", or something along those lines :-).

Mark.

Re: Quality of the internet

[Mark Tinka]

On 18/Jun/20 14:56, Saku Ytti wrote:

> Somewhere between 2000..2005 I personally still delivered customer
> connections that needed that. But we were providing 64kbps still to
> some odd locations, like paper mill in the middle of nowhere. I also
> needed to do MLPPP over 2*64kbps so that serialising single 1500B
> doesn't take too long (PPP could fragment it to two and send parallel,
> improving UX).

VoIP was legalized in South Africa in 2005.

The moment that happened, VoIP operators sprung up, and businesses began
dumping POTS services and moved over to VoIP. In those days, a 64Kbps
leased line was the gold standard; major props if you had anything more
than that; bow-downs if you had 256Kbps or 512Kbps.

We're talking +/- US$1,500/month for a 64Kbps at the time, when the
US$-ZAR exchange rate was 1:6.65.

Customers were willing to pay all that cash back then, because all these
shiny new TDP-based (Tag Distribution Protocol, for the ones who
remember, before it became the LDP standard) MPLS networks were the
guarantors of QoS, and to ensure your VoIP service always received a
steady 16Kbps to deliver two simultaneous phone calls between Jo'burg
and Durban, cash left wallets.

It was still cheaper than paying the telco for an E1. And of course,
there was an eerie eagerness to stick it to the telco :-).

Oh, how far we've come.

Mark.

Re: Router Suggestions

[Warren Kumari]

On Thu, Jun 18, 2020 at 6:26 AM Mark Tinka  wrote:
>
>
>
> On 18/Jun/20 00:50, Warren Kumari wrote:
>
>
> A number of customers (myself included) had 4 hour replacement contracts, 
> which the vendor really could not meet - so we agreed to take a new, much 
> larger/better model as a replacement.
>
>
> It's one of the reasons we never pay for 24/7/365.
>
> In many cases for our experience, with sufficient pre-built redundancy, there 
> isn't much different with 24/7 vs. NBD, apart from the cost. So why pay for 
> the premium.

Ah, because, if you word / negotiate your contract carefully, the
failure to meet the 24/7 SLO can be converted into credit -- either
actual discounts or simply a big stick when negotiating new stuff.
Many years ago I worked for a company who repeatedly tripped over
their own feet - but the one good thing that they actually managed was
to have a good clause in their support contract -- we got free 24/7
support for 3 years in a row because the contact had a "if you miss
the response time more than N% of the time, we ain't gonna pay"
clause. We had many locations in the USA, but also in Bangalore,
Chennai, Hyderabad, Paris, London, Mumbai, and Marseille - for some
reason Marseille was almost always the winner in terms of missing the
SLO.

W

>
> And if a site does not require redundancy, it's cheaper to have a cold 
> standby than to give it 24/7/365, or even NBD.
>
> Mark.



-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

Re: Quality of the internet

[Mark Tinka]

On 18/Jun/20 14:49, Bill Woodcock wrote:

> What time was that?

Back when a 12000 GSR chassis had one line card in slot 0 for the public
Internet, and another in slot 5 for the MPLS backbone. They had to be
that far apart, for safety :-)...

Mark.



signature.asc
Description: OpenPGP digital signature

Re: Quality of the internet

[Saku Ytti]

On Thu, 18 Jun 2020 at 15:49, Bill Woodcock  wrote:

> > On Jun 18, 2020, at 2:28 PM, Saku Ytti  wrote:
> > No one needs strict priority queues anymore, which was absolutely
> > needed at one point in time.
>
> What time was that?

Somewhere between 2000..2005 I personally still delivered customer
connections that needed that. But we were providing 64kbps still to
some odd locations, like paper mill in the middle of nowhere. I also
needed to do MLPPP over 2*64kbps so that serialising single 1500B
doesn't take too long (PPP could fragment it to two and send parallel,
improving UX).

-- 
  ++ytti

Re: Devil's Advocate - Segment Routing, Why?

[Mark Tinka]

On 18/Jun/20 14:30, adamv0...@netconsultings.com wrote:

> Hence our current strategy is to stay on IPv4 control-plane (and IPv4 
> management plane) as it suits, and for the foreseeable future will suite, all 
> our needs (which are to transport v4 data packets via L2 MPLS VPN 
> services), there are simply more important projects than to experiment with 
> v6 control-plane, like for instance perfecting/securing the v6 customer 
> facing services (delivered over the underlying v4 signalled MPLS 
> infrastructure, that no customer really cares about). 

Fair enough.


> But I understand your frustrations case it seems like you're taking the 
> bullet for us late adopters and in a sense you are, cause say in 10 years 
> from now when I decide to migrate to v6 control-plane and management-plane as 
> then it might be viewed as common courtesy, it will be all there on a silver 
> plate waiting for me allowing for a relatively effortless and painless move. 
> All thanks to you fighting the good fight today.

You better hope and pray I don't run out wine. Equipment manufacturers
make me drink, and I like my wine :-).


> And I'd say the future is now, cause there is an actual need for v6 services. 
> But need for v6 control & management plane? - It's not like operators are 
> losing business opportunities not having that. (they might even be viewed as 
> conservative->stable, which might be preferred by some customers). 

Well, the other way to look at it, especially if you are a Broadband or
mobile network operator, is what your plan is when you can no longer
stretch the IPv4 you have, can no longer obtain IPv4 from an RIR, and
can't afford to buy IPv4 on the open market.

For mobile operators, paying US$50 million/year in CGN line cards and
licensing is not even a rounding error on the books. But the telco space
has been under pressure for some time now, further amplified and
accelerated this Coronavirus pandemic. Even though mobile networks are
ATM machines printing money for the shareholders, they probably made
more money in the days of SMS than they do now building and selling
4G/5G packet cores. At some point, that US$50 million/year is going to
start getting some ex-co and Board level visibility, as capex spend
begins to pinch revenue because of the data demands of subscribers, and
the ever-falling ARPU's to go along with it.

Perhaps, at that point, massive IPv6 deployment in the mobile space is
what will wake everyone else up, as the race to grab on to every $$
tightens.

Mark.

Re: Quality of the internet

[Max Tulyev]

Hi,

in our region (CIS, eastern Europe) we still have issues
with overloaded international transport and bad quality of international 
channels from time to time (especially at the beginning of COVID19).


While Internet looks slow, but still usable, this case VoIP goes really bad.

Our regional specific is strong and very cheap internal (inside country) 
connectivity. So one of solution can be join local IXes by dedicated L2 
(DWDM) channels.


Ask me off-list if you want some help/solutions ;)


17.06.20 23:47, Dovid Bender пише:

Hi,

My 9-5 is working for a VoIP provider. When we started in 2006 we had a 
lot of issues with the quality of the internet in eastern europe and 
central Asia. It was not rare for us to have to play around with routing 
to get the quality that we needed. In a review of tickets for the last 
two years it seems as if we barely do any of that these days. Rarely do 
we get a quality complaint that comes back to an issue where a carrier 
or ISP dropping or mangling packets. Has anyone else observed this as well?

Re: Mikrotik RPKI Testing

[Mike Hammett]

This link will take you to their "suggest a feature" section. 


https://help.mikrotik.com/servicedesk/customer/portal/1/create/6 









- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Bryan Fields"  
To: "Nanog"  
Sent: Wednesday, June 17, 2020 9:50:57 PM 
Subject: Re: Mikrotik RPKI Testing 

On 6/17/20 10:38 PM, Musa Stephen Honlue wrote: 
> Did you face any issues with IPv6 on 6.4, I personally have participated in 
> deployment projects on Mikrotik for many large networks. 
> 
> And it worked well in the end. 

The problem I ran into was having it support SLAAC for assignment of IP 
addresses for management to a management vlan. We have a number of them setup 
as bridges, and use ipv4 for management now, but can't seem to make them 
configure IPv6. 

I've run into several issues with them doing bridging as well. Perhaps the 
worst is there's still no way to associate a MAC with a bridge MAC. This 
means we can isolate problem MAC's on an AP level, but then have to dig into 
the FDB of each individual node on that AP. 

These aren't ideal, but at the price point, we put up with the issues. :) 

-- 
Bryan Fields 

727-409-1194 - Voice 
http://bryanfields.net 

Re: Quality of the internet

[Bill Woodcock]

> On Jun 18, 2020, at 2:28 PM, Saku Ytti  wrote:
> No one needs strict priority queues anymore, which was absolutely
> needed at one point in time.

What time was that?

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: Quality of the internet

[Mark Tinka]

On 18/Jun/20 14:28, Saku Ytti wrote:


> ACK. Good Internet is almost an emergent feature, not something we
> really designed for. The main remaining problems are congested
> peerings, which is a silly political problem which ends up hurting
> customers and not helping anyone.

It's easier to keep selling bandwidth to people than to find another
model from which to make money. That, as network operators, is our fate :-).


> No one needs strict priority queues anymore, which was absolutely
> needed at one point in time.
>
> We are not in a market which cares about QoS,...

I was just thinking about this 2 years ago, when we sold more and more
IP services than anything else, despite a market with aggressive price
points, e.t.c., to the extent that while we still build all our nodes
with PHB-DSCP/EXP, with all the usual EF, AF, BE queues and 33% policing
on EF queues and LLQ forwarding on EF queues, and all the rest, in
practice, we don't really need them anymore.

We either over-provision capacity to the point where those QoS policies
never kick in, or (and more likely), all traffic is public Internet,
which lives in BE.

Basic policing/shaping/queueing of customer traffic at the edge is
pretty stable; we haven't needed a new QoS feature in that space for
over 8 years. So when vendors are trying to sell new line cards with
enhanced QoS scale, it makes me wonder. Unless it's for BNG deployments
where millions of customers need to be dumped in specific queues, which
isn't for us, and which I doubt many of the up-and-coming mom & pop FTTH
service providers can afford anyway.


>  yet our BE is globally
> <200us max jitter on a typical day and AF is <50us. Average jitter
> being under 10us. So if I'd have HW timestamping NTP server and
> client, I could synchronise clocks over IP transit cross continents to
> ten microsecond accuracy. I think this is pretty crazy. And I'm sure
> anyone who measures, measures similar numbers, this would have sounded
> scifi 20 years ago.
> As a context, Zoom recommends a jitter of 40ms or better, or 4us.

Which is a good point.

Sitting at my house in Jo'burg, my Zoom calls are typically served out
of some data centre in Paris (161ms from my house) or Amsterdam (176ms
from my house). I've tracked this for months now, my jitter on Zoom
calls is 1ms - 2ms, steady.

The case for EF queues to deliver VoIP calls between a customer and PABX
sitting 1ms apart simply doesn't track anymore. Either the network
already does it due to all the over-engineering, or the traffic goes
over the Public Internet anyway as folk migrate for cost, convenience
and value reasons.

Mark.

Microsoft AS8075 contact?

[Bryan Holloway]

Hello ...

If anyone from Microsoft peering is lurking, I could use an assist.

We have a reachability issue in Chicago.

E-mail to their PeeringDB NOC contact have gone unanswered.

Thank you!

RE: Devil's Advocate - Segment Routing, Why?

[adamv0025]

> From: Mark Tinka 
> Sent: Thursday, June 18, 2020 12:51 PM
> 
> On 18/Jun/20 13:23, adamv0...@netconsultings.com wrote:
> 
> >  is the current state is not the end state, this is a pretty dynamic 
> > industry
> that I'm sure is converging/evolving towards a v4:v6 parity, however the pace
> may be, which is understandable considering the scope of ground to be
> covered.
> 
> Which I am fine with  - if you give me a time line to say LDPv6,
> SR-OSPFv3 and SR-ISISv6 will be available on X date, I can manage my
> operation and expectations accordingly.
> 
> But if you say, "No LDPv6, no SR-OSPFv3, no SR-ISISv6... only SRv6", then
> that's an entirely different issue.
> 
> The good news is there currently is choice on the matter, but upending
> hundreds or thousands of boxes to prove that point should really be a last
> resort, as there are more pressing things we all have to deal with.
> 
Hence our current strategy is to stay on IPv4 control-plane (and IPv4 
management plane) as it suits, and for the foreseeable future will suite, all 
our needs (which are to transport v4 data packets via L2 MPLS VPN 
services), there are simply more important projects than to experiment with v6 
control-plane, like for instance perfecting/securing the v6 customer facing 
services (delivered over the underlying v4 signalled MPLS infrastructure, that 
no customer really cares about). 

But I understand your frustrations case it seems like you're taking the bullet 
for us late adopters and in a sense you are, cause say in 10 years from now 
when I decide to migrate to v6 control-plane and management-plane as then it 
might be viewed as common courtesy, it will be all there on a silver plate 
waiting for me allowing for a relatively effortless and painless move. All 
thanks to you fighting the good fight today.

> 
> >  Yes you're right in acknowledging that we're not living in a perfect world
> and that choices are limited, but it's been like that since ever yet we
> managed to thrive by analysing our options and striving for optimal strategies
> year by year.
> 
> We can thank NAT44, CIDR, DHCP and PPPoE for that strategy over the years
> :-).
> 
> IPv6 is the future, and at some point, we'll have to stop hiding from it.
> 
And I'd say the future is now, cause there is an actual need for v6 services. 
But need for v6 control & management plane? - It's not like operators are 
losing business opportunities not having that. (they might even be viewed as 
conservative->stable, which might be preferred by some customers). 

adam 

Re: Quality of the internet

[Saku Ytti]

> I think, on the whole, as current-production routers have migrated away
> from software-based forwarding in recent years into hardware planes, as

ACK. Good Internet is almost an emergent feature, not something we
really designed for. The main remaining problems are congested
peerings, which is a silly political problem which ends up hurting
customers and not helping anyone.
No one needs strict priority queues anymore, which was absolutely
needed at one point in time.

We are not in a market which cares about QoS, yet our BE is globally
<200us max jitter on a typical day and AF is <50us. Average jitter
being under 10us. So if I'd have HW timestamping NTP server and
client, I could synchronise clocks over IP transit cross continents to
ten microsecond accuracy. I think this is pretty crazy. And I'm sure
anyone who measures, measures similar numbers, this would have sounded
scifi 20 years ago.
As a context, Zoom recommends a jitter of 40ms or better, or 4us.

-- 
  ++ytti

Re: Devil's Advocate - Segment Routing, Why?

[Mark Tinka]

On 18/Jun/20 13:23, adamv0...@netconsultings.com wrote:

> You do have the LDP vs SR choice (in v4 anyways) yes there's not a good 1:1 
> feature parity with v6, but the important point...

But the lack of IPv4/IPv6 parity is a crucial one.

There is only so long we can stretch IPv4, if one can still manage the
tangible and intangible costs of doing so. But that's for another
discussion.


>  is the current state is not the end state, this is a pretty dynamic industry 
> that I'm sure is converging/evolving towards a v4:v6 parity, however the pace 
> may be, which is understandable considering the scope of ground to be covered.

Which I am fine with  - if you give me a time line to say LDPv6,
SR-OSPFv3 and SR-ISISv6 will be available on X date, I can manage my
operation and expectations accordingly.

But if you say, "No LDPv6, no SR-OSPFv3, no SR-ISISv6... only SRv6",
then that's an entirely different issue.

The good news is there currently is choice on the matter, but upending
hundreds or thousands of boxes to prove that point should really be a
last resort, as there are more pressing things we all have to deal with.


>  Yes you're right in acknowledging that we're not living in a perfect world 
> and that choices are limited, but it's been like that since ever yet we 
> managed to thrive by analysing our options and striving for optimal 
> strategies year by year.  

We can thank NAT44, CIDR, DHCP and PPPoE for that strategy over the
years :-).

IPv6 is the future, and at some point, we'll have to stop hiding from it.

Mark.

Re: Hurricane Electric has reached 0 RPKI INVALIDs in our routing table

[Nick Hilliard]

Mark Tinka wrote on 18/06/2020 11:56:

Invalid routes being dropped creates downtime. People respond to
downtime a lot more eagerly.


humanity is a crisis-driven species.

Nick

RE: Devil's Advocate - Segment Routing, Why?

[adamv0025]

> From: NANOG  On Behalf Of Mark Tinka
> Sent: Thursday, June 18, 2020 8:13 AM
> 
> There are probably as many networks running SR-MPLS as there are running
> LDPv6, likely fewer if your SR deployment doesn't yet support OSPFv3 or SR-
> ISISv6. I concede that for some networks looking to go SR-MPLS, label
> distribution state reduction is probably higher up on the agenda than
> MPLSv6 forwarding. For me, I'd like the option to have both, and decide
> whether my network is in a position to handle the additional state required
> for LDPv6, if I feel that I'd prefer to deal with a protocol that has had more
> exposure to the sun.
> 
You do have the LDP vs SR choice (in v4 anyways) yes there's not a good 1:1 
feature parity with v6, but the important point is the current state is not the 
end state, this is a pretty dynamic industry that I'm sure is 
converging/evolving towards a v4:v6 parity, however the pace may be, which is 
understandable considering the scope of ground to be covered. Yes you're right 
in acknowledging that we're not living in a perfect world and that choices are 
limited, but it's been like that since ever yet we managed to thrive by 
analysing our options and striving for optimal strategies year by year.  

adam

Re: Survey on the use of IP blacklists for threat mitigation

[Hank Nussbacher]

  
  
On 16/06/2020 22:08, J. Hellenthal via
  NANOG wrote:


This issue was raised in Reddit and
  Github:
https://www.reddit.com/r/sysadmin/comments/h149em/calls_to_replace_blacklist_whitelist_black_hat/
https://www.techspot.com/news/85631-github-replace-terms-whitelist-blacklist-masterslave-racially-insensitive.html
and is not limited to the term
  "blacklist".


-Hank


Note: the views expressed above are my
  own and do not necessarily reflect the views of my employer



  
  Guess we all better start rewriting all of the documentation out
  there because some PC marketing snowflake wants to get extra
  brownie points and attention for classifying a color in RGB into a
  racial divide for which it never originated.
  
  
  blacklists are not always deny/block/disallow and conformed
of things that allow you to take actions whatever your choosing
upon their contents and your policies.
  
  
  What’s next ? redlisting ? Don’t offend the Russians ... blue
? Don’t want to offend the police ...
  
  
  Leave this crap off the list, it’s not helping anyone.
  
  
  SMH
  


  -- 
   J.
  Hellenthal
  

  The
fact that there's a highway to Hell but only a stairway to
Heaven says a lot about anticipated traffic volume.

  On Jun 16, 2020, at 13:27, Ryan Landry
 wrote:

  


  
In kind, I'd like to encourage the use of
  terms like permit/accept list or deny/block list.
  
  
  Respectfully,
  -Ryan



  On Tue, Jun 16, 2020 at
11:33 AM Rachee Singh 
wrote:
  
  
Hi NANOG community,
  
  We are a group of researchers studying the use of IP
  blacklists as a mechanism to mitigate security threats
  -- particularly over the IPv6 Internet. We would like
  to understand if and how you use IP blacklists to
  secure your networks. Please consider taking our short
  survey: https://forms.gle/ZEsxyiBivJAfLF7e6
  
  The survey will be anonymous unless you choose to
  identify yourself.
  
  Thanks,
  Rachee
  UMass Amherst

  

  

  


  

Re: Hurricane Electric has reached 0 RPKI INVALIDs in our routing table

[Mark Tinka]

On 18/Jun/20 12:51, Nick Hilliard wrote:
 
>
> The customer monitoring system is very reliable and often superior to
> in-house solutions.

What really made the experience great for us is that directly contacting
the remote network (somewhere in Eastern Europe) and getting them to fix
the issue was far more effective than the usual, "Get your customer to
log a case with our customer, who can then log a case with us, since we
have no commercial contract with you".

We had a completely separate second case caused by us rejecting an
Invalid route. It got fixed in 30 minutes as well.

Invalid routes being dropped creates downtime. People respond to
downtime a lot more eagerly.

Mark.

Re: Hurricane Electric has reached 0 RPKI INVALIDs in our routing table

[Nick Hilliard]

Mark Tinka wrote on 18/06/2020 11:16:

On 17/Jun/20 21:16, Tim Warnock wrote:

How did you know? Is there some monitoring system available to let
you know or do you have your own?


The usual way - a customer complained :-).


The customer monitoring system is very reliable and often superior to
in-house solutions.

Nick

Re: Devil's Advocate - Segment Routing, Why?

[Saku Ytti]

On Thu, 18 Jun 2020 at 13:28, Robert Raszuk  wrote:

> To your IGP point let me observe that OSPF runs over IP and ISIS does not. 
> That is first fundamental difference. There are customers using both all over 
> the world and therefore any suggestion to just use OSPFv3 is IMHO quite 
> unrealistic. Keep in mind that OSPF hierarchy is 2 (or 3 with super area) 
> while in IETF there is ongoing work to extend ISIS to 8 levels. There is a 
> lot of fundamental differences between those two (or three) IGPs and I am 
> sure many folks on the lists know them. Last there is a lot of enterprise 
> networks happily using IPv4 RFC1918 all over their global WAN and DCs 
> infrastructure and have no reason to deploy IPv6 there any
time soon.

I view the 802.3 and CLNS as liability, not an asset. People who
actually route CLNS are a dying breed,  think just DCN of a legacy
optical.

Many platforms have no facilities to protect ISIS, any connected
attacker can kill the box. Nokia handles generated packets
classification by assigning DSCP value to  application then DSCP to
forwarding-class, which precludes from configuring ISIS qos. Very few
people understand how ISIS works before ISIS PDU is handed to them,
world from 802.3 to that is largely huge pile of hacks, instead of
complete CLNS stack implementation. There is no standard way to send
large frames over 802.3, so there is non-standard way to encap ISIS
for those links. Also due to lack of LSP roll-over, ISIS is subject to
a horrible attack vector which is very difficult to troubleshoot and
solve.

-- 
  ++ytti

Re: Devil's Advocate - Segment Routing, Why?

[Mark Tinka]

On 18/Jun/20 12:28, Robert Raszuk wrote:

> To your IGP point let me observe that OSPF runs over IP and ISIS does
> not. That is first fundamental difference. There are customers using
> both all over the world and therefore any suggestion to just use
> OSPFv3 is IMHO quite unrealistic.

Are you saying that OSPF houses that want IPv6 should just move to
IS-IS. Don't get me wrong, I support that very much as I think IS-IS is
a great IGP. That said, while it's good to convince OSPF operators to
consider IS-IS, it's not our place to force them to use it.

Also, OSPFv3-only for your dual-stack IGP needs is a supported
capability. Last time I tested it in Juniper in 2010/2011, it worked
well. I don't know if anyone is actually running IPv4 and IPv6 on OSPFv3
only, but it does work.


> Keep in mind that OSPF hierarchy is 2 (or 3 with super area) while in
> IETF there is ongoing work to extend ISIS to 8 levels. There is a lot
> of fundamental differences between those two (or three) IGPs and I am
> sure many folks on the lists know them.

15+ years ago, I'd have said that one protocol may have been suited to a
specific task than another due to the control plane limitations of the day.

In 2020, with the state-of-the-art of control planes today, it near as
makes no difference, IMHO.


> Last there is a lot of enterprise networks happily using IPv4 RFC1918
> all over their global WAN and DCs infrastructure and have no reason to
> deploy IPv6 there any time soon.

No wonder the vendors aren't seeing any LDPv6, SR-ISISv6 or SR-OSPFv3
demand :-).

Mark.

Re: Router Suggestions

[Mark Tinka]

On 18/Jun/20 04:00, Owen DeLong wrote:

> OTOH, I bet if you’d had two of those cards fail, you might
> have been SOL on the second one for a couple of days.

Quite possibly, who knows :-). Perhaps I should ask them, just to get a
squirm :-).

Then again, we had enough redundancy built into the PoP to afford a loss
of 3 out of 4 of them and still be okay.

Mark.

Re: Devil's Advocate - Segment Routing, Why?

[Robert Raszuk]

Hi Saku,

To your IGP point let me observe that OSPF runs over IP and ISIS does not.
That is first fundamental difference. There are customers using both all
over the world and therefore any suggestion to just use OSPFv3 is IMHO
quite unrealistic. Keep in mind that OSPF hierarchy is 2 (or 3 with super
area) while in IETF there is ongoing work to extend ISIS to 8 levels. There
is a lot of fundamental differences between those two (or three) IGPs and I
am sure many folks on the lists know them. Last there is a lot of
enterprise networks happily using IPv4 RFC1918 all over their global WAN
and DCs infrastructure and have no reason to deploy IPv6 there any time
soon.

If you are serious about converging to a single IGP I would rather consider
look towards OpenR type of IGP architecture with message bus underneath.

Thx,
R.

On Thu, Jun 18, 2020 at 7:26 AM Saku Ytti  wrote:

> On Thu, 18 Jun 2020 at 01:17, Mark Tinka  wrote:
>
> > IOS XR does not appear to support SR-OSPFv3.
> > IOS XE does not appear to support SR-ISISv6.
> > IOS XE does not appear to support SR-OSPFv3.
> > Junos does not appear to support SR-OSPFv3.
>
> The IGP mess we are in is horrible, but I can't blame SR for it. It's
> really unacceptable we spend NRE hours developing 3 identical IGP
> (OSPFv2, OSPFv3, ISIS). We all pay a 300-400% premium for a single
> IGP.
>
> In a sane world, we'd retire all of them except OSPFv3 and put all NRE
> focus on there or move some of the NRE dollars to some other problems
> we have, perhaps we would have room to support some different
> non-djikstra IGP.
>
> In a half sane world, IGP code, 90% of your code would be identical,
> then you'd have adapter/ospfv2 adapter/ospfv3 adapter/isis which
> translates internal struct to wire and wire to internal struct. So any
> features you code, come for free to all of them. But no one is doing
> this, it's 300% effort, and we all pay a premium for that.
>
> In a quarter sane world we'd have some CIC, common-igp-container RFC
> and then new features like SR would be specified as CIC-format,
> instead of OSPFv2, OSPFv3, ISIS and BGP. Then each OSPFv2, OSPFv3,
> ISIS and BGP would have CIC-to-x RFC. So people introducing new IGP
> features do not need to write 4 drafts, one is enough.
>
> I would include IPv4+IPv6 my-igp-of-choice SR in my RFP. Luckily ISIS
> is supported on platforms I care about for IPV4+IPV6, so I'm already
> there.
>
> > MPLS/VPN service signaling in IPv6-only networks also has gaps in SR.
>
> I don't understand this.
>
>
> > So for networks that run OSPF and don't run Juniper, they'd need to move
> to IS-IS in order to have SR forward IPv6 traffic in an MPLS encapsulation.
> Seems like a bit of an ask. Yes, code needs to be written, which is fine by
> me, as it also does for LDPv6.
>
> And it's really just adding TLV, if it already does IPv4 all the infra
> should be in place, only  thing missing is transporting the
> information. Adding TLV to IGP is a lot less work than LDPv6.
>
> > I'd be curious to understand what bugs you've suffered with LDP in the
> last 10 or so years, that likely still have open tickets.
>
> 3 within a year.
> - PR1436119
> - PR1428081
> - PR1416032
>
> I don't have IOS-XR LDP bugs within a year, but we had a bunch back
> when going from 4 to 5. And none of these are cosmetic, these are
> blackholing.
>
> I'm not saying LDP is bad, it's just, of course more code lines you
> exercise more bugs you see.
>
> But yes, LDP has a lot of bug surface compared to SR, but in _your
> network_ lot of that bug surface and complexity is amortised
> complexity. So status quo bias is strong to keep running LDP, it is
> simpler _NOW_ as a lot of the tax has been paid and moving to an
> objectively simpler solution carries risk, as its complexity is not
> amortised yet.
>
>
> > Yes, we all love less state, I won't argue that. But it's the same
> question that is being asked less and less with each passing year - what
> scales better in 2020, OSPF or IS-IS. That is becoming less relevant as
> control planes keep getting faster and cheaper.
>
> I don't think it ever was relevant.
>
> > I'm not saying that if you are dealing with 100,000 T-LDP sessions you
> should not consider SR, but if you're not, and SR still requires a bit more
> development (never mind deployment experience), what's wrong with having
> LDPv6? If it makes near-as-no-difference to your control plane in 2020 or
> 2030 as to whether your 10,000-node network is running LDP or SR, why not
> have the choice?
>
> I can't add anything to the upside of going from LDP to SR that I've
> not already said. You get more by spending less, it's win:win. Only
> reason to stay in LDP is status quo bias which makes short term sense.
>
> > Routers, in 2020, still ship with RIPv2. If anyone wants to use it (as I
> am sure there are some that do), who are we to stand in their way, if it
> makes sense for them?
>
> RIP might make sense in some deployments, because it's essentially
> stateless 

Re: Router Suggestions

[Mark Tinka]

On 18/Jun/20 00:50, Warren Kumari wrote:

>
> A number of customers (myself included) had 4 hour replacement
> contracts, which the vendor really could not meet - so we agreed to
> take a new, much larger/better model as a replacement.

It's one of the reasons we never pay for 24/7/365.

In many cases for our experience, with sufficient pre-built redundancy,
there isn't much different with 24/7 vs. NBD, apart from the cost. So
why pay for the premium.

And if a site does not require redundancy, it's cheaper to have a cold
standby than to give it 24/7/365, or even NBD.

Mark.

Re: Quality of the internet

[Mark Tinka]

On 17/Jun/20 22:47, Dovid Bender wrote:
> Hi,
>
> My 9-5 is working for a VoIP provider. When we started in 2006 we had
> a lot of issues with the quality of the internet in eastern europe and
> central Asia. It was not rare for us to have to play around with
> routing to get the quality that we needed. In a review of tickets for
> the last two years it seems as if we barely do any of that these days.
> Rarely do we get a quality complaint that comes back to an issue where
> a carrier or ISP dropping or mangling packets. Has anyone else
> observed this as well?

I think, on the whole, as current-production routers have migrated away
from software-based forwarding in recent years into hardware planes, as
more submarine cables have been laid to all continents, as more exchange
points have been built, as mobile networks have moved from being voice
to becoming data transport networks, and as the cloud and content
providers have shifted the local/regional Internet eco system upon their
arrival, it's not unreasonable to conclude that the overall quality of
the Internet has made a marked improvement.

It feels like I operated a satellite-based IP/MPLS network for a whole
country millions of years ago, and yet it was just as recent as 2007.
It's impressive how much we have moved forward, as a community, in that
space of time.

Mark.

Re: Hurricane Electric has reached 0 RPKI INVALIDs in our routing table

[Mark Tinka]

On 17/Jun/20 21:16, Tim Warnock wrote:

> How did you know? Is there some monitoring system available to let you know 
> or do you have your own?

The usual way - a customer complained :-).

Mark.

Re: Mikrotik RPKI Testing

[Mark Tinka]

On 17/Jun/20 20:31, Bryan Fields wrote:

>
>
> How is IPv6 coming on Mikrotik?  It's a no-go at least for my
> deployment on
> 6.4 code.  Not sure I want to run beta in a quasi-production network.

In my home, basic IPv6 + SLAAC is working fine on Mikrotik, on 6.47.

I have a mate who adds DHCP-PD on his, and he's happy too.

Beyond that, I can't tell you much. It's a home CPE :-).

Mark.

Re: Mikrotik RPKI Testing

[Mark Tinka]

On 17/Jun/20 19:19, Job Snijders wrote:

>
> Our hero Massimiliano Stucchi in Switzerland started doing the legwork.
> He is is sharing the test results here:
>
> http://as58280.net/en/articles/RPKI-on-Mikrotik
>
> Enjoy!

Thanks, and great to see.

Shame IPv6 keeps being sent to the naughty corner, but well :-).

Mark.

RE: Devil's Advocate - Segment Routing, Why?

[adamv0025]

> From: Saku Ytti
> Sent: Thursday, June 18, 2020 6:26 AM
> 
> On Thu, 18 Jun 2020 at 01:17, Mark Tinka  wrote:
> 
> > Yes, we all love less state, I won't argue that. But it's the same question
> that is being asked less and less with each passing year - what scales better 
> in
> 2020, OSPF or IS-IS. That is becoming less relevant as control planes keep
> getting faster and cheaper.
> 
> I don't think it ever was relevant.
> 
In 99% of cases, there are cases however where supporting 1M+ routes in IGP is 
one of the viable options to consider, or running multi-100k of LSPs through a 
core node... 
But these are core MPLS networks that have no boundaries cause these literally 
wrap around the globe, and access to custom code. 
 
adam 

Re: Mikrotik RPKI Testing

[Nick Hilliard]

Musa Stephen Honlue wrote on 18/06/2020 03:38:
Did you face any issues with IPv6 on 6.4, I personally have participated 
in deployment projects on Mikrotik for many large networks.


mikrotik ROS6 doesn't support next-hop recursion for ipv6 routes:

https://forum.mikrotik.com/viewtopic.php?t=42268

It also doesn't support ospfv3 prefixes with the LA-bit set:

https://forum.mikrotik.com/viewtopic.php?t=51124#p319794

I.e. if you originate an ipv6 loopback address from another vendor, the 
Mikrotik will silently drop the prefix on the floor.


Note the dates on these posts: 2010 and 2011.

Nick

Re: Devil's Advocate - Segment Routing, Why?

[Mark Tinka]

On 18/Jun/20 09:30, Saku Ytti wrote:

> Yes work left to be done. Ultimately the root problem is, no one cares
> about IPv6. But perhaps work with vendors in parallel to LDPv6 to get
> them to fix OSPFv3 and/or ISIS.

Yes, this.

Vendor feedback for those not supporting LDPv6 is that there is no
demand for it. And like I said in the previous thread, LDPv6 demand is
not about LDPv6, it's about IPv6.

If the majority of the high-paying vendors' favorite customers that pay
for CGN's continue to do so, what incentive do they have to ask for
IPv6. The T-Mobile US's of the world are few and far between, sadly.

I suppose I would not be unwilling to push the vendors to support
SR-OSPFv3 and SR-ISISv6 as I am also pushing them to support LDPv6 where
it is lacking, because at some point in the future, I do want to deploy
SR-MPLS in the same way I envisioned doing so back in 2014. I just need
to take it on a few dates first before I bring it home to meet the folks
:-).



> FWIW I am definitely saying that, and it should be IGP+BGP. I do
> accept and realise a lot of platforms only did and do Martini not
> Kompella, so reality isn't quite there.

That was me in 2013/2014. Dump LDP, dump RSVP, get SR deployed, forward
IPv4 natively in MPLSv4, and IPv6 natively in MPLSv6. But life happened.

Nonetheless, I will go SR-MPLS in many years to come, after I'm feeling
comfortable about it. That's a promise. But until then, I'd like
trusted, stable IPv4-IPv6 MPLS forwarding parity.

I have never cared much for VPLS because I thought it was a very messy
piece of tech. from Day 1. And while EVPN makes more sense, for our
market, more than 98% of the traffic we sell is IP-based, so we have no
demand for mp2mp Ethernet VPN's. But for those that adore VPLS (or
EVPN), let them have the choice of LDP or BGP, which both Cisco and
Juniper, after years of muscle-flexing, both ended up agreeing on
anyway, despite all the fuss.

So the LDPv6 vs. SR-MPLS vs. SRv6 vs. SRv6+ posturing is a rehash of
those LDP vs. BGP days, which just wastes everyone's time.

Mark.

Re: Devil's Advocate - Segment Routing, Why?

[Saku Ytti]

On Thu, 18 Jun 2020 at 10:13, Mark Tinka  wrote:

> Which is great for you, me, and a ton of other folk that run IS-IS on
> Juniper. What about folk that don't have Juniper, or run OSPF?
>
> I know, not your or my problem, but the Internet isn't just a few networks.

Yes work left to be done. Ultimately the root problem is, no one cares
about IPv6. But perhaps work with vendors in parallel to LDPv6 to get
them to fix OSPFv3 and/or ISIS.

> I'm not saying it should be an SR vs. LDP debate like it was
> BGP-signaling vs. LDP-signaling for VPLS 12+ years ago. All I'm saying

FWIW I am definitely saying that, and it should be IGP+BGP. I do
accept and realise a lot of platforms only did and do Martini not
Kompella, so reality isn't quite there.

-- 
  ++ytti

Re: Devil's Advocate - Segment Routing, Why?

[Mark Tinka]

On 18/Jun/20 07:25, Saku Ytti wrote:

> The IGP mess we are in is horrible, but I can't blame SR for it. It's
> really unacceptable we spend NRE hours developing 3 identical IGP
> (OSPFv2, OSPFv3, ISIS). We all pay a 300-400% premium for a single
> IGP.
>
> In a sane world, we'd retire all of them except OSPFv3 and put all NRE
> focus on there or move some of the NRE dollars to some other problems
> we have, perhaps we would have room to support some different
> non-djikstra IGP.
>
> In a half sane world, IGP code, 90% of your code would be identical,
> then you'd have adapter/ospfv2 adapter/ospfv3 adapter/isis which
> translates internal struct to wire and wire to internal struct. So any
> features you code, come for free to all of them. But no one is doing
> this, it's 300% effort, and we all pay a premium for that.
>
> In a quarter sane world we'd have some CIC, common-igp-container RFC
> and then new features like SR would be specified as CIC-format,
> instead of OSPFv2, OSPFv3, ISIS and BGP. Then each OSPFv2, OSPFv3,
> ISIS and BGP would have CIC-to-x RFC. So people introducing new IGP
> features do not need to write 4 drafts, one is enough.

While I don't have a real opinion on how to fix the IGP mess, the point
is we sit with it now. Getting all these fixed is going to increase the
bug surface area for some time to come as both vendors and operators
work the kinks out, in addition to SR's own kinks.

Yes, it's all par for the course for new features, which is why I'd also
like to have an alternative that has been baked in for many years to
give me an option for stability, as we roll the new kid out.

I probably will deploy SR-MPLS at some point in my lifetime, but I'm not
feeling awfully comfortable to do so right now; and yet I do need MPLSv6
forwarding.


>
> I would include IPv4+IPv6 my-igp-of-choice SR in my RFP. Luckily ISIS
> is supported on platforms I care about for IPV4+IPV6, so I'm already
> there.

Which is great for you, me, and a ton of other folk that run IS-IS on
Juniper. What about folk that don't have Juniper, or run OSPF?

I know, not your or my problem, but the Internet isn't just a few networks.



> I don't understand this.

I mean the same gaps that exist in RFC 7439, for would-be IPv6-only MPLS
networks.



> And it's really just adding TLV, if it already does IPv4 all the infra
> should be in place, only  thing missing is transporting the
> information. Adding TLV to IGP is a lot less work than LDPv6.

What we theorize as "should be easy" can turn out to be a whole
discussion with the vendors about it being months or years of work. Not
being inside their meeting rooms, I can't quite challenge how they
present the task.

Fundamentally, LDPv6 already has 5+ years in implementation (and LDPv4
is 20 years old), inter-op issues seem to be mostly fixed, and for what
we need it to do, it's working very well.

There are probably as many networks running SR-MPLS as there are running
LDPv6, likely fewer if your SR deployment doesn't yet support OSPFv3 or
SR-ISISv6. I concede that for some networks looking to go SR-MPLS, label
distribution state reduction is probably higher up on the agenda than
MPLSv6 forwarding. For me, I'd like the option to have both, and decide
whether my network is in a position to handle the additional state
required for LDPv6, if I feel that I'd prefer to deal with a protocol
that has had more exposure to the sun.

Ultimately, boxes with LDPv6 have been shipping for some time, and we
have a ton of them deployed and running for a while now. If it comes
down to kicking out the 20% that won't support it because of an
all-or-nothing vendor approach on a platform without full SR-MPLS
support for all IGP's, it is what it is.



> 3 within a year.
> - PR1436119
> - PR1428081
> - PR1416032
>
> I don't have IOS-XR LDP bugs within a year, but we had a bunch back
> when going from 4 to 5. And none of these are cosmetic, these are
> blackholing.
>
> I'm not saying LDP is bad, it's just, of course more code lines you
> exercise more bugs you see.
>
> But yes, LDP has a lot of bug surface compared to SR, but in _your
> network_ lot of that bug surface and complexity is amortised
> complexity. So status quo bias is strong to keep running LDP, it is
> simpler _NOW_ as a lot of the tax has been paid and moving to an
> objectively simpler solution carries risk, as its complexity is not
> amortised yet.

And FWIW, if some operators are willing to benefit from all the
experience that has gone into developing and maintaining LDP, while we
let SR settle down, I don't see why that choice shouldn't be there.

I'm not saying it should be an SR vs. LDP debate like it was
BGP-signaling vs. LDP-signaling for VPLS 12+ years ago. All I'm saying
is for those who want to go bleeding edge with SR, go for it. For those
who prefer to gracefully transition toward SR over time by settling on
LDP that has been in the field for a minute, go for it too.

I won't claim to know whether LDP or SR 

Re: [c-nsp] Devil's Advocate - Segment Routing, Why?

[Mark Tinka]

On 18/Jun/20 02:32, Phil Bedard wrote:
> I look at the basic SR via IGP extensions like VPLS vs. EVPN.   If we had a 
> way to go back in history I'm not sure anyone would have said VPLS was a good 
> idea vs. EVPN.  
>
> There were reasons back in the day why something like SR wasn't done.  The 
> thought of global MPLS labels scared people and source routing was also evil. 
>  So LDP was created to distribute labels hop by hop, while still relying 100% 
> on the IGP to do so.  It kind of defies common sense when you look at it now, 
> but there were supposedly good reasons for it back then.  
>
> SR-MPLS on an existing device supporting MPLS forwarding is a control-plane 
> change, meaning almost any device could support SR-MPLS.  
>
> SR is meant to be data plane agnostic, the SID is just an identifier.  Most 
> support MPLS, some support IPv6.  

Fair enough.

There's still a whole IGP mess to sort out though, not to mention many
years of field experience to bake in.

Mark.

Re: Devil's Advocate - Segment Routing, Why?

[Mark Tinka]

On 18/Jun/20 00:29, Robert Raszuk wrote:

>
> Example: If your hardware ASICs do not support IPv6 while support IPv4
> - LDPv4 will work just fine while LDPv6 will have a rather a bit of
> hard time :)

Well, safe to say that if your box doesn't support IPv6, MPLSv6 is
probably the least of your worries :-).

Mark.