Re: Virginia voter registration down due to cable cut

[Sean Donelan]

On Tue, 13 Oct 2020, Valdis Klētnieks wrote:

my reaction was more like

Surprise, surprise, surprise...



S.N.A.F.U.

Other SNAFUs, Georgia had technical problems with its voter database 
systems during the first couple of days of early voting. Expect all sorts 
of minor problems throughout the election and afterwards. Nonetheless they 
are unlikely to significantly impact the results (hopefully), but will 
generate lots of noise.


Its not just underfunded state I.T. systems.  Even very large social media 
companies can have technical Oopsies.  Again, hopefully Twitter won't 
fall down again during the evening of November 3rd.  The digeratti will 
lose thier minds.


Even if Twitter or another major social media platform does go belly-up, 
most likely it will be a normal technical problem.  Wishing the FBI & CISA 
& OGAs watch officers a very boring night on November 3rd.




In other news, New Zealand is having national elections this weekend.  New 
Zealand is usually ranked in the top 10 best election administrations 
worldwide. NZ expects to have the majority of ballots counted within 2 
hours of their polls closing on Saturday evening.


Jealous of the Kiwis and their competently run elections. :-)

RFC 2468

[Rodney Joffe]

It is especially fitting whenever the NANOG/ARIN joint meetings occur in the 
same week that we “remember IANA”.

As time has gone on, fewer and fewer of us actually know who J. Postel is - 
that name that appears at the end of so many RFC’s we refer to every day. The 
same person who also guided the management of names and numbers in the “early” 
days of this grand experiment we’re still struggling to get “right”.

Today (Friday, October 16) is 22 years since Jon Postel passed away. I won’t 
start to list the rest of the pioneers we’ve lost since then - its obviously 
getting longer and longer. But I think its worth pointing “newcomers" at Vint’s 
RFC2468 (https://tools.ietf.org/rfc/rfc2468.txt) as the starting point for them 
(you) to understand the importance of Jon’s legacy as a moral compass to help 
guide some of the decisions being made or ignored during this week. And 
obviously other weeks and decisions that follow.

Jon was my mentor, colleague, business partner, and friend. And along with his 
other friends still on this list, I miss him a lot. It hasn’t been the same 
without him.

/rlj

Re: Cogent Layer 2

[Brandon Martin]

On 10/15/20 6:15 PM, Robert Blayzor wrote:

On 10/14/20 1:56 PM, Shawn L via NANOG wrote:

When I last spoke to them, it sounded like they were using a bunch of
LAG groups based on ip address because they _really_ wanted to know how
many ip addresses we had and what kind of traffic we would be expecting
(eyeball networks, big data transport, etc).



Not why IP addresses are even an issue on an a "Layer 2" service. But
then again, this is Cogent we're talking about here.



Hashing on LAGs within their core.  Even otherwise fairly braindead 
Ethernet switches often hash on L3 to try and get more entropy.


I hope they hash on L2 MAC, as well, but a pretty common scenario for an 
L2 interconnect only has one MAC on each end of the link, so that 
doesn't help much.  They rallly don't want all your traffic ending 
up on one side of a LAG.

--
Brandon Martin

Cox contact?

[Fred Baker]

Would an engineer from Cox please contact me privately?

Re: Cogent Layer 2

[Robert Blayzor]

On 10/14/20 1:56 PM, Shawn L via NANOG wrote:
> When I last spoke to them, it sounded like they were using a bunch of
> LAG groups based on ip address because they _really_ wanted to know how
> many ip addresses we had and what kind of traffic we would be expecting
> (eyeball networks, big data transport, etc).


Not why IP addresses are even an issue on an a "Layer 2" service. But
then again, this is Cogent we're talking about here.

-- 
inoc.net!rblayzor
XMPP: rblayzor.AT.inoc.net
PGP:  https://pgp.inoc.net/rblayzor/

Re: Residential GPON last mile for network engineers (Telus AS852 and others)

[Paul Nash]

I have a Bell Canada gig fibre connection.  My first attempt was to bridge 
their all-in-one box (disaster, unreliable as all hell), second was to set a 
bunch of rules for inbound traffic.  Apart from inbound access being *very* 
iffy, their device was s_l_o_w.

So I pulled the fibre GBIC, used a small switch to grab the correct VLAN and 
pointed that at a small Cisco box.  Way more flexible, faster and more reliable 
than Bell’s box.  DSLreports had all the info needed to get the correct VLANs

YMMV

> On Oct 13, 2020, at 9:56 PM, Eric Kuhnke  wrote:
> 
> Very interesting. Looks like the intention is to bypass the ONT entirely and 
> use a GPON ONT SFP in ones own choice of small home router. If the ISP wants 
> to do some weird TR069 provisioning or other stuff it could be seen as 
> interfering with the proper management of their network if you remove the CPE 
> entirely.
> 
> In an ideal world, personally I would be totally fine with keeping a telco 
> provided small ONT configured as a dumb L2 bridge, with one optical interface 
> single strand (SC/APC) going to the ISP, and 1000BaseT to my own router.
> 
> On Tue, Oct 13, 2020 at 6:51 PM Eric Dugas  wrote:
> I don't have any particular insights for Telus, but there is a huge thread 
> about bypassing Bell ONTs on DSLReports: 
> https://www.dslreports.com/forum/r32230041-Internet-Bypassing-the-HH3K-up-to-2-5Gbps-using-a-BCM57810S-NIC
> Cheers,
> Eric
> On Oct 13 2020, at 9:38 pm, Eric Kuhnke  wrote:
> With the growth of gigabit class single fiber GPON last mile services, I 
> imagine a number of people reading the list must have subscribed to such by 
> now.
> 
> Something that I have observed, and shared observations with a number of 
> colleagues, is that very often a person who works for ($someAS) lives in a 
> location where you are effectively singlehomed to ($someotherAS). Maybe you 
> bought your house before you got a job with your current employer, or maybe 
> the network you work for doesn't do residential last mile service at all. 
> Perhaps you work remotely for a regional sized entity that's a long distance 
> away from where you live.
> 
> Therefore necessitating a choice of service from whatever facilities based 
> consumer-facing ISP happens to service your home.
> 
> For example, in Seattle, a number of people discovered that they could keep 
> the Centurylink GPON ONT, and remove the centurylink-provided router/modem 
> combo device. Provided that they were able to configure their own router 
> (small vyatta, pfsense box, mikrotik, whatever) to speak a certain VLAN tag 
> on its WAN interface and be a normal PPPoE / DHCP client.
> 
> I'm sure there are a lot of people who prefer to run their own home router 
> and wifi devices, and not rely upon a ($big_residential_isp) provided 
> all-in-one router/nat/wifi box with opaque configuration parameters, or no 
> ability to change configuration at all.
> 
> Any insights as to what the configuration of the Telus AS852 GPON network 
> looks would be helpful. Or other observations in general on 
> technically-oriented persons who are doing similar with other ILECs.

RE: Ingress filtering on transits, peers, and IX ports

[adamv0025]

> Chris Adams
> Sent: Thursday, October 15, 2020 3:59 PM
> 
> Once upon a time, adamv0...@netconsultings.com
>  said:
> > Actually ideally there would be a feature/knob to automatically sync BGP
> (and static routes) with packet filters.
> 
> Junos has prefix-lists that can be referenced in both BGP policy and
firewall
> statements.
>
What I mean was a firewall filter that would get updated with accordance to
prefixes received/advertised via BGP, even better if this was a default
behaviour.


adam 

Shopify Network Admin ?

[John Rees]

Hi Nanog,
I am troubleshooting an issue where it appears that users orginitating from
a certain subnet that I manage are unable to access websites hosted by
Shopify.  We have contacted Shopify support and are still waiting for
resolution.

If anybody here is from Shopify - I would like to get some guidance on how
to proceed with the ticket so we can fix this issue for our mutual
customers.

Please contact me directly for additional information.

Thank you,
John

Looking for a contact at Twitter

[Ariën Vijn via NANOG]

Greetings,

I am looking for somebody working for Twitter. I am working for a small ISP in 
the Netherlands (AS 206238).

Our problem is that Twitter's geolocation database still situates some our IPv4 
blocks in the United Arab Emirates. This renders Twitter unusable for some of 
our customers.

We are looking for somebody that can help us with this Please contact me 
outside the ML. 

Thanks in advance,

-- Ariën

Re: Ingress filtering on transits, peers, and IX ports

[Blake Hudson]

Speaking as an ISP:
    Most of the ISP networks I manage are multi-homed, and I don't 
think uRPF provides the knobs to ensure legitimate traffic doesn't get 
dropped in some cases, so we use static ACLs at the upstream edge on 
ingress (and egress). These need updated any time new IP space is added 
to the network (not very often).
        Ingress ACL: Discard if source or destination is a bogon, RFC 
1912, RFC1918; discard any traffic sourced from our own IP space 
(spoofed); discard any traffic that is not destined for our IP space 
(bad/mis-routed).
        Egress ACL: Discard if source or destination is a bogon, RFC 
1912, RFC1918; discard any traffic that is not sourced from our IP space 
(spoofed).
        I believe in a policy of non-blocking and being net neutral, 
but if any TCP/UDP ports, IP protocols, or IP options are blocked you 
might add them to the above ACLs.


    On customer facing ports we use uRPF strict. Why? It's easy (one 
line to implement, zero maintenance, and it works well on our Cisco 
ASR1k/9k platforms)! Our customers are all single homed.


For a single homed enterprises, service providers, and end-users I'd 
recommend uRPF strict. Why? Again, it's dead simple. I would love to see 
this be the default on all home, prosumer, and branch office oriented 
platform. Linux does this by default with the rp_filter kernel option. I 
suspect that networking gear based on Linux probably leaves this at the 
default setting (strict mode).


In practice I don't know that I've ever used uRPF loose mode. ACLs have 
counters to verify they are working. I might have confirmed uRPF was 
working as intended the first time I implemented it, but otherwise I've 
used ACLs often enough to trust they are working as configured and trust 
the same for uRPF.


--Blake


On 10/13/2020 5:13 PM, Brian Knight via NANOG wrote:
We recently received an email notice from a group of security 
researchers who are looking at the feasibility of attacks using 
spoofed traffic.  Their methodology, in broad strokes, was to send 
traffic to our DNS servers with a source IP that looked like it came 
from our network.  Their attacks were successful, and they included a 
summary of what they found.  So this message has started an internal 
conversation on what traffic we should be filtering and how.


This security test was not about BCP 38 for ingress traffic from our 
customers, nor was it about BGP ingress filtering.  This tested our 
ingress filtering from the rest of the Internet.


It seems to me like we should be filtering traffic with spoofed IPs on 
our transit, IX, and peering links.  I have done this filtering in the 
enterprise world extensively, and it's very helpful to keep out bad 
traffic.  BCP 84 also discusses ingress filtering for SP's briefly and 
seems to advocate for it.


We have about 15 IP blocks allocated to us.  With a network as small 
as ours, I chose to go with a static ACL approach to start the 
conversation.  I crafted a static ACL, blocking all ingress traffic 
sourced from any of our assigned IP ranges.  I made sure to include:


* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)

The questions I wanted to ask the SP community are:

* What traffic filtering do you do on your transits, on IX ports, and 
your direct peering links?

* How is it accomplished?  Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like?  
What makes it easy or difficult to update?

* How did you test your filters when they were implemented?

Thanks a lot,

-Brian

Re: Cogent Layer 2

[Saku Ytti]

On Thu, 15 Oct 2020 at 17:49, Ryan Hamel  wrote:

> > So you're dropping in every edge all UDP packets towards these three ports? 
> > Your customers may not appreciate.
> You must not be familiar with JUNOS' ACL handling. This would be applied to 
> interface lo0, which is specifically for control planes. No data plane 
> traffic to customers would be hit.

I'm sure there are some gaps in knowledge at play here.

There are many reasons why packets hit the control-plane and not be
subject to lo0 filter, for example TTL expiry. Also, as I tried to
communicate with little success, BFD is implemented in NPU ucode and
you are subjected to NPU ucode bugs.
The bug I'm talking about, does not require you using or configuring
BFD, it just needs NPU to parse it, and your FPC is gone. Same deal
with Cisco issue I'm talking about.

I've not yet seen single non-broken junos control-plane protection,
everyone has terribly poorly written lo0 filters, no one has any idea
how to configure ddos-protection. If you some canonical sources to do
this, like Cymru or Juniper's MX book as source, you'll get it all
wrong, as they both contain trivial and naive errors.

But if you do manage to configure lo0 and ddos-protection correctly,
you're still exposed to wide array of packet-of-death style vectors.
Just yesterday on Junos SIRT-day bug where your KRT will become wedged
if you sample (IPFIX) specifically crafted packet, this will be
transit packet.

Problems become increasingly simple the less you understand them.

-- 
  ++ytti

Re: Ingress filtering on transits, peers, and IX ports

[Chris Adams]

Once upon a time, adamv0...@netconsultings.com  
said:
> Actually ideally there would be a feature/knob to automatically sync BGP (and 
> static routes) with packet filters.

Junos has prefix-lists that can be referenced in both BGP policy and
firewall statements.
-- 
Chris Adams 

Re: Cogent Layer 2

[Ryan Hamel]

> Do you want your martini emulated backbone link to fail when operator 
> reroutes their own LSR-LSR link failure?
As I said, it's an acceptable loss for my employers network, as we have a BGP 
failover mechanism in place that works perfectly.

> So you're dropping in every edge all UDP packets towards these three ports? 
> Your customers may not appreciate.
You must not be familiar with JUNOS' ACL handling. This would be applied to 
interface lo0, which is specifically for control planes. No data plane traffic 
to customers would be hit.

Ryan
On Oct 15 2020, at 1:03 am, Saku Ytti  wrote:
> On Thu, 15 Oct 2020 at 10:28, Ryan Hamel  wrote:
>
> > My experience with multiple carriers is that reroutes happen in under a 
> > minute but rarely happen, I also have redundant backup circuits to another 
> > datacenter, so no traffic is truly lost. If an outage lasts longer than 5 
> > minutes, or it's flapping very frequently, then I call the carrier. Last 
> > mile carriers install CPE equipment at the sites, which makes BFD a 
> > requirement to account for the fiber uplink on it going down, or an issue 
> > upstream.
> I think I may have spoken ambiguously and confusingly based on that
> statement. Rerouting inside operator network, such as their LSR-LSR
> link dropping is ostensibly invisible to the customer, can be tens of
> milliseconds outage can be 10s outage.
> Do you want your martini emulated backbone link to fail when operator
> reroutes their own LSR-LSR link failure?
>
> > As for security vulnerabilities, none can be leveraged if they are using 
> > internal IPs, and if not, a quick ACL can drop BFD traffic from unknown 
> > sources the same way BGP sessions are filtered.
> > In Juniper speak, the ACL would look like:
> > term deny_bfd {
> > from {
> > protocol udp;
> > destination-port [ 3784 3785 4784 ];
> > }
> > then discard;
>
> So you're dropping in every edge all UDP packets towards these three
> ports? Your customers may not appreciate.
>
> --
> ++ytti
>

Re: Ingress filtering on transits, peers, and IX ports

[Tim Durack]

On Thu, Oct 15, 2020 at 10:30 AM Saku Ytti  wrote:

> On Thu, 15 Oct 2020 at 17:22, Tim Durack  wrote:
>
>
> > We deploy urpf strict on all customer end-host and broadband circuits.
> In this scenario urpf = ingress acl I don't have to think about.
>
> But you have to think about what prefixes a customer has. If BGP you
> need to generate prefix-list, if static you need to generate a static
> route. As you already have to know and manage this information, what
> is the incremental cost to also emit an ACL?
>
> --
>   ++ytti
>

"You might argue that ingress packet acl would be operationally simpler on
customer and upstream, as you could cover all scenarios."

Although for a static customer urpf is hard to beat...

-- 
Tim:>

RE: Ingress filtering on transits, peers, and IX ports

[adamv0025]

> From: Saku Ytti 
> Sent: Thursday, October 15, 2020 3:30 PM
> 
> On Thu, 15 Oct 2020 at 17:22, Tim Durack  wrote:
> 
> 
> > We deploy urpf strict on all customer end-host and broadband circuits. In
> this scenario urpf = ingress acl I don't have to think about.
> 
> But you have to think about what prefixes a customer has. If BGP you need
> to generate prefix-list, if static you need to generate a static route. As you
> already have to know and manage this information, what is the incremental
> cost to also emit an ACL?
> 
Actually ideally there would be a feature/knob to automatically sync BGP (and 
static routes) with packet filters.

adam 

Re: Ingress filtering on transits, peers, and IX ports

[Nick Hilliard]

Saku Ytti wrote on 15/10/2020 15:29:

But you have to think about what prefixes a customer has. If BGP you
need to generate prefix-list, if static you need to generate a static
route. As you already have to know and manage this information, what
is the incremental cost to also emit an ACL?


the unfortunate reality is that many networks are run by CLI jockeys, so 
the incremental cost of this can be high.  There are no good 
general-purpose networking sources of truth, which means that usually 
provisioning databases need to be highly customised, which is only worth 
it if the scale merits it.


Nick

Re: Ingress filtering on transits, peers, and IX ports

[Saku Ytti]

On Thu, 15 Oct 2020 at 17:22, Tim Durack  wrote:


> We deploy urpf strict on all customer end-host and broadband circuits. In 
> this scenario urpf = ingress acl I don't have to think about.

But you have to think about what prefixes a customer has. If BGP you
need to generate prefix-list, if static you need to generate a static
route. As you already have to know and manage this information, what
is the incremental cost to also emit an ACL?

-- 
  ++ytti

Re: Ingress filtering on transits, peers, and IX ports

[Tim Durack]

We deploy urpf strict on all customer end-host and broadband circuits. In
this scenario urpf = ingress acl I don't have to think about.

We deploy urpf loose on all customer multihomed DIA circuits. I dont this
makes sense - ingress packet acl would be more sane.

Any flavour of urpf on upstream transit or peering would be challenging.
Ingress packet acl dropping source = own+customer prefix might make sense
depending on your AS topology.

You might argue that ingress packet acl would be operationally simpler on
customer and upstream, as you could cover all scenarios.

On Thu, Oct 15, 2020 at 10:05 AM Saku Ytti  wrote:

> On Thu, 15 Oct 2020 at 15:14,  wrote:
>
>
> > Yes one should absolutely do that, but...
> > But considering to become a good netizen what is more work?
> > a) Testing and the enabling uRPF on every customer facing box or setting
> up precise ACLs on every customer facing port, and then maintaining all
> that?
> > b) Gathering  all your PAs (potentially PIs) (hint: show bgp nei x.x.x.x
> advertised routes) crafting an ACL and apply it on several peering/transit
> links?
> > One of them is couple of weeks work and one is an afternoon job.
>
> I am not fan of uRPF, expensive for what it does. But I don't view it
> as an alternative here, I view it as either adding an ACE on all
> egresses on egress direction or adding ACE on the ingress where
> customer is on ingress direction.
>
> To me these options seem equally complex but the latter one seems superior.
>
> --
>   ++ytti
>


-- 
Tim:>

Re: Ingress filtering on transits, peers, and IX ports

[Saku Ytti]

On Thu, 15 Oct 2020 at 15:14,  wrote:


> Yes one should absolutely do that, but...
> But considering to become a good netizen what is more work?
> a) Testing and the enabling uRPF on every customer facing box or setting up 
> precise ACLs on every customer facing port, and then maintaining all that?
> b) Gathering  all your PAs (potentially PIs) (hint: show bgp nei x.x.x.x 
> advertised routes) crafting an ACL and apply it on several peering/transit 
> links?
> One of them is couple of weeks work and one is an afternoon job.

I am not fan of uRPF, expensive for what it does. But I don't view it
as an alternative here, I view it as either adding an ACE on all
egresses on egress direction or adding ACE on the ingress where
customer is on ingress direction.

To me these options seem equally complex but the latter one seems superior.

-- 
  ++ytti

RE: Ingress filtering on transits, peers, and IX ports

[adamv0025]

> From: Saku Ytti 
> Sent: Thursday, October 15, 2020 11:12 AM
> 
> Hey,
> 
Hey Saku,

> > All stub autonomous systems should have a simple egress ACL allowing
> only PI of their customers and their own PAs -it’s a simple ACL at each 
> AS-Exit
> points (towards transits/peers), that’s it.
> >
> > -not sure why this isn’t the first sentence in every BCP and “security
> > bulletin”…
> 
> I will venture a guess.
> 
>   1) it's very specific scenario to be stubby and have downstream PI
Yes it is and in most cases it's about "I have these few /XYs from which I 
address my customers (eyeballs) so nothing outside of these should ever leave 
my AS" -as simple as that.

>   2) it won't address customers spoofing each other arbitrarily and
> customer1 spoofing as customer2 on the internet, giving large chunk of the
> utility of spoofing even with protection in place
> 
Yes its not 100% effective as you pointed out,
However if every stub AS implemented this egress ACL on the few of their 
transit/peering links, there would be a lot less inter-AS garbage floating 
around.

> How do you maintain that ACL? 
Well you're gonna update the ACL every time you acquire a new PA space (every 
now and then), or when another customer wants you route their PI (rare, simply 
cause you're a stub AS with mostly eyeballs).
It's one ACL you need to update on every box with AS-exit links -usually there 
are not that many in a stub-AS. 
  
> Why doesn't that same mechanism allow
> ingress ACL on the customer port? Your proposal looks low utility for work
> needed.
> 
Yes one should absolutely do that, but...
But considering to become a good netizen what is more work? 
a) Testing and the enabling uRPF on every customer facing box or setting up 
precise ACLs on every customer facing port, and then maintaining all that?
b) Gathering  all your PAs (potentially PIs) (hint: show bgp nei x.x.x.x 
advertised routes) crafting an ACL and apply it on several peering/transit 
links?
One of them is couple of weeks work and one is an afternoon job.

adam

RE: Ingress filtering on transits, peers, and IX ports

[Jean St-Laurent via NANOG]

Hi Brian,

"However, I recognized a SP-specific case where we could receive legitimate 
traffic sourcing from our own IP blocks: customers running multi-homed BGP 
where we have assigned PA space to them.  So I added "permit" statements for 
traffic sourcing from these blocks."

If your customers are not using your DNS in your ip space, you could deny 
traffic sourcing from your ip block coming from these customers based on the 
UDP port DST 53. Then, accept all the rest. It's just one more line in your 
ACL. If they are using your DNS though, this won't work. You might want to add 
all the amplification ports like 123, 161, etc

Jean

-Original Message-
From: NANOG  On Behalf Of Brian 
Knight via NANOG
Sent: Wednesday, October 14, 2020 6:43 PM
To: nanog 
Subject: Re: Ingress filtering on transits, peers, and IX ports

So I have put together what I think is a reasonable and complete ACL.  
 From my time in the enterprise world, I know that a good ingress ACL filters 
out traffic sourcing from:

* Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc 
(well-documented in
https://team-cymru.com/community-services/bogon-reference/bogon-reference-http/)
* RIR-assigned blocks I am announcing to the rest of the world

However, I recognized a SP-specific case where we could receive legitimate 
traffic sourcing from our own IP blocks: customers running multi-homed BGP 
where we have assigned PA space to them.  So I added "permit" statements for 
traffic sourcing from these blocks.

Also, we have direct peering links that are numbered within our assigned 
prefixes.  So we can use the same ACL with these peer interfaces and continue 
to have BGP work, I added "permit" statements for these point-to-point subnets.

So the order of the statements is:

* Permit where source is direct peer PtP networks
* Permit where source is BGP customer PA prefix
* Deny where source is bogon
* Deny where source is our advertised prefixes
* Permit all other traffic

I considered BGP customer PI prefixes to be out of scope for ingress filtering, 
since the customer is likely to be multi-homing.  Should we consider filtering 
them?

The Team Cymru Secure IOS Template
[https://www.cymru.com/Documents/secure-ios-template.html] also references an 
ICMP fragment drop entry on the ingress ACL.  I think that's good for an 
enterprise network, but as an SP, I'm very hesitant to include this.  Is this 
included in anyone else's transit / peer / IX ACL?

Is there anything else that I'm not thinking of?

Thanks,

-Brian


On 2020-10-14 09:25, Brian Knight via NANOG wrote:
> Hi Marcos,
> 
> Thanks for your reply.  But I am looking for guidance on traffic 
> filtering, not BGP prefix filtering.
> 
> I have looked at BCP 84, and it's a good overview of the methods 
> available to an ISP.  My questions are more operational in nature and 
> are not covered by the document.  Of the choices presented in BCP 84, 
> what do folks really use?  If it's an ACL, what challenges have there 
> been with updates?  etc.
> 
> -Brian
> 
> 
> On 2020-10-13 18:52, Marcos Manoni wrote:
>> Hi, Brian
>> 
>> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks 
>> (Updated
>> by: RFC8704 Enhanced Feasible-Path uRPF).
>> 
>> Ingress Access Lists require typically manual maintenance, but are
>> the most bulletproof when done properly; typically, ingress access
>> lists are best fit between the edge and the ISP when the
>> configuration is not too dynamic if strict RPF is not an option,
>> between ISPs if the number of used prefixes is low, or as an
>> additional layer of protection
>> 
>> 
>> Ingress filters Best Practices
>> https://www.ripe.net/support/training/material/bgp-operations-and-sec
>> urity-training-course/BGP-Slides-Single.pdf
>> *Don’t accept BOGON ASNs
>> *Don’t accept BOGON prefixes
>> *Don’t accept your own prefix
>> *Don’t accept default (unless you requested it) *Don’t accept 
>> prefixes that are too specific *Don’t accept if AS Path is too long 
>> *Create filters based on Internet Routing Registries
>> 
>> And also BGP Best Current Practices by Philip Smith 
>> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
>> 
>> Regards.
>> 
>> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
>> () escribió:
>>> 
>>> Hi Mel,
>>> 
>>> My understanding of uRPF is:
>>> 
>>> * Strict mode will permit a packet only if there is a route for the 
>>> source IP in the RIB, and that route points to the interface where 
>>> the packet was received
>>> 
>>> * Loose mode will permit a packet if there is a route for the source 
>>> IP in the RIB.  It does not matter where the route is pointed.
>>> 
>>> Strict mode won't work for us, because with our multi-homed transits 
>>> and IX peers, we will almost certainly drop a legitimate packet 
>>> because the best route is through another transit.
>>> 
>>> Loose mode won't work for us, because all of our own prefixes are in 
>>> our RIB, and thus the uRPF check o

Re: Ingress filtering on transits, peers, and IX ports

[Baldur Norddahl]

This is about ingress ACL not egress.

tor. 15. okt. 2020 12.00 skrev :

> Simple,
>
> All stub autonomous systems should have a simple egress ACL allowing only
> PI of their customers and their own PAs -it’s a simple ACL at each AS-Exit
> points (towards transits/peers), that’s it.
>
> -not sure why this isn’t the first sentence in every BCP and “security
> bulletin”…
>
>
>
>
>
> adam
>
>
>
> *From:* NANOG  *On
> Behalf Of *Baldur Norddahl
> *Sent:* Thursday, October 15, 2020 8:38 AM
> *To:* nanog@nanog.org
> *Subject:* Re: Ingress filtering on transits, peers, and IX ports
>
>
>
> All DNS resolvers discovered on our network belong to customers. Our own
> resolvers, running unbound, were not discovered.
>
>
>
> While filtering same AS on ingress could help those customers (but only
> one was a open relay), filtering bogons is something the customer can also
> do. Or the software can be fixed. Do we really expect the ISP to implement
> firewalls instead of customers upgrading software?
>
>
>
> I also note that apparently our own ISPs (transits) do not filter bogons
> either.
>
>
>
> The above is a principal question. I am going to filter bogons, it just is
> not very high on my long list of stuff to do.
>
>
>
> Regards
>
>
>
> Baldur
>
>
>
>
>
> ons. 14. okt. 2020 20.53 skrev Casey Deccio :
>
> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway  wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be
> available in a few weeks.  Happy to post it here if folks are interested.
> Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet.
> Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey
>
>

Re: Ingress filtering on transits, peers, and IX ports

[Saku Ytti]

Hey,

> All stub autonomous systems should have a simple egress ACL allowing only PI 
> of their customers and their own PAs -it’s a simple ACL at each AS-Exit 
> points (towards transits/peers), that’s it.
>
> -not sure why this isn’t the first sentence in every BCP and “security 
> bulletin”…

I will venture a guess.

  1) it's very specific scenario to be stubby and have downstream PI
  2) it won't address customers spoofing each other arbitrarily and
customer1 spoofing as customer2 on the internet, giving large chunk of
the utility of spoofing even with protection in place

How do you maintain that ACL? Why doesn't that same mechanism allow
ingress ACL on the customer port? Your proposal looks low utility for
work needed.


-- 
  ++ytti

RE: Ingress filtering on transits, peers, and IX ports

[adamv0025]

Simple, 

All stub autonomous systems should have a simple egress ACL allowing only PI of 
their customers and their own PAs -it’s a simple ACL at each AS-Exit points 
(towards transits/peers), that’s it.

-not sure why this isn’t the first sentence in every BCP and “security 
bulletin”…

 

 

adam 

 

From: NANOG  On Behalf Of 
Baldur Norddahl
Sent: Thursday, October 15, 2020 8:38 AM
To: nanog@nanog.org
Subject: Re: Ingress filtering on transits, peers, and IX ports

 

All DNS resolvers discovered on our network belong to customers. Our own 
resolvers, running unbound, were not discovered. 

 

While filtering same AS on ingress could help those customers (but only one was 
a open relay), filtering bogons is something the customer can also do. Or the 
software can be fixed. Do we really expect the ISP to implement firewalls 
instead of customers upgrading software?

 

I also note that apparently our own ISPs (transits) do not filter bogons 
either. 

 

The above is a principal question. I am going to filter bogons, it just is not 
very high on my long list of stuff to do. 

 

Regards 

 

Baldur 

 

 

ons. 14. okt. 2020 20.53 skrev Casey Deccio mailto:ca...@deccio.net> >:

Hi Bryan,

> On Oct 14, 2020, at 12:43 PM, Bryan Holloway   > wrote:
> 
> I too would like to know more about their methodology

We've written up our methodology and results in a paper that will be available 
in a few weeks.  Happy to post it here if folks are interested.  Obviously, no 
networks are individually identified; it's all aggregate.

Also, we're working on a self-test tool, but it's not quite ready yet.  Sorry.

> and actual tangibles ideally in the form of PCAPs.

What do you mean by "tangibles in the form of PCAPs"?

Casey

Re: FCC FUSF charges clarification

[Nuno Vieira via NANOG]

Thanks all who replied. Yes in fact it is "ayo"-ending one, and i do
have others in the very same location and this doesn't happen at all.

Matter handed over to legal team.

cheers

/Nuno

On Wed, 2020-10-14 at 16:58 -0700, Robert L Mathews wrote:
> On 10/14/20 2:14 PM, Nuno Vieira via NANOG wrote:
> 
> > Company Z charges company A on top of agreed services:
> 
> It may be just a coincidence, but I had the same problem with a
> company
> that begins with "Z" (and ends with "ayo").
> 
> The quote I got was for a certain exact dollar amount with some tiny
> boilerplate-looking text that says they may pass on any taxes. Since
> I've used several providers in the same buildings who have never
> mentioned any taxes, I ignored it.
> 
> When the actual bill came, it was 8% higher due to absurd things like
> "property tax surcharge". This is simply a fee they charge to cover
> their own property taxes. The other extras were similar.
> 
> These are not taxes they are required to charge you separately.
> They're
> just adding money to the bill to get you to contribute towards
> *their*
> taxes. I complained long and hard about it, but they didn't care.
> 
> 

Re: Hurricane Electric AS6939

[Radu-Adrian Feurdean]

On Wed, Oct 14, 2020, at 22:40, Darin Steffl wrote:
> For 1G or less, ethernet 
> might be cheaper with some protection already 

Not to mention that 1G waves are becoming less and less comon those days. In 
this part of the world waves tend to start at 10G.

Re: Cogent Layer 2

[Saku Ytti]

On Thu, 15 Oct 2020 at 10:28, Ryan Hamel  wrote:

> My experience with multiple carriers is that reroutes happen in under a 
> minute but rarely happen, I also have redundant backup circuits to another 
> datacenter, so no traffic is truly lost. If an outage lasts longer than 5 
> minutes, or it's flapping very frequently, then I call the carrier. Last mile 
> carriers install CPE equipment at the sites, which makes BFD a requirement to 
> account for the fiber uplink on it going down, or an issue upstream.

I think I may have spoken ambiguously and confusingly based on that
statement. Rerouting inside operator network, such as their LSR-LSR
link dropping is ostensibly invisible to the customer, can be tens of
milliseconds outage can be 10s outage.
Do you want your martini emulated backbone link to fail when operator
reroutes their own LSR-LSR link failure?

> As for security vulnerabilities, none can be leveraged if they are using 
> internal IPs, and if not, a quick ACL can drop BFD traffic from unknown 
> sources the same way BGP sessions are filtered.

> In Juniper speak, the ACL would look like:

> term deny_bfd {
> from {
> protocol udp;
> destination-port [ 3784 3785 4784 ];
> }
> then discard;

So you're dropping in every edge all UDP packets towards these three
ports? Your customers may not appreciate.

-- 
  ++ytti

Re: Ingress filtering on transits, peers, and IX ports

[Baldur Norddahl]

All DNS resolvers discovered on our network belong to customers. Our own
resolvers, running unbound, were not discovered.

While filtering same AS on ingress could help those customers (but only one
was a open relay), filtering bogons is something the customer can also do.
Or the software can be fixed. Do we really expect the ISP to implement
firewalls instead of customers upgrading software?

I also note that apparently our own ISPs (transits) do not filter bogons
either.

The above is a principal question. I am going to filter bogons, it just is
not very high on my long list of stuff to do.

Regards

Baldur


ons. 14. okt. 2020 20.53 skrev Casey Deccio :

> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway  wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be
> available in a few weeks.  Happy to post it here if folks are interested.
> Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet.
> Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey

Re: Cogent Layer 2

[Ryan Hamel]

Saku,

My experience with multiple carriers is that reroutes happen in under a minute 
but rarely happen, I also have redundant backup circuits to another datacenter, 
so no traffic is truly lost. If an outage lasts longer than 5 minutes, or it's 
flapping very frequently, then I call the carrier. Last mile carriers install 
CPE equipment at the sites, which makes BFD a requirement to account for the 
fiber uplink on it going down, or an issue upstream.
As for security vulnerabilities, none can be leveraged if they are using 
internal IPs, and if not, a quick ACL can drop BFD traffic from unknown sources 
the same way BGP sessions are filtered.
In Juniper speak, the ACL would look like:
(under policy-options)
prefix-list bgp_hosts {
apply-path "protocols bgp group <*> neighbor <*>";
}

(under firewall family inet(6) filter mgmt_acl)
term allow_bfd {
from {
protocol udp;
destination-port [ 3784 3785 4784 ];
source-prefix-list bgp_hosts;
}
then accept;
}
term deny_bfd {
from {
protocol udp;
destination-port [ 3784 3785 4784 ];
}
then discard;
}

Ryan
On Oct 14 2020, at 11:29 pm, Saku Ytti  wrote:
> On Thu, 15 Oct 2020 at 09:11, Ryan Hamel  (mailto:r...@rkhtech.org)> wrote:
>
>
> > Yep. Make sure you run BFD with your peering protocols, to catch outages 
> > very quickly.
>
> Make sure you get higher availability with BFD than without it, it is easy to 
> get this wrong and end up losing availability.
>
> First issue is that BFD has quite a lot of bug surface, because unlike most 
> of your control-plane protocols, BFD is implemented in your NPU ucode when 
> done right.
> We've had the entire linecard down on ASR9k due to BFD, their BFD-of-death 
> packet you can send over the internet to crash JNPR FPC.
> When done in a control-plane, poor scheduling can cause false positives more 
> often than it protects from actual outages (CISCO7600).
>
> In a world where BFD is perfect you still need to consider what you are 
> protecting yourself from, so you bought Martini from someone and run your 
> backbone over that Martini. What is an outage? Is your provider IGP rerouting 
> due to backbone outage an outage to you? Or would you rather the provider 
> convergees their network and you don't converge, you take the outage?
> If provider rerouting is not an outage, you need to know what their SLA is 
> regarding rerouting time and make BFD less aggressive than that. If provider 
> rerouting is an outage, you can of course run as aggressive timers as you 
> want, but you probably have lower availability than without BFD.
>
> Also, don't add complexity to solve problems you don't have. If you don't 
> know if BFD improved your availability, you didn't need it.
> Networking is full of belief practices, we do things because we believe they 
> help and faux data is used often to dress the beliefs as science. The problem 
> space tends to be complex and good quality data is sparse to come by, we do 
> necessarily fly a lot by the seat of our pants, if we admit or not.
> My belief is the majority of BFD implementations in real life on average 
> reduce availability, my belief is you need frequently failing link which does 
> not propagate link-down to reliability improve availability by deploying BFD.
>
>
>
>
>
> --
> ++ytti
>